Monitor OpenAI APIs

AppDynamics Python Agent helps you monitor the OpenAI and Azure OpenAI usage and related metrics. It monitors the ChatCompletion, Completion, Moderation, and Embeddings API type requests created using the openai-python library to the OpenAI backend. The agent also captures the prompts and responses used by the OpenAI API models.

The following metrics are reported at the application, tier, and node levels:

  • Calls per minute
  • Errors per minute
  • Tokens
  • Input Tokens
  • Completion Tokens

The following are the example metric paths:

  • Per model metric path - Application Infrastructure Performance|OpenAI_Tier|Agent|OpenAI|ada|Tokens
  • All model cumulative metrics path - Application Infrastructure Performance|Agent|OpenAI_Tier|OpenAI|Tokens - All Models
  • Net application level metrics path - Overall Application Summary|Agent|OpenAI - Tokens - ada
Note: These metrics are reported even if the exit call is outside the business transaction. However, the Average Response Time(ms) metric is not reported when the exit call is outside the business transaction.

OpenAI Moderation API

From version 24.2.0, Python Agent supports OpenAI Moderation API. The moderation API will prompt the users if the queried statement is policy compliant or not. The Python Agent captures the following metrics:

  • Calls to moderations API: The number of times moderation API is used every minute. (OpenAI|moderations|Calls per minute)
  • Flagged category metrics: The number of times specific types of content, like hate speech, are flagged. (OpenAI|moderations|Flagged Calls by category|hate)
  • Flagged queries count: The total amount of flagged content. (OpenAI|moderations|Flagged queries)
  • Total queries count: The overall number of queries processed. (OpenAI|moderations|Total queries)

Prerequisites

Ensure that you meet the following prerequisites to monitor the OpenAI API exit calls with AppDynamics:

  • Install Python Agent 23.8.0 or later. See Install the Python Agent.
  • Import custom dashboards and templates.
  • Set the enable-openai flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].
  • To capture OpenAI request and response in snapshot exit call detail, set the enable-genai-data-capture flag to true in the /path/to/appdynamics.cfg file of the Python Agent. See [instrumentation].

Create a Custom Dashboard

  1. Download Custom_dashboard_script.zip.
    It contains:
    Directory Description
    templates Contains json templates to create the dashboard files.
    dashboard Dashboard json files are created here. It contains the sample json files.
    main.py Python script to create the dashboards.
    readme.md Description of the solution.
  2. Run the following command:
    CODE 
    cd Custom_dashboard_script
python3 main.py
    The preceding command prompts you to specify the dashboard type (application or tier) that you want to create.
  3. Based on your selected dashboard type, you will be prompted to specify the following dashboard details.
    Application Level Dashboard Details
    Variable Description Default Possible Values Required
    Dashboard Name The dashboard name to be displayed on Dashboard & Reports in the Controller UI. OpenAI_App_dashboard app_dashboard Optional
    Application Name Name of the Application with OpenAI backend calls. none openai_app_1 Yes
    OpenAI Endpoints Host Name The hostname of the OpenAI endpoint. api.openai.com api.openai.com Optional
    OpenAI Endpoints Port The port of the OpenAI endpoint. 443 443 Optional
    Tier Level Dashboard Details
    Variable Description Default Possible Values Required
    Dashboard Name The dashboard name to be displayed on Dashboard & Reports in the Controller UI. tier_dashboard tier_dashboard Optional
    Application Name Name of the Application with OpenAI backend calls. none openai_app_1 Yes
    Tier Name Name of the Tier exposed to OpenAI backend calls. none openai_tier_1 Yes
    Business Transaction Name The name of the business transaction reported to the host. This is to capture API Response time metric. none openai_bt Yes
    OpenAI Endpoints Host Name The hostname of the OpenAI endpoint. api.openai.com api.openai.com Optional
    OpenAI Endpoints Port The port of the OpenAI endpoint. 443 443 Optional
    Application Level Moderation API Dashboard Details
    Variable Description Default Possible Values Required
    Dashboard Name The dashboard name to be displayed on Dashboard & Reports in the Controller UI. OpenAI_Moderation_App moderation_app_dashboard Optional
    Application Name Name of the Application with OpenAI backend calls. none openai_app_1 Yes
    Tier Level Moderation API Dashboard Details
    Variable Description Default Possible Values Required
    Dashboard Name The dashboard name to be displayed on Dashboard & Reports in the Controller UI. OpenAI_Moderation_Tier moderation_tier_dashboard Optional
    Application Name Name of the Application with OpenAI backend calls. none openai_app_1 Yes
    Tier Name Name of the Tier exposed to OpenAI backend calls. none openai_tier_1 Yes
  4. Import the dashboard in the Controller UI.
    See Create and Manage Custom Dashboards and Templates.

Example Dashboard

Example Dashboard
Note: Navigate to Tiers & Nodes > My Dashboards and create a dashboard for the selected application by copying and pasting widgets from the existing dashboard for easier navigation.

Create a Custom Cost Metric Expression

You can create a custom cost metric expression by multiplying the Input Tokens and Completion Tokens metrics by their respective costs.

For example, the cost per one million tokens for gpt-3.5-turbo-0125 is:

  • Input Tokens = 0.5$
  • Completion Tokens = 1.5$
In this scenario, you can create a custom expression for gpt-3.5-turbo-0125 to calculate the cost as follows:
CODE 
({input Tokens}*0.5 + {Completion Tokens}*1.5)/1000000

Use this custom expression to create a pie chart for cost distribution across different models or an overall cost widget that includes the cost function for all models. Complete the following steps to create a cost per model widget to the existing OpenAI Dashboard:

  1. Add a pie chart widget to an existing openai dashboard, see Metric Expression.
  2. Specify the input_token and completion_token cost and multiply them by the token count.
    1. Data Source: Select the application name.
    2. Metric Category: Select Tier / Node Health -Hardware, JVM, CLR (cpu,heap, disk I/O, etc).
    3. Metric: Select Metric Expression.
    4. Edit the metric Expression.
    Edit the metric expression
  3. In Metric Expression, add two variables and assign names to them.
  4. In Metric Selection, specify the relative metric path for input and completion tokens:Specify the relative path for metric selection
    • Input Tokens: Agent|OpenAl|gpt-3.5-turbo|Input Tokens
    • Completion Tokens: Agent|OpenAI|gpt-3.5-turbo|Completion Tokens
  5. In Metric Expression, define the metric expression as follows:
    CODE 
    ({input_token_gpt3.5)*0.5 + {completion_token_gpt3.5)*1.5)/1000000
Metric Expression

This metric expression includes the cost multiplier per million tokens for both types.