Rate Limiting Policy in Azure API Management
Rate limiting in Azure API Management is a policy that restricts the number of requests a user can make to an API within a certain timeframe, ensuring cost control, fair usage and protecting the API from overuse and abuse. Just as we have rate limits at the OpenAI API level discussed above with TPM and RPM, we can also set rate limiting policies in Azure API management too. This has several benefits as mentioned below –
- Prevents Overuse: Ensures no single user can monopolize API resources by making too many requests.
- Manages Resources: Helps in evenly distributing server resources to maintain service reliability.
- Controls Costs: Avoids unexpected spikes in usage that could lead to higher operational costs.
- Enhances Security: Acts as a defense layer against attacks, such as Denial of Service (DoS), by limiting request rates.
- Ensures Quality of Service: Guarantees fair resource distribution among all users to maintain expected service levels.
- Promotes Operational Stability: Contributes to the API’s stability and predictability by allowing for effective resource planning.
Now that we have a good grasp on fundamental components of scaling and strategies with our special scaling sauce on Azure API Management, let’s turn our attention to Monitoring and Logging capabilities that can help build telemetry on our Gen AI application that can help you measure critical metrics to determine the performance and availability of your application.
Monitoring, logging, and HTTP return codes
As we have learned in the previous sections, both limits and how we manage these limits using various scaling techniques can help us provide a robust, enterprise-class, highly scalable cloud GenAI service to many thousands of users/demanding enterprise applications.
But as with any good enterprise-class service, it’s important to configure and deploy the basic telemetry data provided by monitoring and logging to ensure optimal performance and timely notifications in case of issues.
Monitoring and logging
One of the most critical operational categories that is required for any robust enterprise service or solution that’s designed to be enterprise-ready are monitoring/instrumentation/observability and logging of the solution.
These components are required for any enterprise-level service, and you may already be familiar with the concepts or have a lot of experience in these areas, so we will not cover this extensively, only how monitoring and logging pertain to running a GenAI/ChatGPT-based cloud service, as well as some best practices.
Any enterprise monitoring solution can be used for health-checking applications and services, as well as setting up alerts to be notified if certain thresholds are reached or exceeded, such as protection against automated and high volume misuse or other anomalies related to unusual usage patterns. Two very well and broadly used services, Azure Monitoring and DataDog, both have operational modules for use with OpenAI/Azure OpenAI. These enterprise tools know which metrics are important to collect, display, and alert on for the success and optimal health of your cloud GenAI service.
Monitoring transactional events, such as TokenTransaction, Latency, or TotalError to name a few, can provide valuable insight into how your Cloud ChatGPT service is operating, or alert you if settings or conditions are not within your ideal parameters. The alerting and notification of these available metrics are highly configurable. You can find the complete list of metrics here: https://learn. microsoft.com/en-us/azure/ai-services/openai/how-to/monitoring#azure-openai-metrics.
For more information about OpenAI monitoring by Datadog, check out https://www.datadoghq.
com/solutions/openai/.
On a related note, application logging is critical to the success of reviewing events either in real time or after they have occurred. All metrics described previously can be collected and stored, reported in real-time for historical analysis, and output to visualization tools such as Fabric (Power BI) using Log Analytics Workspace in Azure, for example.
Every cloud GenAI application will have different logging requirements defined by the business/ organization. As such, Microsoft has created a monitoring and logging AOAI best practices guide, a link to which you can find at the end of this chapter.