Application Layer

• API Parameters: These are configurable settings or values used to customize the behavior of an API, allowing users to control aspects such as data processing, request format, and response content. Setting appropriate parameters ensures efficient utilization of resources and optimal interaction with the API.

• Token Size: Always set the max_tokens parameter to control the token size per API call.

• Batch requests: Instead of individual requests consider sending batch requests to reduce the overall costs.

• Caching: For applications where the same inputs might result in the same outputs frequently, implement caching mechanisms to save on compute costs by serving cached results instead of regenerating them.

• Prompt Guide: Offer users guidance on crafting effective prompts with a sample prompt guide/ collection. This approach ensures users can achieve their desired outcomes.with minimal iterations.

• Context Window: Despite the rise of context window lengths up to a million in LLMs, it’s crucial not to default to utilizing the full extent in every instance. Especially in RAG applications, strategically optimizing to use only a minimal number of tokens is key for cost efficiency.

Infrastructure Layer

• Cloud infrastructure: Leverage cloud platforms that offer flexible pricing options and pay-as-you-go models. This allows you to scale your resources up or down based on your needs, avoiding unnecessary costs. Consider using managed services like Autoscaling and terminate compute instances when idle.

• Spot VMs or Preemptimble VMs: If not using PaaS services, then look for Spot or Low Priority VMs for model training or fine-tuning to benefit from lower pricing.

• Reserved Instances: If you have predictable, steady-state workloads, purchasing reserved instances can offer significant savings over on-demand pricing in exchange for committing to a one-year or three-year term. This is beneficial for customer facing workloads where predictable performance is important. E.g Azure PTUs

• Rate Limiting: Rate limiting in Azure API Management is a policy to control the number of processed requests by a client within a specified time period, ensuring fair usage and preventing abuse of the API. This can help control costs too.

• Monitoring and logging: Continuously monitor your model’s performance and resource usage. This helps you identify areas for optimization and potential cost savings. You can build this telemetry using Azure API Management and Azure Cost Monitor.

Note:

We advise implementing a telemetry solution early to monitor your application’s token usage for prompts and completions. This allows for informed decisions between PTU and Pay-As-You-Go as your workload grows. Gradually scaling your solution to production through a ramp-up approach is recommended for cost-effective management.

• Data Transfer Costs/Egress Costs: In a multi-cloud and/or a multi-region setup, monitoring egress usage and fees is crucial to managing total solution costs effectively. The traditional observability
• Data Storage: Store your training datasets or files generated from AI applications in lower cost object storage like Azure Blob, S3 or Google Cloud Storage when possible. Utilize compression techniques to reduce storage costs.