Training

You have already started your journey of training for ChatGPT and OpenAI, especially if you have read this book thus far. There are many forms of learning and training that we already know about, but the key point here is that it is important to be knowledgeable or have staff/colleagues trained in not only the ChatGPT services themselves but other related services as well. We mentioned a few of these other services in the previous chapter, such as the APIM service, enterprise monitoring, instrumentation, logging, application and web development and management, and data science and analytics to name a few.

Another aspect of training may include database management training, especially a NoSQL type of enterprise service such as Azure CosmosDB. Why? Typically, a large organization would want to save their prompt and completion history, for example, so that they can retrieve it later or search without having to resend the same prompts again. This does make for a highly efficient and optimized ChatGPT cloud service, with all the benefits a NoSQL database, such as CosmosDB, can provide – such as being highly performant, having lower costs, and being a globally scalable service. Based on our experience, we have found that CosmosDB can be beneficial for Caching and Session Management of Conversational generative AI applications.

Of course, no one person can run an enterprise solution, so you are not expected to know the intricate details and job tasks for every service – this is what an enterprise cloud team does… and does it as a team! However, identifying training requirements for the enterprise services you will run and identifying any gaps early in the service planning life cycle is highly recommended and a best practice.

Support

Just like training is a critical part of designing and scaling a ChatGPT for cloud solutions, so is supporting this enterprise solution/service.

Many aspects of support need to be considered: internal technical support for the end users who may be using your enterprise-ready service and the internal support provided by various workload owners, including both primary and ancillary services, as described earlier.

However, this is not only internal support, but also any external, third-party, and vendor cloud support you will need to consider. Both OpenAI and Azure provide many tiers of support, whether it is free-to-low-cost self-service forums, where communities support each other, or paid support by trained personnel who can quickly resolve an enterprise issue, and they have personnel trained in all aspects (components) of the service. These paid support services can have many tiers of support, depending on how quickly you want the solution to be resolved based on your internal SLAs.

When designing and scaling a ChatGPT for cloud solutions, ensure “support” is on your checklist of items for a successful, robust deployment. This category cannot be overlooked or skipped.

Summary

In this chapter on deploying GenAI in the cloud, we learned how to design and scale a robust, enterprise ready GenAI cloud solution. We covered what limits exist within each of the models and how to overcome these limits either by adding additional (Azure) OpenAI accounts and/or using an Azure APIM service.

APIM, with its very important exponential interval retry setting, is yet another way to help organizations scale up to meet business and user requirements.

Reserved capacity, known as PTUs in Microsoft Azure, is another way an enterprise can scale up to meet business requirements. We described how additional PTUs can be added and scaled by increasing the number of PTUs.

During our cloud scaling journey, we learned how to scale across multiple geographies, or multi-regions, to support broader scale globally, while also supporting our enterprise DR scenarios.

We now understand how to handle various response and error codes when making API calls against our generative AI models, and we also know about best practices such as always configuring error checking the size of the prompt against the model this prompt is intended for first for a more optimized experience.

Then, you learned about the scaling special sauce, an insightful technique that ensures both a large-scale and seamless experience by using a retry pattern known as retries with exponential backoff. With this technique, scaling at extremely large user and prompt counts can be achieved.

As we wrapped up, we described how monitoring/instrumentation/observability plays a critical part in the overall solution by providing alerting notifications and deeper insights into the operational side of the service. Logging further supports the operational requirements for the enterprise, such as using logs for real-time analytics or historical data, so that it can be presented in reports.

Finally, we covered categories that will require further investigation as you design a scalable and robust enterprise ChatGPT cloud solution – training, support, and costs.

In the next chapter, we will learn about another important aspect for enterprises that want to scale and deploy ChatGPT in the cloud: security. We will look at some of the critical security considerations or concerns for deploying ChatGPT for cloud solutions, as well as how to best address them for a continued robust, enterprise-ready cloud solution.