Evaluating Generative AI Models Using Amazon Bedrock

Amazon Bedrock is an offering from AWS that enables you to use Generative AI models as a service. When using Bedrock, you don't have to worry about the technical complexity of training, hosting, and scaling models. Instead, you can focus on integrating the models into your applications and workflows.

Learning how to use Amazon Bedrock's model evaluation feature is beneficial to anyone looking to gain confidence in the quality of the models they choose to use.

In this hands-on lab, you will configure an Amazon S3 bucket for use with Amazon Bedrock and run a model evaluation job to compare the performance of a Generative AI model.

Learning objectives

Upon completion of this beginner-level lab, you will be able to:

• Add a CORS configuration to an Amazon S3 bucket
• Create an Amazon Bedrock model evaluation job
• Run an Amazon Bedrock model evaluation job and interpret the results

Intended audience

• Anyone looking to make use of Generative AI models in the public AWS cloud
• Cloud Architects
• Data Engineers
• DevOps Engineers
• Machine Learning Engineers
• Software Engineers

Prerequisites

Familiarity with the following will be beneficial but is not required:

• Amazon S3
• Amazon Bedrock

The following content can be used to fulfill the prerequisites:

• Increasing Your Security Posture when Using Amazon S3
• Interacting with Amazon Bedrock Foundation Models using APIs

Updates

July 21st, 2025 - Updated model from Titan to Nova

January 16th, 2025 - Updated the instructions and screenshots to reflect the latest UI