Skip to content
hands-on lab

Collecting Log Data with Kinesis Agent and Querying with Amazon Athena

Difficulty: Beginner
Duration: Up to 1 hour and 30 minutes
Students: 1,555
Rating: 4.2/5
Start lab
On average, students complete this lab in45m
Get guided in a real environmentPractice with a step-by-step scenario in a real, provisioned environment.
Learn and validateUse validations to check your solutions every step of the way.
See resultsTrack your knowledge and monitor your progress.
About
Author

Description

Amazon Kinesis Agent is an application that continuously monitors files and sends data to a Amazon Kinesis Data Firehose Delivery Stream or a Kinesis Data Stream. The agent handles rotating files, checkpointing, and retrying upon a failure.

The primary use-case for Kinesis agent is log collection, it's designed to deliver your data quickly and reliably, and also integrates with Amazon CloudWatch to enable you to monitor the health of, and failures in, the streaming process.

In this lab, you will install and configure Kinesis Agent, use it to collect log entries, and query the log entries with Amazon Athena.

Learning Objectives

This is a beginner-level lab. Upon completion of this lab you will be able to:

  • Create a Kinesis Firehose Delivery Stream to deliver data to an S3 bucket
  • Install and configure the Kinesis Agent on a Linux EC2 instance to send data to your delivery stream
  • Query your log data stored in S3 using Amazon Athena

Intended Audience

  • Candidates for the AWS Data Analytics Speciality Certification
  • Data Engineers
  • Cloud Engineers

Prerequisites

Familiarity with Analytics in AWS and the Linux command-line will be beneficial but is not required.

The following courses can be used to fulfill the prerequisites:

Updates

January 22nd, 2025 - Updated screenshots & instructions to reflect the latest UI

February 20th, 2024 - Updated screenshots & instructions to reflect the latest UI

April 25th, 2023 - Updated screenshots & instructions to reflect the latest UI

November 27th, 2022 - Updated screenshots & instructions to reflect the latest UI

June 23rd, 2022 - Resolved an issue with the last lab step and improved instructions

May 2nd, 2022 - Updated instructions and screenshots for accuracy

February 14th, 2022 - Updated the instructions and screenshots to reflect the latest Athena UI

December 21st, 2021 - Fixed an issue preventing the Kinesis Agent from being installed

Environment before

Environment after

Covered topics

Amazon Web Services
Analytics for AWS
Development for AWS
Amazon Kinesis
Amazon Athena
App Streaming for AWS
App Streaming
Analytics
Development
Hands-on Lab UUID

Lab steps

Logging In to the Amazon Web Services Console
Creating a Kinesis Firehose Delivery Stream
Connecting to the Virtual Machine using EC2 Instance Connect
Installing and Configuring the Kinesis Agent Application
Querying the Log Data using Amazon Athena

Lab Rules