Serverless observability with OpenTelemetry

What you will learn

  • What is observability and distributed tracing.
  • What is X-Ray.
  • What is OpenTelemetry.
  • AWS Distro for OpenTelemetry.
  • How to auto-instrument Lambda function.


There are three primary types of telemetry data through which systems are made observable:

  • Logs – structured or unstructured lines of text that are emitted by an application in response to some event in the code.
  • Metrics – values that expresses some data about a system.
  • Traces – show the activity for an individual transaction or request as if flows through an application.

Distributed tracing

Single application could be composed of many services deployed across multiple networks, multiple availability zones or even multiple regions. Each service performs multiple operations and handles requests to external services. It is a challenge to monitor and profile such applications. Distributed tracing helps pinpoint where failures occur and what causes poor performance.


AWS X-Ray is a service that collects data about your application requests, and provides tools you can use to view, filter, and gain insights into that data to identify issues and opportunities for optimization. You can see request and response data and information about calls that your application makes to downstream AWS resources, microservices, databases and HTTP web APIs.

AWS X-Ray creates a map of services used by your application with trace data that you can use to drill down into specific services or issues. This provides a view of connections between services in your application and aggregated data for each service, including average latency and failure rates.
Within the  service map, the health of each node is represented by coloring on the ratio of successful calls to errors and faults:

  • Green = successful calls.
  • Red = server faults (500 series errors).
  • Yellow = client errors (400 series errors).
  • Purple = throttling errors (429 Too Many Requests).


OpenTelemetry is a set of APIs, SDKs, tooling and integrations that are designed for the creation and management of telemetry data such as traces, metrics, and logs. The project provides a vendor-agnostic implementation that can be configured to send telemetry data to the backend(s) of your choice (for example AWS X-Ray). It also supports a variety of popular open-source projects including Jaeger and Prometheus.

AWS Distro for OpenTelemetry

AWS Distro for OpenTelemetry is a secure, production-ready, AWS-supported distribution of the OpenTelemetry project. Part of the Cloud Native Computing Foundation, OpenTelemetry provides open source APIs, libraries, and agents to collect distributed traces and metrics for application monitoring. AWS contributes code to the OpenTelemetry upstream project first.

Lambda instrumentation

The AWS Distro for OpenTelemetry supports AWS managed Lambda layers. AWS managed Lambda layers for ADOT provides a plug-and-play user experience by automatically instrumenting a Lambda function, by packaging OpenTelemetry together with an out-of-the-box configuration for AWS Lambda and AWS X-Ray in an easy to setup layer. Users can enable and disable OpenTelemetry for their Lambda function without changing code.

Lab should be done in Ireland (eu-west-1) region.
1. Click on Start Lab button.
2. Wait until lab creates. It can take up to 15 minutes, depending on the lab.
3. Click AWS Console button to open the console in new tab.
4. Copy username and password and paste them on the AWS web page and log in.
5. Each lab has specified time when you should finish it, visible on the countdown timer below.
After time passes you will be logged out from the console. Each lab can be launched only once.

Lab time: 90 minutes

Lab scenario

You wrote an application that requests data from Facebook and saves it to a file.  The application is made up of API Gateway, Lambda function and S3 bucket, where the file is saved. You noticed that it takes really long to execute. Enable distributed tracing using AWS Distro for Opentelemetry and fix the issue. Application in already precreted for you.


  1. Test your application by invoking opentelemetry API Gateway URL. How long it takes to execute?
  2. Verify that no traces are present in X-Ray.
  3. Enable tracing for Lambda function and API Gateway and execute your application again.
  4. Verify X-Ray traces. What spans can you see?
  5. Add AWS Distro for OpenTelemetry layer to your Lambda function.
    Layer ARN:arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-python38-ver-1-5-0:3
  6. Add the environment variable AWS_LAMBDA_EXEC_WRAPPER and set it to /opt/otel-instrument.
  7. Test your application again and verify X-Ray trace. What additional info can you see?
  8. Now fix your function by removing unnecessary code.
  9. Test you application and check traces.

By default, the layer is configured to export traces to AWS X-Ray. Your function role must have the required AWS X-Ray permissions (precreated for you).

Expected results



Learning through challenge philosophy

We believe that only by solving challenges you can grow quickly and unleash your full potential. When faced with a challenge, you will leverage your experience, harness provided and external resources, develop a plan and push forward to find the best solution. Along the way, there is experimentation, failure, and ultimately success. First, try to finish this lab by yourself, using information in Theory section and recommended learning materials. Got stuck? No problem, check this step by step instructions.

  1. Go to eu-west-1 region and opentelemetry-otel function.
  2. Select Test tab and click Test.
  3. Test you function by executing API Gateway endpoint. Go to API Gateway service and select opentelemetry API. Click Stages -> Prod and click Invoke URL.

  4. Your function should execute successfully.
  5. Verify that no traces are present in X-Ray. Go back to Lambda service and click Monitoring tab. Check traces.
  6. Now enable tracing for Lambda function. Go to Configuration tab and click Edit.
  7. Click Activate tracing and click Save.
  8. Now enable tracing for API Gateway. Go to API Gateway service and select opentelemetry API. Click Stages -> Prod and click Logs/Tracing. Select Enable X-Ray Tracing.
  9. Execute you application again by Invoking URL. Go to X-Ray service and click Service Map. You should see traces from API Gateway to Lambda function.
  10. Click latest trace and check application timeline.
  11. Add AWS Distro for OpenTelemetry layer to your Lambda function. Go to your Lambda function and click Add a Layer.
  12. Select Specify an ARN: arn:aws:lambda:eu-west-1:901920570463:layer:aws-otel-python38-ver-1-5-0:3
    Click Add.
  13. Go to Configuration tab and click Environment variables. Add the environment variable AWS_LAMBDA_EXEC_WRAPPER and set it to /opt/otel-instrument
  14. Test your application again by invoking URL. Go to X-Ray service and check traces. You can see that Lambda HTTP and S3 request takes under second but Lambda executes over 10 seconds. This may indicate some internal code issue.
  15. Go to your Lambda function code and remove line #13. It adds delay to the function.
  16. Test your application again by invoking URL. Go to X-Ray service and check newest trace. Additional delay should be removed.

CloudWatch Metrics

  1. Go to your Lambda function and click Monitor tab.

  2. Check if your function was executed. Go to Metric tab. You should see invocation metric showing value bigger than 1. Are there any errors?

    Note: Metrics are send to CloudWatch after function finishes execution. It take some time to deliver metrics to CloudWatch. If you do not see any invocations refresh metrics.

CloudWatch Logs

  1. Verify logs of your Lambda function.  Go to Logs tab. Click View logs in CloudWatch button.

  2. You will be redirected to CloudWatch Logs service. You should see your function logs. Can you identify any error?

Copyright © - 2021

Privacy Policy