AWS Lambda: libstdc++.so.6 version GLIBCXX_3.4.20 not found error

January 16, 2019 · 5 min read

Step-by-step guidance to fix the /usr/lib64/libstdc++.so.6: version 'GLIBCXX_3.4.20' not found error in AWS Lambda when using a custom binary.

Unable to import module 'handler': /usr/lib64/libstdc++.so.6: version
`GLIBCXX_3.4.20' not found

I ran into this issue when I tried to run the static website generator Hugo on Lambda. The solution was not trivial, so hopefully this post will save you some time.

Root cause

Let's first understand why this issue happens.

The binary you're trying to run has a dependency on libstdc++, or “GNU Standard C++ Library”, a runtime library for C++ programs built with the GNU compiler. The Lambda execution environment doesn't have it pre-installed, so the dependency is missing and the binary fails to run.

Solution

To fix the problem, we need to package the missing library along with the Lambda function code or as a Lambda layer.

But first we need libstc++.so.6 compiled for Amazon Linux, so that it's compatible with the Lambda execution environment. To do that, we'll obtain a Lambda-compatible libstc++.so.6 using an Amazon EC2 instance. Then we'll create a Lambda Layer to make the library available to the function.

Step-by-step guide

1. Create an S3 bucket

To retrieve a file from an EC2 instance, we'll need to create an S3 bucket to upload the file to.

Create an S3 bucket. Follow S3 documentation if you need help.
Copy the bucket name, you'll need it later.

2. Create an IAM user

We will create an IAM user that the EC2 instance will use to upload a file to S3.

Create an IAM user. Follow IAM documentation if you need help. In the wizard, select Programmatic access and attach the AmazonS3FullAccess policy in the Set permissions screen*.
When the user is created, you will see a table with Access key ID and Secret access key. Don't close the page, you'll need these values in the next step.

* Note: This role will have full access to all of your S3 buckets. I use it for demonstration purposes only. I highly recommend removing the user after you've completed this tutorial.

3. Start an EC2 instance, SSH into it and configure AWS CLI

To obtain libstc++.so.6 compiled for Amazon Linux, we'll need to use an Amazon EC2 instance. Note that Lambda runs on Amazon Linux 1 but compiling for a newer Amazon Linux 2 also works.

Start an EC2 instance using the Amazon Linux 2 AMI (HVM), SSD Volume Type Amazon Machine Image (AMI). Follow Step 1 in EC2 documentation if you need help.
Connect to your instance using SSH. Follow EC2 documentation if you need help.
Configure AWS CLI on the EC2 instance by running aws configure in the instance's shell. We will need the CLI to retrieve the file from the instance. Use credentials (secret and access keys) for the IAM role you created in the previous step. Follow AWS CLI documentation if you need help.

4. Install the gcc-c++ package

Run the following command in EC2 shell to install GCC C++ packages, which libstc++.so.6 is part of. Installing it with the yum package manager ensures that you get the package compatible with the Amazon Linux 2 distribution.

sudo yum install gcc-c++

libstc++.so.6 is now installed in /usr/lib64/. To verify that it contains the GLIBCXX_3.4.20 object, you can run the following commands. The first one installs binutils and the second one looks up GLIBCXX_3.4.20 in libstc++.so.6.

sudo yum install binutils
objdump -x /usr/lib64/libstdc++.so.6 | grep GLIBCXX_3.4.20

The command should return the following result confirming there's a match.

22 0x00 0x0297f870 GLIBCXX_3.4.20
	GLIBCXX_3.4.20

5. Copy libstc++.so.6 to S3

Run the following command in EC2 shell to copy libstc++.so.6 to the S3 bucket. Use the bucket name from Step 1 instead of BUCKET_NAME.

aws s3 cp /usr/lib64/libstdc++.so.6 s3://BUCKET_NAME/libstc++/

Run the following command to verify that the file has been uploaded. You should see libstc++.so.6 in the output.

aws s3 ls BUCKET_NAME/libstc++/

6. Create a Lambda Layer with libstc++.so.6

Download libstc++.so.6 from S3 bucket using the console or AWS CLI.
In the Lambda console, click on Layers* in the left navigation, click Create layer and follow the wizard to create a layer. It's self-explanatory.
Then navigate to your Lambda function in the console, click on Layers in the Designer section, and add the layer you've just created to the function. If you use CloudFormation or SAM, add the Layer ARN under Layers section in the template.
The function should now execute successfully.

* Layers is the way to go with dependencies like this but you can also do it “the old way” and include libstc++.so.6 along with your Lambda function zip file. It will work either way.

7. Clean up

Terminate the EC2 instance you created in Step 3 to stop incurring charges. Follow EC2 documentation if you need help.
Delete the IAM user you created in Step 2. Follow IAM documentation if you need help.

References

This post would not have been possible without other resources I used to troubleshoot the issue.

Disclosure

At the time of this writing, I work as a Principal Product Manager at AWS. This post is about my personal project and is not endorsed by AWS.