Running the stock NGINX container image with AWS Lambda

calendar Nov 6, 2021 · 8 min read  ·
Share on: linkedin copy

Part of my job at AWS is to explore the art of possible. A few weeks ago I came across an open source project called re:Web. What intrigued me about re:Web is that it allows a traditional container image (wrapping a traditional “web service” application) to be repurposed and deployed to AWS Lambda. The idea for this blog was sparked by an issue that Aidan Steele opened on the re:Web project. The technique that re:Web implements was originally pioneered by Aidan himself with his Serverlessish prototype. This blog will focus on re:Web but the outcome could be implemented in other ways, including Serverlessish. I’d like also to thank Aidan for his help with the prototype discussed in this blog (without his support I’d still be here trying to figure out how to map cache files to temp folders - who knew about /etc/nginx/conf.d/cachepaths.conf?!?).

Now that we are done praising Aidan (no, we are never done), let’s switch gears and talk about... how to run the stock NGINX container image in Lambda.

The way re:Web works is that it injects itself between Lambda and the actual web service application. The long story short is that, after a lot of trials and errors, the following Dockerfile is what you need to re-package the stock NGINX image to make it run in Lambda:

 1# syntax=docker/dockerfile:1.3-labs
 2
 3FROM public.ecr.aws/apparentorder/reweb as reweb
 4
 5FROM public.ecr.aws/nginx/nginx:latest
 6COPY --from=reweb /reweb /reweb
 7
 8# setup the local lambda runtime (to run the image locally)
 9RUN curl -L -o /usr/bin/lambda_rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/download/v1.2/aws-lambda-rie-x86_64
10RUN chmod +x /usr/bin/lambda_rie
11
12###############################################################
13########## start of custom tweaks - NGINX specific ############ 
14###############################################################
15
16# make nginx listin on 8090
17RUN sed -i "s/listen       80/listen       8090/g" /etc/nginx/conf.d/default.conf
18
19# move the nginx pid file to a directory that can be written 
20RUN sed -i "s,pid        /var/run/nginx.pid;,pid        /tmp/nginx.pid;,g" /etc/nginx/nginx.conf
21
22# put the nginx logs to stdout and stderr (which also avoids writing to non writable folders)
23RUN ln -sf /dev/stdout /var/log/nginx/access.log && \
24    ln -sf /dev/stderr /var/log/nginx/error.log
25
26# redirect all cache files to /tmp (writable)
27COPY <<EOF /etc/nginx/conf.d/cachepaths.conf
28client_body_temp_path /tmp/client_temp;
29proxy_temp_path /tmp/proxy_temp_path;
30fastcgi_temp_path /tmp/fastcgi_temp;
31uwsgi_temp_path /tmp/uwsgi_temp;
32scgi_temp_path /tmp/scgi_temp;
33EOF
34
35###############################################################
36########### end of custom tweaks - NGINX specific ############# 
37###############################################################
38
39# reweb environment variables
40ENV REWEB_APPLICATION_EXEC nginx
41ENV REWEB_APPLICATION_PORT 8090
42ENV REWEB_WAIT_CODE 200
43
44ENTRYPOINT ["/reweb"]

This is what this (multi-stage) Dockerfile does:

  • it gets (FROM) the re:Web image to source the reweb binary
  • it gets (FROM) the stock NGINX image
  • it copies the re:Web binary into the NGINX image
  • it pulls the Lambda RIE (for local execution - only required if testing Lambda locally - highly recommended)
  • it tweaks the NGINX image to bypass (current) Lambda limitations:
    • /tmp is the only writable directory
    • can’t bind processes to ports <1024
  • it sets ENV variables to configure re:Web (e.g. note that 8090 is the port NGINX responds to)
  • it runs /reweb as the entrypoint

Please note that while this example talks about NGINX , you can almost extract a common pattern from the above. All these steps are required (and mostly the same) to potentially make any stock container image exposing a web service work in Lambda. The notable exception is the “custom tweaks” section which is very container image specific.

Now onto the action. To complete the following steps you need an AWS account, Docker Desktop or Docker Engine installed locally (or anything that can build, push, run a container image really) as well as the AWS CLI installed and configured.

Local testing

The image can be built as follows:

1$ docker build -t lambdanginx:latest .

You can now run the image locally using the Lambda Runtime Interface Emulator (RIE). You can do so by modifying the entrypoint to call the rie binary and adding the CMD to call the reweb binary:

1$ docker run -it -p 9000:8080 --entrypoint /usr/bin/lambda_rie lambdanginx /reweb

This image runs fine locally (this log includes the launch + 3 invocations from another terminal):

 1$ docker run -it -p 9000:8080 --entrypoint /usr/bin/lambda_rie lambdanginx /reweb  
 2INFO[0000] exec '/reweb' (cwd=/, handler=)              
 3INFO[0012] extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory 
 4WARN[0012] Cannot list external agents                   error="open /opt/extensions: no such file or directory"
 5START RequestId: 76ccd182-f70d-4fc6-93ed-a6dfb3aea8c8 Version: $LATEST
 6re:Web -- SERVICE NOT UP: Get "http://localhost:80/": dial tcp 127.0.0.1:80: connect: connection refused
 72021/10/30 16:29:52 [notice] 21#21: using the "epoll" event method
 82021/10/30 16:29:52 [notice] 21#21: nginx/1.21.3
 92021/10/30 16:29:52 [notice] 21#21: built by gcc 8.3.0 (Debian 8.3.0-6) 
102021/10/30 16:29:52 [notice] 21#21: OS: Linux 5.10.47-linuxkit
112021/10/30 16:29:52 [notice] 21#21: getrlimit(RLIMIT_NOFILE): 1048576:1048576
122021/10/30 16:29:52 [notice] 22#22: start worker processes
132021/10/30 16:29:52 [notice] 22#22: start worker process 23
142021/10/30 16:29:52 [notice] 22#22: start worker process 24
152021/10/30 16:29:52 [notice] 22#22: start worker process 25
162021/10/30 16:29:52 [notice] 22#22: start worker process 26
172021/10/30 16:29:52 [notice] 22#22: start worker process 27
182021/10/30 16:29:52 [notice] 22#22: start worker process 28
19127.0.0.1 - - [30/Oct/2021:16:29:52 +0000] "GET / HTTP/1.1" 200 615 "-" "Go-http-client/1.1" "-"
20re:Web -- SERVICE UP: 200 OK
21127.0.0.1 - - [30/Oct/2021:16:29:52 +0000] "GET / HTTP/1.1" 200 615 "-" "Go-http-client/1.1" "-"
22END RequestId: 76ccd182-f70d-4fc6-93ed-a6dfb3aea8c8
23REPORT RequestId: 76ccd182-f70d-4fc6-93ed-a6dfb3aea8c8    Init Duration: 0.46 ms    Duration: 66.09 ms    Billed Duration: 67 ms    Memory Size: 3008 MB    Max Memory Used: 3008 MB    
24START RequestId: b65d72bf-519a-4583-9514-44b9a87278dd Version: $LATEST
25127.0.0.1 - - [30/Oct/2021:16:37:56 +0000] "GET / HTTP/1.1" 200 615 "-" "Go-http-client/1.1" "-"
26END RequestId: b65d72bf-519a-4583-9514-44b9a87278dd
27REPORT RequestId: b65d72bf-519a-4583-9514-44b9a87278dd    Duration: 2.92 ms    Billed Duration: 3 ms    Memory Size: 3008 MB    Max Memory Used: 3008 MB    
28START RequestId: 3fb34af9-7b4e-47b2-9c35-a5a127e1e835 Version: $LATEST
29127.0.0.1 - - [30/Oct/2021:16:37:57 +0000] "GET / HTTP/1.1" 200 615 "-" "Go-http-client/1.1" "-"
30END RequestId: 3fb34af9-7b4e-47b2-9c35-a5a127e1e835
31REPORT RequestId: 3fb34af9-7b4e-47b2-9c35-a5a127e1e835    Duration: 1.92 ms    Billed Duration: 2 ms    Memory Size: 3008 MB    Max Memory Used: 3008 MB
32

The locally running Lambda function can be invoked using a specific path and endpoint. This is how the function responds (with the nginx default home page):

 1$ curl -X POST -d '{}' http://localhost:9000/2015-03-31/functions/function/invocations | jq -r '.body' | base64 -D
 2  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
 3                                 Dload  Upload   Total   Spent    Left  Speed
 4100  1186  100  1184  100     2   231k    400 --:--:-- --:--:-- --:--:--  231k
 5<!DOCTYPE html>
 6<html>
 7<head>
 8<title>Welcome to nginx!</title>
 9<style>
10html { color-scheme: light dark; }
11body { width: 35em; margin: 0 auto;
12font-family: Tahoma, Verdana, Arial, sans-serif; }
13</style>
14</head>
15<body>
16<h1>Welcome to nginx!</h1>
17<p>If you see this page, the nginx web server is successfully installed and
18working. Further configuration is required.</p>
19
20<p>For online documentation and support please refer to
21<a href="http://nginx.org/">nginx.org</a>.<br/>
22Commercial support is available at
23<a href="http://nginx.com/">nginx.com</a>.</p>
24
25<p><em>Thank you for using nginx.</em></p>
26</body>
27</html>
28$

Cloud testing

Now onto the real thing.

Note I am using us-west-2 as the region in the example below. Change it as you see fit. Also remember to change the AWS account placeholder (123456789) to your real account.

Before we can create the Lambda function we need to upload the new container image to ECR. These 4 commands will:

  • create the ECR repository
  • login to ECR
  • tag the image
  • push the image to the repository
1$ aws ecr create-repository --repository-name lambda-nginx --region us-west-2 
2$ aws ecr get-login-password --region us-west-2 | docker login --username AWS --password-stdin 123456789.dkr.ecr.us-west-2.amazonaws.com
3$ docker tag lambdanginx:latest 123456789.dkr.ecr.us-west-2.amazonaws.com/lambdanginx:latest
4$ docker push 123456789.dkr.ecr.us-west-2.amazonaws.com/lambdanginx:latest

This is a CloudFormation stack that deploys the Lambda (courtesy of Aidan, again!). Save this file as cfn-nginx-lambda.yaml.

 1"Description" : "Running the NGINX image as a Lambda function"
 2
 3Transform: AWS::Serverless-2016-10-31
 4
 5Parameters:
 6  ImageUri:
 7    Type: String
 8    Description: "ECR image uri"
 9
10Resources:
11  Function:
12    Type: AWS::Serverless::Function
13    Properties:
14      PackageType: Image
15      ImageUri: !Ref ImageUri
16      Timeout: 10
17      AutoPublishAlias: live
18      Events:
19        Http:
20          Type: HttpApi
21
22Outputs:
23  Function:
24    Value: !Ref Function.Version
25  Url:
26    Value: !GetAtt ServerlessHttpApi.ApiEndpoint

The template can be deployed with the following command (again, remember to check region and account ID):

1$ aws cloudformation create-stack \
2    --template-body file://./cfn-nginx-lambda.yaml \
3    --parameters ParameterKey=ImageUri,ParameterValue="123456789.dkr.ecr.us-west-2.amazonaws.com/lambdanginx:latest" \
4    --capabilities CAPABILITY_IAM CAPABILITY_NAMED_IAM CAPABILITY_AUTO_EXPAND \
5    --stack-name nginx-lambda \
6    --region us-west-2

This stack has created a Lambda function using the container image you have just created and put an API Gateway interface in front of it.

You can find the API Gateway endpoint by querying the stack:

1$ aws cloudformation describe-stacks --stack-name nginx-lambda --query "Stacks[0].Outputs[1].OutputValue"
2"https://zi0m7aklv9.execute-api.us-west-2.amazonaws.com"

And then hit the endpoint with curl and enjoy the NGINX default page coming off of a Lambda function (through said API Gateway):

 1$ curl https://zi0m7aklv9.execute-api.us-west-2.amazonaws.com 
 2<!DOCTYPE html>
 3<html>
 4<head>
 5<title>Welcome to nginx!</title>
 6<style>
 7html { color-scheme: light dark; }
 8body { width: 35em; margin: 0 auto;
 9font-family: Tahoma, Verdana, Arial, sans-serif; }
10</style>
11</head>
12<body>
13<h1>Welcome to nginx!</h1>
14<p>If you see this page, the nginx web server is successfully installed and
15working. Further configuration is required.</p>
16
17<p>For online documentation and support please refer to
18<a href="http://nginx.org/">nginx.org</a>.<br/>
19Commercial support is available at
20<a href="http://nginx.com/">nginx.com</a>.</p>
21
22<p><em>Thank you for using nginx.</em></p>
23</body>
24</html>

Alternatively, you can hit the same endpoint via a browser:

From here, you could do a quick scaling test and observe how Lambda responds to requests. Below I have used Apache ab to hit the API Gateway endpoint with a couple of different profiles:

1while TRUE; do ab -n 10 -c 5 https://zi0m7aklv9.execute-api.us-west-2.amazonaws.com/; sleep 2; done
2
3while TRUE; do ab -n 100 -c 50 https://zi0m7aklv9.execute-api.us-west-2.amazonaws.com/; sleep 2; done

I ran the first test profile (10 requests with concurrency 5) every 2 seconds in a loop for roughly 30 minutes and the second test profile (100 requests with concurrency 50) for another 30-ish minutes. And this is how our Lambda reacted:

Conclusions

In this post I have tried to demonstrate how easy it is to wrap a stock nginx image and run it with AWS Lambda. I did not have a particular use case in mind for this and I was just exploring the art of possible through some hacking. If this is of interest to you, and you want to chat about how this, please reach out! I want to hear how you are thinking about it.