Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly
Appearance settings

Commit c3d6f55

Browse filesBrowse files
mhartmhart
committed
Init from local repo
0 parents  commit c3d6f55
Copy full SHA for c3d6f55
Expand file treeCollapse file tree

40 files changed

+1345
-0
lines changed
Open diff view settings
Collapse file

‎.eslintignore‎

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
node_modules
Collapse file

‎.gitignore‎

Copy file name to clipboard
+1Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
node_modules
Collapse file

‎.npmignore‎

Copy file name to clipboard
+5Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
base
2+
examples
3+
nodejs
4+
nodejs4.3
5+
python2.7
Collapse file

‎LICENSE‎

Copy file name to clipboard
+19Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
Copyright 2016 Michael Hart
2+
3+
Permission is hereby granted, free of charge, to any person obtaining a copy of
4+
this software and associated documentation files (the "Software"), to deal in
5+
the Software without restriction, including without limitation the rights to
6+
use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
7+
of the Software, and to permit persons to whom the Software is furnished to do
8+
so, subject to the following conditions:
9+
10+
The above copyright notice and this permission notice shall be included in all
11+
copies or substantial portions of the Software.
12+
13+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
14+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
15+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
16+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
17+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
18+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
19+
SOFTWARE.
Collapse file

‎README.md‎

Copy file name to clipboard
+153Lines changed: 153 additions & 0 deletions
  • Display the source diff
  • Display the rich diff
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,153 @@
1+
docker-lambda
2+
-------------
3+
4+
A sandboxed local environment that replicates the live [AWS Lambda](https://aws.amazon.com/lambda/)
5+
environment almost identically – including installed software and libraries,
6+
file structure and permissions, environment variables, context objects and
7+
behaviors – even the user and running process are the same.
8+
9+
You can use it for testing your functions in the same strict Lambda environment,
10+
knowing that they'll exhibit the same behavior when deployed live. You can
11+
also use it to compile native dependencies knowing that you're linking to the
12+
same library versions that exist on AWS Lambda and then deploy using
13+
the [AWS CLI](https://aws.amazon.com/cli/).
14+
15+
This project consists of a set of Docker images for each of the supported Lambda runtimes
16+
(Node.js 0.10 and 4.3, Python 2.7\* and Java 8\*) – as well as build
17+
images that include packages like gcc-c++, git, zip and the aws-cli for
18+
compiling and deploying.
19+
20+
There's also an npm module to make it convenient to invoke from Node.js
21+
22+
\* NB: Python 2.7 and Java 8 test runners are not yet complete, but both
23+
languages are installed in the images so can be manually tested
24+
25+
Prerequisites
26+
-------------
27+
28+
You'll need [Docker](https://www.docker.com) installed
29+
30+
Example
31+
-------
32+
33+
You can perform actions with the current directory using the `-v` arg with
34+
`docker run` – logging goes to stderr and the callback result goes to stdout:
35+
36+
```console
37+
# Test an index.handler function from the current directory on Node.js v4.3
38+
docker run -v "$PWD":/var/task lambci/lambda
39+
40+
# If using a function other than index.handler, with a custom event
41+
docker run -v "$PWD":/var/task lambci/lambda index.myHandler '{"some": "event"}'
42+
43+
# Use the original Node.js v0.10 runtime
44+
docker run -v "$PWD":/var/task lambci/lambda:nodejs
45+
46+
# To compile native deps in node_modules (runs `npm rebuild`)
47+
docker run -v "$PWD":/var/task lambci/lambda:build
48+
49+
# Run custom commands on the build container
50+
docker run lambci/lambda:build java -version
51+
52+
# To run an interactive session on the build container
53+
docker run -it lambci/lambda:build bash
54+
```
55+
56+
Using the Node.js module (`npm install docker-lambda`) – for example in tests:
57+
58+
```js
59+
var dockerLambda = require('docker-lambda')
60+
61+
// Spawns synchronously, uses current dir – will throw if it fails
62+
var lambdaCallbackResult = dockerLambda({event: {some: 'event'}})
63+
64+
// Manually specify directory and custom args
65+
lambdaCallbackResult = dockerLambda({taskDir: __dirname, dockerArgs: ['-m', '1.5G']})
66+
```
67+
68+
Create your own Docker image for finer control:
69+
70+
```dockerfile
71+
FROM lambci/lambda:build
72+
73+
ENV AWS_DEFAULT_REGION us-east-1
74+
75+
ADD . .
76+
77+
RUN npm install
78+
79+
CMD cat .lambdaignore | xargs zip -9qyr lambda.zip . -x && \
80+
aws lambda update-function-code --function-name mylambda --zip-file fileb://lambda.zip
81+
82+
# docker build -t mylambda .
83+
# docker run -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY mylambda
84+
```
85+
86+
87+
Questions
88+
---------
89+
90+
* *When should I use this?*
91+
92+
When you want fast local reproducibility. When you don't want to spin up an
93+
Amazon Linux EC2 instance (indeed, network aside, this is closer to the real
94+
Lambda environment because there are a number of different files, permissions
95+
and libraries on a default Amazon Linux instance). When you don't want to
96+
invoke a live Lambda just to test your Lambda package – you can do it locally
97+
from your dev machine or run tests on your CI system (assuming it has Docker
98+
support!)
99+
100+
101+
* *Wut, how?*
102+
103+
By tarring the full filesystem in Lambda, uploading that to S3, and then
104+
piping into Docker to create a new image from scratch – then creating
105+
mock modules that will be required/included in place of the actual native
106+
modules that communicate with the real Lambda coordinating services. Only the
107+
native modules are mocked out – the actual parent JS/PY runner files are left
108+
alone, so their behaviors don't need to be replicated (like the
109+
overriding of `console.log`, and custom defined properties like
110+
`callbackWaitsForEmptyEventLoop`)
111+
112+
* *What's missing from the images?*
113+
114+
Hard to tell – anything that's not readable – so at least `/root/*` –
115+
but probably a little more than that – hopefully nothing important, after all,
116+
it's not readable by Lambda, so how could it be!
117+
118+
* *Is it really necessary to replicate exactly to this degree?*
119+
120+
Not for many scenarios – some compiled Linux binaries work out of the box
121+
and a CentOS Docker image can compile some binaries that work on Lambda too,
122+
for example – but for testing it's great to be able to reliably verify
123+
permissions issues, library linking issues, etc.
124+
125+
* *What's this got to do with LambCI?*
126+
127+
Technically nothing – it's just been incredibly useful during the building
128+
and testing of LambCI.
129+
130+
Documentation
131+
------------
132+
133+
TODO
134+
135+
lambci/lambda
136+
- uses ENTRYPOINT, override with `--entrypoint`
137+
lambci/lambda:build
138+
- uses CMD
139+
140+
'AWS_LAMBDA_FUNCTION_NAME',
141+
'AWS_LAMBDA_FUNCTION_VERSION',
142+
'AWS_LAMBDA_FUNCTION_MEMORY_SIZE',
143+
'AWS_LAMBDA_FUNCTION_TIMEOUT',
144+
'AWS_LAMBDA_FUNCTION_HANDLER',
145+
'AWS_LAMBDA_EVENT_BODY',
146+
147+
'AWS_REGION',
148+
'AWS_DEFAULT_REGION',
149+
'AWS_ACCOUNT_ID',
150+
'AWS_ACCESS_KEY_ID',
151+
'AWS_SECRET_ACCESS_KEY',
152+
'AWS_SESSION_TOKEN',
153+
Collapse file

‎base/Dockerfile‎

Copy file name to clipboard
+17Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
FROM lambci/lambda-base
2+
3+
ENV PATH=/usr/local/lib64/node-v4.3.x/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
4+
LD_LIBRARY_PATH=/usr/local/lib64/node-v4.3.x/lib:/lib64:/usr/lib64:/var/runtime:/var/task:/var/task/lib
5+
6+
WORKDIR /var/task
7+
8+
ADD yum.conf /etc/yum.conf
9+
10+
# A couple of packages are either missing critical-ish files, or didn't make it into the tar
11+
# Reinstalling filesystem might not succeed fully, but continue anyway
12+
RUN yum reinstall -y filesystem; \
13+
yum reinstall -y shadow-utils && \
14+
yum install -y aws-cli zip git vim docker gcc-c++ clang openssl-devel cmake autoconf automake libtool && \
15+
rm -rf /var/cache/yum /var/lib/rpm/__db.* && \
16+
> /var/log/yum.log
17+
Collapse file

‎base/create-base.sh‎

Copy file name to clipboard
+14Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
#!/bin/bash
2+
3+
IMAGE_NAME=lambci/lambda-base
4+
5+
curl http://lambci.s3.amazonaws.com/fs/nodejs4.3.tgz | gzip -d | docker import - $IMAGE_NAME
6+
7+
curl http://lambci.s3.amazonaws.com/fs/nodejs.tgz -o ../nodejs/run/nodejs.tgz
8+
cp ../nodejs/run/nodejs.tgz ../nodejs/build/
9+
10+
curl http://lambci.s3.amazonaws.com/fs/python2.7.tgz -o ../python2.7/run/python2.7.tgz
11+
cp ../python2.7/run/python2.7.tgz ../python2.7/build/
12+
13+
echo "Sandbox user is: $(docker run $IMAGE_NAME stat -c '%U' /tmp)"
14+
Collapse file

‎base/create-build.sh‎

Copy file name to clipboard
+6Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
#!/bin/bash
2+
3+
IMAGE_NAME=lambci/lambda-base:build
4+
5+
docker build $BUILD_ARG -t ${IMAGE_NAME} .
6+
Collapse file

‎base/dump-nodejs.js‎

Copy file name to clipboard
+64Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
var fs = require('fs')
2+
var spawn = require('child_process').spawn
3+
var AWS = require('aws-sdk')
4+
var s3 = new AWS.S3()
5+
6+
exports.handler = function(event, context) {
7+
var filename = 'nodejs.tgz'
8+
var cmd = 'tar -cvpzf /tmp/' + filename + ' --numeric-owner --ignore-failed-read /var/runtime'
9+
10+
var child = spawn('sh', ['-c', event.cmd || cmd])
11+
child.stdout.setEncoding('utf8')
12+
child.stderr.setEncoding('utf8')
13+
child.stdout.on('data', console.log.bind(console))
14+
child.stderr.on('data', console.error.bind(console))
15+
child.on('error', context.done.bind(context))
16+
17+
child.on('close', function() {
18+
if (event.cmd) return context.done()
19+
20+
console.log('Zipping done! Uploading...')
21+
22+
s3.upload({
23+
Bucket: 'lambci',
24+
Key: 'fs/' + filename,
25+
Body: fs.createReadStream('/tmp/' + filename),
26+
ACL: 'public-read',
27+
}, function(err, data) {
28+
if (err) return context.done(err)
29+
30+
console.log('Uploading done!')
31+
32+
console.log(process.execPath)
33+
console.log(process.execArgv)
34+
console.log(process.argv)
35+
console.log(process.cwd())
36+
console.log(__filename)
37+
console.log(process.env)
38+
39+
context.done(null, data)
40+
})
41+
})
42+
}
43+
44+
// /usr/bin/node
45+
// [ '--max-old-space-size=1229', '--max-new-space-size=153', '--max-executable-size=153' ]
46+
// [ 'node', '/var/runtime/node_modules/.bin/awslambda' ]
47+
// /var/task
48+
// /var/task/index.js
49+
// {
50+
// PATH: '/usr/local/bin:/usr/bin/:/bin',
51+
// LAMBDA_TASK_ROOT: '/var/task',
52+
// LAMBDA_RUNTIME_DIR: '/var/runtime',
53+
// AWS_REGION: 'us-east-1',
54+
// AWS_DEFAULT_REGION: 'us-east-1',
55+
// AWS_LAMBDA_LOG_GROUP_NAME: '/aws/lambda/dump-nodejs',
56+
// AWS_LAMBDA_LOG_STREAM_NAME: '2016/05/18/[$LATEST]85da517...0ec8b49e',
57+
// AWS_LAMBDA_FUNCTION_NAME: 'dump-nodejs', AWS_LAMBDA_FUNCTION_MEMORY_SIZE: '1536',
58+
// AWS_LAMBDA_FUNCTION_VERSION: '$LATEST',
59+
// LD_LIBRARY_PATH: '/lib64:/usr/lib64:/var/runtime:/var/task:/var/task/lib',
60+
// NODE_PATH: '/var/runtime:/var/task:/var/runtime/node_modules',
61+
// AWS_ACCESS_KEY_ID: 'ASIA...C37A',
62+
// AWS_SECRET_ACCESS_KEY: 'JZvD...BDZ4L',
63+
// AWS_SESSION_TOKEN: 'FQoDYXdzEMb//////////...0oog7bzuQU='
64+
// }
Collapse file

‎base/dump-nodejs43.js‎

Copy file name to clipboard
+67Lines changed: 67 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
var fs = require('fs')
2+
var spawn = require('child_process').spawn
3+
var AWS = require('aws-sdk')
4+
var s3 = new AWS.S3()
5+
6+
exports.handler = function(event, context, cb) {
7+
var filename = 'nodejs4.3.tgz'
8+
var cmd = 'tar -cvpzf /tmp/' + filename + ' -C / ' +
9+
'--exclude=/proc --exclude=/sys --exclude=/tmp/* --exclude=/var/task/* ' +
10+
'--numeric-owner --ignore-failed-read /'
11+
12+
var child = spawn('sh', ['-c', event.cmd || cmd])
13+
child.stdout.setEncoding('utf8')
14+
child.stderr.setEncoding('utf8')
15+
child.stdout.on('data', console.log.bind(console))
16+
child.stderr.on('data', console.error.bind(console))
17+
child.on('error', cb)
18+
19+
child.on('close', function() {
20+
if (event.cmd) return cb()
21+
22+
console.log('Zipping done! Uploading...')
23+
24+
s3.upload({
25+
Bucket: 'lambci',
26+
Key: 'fs/' + filename,
27+
Body: fs.createReadStream('/tmp/' + filename),
28+
ACL: 'public-read',
29+
}, function(err, data) {
30+
if (err) return cb(err)
31+
32+
console.log('Uploading done!')
33+
34+
console.log(process.execPath)
35+
console.log(process.execArgv)
36+
console.log(process.argv)
37+
console.log(process.cwd())
38+
console.log(__filename)
39+
console.log(process.env)
40+
41+
cb(null, data)
42+
})
43+
})
44+
}
45+
46+
// /usr/local/lib64/node-v4.3.x/bin/node
47+
// [ '--max-old-space-size=1229', '--max-semi-space-size=76', '--max-executable-size=153' ]
48+
// [ '/usr/local/lib64/node-v4.3.x/bin/node', '/var/runtime/node_modules/awslambda/index.js' ]
49+
// /var/task
50+
// /var/task/index.js
51+
// {
52+
// PATH: '/usr/local/lib64/node-v4.3.x/bin:/usr/local/bin:/usr/bin/:/bin',
53+
// LD_LIBRARY_PATH: '/usr/local/lib64/node-v4.3.x/lib:/lib64:/usr/lib64:/var/runtime:/var/task:/var/task/lib',
54+
// NODE_PATH: '/var/runtime:/var/task:/var/runtime/node_modules',
55+
// LAMBDA_TASK_ROOT: '/var/task',
56+
// LAMBDA_RUNTIME_DIR: '/var/runtime',
57+
// AWS_REGION: 'us-east-1',
58+
// AWS_DEFAULT_REGION: 'us-east-1',
59+
// AWS_LAMBDA_LOG_GROUP_NAME: '/aws/lambda/dump-nodejs43',
60+
// AWS_LAMBDA_LOG_STREAM_NAME: '2016/05/18/[$LATEST]c079a84d433534434534ef0ddc99d00f',
61+
// AWS_LAMBDA_FUNCTION_NAME: 'dump-nodejs43',
62+
// AWS_LAMBDA_FUNCTION_MEMORY_SIZE: '1536',
63+
// AWS_LAMBDA_FUNCTION_VERSION: '$LATEST',
64+
// AWS_ACCESS_KEY_ID: 'ASIA...C37A',
65+
// AWS_SECRET_ACCESS_KEY: 'JZvD...BDZ4L',
66+
// AWS_SESSION_TOKEN: 'FQoDYXdzEMb//////////...0oog7bzuQU='
67+
// }

0 commit comments

Comments
0 (0)
Morty Proxy This is a proxified and sanitized view of the page, visit original site.