diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 8f2e39c..606bf24 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -76,12 +76,17 @@ ironcli is built from source. To make sure you can hack on the `lambda` package within the vendoring workflow, you should do something like this. * Clone ironcli. -* Fork this repository. -* Edit the glide.yaml file to change the `lambda` package import path from - `github.com/iron-io/lambda/lambda` to `github.com//lambda/lambda`. - You'll also need to fix the imports in the ironcli lambda.go file. -* Run `glide i` to get the new package. -* Now `ironcli/vendor/github.com//lambda` will have your fork. Hack - in here and submit a PR. -* If you have a better idea of a workflow, we would appreciate leaving us - a note. +* Run `glide i` to install dependencies. +* Now `ironcli/vendor/github.com/iron-io/lambda` will have a clone of the + master branch of lambda. +* Create a new branch, hack hack hack. +* Make sure relevant code compiles and ironcli builds. +* Fork the lambda repository on Github. +* Add your fork as a remote: `git remote add git@github.com:/lambda` +* Push your branch to your fork: `git push ` +* Submit a PR. + +### Improving Lambda Docker images + +Simply clone this repository, make changes to the Dockerfile for individual +images in the `images/` directories and submit a Pull Request. diff --git a/LICENSE b/LICENSE new file mode 100644 index 0000000..497dc87 --- /dev/null +++ b/LICENSE @@ -0,0 +1,202 @@ + + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright 2016 Iron.io Inc. All rights reserved. + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/docs/aws.md b/docs/aws.md index 6931542..0da9f92 100644 --- a/docs/aws.md +++ b/docs/aws.md @@ -41,21 +41,21 @@ Assuming you [packaged this function](./introduction.md) into a Docker image `iron/s3-write` and pushed it to Docker Hub. Instead of just registering with IronWorker as: - ironcli register /s3-write + iron register /s3-write do this instead: - ironcli register -e AWS_ACCESS_KEY_ID= \ - -e AWS_SECRET_ACCESS_KEY= \ - /s3-write + iron register -e AWS_ACCESS_KEY_ID= \ + -e AWS_SECRET_ACCESS_KEY= \ + /s3-write -Alternatively, if you use `ironcli publish-function`, it will automatically +Alternatively, if you use `iron publish-function`, it will automatically pick up the environment variables and forward them if valid ones are found. ```sh export AWS_ACCESS_KEY_ID= export AWS_SECRET_ACCESS_KEY= -ironcli publish-function -function-name /s3-write +iron publish-function -function-name /s3-write ``` If you have an existing image with the same name registered with IronWorker, @@ -222,15 +222,13 @@ With this function ready, we can Dockerize it and publish it to actually try it out with SNS. ```sh -ironcli lambda create-function -function-name /sns-example -runtime +iron lambda create-function -function-name /sns-example -runtime nodejs -handler sns.handler sns.js ``` This will create a local docker image. The `publish-function` command will upload this to Docker Hub and register it with IronWorker. -FIXME(nikhil): AWS credentials bit. - To be able to use the AWS SDK, you'll also need to set two environment variables. The values must be your AWS credentials. @@ -238,7 +236,7 @@ variables. The values must be your AWS credentials. AWS_ACCESS_KEY_ID= AWS_SECRET_ACCESS_KEY= -ironcli publish-function -function-name /sns-example:latest +iron publish-function -function-name /sns-example:latest ``` Visit the published function's code page in the [IronWorker control diff --git a/docs/getting-started.md b/docs/getting-started.md index 4c7c560..6922250 100644 --- a/docs/getting-started.md +++ b/docs/getting-started.md @@ -4,19 +4,8 @@ This guide will walk you through creating and testing a simple Lambda function. We will then upload it to IronWorker and run it. -## Prerequisites - -TODO: Update these to use real ironcli static build. Also add ironcli -installation instructions. - -We are going to use a development branch of `ironcli` instead of the official -release. TODO: `go install` will fail due to (lack of) vendoring. - - $ cd $GOPATH - $ go get github.com/iron-io/ironcli - $ cd src/github.com/iron-io/ironcli - $ git checkout -t origin/lambda - $ go install . +We need the the `ironcli` tool for the rest of this guide. You can install it +by following [these instructions](https://github.com/iron-io/ironcli). ## Creating the function @@ -46,7 +35,7 @@ Create an empty directory for your project and save this code in a file called Now let's use `ironcli`'s Lambda functionality to create a Docker image. We can then run the Docker image with a payload to execute the Lambda function. - $ ironcli lambda create-function --function-name irontest/node-exec:1 --runtime nodejs --handler node_exec.handler node_exec.js + $ iron lambda create-function --function-name irontest/node-exec:1 --runtime nodejs --handler node_exec.handler node_exec.js Image output Step 1 : FROM iron/lambda-nodejs ---> 66fb7af42230 Step 2 : ADD node_exec.js ./node_exec.js @@ -79,7 +68,7 @@ You should now see the generated Docker image. The `test-function` subcommand can launch the Dockerized function with the right parameters. - $ ironcli lambda test-function --function-name irontest/node-exec:1 --payload '{ "cmd": "echo Dockerized Lambda" }' + $ iron lambda test-function --function-name irontest/node-exec:1 --payload '{ "cmd": "echo Dockerized Lambda" }' Dockerized Lambda! You should see the output. Try changing the command to `date` or something more @@ -107,7 +96,7 @@ The `publish-function` command first uploads the image to Docker Hub, then registers it with IronWorker so you can queue tasks or launch a task in response to a webhook. - $ $GOPATH/bin/ironcli lambda publish-function --function-name irontest/node-exec:1 + $ iron lambda publish-function --function-name irontest/node-exec:1 -----> Configuring client Project '' with id='' -----> Registering worker 'irontest/node-exec' @@ -121,7 +110,7 @@ page. We can now run this from the command line. - $ $GOPATH/bin/ironcli worker queue -payload '{ "cmd": "echo Dockerized Lambda" }' -wait irontest/node-exec + $ iron worker queue -payload '{ "cmd": "echo Dockerized Lambda" }' -wait irontest/node-exec -----> Configuring client Project '' with id='' -----> Queueing task 'irontest/node-exec' diff --git a/docs/import.md b/docs/import.md new file mode 100644 index 0000000..2ed9caa --- /dev/null +++ b/docs/import.md @@ -0,0 +1,42 @@ +Import existing AWS Lambda functions +==================================== + +The [ironcli](https://github.com/iron-io/ironcli/) tool includes a set of +commands to act on Lambda functions. Most of these are described in +[getting-started](./getting-started.md). One more subcommand is `aws-import`. + +If you have an existing AWS Lambda function, you can use this command to +automatically convert it to a Docker image that is ready to be deployed on +other platforms. + +### Credentials + +To use this, either have your AWS access key and secret key set in config +files, or in environment variables. In addition, you'll want to set a default +region. You can use the `aws` tool to set this up. Full instructions are in the +[AWS documentation][awscli]. + +[awscli]: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html#cli-config-files + +### Importing + +Assuming you have a lambda function named `my-function`, the following command: + +```sh +ironcli lambda aws-import my-function +``` + +will import the function code to a directory called `./my-function`. It will +then create a docker image called `my-function`. + +Using Lambda with Docker Hub and Iron Worker requires that the Docker image be +named `/`. This is used to uniquely identify +images on Docker Hub. Please use the `-image /` flag to `aws-import` to create a correctly named image. + +If you only want to download the code, pass the `-download-only` flag. The +`-region` and `-profile` flags are available similar to the `aws` tool to help +you tweak the settings on a command level. If you want to call the docker image +something other than `my-function`, pass the `-image ` flag. Finally, +you can import a different version of your lambda function than the latest one +by passing `-version .` diff --git a/docs/introduction.md b/docs/introduction.md index dd25ec4..cd1114f 100644 --- a/docs/introduction.md +++ b/docs/introduction.md @@ -41,6 +41,7 @@ guide](./getting-started.md). [Here is the environment](./environment.md) that Lambda provides. `ironcli lambda` lists the commands to work with Lambda functions locally. +You can [import](./import.md) existing Lambda functions hosted on Amazon! The Docker environment required to run Lambda functions is described [here](./docker.md). diff --git a/examples/s3/example.js b/examples/s3/example.js index 8b1e7c4..2f4c3de 100644 --- a/examples/s3/example.js +++ b/examples/s3/example.js @@ -59,10 +59,10 @@ exports.run = function(event, context) { ACL: 'public-read', }, function (err, data) { if (err) throw err; - context.done() + context.succeed("Image updated"); }); } else { - context.done(); + context.succeed("Image not updated"); } }); }); diff --git a/images/examples/java/src/main/java/example/Hello.java b/images/examples/java/src/main/java/example/Hello.java index 7d4054c..3b98d2f 100644 --- a/images/examples/java/src/main/java/example/Hello.java +++ b/images/examples/java/src/main/java/example/Hello.java @@ -49,7 +49,7 @@ public ArrayList myHandlerList(ArrayList arrayList, Context return arrayList; } - public static void myHandlerPOJO(RequestClass request, Context context){ - System.out.println(String.format("Hello %s %s" , request.firstName, request.lastName)); + public static String myHandlerPOJO(RequestClass request, Context context){ + return String.format("Hello %s %s" , request.firstName, request.lastName); } } diff --git a/images/java/LambdaLauncher.sh b/images/java/LambdaLauncher.sh index 870412c..fc50294 100755 --- a/images/java/LambdaLauncher.sh +++ b/images/java/LambdaLauncher.sh @@ -17,6 +17,13 @@ else exit 1 fi +# set env variables from CONFIG_* prefixed ones +for i in $(env); do + if [[ $i == CONFIG_* ]]; then + export ${i:7} + fi +done + java -jar lambda.jar $2 exit 0 diff --git a/images/java/src/main/java/io/iron/lambda/Launcher.java b/images/java/src/main/java/io/iron/lambda/Launcher.java index c4ea43b..6d63ef7 100644 --- a/images/java/src/main/java/io/iron/lambda/Launcher.java +++ b/images/java/src/main/java/io/iron/lambda/Launcher.java @@ -12,16 +12,40 @@ import io.github.lukehutch.fastclasspathscanner.FastClasspathScanner; public class Launcher { + + PrintStream oldout; + + public Launcher() { + oldout = System.out; + System.setOut(System.err); + } + public static void main(String[] args) { String handler = args[0]; String payload = ""; - try { - String file = System.getenv("PAYLOAD_FILE"); - if (file != null) { + String file = System.getenv("PAYLOAD_FILE"); + if (file != null && !file.isEmpty()) { + try { payload = new String(Files.readAllBytes(Paths.get(file))); + } catch (IOException ioe) { + System.err.println("bootstrap: Could not read payload file " + ioe); + System.exit(1); + } + } else { + try { + ByteArrayOutputStream os = new ByteArrayOutputStream(); + byte[] buffer = new byte[4096]; + int n; + while ((n = System.in.read(buffer)) != -1) { + os.write(buffer, 0, n); + } + os.close(); + + payload = os.toString("UTF-8"); + } catch (IOException ioe) { + System.err.println("bootstrap: Could not read stdin " + ioe); + System.exit(1); } - } catch (IOException ioe) { - // Should probably log this somewhere useful but not in the output. } try { @@ -115,6 +139,15 @@ private void runMethod(Method lambdaMethod, String payload, Class cls, String ha if (!processed) { System.err.println(String.format("Handler %s with simple, POJO, or IO(input/output) types not found", handlerName)); } + + try { + String jsonInString = ClassTypeHelper.gson.toJson(result); + oldout.println(jsonInString); + } + catch (Exception e) { + oldout.println("{\"error\": \"" + ClassTypeHelper.gson.toJson(e.toString()) + "\"}"); + System.exit(1); + } } private void launchMethod(String[] packageHandler, String payload) throws Exception { @@ -179,10 +212,6 @@ private void launchMethod(String[] packageHandler, String payload) throws Except }); Class returnType = matchedArray[0].getReturnType(); - if (!returnType.getTypeName().equals("void")) { - System.err.println(String.format("Handler can only have 'void' return type. Found '%s'.", returnType.getTypeName())); - System.exit(1); - } runMethod(matchedArray[0], payload, cls, handlerName); } diff --git a/images/node/README.md b/images/node/README.md deleted file mode 100644 index ee5afe8..0000000 --- a/images/node/README.md +++ /dev/null @@ -1,28 +0,0 @@ -Support for running nodejs Lambda functions. - -Create an image with - - docker build -t iron/node-lambda . - -This sets up a node stack and installs some deps to provide the Lambda runtime. - -Right now we use [node-lambda](https://github.com/motdotla/node-lambda) to exec -the handler and provide the function. Unfortunately it requires -a `package.json`, hence the `package.json.stupid`. Eventually we should break -the `node-lambda run` part of it out and use that. - -Running -------- - -Does not support payload (AWS Lambda 'event') yet. Expects the lambda zip file -to be called `function.zip` and available at `/mnt/function.zip`. HANDLER -should be set to `module.*export*`. So - - // fancy.js inside function.zip - exports.fancyFunction = function(event, context) {} - -would be launched as: - - docker run --rm -it -v /path/to/dir/containing/function.zip:/mnt -e HANDLER=fancy.fancyFunction iron/node-lambda - -In Lambda you'd submit these parameters in the call to `create-function`. diff --git a/images/node/bootstrap.js b/images/node/bootstrap.js index 0e41b6c..a080ed6 100644 --- a/images/node/bootstrap.js +++ b/images/node/bootstrap.js @@ -2,6 +2,9 @@ var fs = require('fs'); +var oldlog = console.log +console.log = console.error + // Some notes on the semantics of the succeed(), fail() and done() methods. // Tests are the source of truth! // First call wins in terms of deciding the result of the function. BUT, @@ -64,11 +67,10 @@ var Context = function() { result = null } - var str; var failed = false; try { - str = JSON.stringify(result) - // Succeed does not output to log, it only responds to the HTTP request. + // Output result to log + oldlog(JSON.stringify(result)); } catch(e) { // Set X-Amz-Function-Error: Unhandled header console.log("Unable to stringify body as json: " + e); @@ -108,10 +110,10 @@ var Context = function() { } else { errstr = error.toString() } - console.log(JSON.stringify({"errorMessage": errstr })) + oldlog(JSON.stringify({"errorMessage": errstr })) } catch(e) { // Set X-Amz-Function-Error: Unhandled header - console.log(errstr) + oldlog(errstr) } } @@ -223,52 +225,92 @@ var makeCtx = function() { return ctx; } +var setEnvFromHeader = function () { + var headerPrefix = "CONFIG_"; + var newEnvVars = {}; + for (var key in process.env) { + if (key.indexOf(headerPrefix) == 0) { + newEnvVars[key.slice(headerPrefix.length)] = process.env[key]; + } + } + + for (var key in newEnvVars) { + process.env[key] = newEnvVars[key]; + } +} + + function run() { + setEnvFromHeader(); // FIXME(nikhil): Check for file existence and allow non-payload. - var payload = {}; var path = process.env["PAYLOAD_FILE"]; + var stream = process.stdin; if (path) { try { - var contents = fs.readFileSync(path, { encoding: 'utf8' }); - payload = JSON.parse(contents); + stream = fs.createReadStream(path); } catch(e) { - console.error("bootstrap: Error reading payload file", e) + console.error("bootstrap: Error opening payload file", e) + process.exit(1); } } - if (process.argv.length > 2) { - var handler = process.argv[2]; - var parts = handler.split('.'); - // FIXME(nikhil): Error checking. - var script = parts[0]; - var entry = parts[1]; - var started = false; - try { - var mod = require('./'+script); - var func = mod[entry]; - if (func === undefined) { - console.log("Handler '" + entry + "' missing on module '" + script + "'"); - return; - } + var input = ""; + stream.setEncoding('utf8'); + stream.on('data', function(chunk) { + input += chunk; + }); - if (typeof func !== 'function') { - throw "TypeError: " + (typeof func) + " is not a function"; + stream.on('error', function(err) { + console.error("bootstrap: Error reading payload stream", err); + process.exit(1); + }); + + stream.on('end', function() { + var payload = {} + try { + if (input.length > 0) { + payload = JSON.parse(input); } - started = true; - mod[entry](payload, makeCtx()) } catch(e) { - if (typeof e === 'string') { - console.log(e) - } else { - console.log(e.message) - } - if (!started) { - console.log("Process exited before completing request\n") + console.error("bootstrap: Error parsing JSON", e); + process.exit(1); + } + + if (process.argv.length > 2) { + var handler = process.argv[2]; + var parts = handler.split('.'); + // FIXME(nikhil): Error checking. + var script = parts[0]; + var entry = parts[1]; + var started = false; + try { + var mod = require('./'+script); + var func = mod[entry]; + if (func === undefined) { + oldlog("Handler '" + entry + "' missing on module '" + script + "'"); + return; + } + + if (typeof func !== 'function') { + throw "TypeError: " + (typeof func) + " is not a function"; + } + started = true; + mod[entry](payload, makeCtx()) + } catch(e) { + if (typeof e === 'string') { + oldlog(e) + } else { + oldlog(e.message) + } + if (!started) { + oldlog("Process exited before completing request\n") + } } + } else { + console.error("bootstrap: No script specified") + process.exit(1); } - } else { - console.error("bootstrap: No script specified") - } + }) } run() diff --git a/images/python/Dockerfile b/images/python/Dockerfile index 8190521..fc8330e 100644 --- a/images/python/Dockerfile +++ b/images/python/Dockerfile @@ -13,8 +13,8 @@ RUN \ && apk --no-cache add py-curl \ && easy_install-2.7 "urlgrabber==3.9.1" \ \ - && easy_install-2.7 "botocore==1.4.1" \ - && easy_install-2.7 "boto3==1.2.6" \ + && easy_install-2.7 "botocore==1.4.17" \ + && easy_install-2.7 "boto3==1.3.1" \ \ && rm -rf /tmp/* diff --git a/images/python/bootstrap.py b/images/python/bootstrap.py index 449a7cd..b575caf 100644 --- a/images/python/bootstrap.py +++ b/images/python/bootstrap.py @@ -8,6 +8,8 @@ import time import uuid +oldstdout = sys.stdout +sys.stdout = sys.stderr debugging = False @@ -33,7 +35,7 @@ def __init__(self): self.memory_limit_in_mb = int(getTASK_MAXRAM() / 1024 / 1024) def get_remaining_time_in_millis(self): - remaining = plannedEnd - time.time() + remaining = plannedEnd - int(time.time()) if remaining < 0: remaining = 0 return remaining * 1000 @@ -112,8 +114,10 @@ def getPAYLOAD_FILE(): def getTASK_TIMEOUT(): - return os.environ.get('TASK_TIMEOUT') or 3600 - + try: + return int(os.environ.get('TASK_TIMEOUT', '')) + except ValueError: + return 3600 def getTASK_MAXRAM(): # IronWorker uses MAXMEM, Hybrid uses MAXRAM. @@ -190,7 +194,19 @@ def filter(self, record): } logging.config.dictConfig(loggingConfig) -plannedEnd = time.time() + getTASK_TIMEOUT() +def setEnvFromHeader(): + headerPrefix = "CONFIG_" + newEnvVars = {} + for key, value in os.environ.iteritems(): + if key.startswith(headerPrefix): + newEnvVars[key[len(headerPrefix):]] = value + + for key, value in newEnvVars.iteritems(): + os.environ[key] = value + + +setEnvFromHeader() +plannedEnd = int(time.time()) + getTASK_TIMEOUT() debugging and print ('os.environ = ', os.environ) debugging and print ('/mnt content = ', os.listdir("/mnt")) @@ -211,11 +227,6 @@ def filter(self, record): if handlerName is None: stopWithError("handlerName arg is not specified") -if payloadFileName is None: - stopWithError("PAYLOAD_FILE variable is not specified") - -if not os.path.isfile(payloadFileName): - stopWithError("No payload present") handlerParts = string.rsplit(handlerName, ".", 2) @@ -232,15 +243,20 @@ def filter(self, record): stopWithError("Function name is not defined") try: - with file(payloadFileName) as f: + if payloadFileName: + payloadFile = file(payloadFileName, 'r') + else: + payloadFile = sys.stdin + + with payloadFile as f: payload = f.read() -except: - stopWithError("Failed to read {payloadFileName}".format(payloadFileName=payloadFileName)) -debugging and print ('payload loaded') +except Exception, e: + stopWithError("Failed to read {payloadFileName}. err={err}".format(payloadFileName=(payloadFileName or ''), err=e)) try: - payload = json.loads(payload) + if len(payload) > 0: + payload = json.loads(payload) except: debugging and print ('payload is ') and print (payload) stopWithError('Payload is not JSON') @@ -260,7 +276,7 @@ def filter(self, record): try: result = caller.call(payload, context) - #FIXME where to put result in async mode? + oldstdout.write(json.dumps(result)) except Exception as e: stopWithError(e) diff --git a/test-suite/glide.lock b/test-suite/glide.lock index 0aa23fd..f8d09ac 100644 --- a/test-suite/glide.lock +++ b/test-suite/glide.lock @@ -1,8 +1,8 @@ -hash: 53e196d6ce241dae228e214717004a4ed3ce39691e8ffee195e37e76f6e9e265 -updated: 2016-03-22T10:32:17.613767945-07:00 +hash: 144bd93c4247f3246e459717b36598951283499fa842ff3bed8ad23dfea2478d +updated: 2016-10-18T11:20:43.851037791-07:00 imports: - name: github.com/aws/aws-sdk-go - version: 4da0bec8953a0a540f391930a946917b12a95671 + version: 8f2725740345a0561fa09669f5f75f905404ef8b subpackages: - aws - aws/awserr @@ -26,39 +26,54 @@ imports: - private/protocol/json/jsonutil - private/protocol/rest - name: github.com/Azure/go-ansiterm - version: 70b2c90b260171e829f1ebd7c17f600c11858dbe + version: fa152c58bc15761d0200cb75fe958b89a9d4888e subpackages: - winterm - name: github.com/docker/docker - version: 0dcdd1c5b84c4a066882ea52dd81b5b3a624897a + version: 20f81dde9bd97c86b2d0e33bbbf1388018611929 subpackages: - pkg/jsonmessage - pkg/jsonlog - pkg/term - pkg/system - pkg/term/windows +- name: github.com/docker/engine-api + version: 4290f40c056686fcaa5c9caf02eac1dde9315adf + subpackages: + - types/filters + - types/swarm - name: github.com/docker/go-units - version: 5d2041e26a699eaca682e2ea41c8f891e1060444 + version: f2145db703495b2e525c59662db69a7344b00bb8 - name: github.com/go-ini/ini - version: 12f418cc7edc5a618a51407b7ac1f1f512139df3 + version: 6e4869b434bd001f6983749881c7ead3545887d8 +- name: github.com/hashicorp/go-cleanhttp + version: ad28ea4487f05916463e2423a55166280e8254b5 - name: github.com/iron-io/iron_go3 - version: d1197926301489b15e418c49a0b17a9c42817acd + version: cd9cc95ce2d2bb25d2e4e10cd62fff1d97ad1906 subpackages: - worker - api - config +- name: github.com/iron-io/lambda + version: "" + subpackages: + - lambda - name: github.com/jmespath/go-jmespath - version: 0b12d6b521d83fc7f755e7cfc1b1fbdd35a01a74 + version: bd40a432e4c76585ef6b72d3fd96fb9b6dc7b68d +- name: github.com/opencontainers/runc + version: bf77e5976ad42b6167a167a9f3093748f7c154db + subpackages: + - libcontainer/user - name: github.com/satori/go.uuid - version: e673fdd4dea8a7334adbbe7f57b7e4b00bdc5502 + version: b061729afc07e77a8aa4fad0a2fd840958f1942a - name: github.com/sendgrid/sendgrid-go version: 0bf6332788d0230b7da84a1ae68d7531073200e1 - name: github.com/sendgrid/smtpapi-go - version: 08102729bcf72bfe74dde6c0da8e0423e94090ca + version: 072b6f477c501c30348f442559997853f12b364e - name: github.com/Sirupsen/logrus - version: 4b6ea7319e214d98c938f12692336f7ca9348d6b + version: 3ec0642a7fb6488f65b06f9040adc67e3990296a - name: golang.org/x/sys - version: 9d4e42a20653790449273b3c85e67d6d8bae6e2e + version: 002cbb5f952456d0c50e0d2aff17ea5eca716979 subpackages: - unix devImports: [] diff --git a/test-suite/glide.yaml b/test-suite/glide.yaml index 5b60b51..cfc5b3d 100644 --- a/test-suite/glide.yaml +++ b/test-suite/glide.yaml @@ -21,3 +21,11 @@ import: version: ~1.10.3 subpackages: - pkg/jsonmessage +- package: github.com/docker/engine-api + subpackages: + - types/filters + - types/swarm +- package: github.com/hashicorp/go-cleanhttp +- package: github.com/opencontainers/runc + subpackages: + - libcontainer/user diff --git a/test-suite/tests/python/test-context-values/test.py b/test-suite/tests/python/test-context-values/test.py index ca48a88..f44b5e8 100644 --- a/test-suite/tests/python/test-context-values/test.py +++ b/test-suite/tests/python/test-context-values/test.py @@ -5,7 +5,7 @@ def run(event, context): print("Function name is set:", not (context.function_name is None)) print("Function version is set:", not (context.function_version is None)) - print("Memory limit in MB (grater or equal than 100):", context.memory_limit_in_mb > 100) + print("Memory limit in MB is positive:", context.memory_limit_in_mb > 0) print("AWS request ID is set:", not (context.aws_request_id is None)) remaining1 = context.get_remaining_time_in_millis()