cargo watch/invoke: hangs when registering internal extensions from inside function binary #848
Description
Currently this aws-lambda-rust-runtime example doesn't work with cargo lambda watch:
https://github.com/awslabs/aws-lambda-rust-runtime/blob/main/examples/extension-internal-flush/src/main.rs
Locally, the example hangs when it attempts to register an internal extension. In AWS, the registration resolves, and the extension begins receiving events once run.
Reproduction
git clone https://github.com/awslabs/aws-lambda-rust-runtime.git
cd aws-lambda-rust-runtime/examples/extension-internal-flush
cargo lambda watch -v
Outputs:
DEBUG connecting to [::]:9000
DEBUG connected to [::]:9000
DEBUG http invocation received path=/.rt/_/2020-01-01/extension/register method=POST
DEBUG request stack increased function_name="_"
And then cargo lambda invokes show up in the orchestrator logs but aren't processed.
> cargo lambda invoke --data-ascii '{}'
DEBUG invocation received function_name=_
DEBUG request stack increased function_name="_"
The function runs fine when invoked in AWS:
cargo lambda build --release
cargo lambda deploy --release
cargo lambda invoke extension-internal-flush --remote --data-ascii '{"Records":[{"messageId":"MessageID_1","receiptHandle":"MessageReceiptHandle","body":"{\"a\":\"Test\",\"b\":123}"}]}'
Outputs:
{"batchItemFailures":[]}
(and the AWS logs do show the extension running):
EXTENSION Name: internal-flush State: Ready Events: [INVOKE]
START RequestId: a6bd9cf2-79f9-4301-906c-37d70c19fc7f Version: $LATEST
[runtime] received event Data { a: "Test", b: 123 }
[extension] waiting for event to be processed
[extension] flushing logs and telemetry
END RequestId: a6bd9cf2-79f9-4301-906c-37d70c19fc7f
REPORT RequestId: a6bd9cf2-79f9-4301-906c-37d70c19fc7f Duration: 35.16 ms Billed Duration: 36 ms Memory Size: 128 MB Max Memory Used: 13 MB
Meanwhile a successful extension registration looks like this on the extension process (run with cargo run -v):
DEBUG connecting to [::]:9000
DEBUG connected to [::]:9000
DEBUG pooling idle connection for ("http", [::]:9000)
DEBUG connecting to [::]:9000
DEBUG connected to [::]:9000
And this on the process running cargo lambda watch
DEBUG connecting to [::]:9000
DEBUG connected to [::]:9000
DEBUG registering extension payload=EventsRequest { events: ["INVOKE", "SHUTDOWN"] }
DEBUG extension waiting for next event extension_id=f5ca0c6c-2fff-4812-b88f-fbd393b2f936