Solvedserverless chrome Reliability on lambda

First of all, thanks @adieuadieu for your work on this project. It's been really helpful to me!

I'm opening this ticket in hopes that we can have some discussion around the reliability of launching & connecting to chrome on lambda. I'm working with the latest packages from the develop branch, so I realize/acknowledge that there may be bugs.

Really what I'm hoping to figure out is if the problems I'm having are specific to me, or if there are general problems that are known and need work. If we can identify what those problems are, perhaps we, the community, can help!

I can get real specific and share application code & backtraces, but before we get to that, perhaps we can establish if this behavior is common. If I'm the only one experiencing these problems, then this ticket should be handled differently (as a cry for help!).

  1. The first few invocations of my function typically work as expected (hurray!).
  2. If I invoke the function multiple times in a row (quickly), the service begins to fail with ECONNREFUSED. Here's an example from the log:

https://gist.github.com/benjaminwood/21dbdb95cbd140dae67460f200cbf7ec

It is common that after it fails like this, all additional invocations fail the same way until a new container is spun up (with time or by deploying new code). It appears that a chrome process is running, but it may be hung/frozen.

For your reference, my function cleans up after itself like this:

await CDP.Close({id: client.target.id})
await client.close()

I'm not killing the chrome process, but I am closing the tab and the connection (client). I believe this is the correct thing to do, but feel free to give input, I'd appreciate it!

So, is this happening to anyone else? I did try running some of the examples here in the repo and ran into the same problems. So at first review, it appears at though the problem is not 100% specific to my situation.

52 Answers

โœ”๏ธAccepted Answer

I am using the latest version 1.0.0.34 and got this error, sometimes after spawning new headless Chrome on AWS Lambda, we got this errors:

2018-01-23T04:30:46.009Z	@serverless-chrome/lambda: Spawning headless shell
2018-01-23T04:30:46.021Z	        @serverless-chrome/lambda: ChromeLauncher No debugging port found on port 9222, launching a new Chrome.
2018-01-23T04:30:46.049Z	@serverless-chrome/lambda: Launcher Chrome running with pid 13 on port 9222.
2018-01-23T04:30:46.050Z	@serverless-chrome/lambda: Waiting for Chrome 0
2018-01-23T04:30:46.557Z	@serverless-chrome/lambda: Waiting for Chrome 1
2018-01-23T04:30:46.565Z	@serverless-chrome/lambda: Started Chrome
2018-01-23T04:30:46.566Z	@serverless-chrome/lambda: It took 557ms to spawn chrome.
2018-01-23T04:30:56.750Z	        (node:1) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 1): Error: socket hang up
END RequestId: 2d503be3-fff6-11e7-b7aa-23030585cee8

The error was:

Error: socket hang up

Does anyone know how to fix this? We tried to chrome.kill() for each request but it still doesn't resolve the problem

Thanks a lot :)

Other Answers:

I can echo this issue, and will start another discussion on headless-dev tomorrow morning.

This morning I spent some time playing around with Headless Chrome target APIs, hoping that maybe tab management was the issue. Doesn't seem to be the case, as we can see at https://github.com/momer/chrome-headless-lambda-rl/blob/no-zombielord/src/requestLogger.ts

:(

Hopefully this or next week I'll have some time to do some profiling on the lambda container (if that's possible?) to further dig in.

Thanks for following up @adieuadieu. I agree that the first step is to establish a pattern that causes it to fail. I've been able to get it to fail pretty consistently calling the function multiple times in a row. I'll do some more tests next week and will report back here.

I'm hoping to use headless chrome on lambda for a production application, so figuring this out is important to me and I'm happy to help however I can!

@quangbuule no I haven't yet directly contacted AWS about it. I feel like before I do, I need to put together a very succinct test case and I just haven't had the time to do that yet.

I'm about ready to throw in the towel on getting Chrome to play nice with the Linux Kernel Freezer Subsystem on Lambda.

I do at least have one thing to show for it: a new headless_shell build based on the latest Chrome Master code. I've tested it against the Alexa Top 500 websites and it works at least as well as previous builds, and might produce better screenshots/crash less since it includes a number of additional font-related dependencies. It also now supports connecting to the remote debugger via fd socket, which I'm not sure was available in the previous build. YMMV.

https://github.com/neekolas/chromeless-testbed/releases/tag/0.0.1
/cc @adieuadieu

Related Issues:

81
serverless chrome NSS_VersionCheck("3.26") failed
I have done to fix this in Ubuntu 16.04 by reinstalling libnss3 Hello I get the below at runtime whe...
11
serverless chrome Unable to start Chrome in NodeJS10.x
Why closing this without any solution ? I an running into the error Unable to start Chrome when runn...
5
serverless chrome Reliability on lambda
I am using the latest version 1.0.0.34 and got this error sometimes after spawning new headless Chro...
480
aws cli Invalid Syntax Error when running any AWS command.
Leaving this here in case someone comes across this via google like me: I had the same issue with th...
328
amplify js Uncaught ReferenceError: global is not defined in latest Angular 6 RC
Just for reference I have passed through this issue with adding these lines on my index.html head: ...
219
aws cli aws s3api create-bucket throws error for us-east-2
For those of you looking to create a bucket via the CLI run this: aws s3api create-bucket --bucket d...
210
aws sdk js Typescript error: Cannot find name 'Buffer'/'http'/'https'
@pvamshi So I was able to reproduce your issue Hi! First I want to thank you for this SDK ...
171
aws cli aws ssm put-parameter performs an HTTP GET request when the value param is an url
Just kill this 'feature' seriously Even apart from security concerns it certainly violates the princ...
151
serverless How do I enable CORS?
@lakinducker Thanks No problem! I updated your comment with the corresponding markdown and now the i...
136
terragrunt Upgrading to Terraform 0.12: separate configuration file for Terragrunt?
Hi all Hi! I'm one of the engineers at HashiCorp who works on Terraform Core As you might be aware ...
122
amplify cli Many-To-Many
You can implement many to many yourself using two 1-M @connections and a joining @model ...
120
aws cli How to describe instances from all regions?
For people arriving here from Google here's one way to list all your instances across all regions wi...
114
amplify js fetch is not defined
nodejs fix: I'm using amazon-cognito-auth-js with my express app and I'm following the case 1 exampl...
109
terraform aws eks Error: Post "http://localhost/api/v1/namespaces/kube-system/configmaps": dial tcp 127.0.0.1:80: connect: connection refused
this fixed it for me thanks @cidesaasoptics I am started getting this issue: All my code were workin...
106
amplify js Error: No credentials, applicationId or region
I had the same issue (running on the latest Amplify v3) and worked around it by changing the followi...
104
amplify js Is it possible to get cognito user attributes in Lambda/cloud logic ?
I have been looking around for a while I feel the answers here didn't really answer the problem ...
96
sops Cannot decrypt with GPG 2.2.5 and SOPS 3.0.0
The problem suddenly re-occured.. I think it has to do with the gpg-agent For the moment this solved...
93
serverless Error: spawn java ENOENT
Have you tried running sls dynamodb install? This downloads the DynamoDb libs you need. ...
88
serverless Schedule event not created when supplying options
Sorry this was user error and I actually just had the wrong indentation Correct indentation ...
83
amplify cli aws-exports.js is not generated
Even after the third read I find it utterly confusing and I have usability issues too ...
83
amplify js Auth Error: Amplify has not been configured correctly using Nuxt.js
I 'm having the same issue in aws-amplify: ^3.0.11 I found out Auth module didn't load configs of aw...
77
kubespray After the certificate expires how use kubespray to renew certificate
@kerOssinas you are right the upgrade-cluster.yml of Kubespray will also rotate the certificates ...
74
terraform aws vpc Terraform 0.12 + vpc module v2.2 (Inappropriate value for attribute "subnet_ids": element 0: string required.)
@sonianara You can always unwrap value between [ and ] to make it like this: I am trying to upgrade ...
70
amplify js How to refresh Cognito tokens
It will refresh if you call the SDK for it e.g. with Auth.currentSession() and it finds an expired t...
69
aws cdk TS - Argument of type 'this' is not assignable to parameter of type 'Construct'
Hi @benswinburne I have a created a simple Stack that creates a VPC with three simple subnets ...
66
serverless Serverless using AWS profiles only half working
You can use AWS Profiles with Severless including IAM cross-account role assumption ...
62
serverless Narrowing the Serverless IAM Deployment Policy
I think I have the serverless deployment policy nailed at this point A bit more testing is in order ...
60
aws cli Can't install / configure aws cli
Interesting It looks like it's failing to parse ~/.aws/config Can you try putting in fake values? He...
59
aws sam cli Error when installing python version on MACOS
Try with --user flag? Description: I got an error when trying to install the new version on MacOS St...
59
aws iam authenticator QUESTION: How to create the k8s users and groups to map to?
I got it to work! For anyone else finding this issue here's what I did In my configmap for aws-iam-a...
57
containers roadmap [EKS] [request]: Remove requirement of public IPs on EKS managed worker nodes
Yea thats the part in your documentation I was surprised by Community Note Please vote on this issue...
57
terraform provider aws Error: "policy" contains an invalid JSON: invalid character '}' looking for beginning of object key string
@JayMaree I think your error is here: I had the same issue which brought me here but mine was just a...
56
amplify cli @auth public/private IAM roles and other Providers
ok my bad was actually quite easy just do : and add a auth provider in my case was IAM ...
55
amplify js Sign up multiple different accounts with the same email
The pre-signup trigger can be used to prevent the new signup from being created when there's an exis...
54
terraform provider aws Creating aws_elasticsearch_domain can't be done due to absence of AWSServiceRoleForAmazonElasticsearchService role
You can just add this ressource before creating your domain: This will create the needed role for ES...
52
amplify js Amplify Console 200 (Rewrite) fails on SPA React (Router) Application
This worked for me source: </^((?!.(css|gif|ico|jpg|js|png|txt|svg|woff|ttf)$).)*$/> target address:...
51
amplify js aws-amplify 0.3.0: "Uncaught ReferenceError: require is not defined" when packaged with webpack
Ok I found something that helped Graphql-js uses .mjs as file extension which caused issues with the...
51
aws cli Code Deploy - Unhandled exception - ZIP does not support timestamps before 1980
eb deploy gave me find -mtime +10950 -print -exec touch {} \; solved the problem. ...
50
aws cli PyYAML requires python-dev dependency
We have had some success here by running Hi It seems the latest release of awscli requires PyYAML as...
50
terraform provider aws Feature Request: WAFv2 Web ACL Resource
@briensherman @shadbi I'm currently busy implementing the resources already did #12119 #12284 and I'...
47
aws cli aws ecr get-login error: argument operation: Invalid choice
If you still want to use cli (maybe for automation) this is the full command according to the offici...
47
aws iam authenticator error: You must be logged in to the server (Unauthorized) -- same IAM user created cluster
You need to map IAM users or roles into the cluster using the aws-auth ConfigMap This is done automa...
46
aws load balancer controller Create option to reuse an existing ALB instead of creating a new ALB per Ingress
I've created another ingress controller that combines multiple ingress resources into a new one => h...
46
serverless Ignore check for unchanged files after failed deployment
Have you tried to use sls deploy --force @tom10271 ? After a failed deployment due to remote (CloudF...
45
amplify js Identity providers authentication against User Pools WITHOUT hosted UI
@martimarkov we find a solution for you to use the customized button to do that ...
45
serverless application model cloudformation deploy CLI exits with return code 255 if stack exists
Adding --no-fail-on-empty-changeset seems to work i.e. allow bash script execution to continue even ...
45
terraform provider aws Support AWS CLI v2 AWS Single Sign-On
Support for AWS Single-Sign On (SSO) cached credentials has been merged and will release with versio...
44
amplify cli jest-haste-map: Haste module naming collision: -> namefunction <-
For React Native 0.6x configure the blacklist in metro.config.js instead of rn-cli.config.js as per ...
44
amplify js Getting "no current user" after successful login to Cognito UserPool
I had the same problem but for me removing the cookie storage configuration in aws-exports.js solved...
43
amplify js How to add user to Group
๐Ÿ‘† Also we have achieved this using the Post Confirmation Lambda trigger Very simplified from our im...