SolvedZappa Error loading psycopg2 module on Python 3.6 (when on Lambda)

Context

Shiny new Zappa 0.41 and Python 3.6

Django==1.11
djangorestframework==3.6.2
psycopg2==2.6.1
zappa==0.41.0
...

Expected Behavior

It works fine locally, but on Lambda the following happens:

[1492873312435] File "/var/task/django_zappa_app.py", line 20, in get_django_wsgi
[1492873312435] return get_wsgi_application()
[1492873312435] File "/var/task/django/core/wsgi.py", line 13, in get_wsgi_application
[1492873312435] django.setup(set_prefix=False)
[1492873312435] File "/var/task/django/__init__.py", line 27, in setup
[1492873312435] apps.populate(settings.INSTALLED_APPS)
[1492873312435] File "/var/task/django/apps/registry.py", line 108, in populate
[1492873312435] app_config.import_models()
[1492873312435] File "/var/task/django/apps/config.py", line 202, in import_models
[1492873312436] self.models_module = import_module(models_module_name)
[1492873312436] File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
[1492873312436] return _bootstrap._gcd_import(name[level:], package, level)
[1492873312436] File "<frozen importlib._bootstrap>", line 978, in _gcd_import
[1492873312436] File "<frozen importlib._bootstrap>", line 961, in _find_and_load
[1492873312436] File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
[1492873312436] File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
[1492873312436] File "<frozen importlib._bootstrap_external>", line 678, in exec_module
[1492873312436] File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
[1492873312436] File "/var/task/django/contrib/auth/models.py", line 4, in <module>
[1492873312436] from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
[1492873312436] File "/var/task/django/contrib/auth/base_user.py", line 52, in <module>
[1492873312436] class AbstractBaseUser(models.Model):
[1492873312436] File "/var/task/django/db/models/base.py", line 124, in __new__
[1492873312436] new_class.add_to_class('_meta', Options(meta, app_label))
[1492873312436] File "/var/task/django/db/models/base.py", line 330, in add_to_class
[1492873312437] value.contribute_to_class(cls, name)
[1492873312437] File "/var/task/django/db/models/options.py", line 214, in contribute_to_class
[1492873312437] self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
[1492873312437] File "/var/task/django/db/__init__.py", line 33, in __getattr__
[1492873312437] return getattr(connections[DEFAULT_DB_ALIAS], item)
[1492873312437] File "/var/task/django/db/utils.py", line 211, in __getitem__
[1492873312437] backend = load_backend(db['ENGINE'])
[1492873312437] File "/var/task/django/db/utils.py", line 115, in load_backend
[1492873312437] return import_module('%s.base' % backend_name)
[1492873312437] File "/var/lang/lib/python3.6/importlib/__init__.py", line 126, in import_module
[1492873312437] return _bootstrap._gcd_import(name[level:], package, level)
[1492873312437] File "/var/task/django/db/backends/postgresql/base.py", line 25, in <module>
[1492873312437] raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
[1492873312437] django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dynamic module does not define module export function (PyInit__psycopg)
[1492873312437] During handling of the above exception, another exception occurred:
[1492873312437] Traceback (most recent call last):
[1492873312437] File "/var/runtime/awslambda/bootstrap.py", line 466, in <module>
[1492873312437] main()
[1492873312437] File "/var/runtime/awslambda/bootstrap.py", line 462, in main
[1492873312438] handle_event_request(request_handler, invokeid, event_body, context_objs, invoked_function_arn)
[1492873312438] File "/var/runtime/awslambda/bootstrap.py", line 248, in handle_event_request
[1492873312438] result = to_json(result)
[1492873312438] File "/var/runtime/awslambda/bootstrap.py", line 224, in to_json
[1492873312438] return json.dumps(obj, default=decimal_serializer)
[1492873312438] File "/var/lang/lib/python3.6/json/__init__.py", line 238, in dumps
[1492873312438] **kw).encode(obj)
[1492873312438] File "/var/lang/lib/python3.6/json/encoder.py", line 199, in encode
[1492873312438] chunks = self.iterencode(o, _one_shot=True)
[1492873312438] File "/var/lang/lib/python3.6/json/encoder.py", line 257, in iterencode
[1492873312438] return _iterencode(o, 0)
[1492873312438] File "/var/runtime/awslambda/bootstrap.py", line 42, in decimal_serializer
[1492873312439] raise TypeError(repr(o) + " is not JSON serializable")
[1492873312439] TypeError: <FrameSummary file /var/task/handler.py, line 514 in lambda_handler> is not JSON serializable
[1492873312598] Error loading psycopg2 module: dynamic module does not define module export function (PyInit__psycopg)
ImproperlyConfigured

Possible Fix

No idea yet, but before I dig further, I was wondering if someone had any pointers. I assume its something to do with pre-compiled C-extension packages not working on Python 3.6 yet.

31 Answers

✔️Accepted Answer

I solved the issue by installing psycopg2-binary

Other Answers:

psycopg2-binary worked for me.

i was having this issue:

RuntimeError: populate() isn't reentrant

#324

thank you!

pipenv install psycopg2-binary also solved the issue for me, but I have 2 questions.

  1. Is using psycopg2-binary suitable for a production environment? Seems like compiled code would be a good thing, but the docs at https://pypi.org/project/psycopg2-binary/ currently read "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources."

  2. If it is necessary for Django to work in a lambda using Zappa, should psycopg2-binary be added (or verified) as a dependency in the zappa init command. That would've saved me time tracking this down.

I'm still getting No module named 'psycopg2._psycopg' with zappa==0.41.2 and psycopg2==2.7.1

Related Issues:

373
Zappa bad magic number in 'application': b'\x03\xf3\r\n': ImportError
I was able to fix this too by doing find -name \*.pyc -delete Mine definitely wasn't caused by PY2 -...
94
Zappa AttributeError: 'module' has no attribute 'get_installed_distributions'
You probably have a newer pip they made changes to their public api top level functions recently Try...
13
Zappa Error loading psycopg2 module on Python 3.6 (when on Lambda)
I solved the issue by installing psycopg2-binary Context Shiny new Zappa 0.41 and Python 3.6 Django=...
59
aws sam cli Error when installing python version on MACOS
Try with --user flag? Description: I got an error when trying to install the new version on MacOS St...
27
aws sam cli AttributeError in arn_generator.py
I am a new user following the hello world tutorial linked from the README and hit this issue I don't...
26
aws sam cli fork/exec /var/task/main: permission denied
Not sure if this is the case here I successfully deployed a golang lambda api service using sam pack...
24
aws sam cli AWS::Serverless::SimpleTable not Created in DynamoDB Local
@PaulMaddox wrote: Sorry for the confusion SAM Local currently only does anything with AWS::Serverle...
23
aws sam cli sam build fails when required to install some 3rd party libraries in requirements.txt
For those that don't want to run the build inside a container: pip install wheel fixed it for me wit...
22
kong how to let kong start automatically
you can use systemd for Control kong service First create kong.service file : and then put this line...
19
serverless express [Question] Internal server error when request method OPTIONS
With the help from staff on AWS forums I've resolved it by adding: which now returns a correct 200 r...
16
aws sam cli sam init errors with "No module named functools_lru_cache"
I had the same problem today Resolved be uninstalling and installing backports.functools_lru_cache ...
15
aws sam cli Local API Gateway responds with Internal Server error
Your timeout looks 3s try to increase in your template. Hi ...
14
spring cloud gateway Spring-cloud-gateway application not starting up.
Gateway is not compatible with org.springframework.boot:spring-boot-starter-web ...
14
spring cloud gateway io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:
You usually see this when you have a http vs https mismatch I encountered this problem when my ssl r...
14
serverless express Async handler doesn't work on node 8.10
PR for this here #173 Note that I plan on improving the interface in a future breaking change (aws-s...
13
emissary Dev Portal attempts to find docs for every mapping by default
+1 for this ticket I would also like to disable this feature as it is generating large numbers of lo...
13
spring cloud gateway Cors Pre Flight Request
To version pre to 2.1.3.RELEASE this works for me: Spring Cloud Version: Greenwich.RELEASE To make m...
12
serverless next.js Started getting the Error: Cannot find module '@sls-next/next-aws-cloudfront'
Hi folks please pin the version number in your serverless.yml like it states in the README For examp...
12
serverless express Can't get the API Gateway event object
In case any other lost souls who are encountering Missing x-apigateway-event or x-apigateway-context...
11
spring cloud gateway Doubled CORS headers after upgrade to Greenwich
This works for me No need to add any bean. Some of the legacy back ends behind our gateway have thei...
10
spring cloud gateway Preflight Request
@hnxuruochen Just did a workaroundd below to set RoutePredicateHandlerMapping CORS configuration ...
9
aws sam cli sam local: --env-vars parameter does not work with "start-api" subcommand
If it helps anyone I found out that by setting empty environment variables in the template.yml ...
7
serverless next.js Using webpack 5 causes 503
Same for 10.2.3 Using webpack 5 returns the following from Cloudfront: Using versions: ...
6
serverless next.js Error: Source and destination must not be the same. When using useServerlessTraceTarget
I was also facing the same issue (getting Error: Source and destination must not be the same) and sp...
4
kong request-transformer plugin fails if multipart/form-data contains file
@arvileino that's really valuable Summary When request-transformer plugin enabled multipart/form-dat...
4
spring cloud gateway How to modify request / response body?
I didn't get how to change the response body I have a json in the body and I want to add a property ...
4
Zappa Error when deploying AWS Lambda with Zappa package
I seems that https://pypi.org/project/troposphere/ causes the problem: release 3.0.0 does not instal...
3
aws sam cli Conditionals, Refs, etc. in template yaml don't appear to be evaluated
Should this caveat be mentioned in the documentation? It took a lot of searching to hunt down this c...
3
serverless next.js 503 when using image optimization
It should be fixed now in@sls-next/serverless-component@1.19.0-alpha.19 after also include 7256ab9 W...
151
serverless How do I enable CORS?
@lakinducker Thanks No problem! I updated your comment with the corresponding markdown and now the i...
122
amplify cli Many-To-Many
You can implement many to many yourself using two 1-M @connections and a joining @model ...
93
serverless Error: spawn java ENOENT
Have you tried running sls dynamodb install? This downloads the DynamoDb libs you need. ...
88
serverless Schedule event not created when supplying options
Sorry this was user error and I actually just had the wrong indentation Correct indentation ...
83
amplify cli aws-exports.js is not generated
Even after the third read I find it utterly confusing and I have usability issues too ...
81
serverless chrome NSS_VersionCheck("3.26") failed
I have done to fix this in Ubuntu 16.04 by reinstalling libnss3 Hello I get the below at runtime whe...
66
serverless Serverless using AWS profiles only half working
You can use AWS Profiles with Severless including IAM cross-account role assumption ...
62
serverless Narrowing the Serverless IAM Deployment Policy
I think I have the serverless deployment policy nailed at this point A bit more testing is in order ...
56
amplify cli @auth public/private IAM roles and other Providers
ok my bad was actually quite easy just do : and add a auth provider in my case was IAM ...
46
serverless Ignore check for unchanged files after failed deployment
Have you tried to use sls deploy --force @tom10271 ? After a failed deployment due to remote (CloudF...
44
amplify cli jest-haste-map: Haste module naming collision: -> namefunction <-
For React Native 0.6x configure the blacklist in metro.config.js instead of rn-cli.config.js as per ...
42
serverless Serverless Error - Function not found...
I had the same problem after killing a running sls deploy @mrosett's workaround worked for me as wel...
40
serverless Cannot deploy/remove stack because S3 bucket is gone
Here is a workaround that's been working for me without needing --force: Run sls deploy to generate ...
40
serverless Allow explicitly specifying serverless.yml file in CLI options or env var
We should have a command like sls deploy --file serverless-foo.yml This is a (Feature Proposal) Desc...
40
serverless Unable to deploy to Serverless due to 'empty zip'
Yeah Node version was the problem Here's a quick script that I made if anyone needs it: ...
38
serverless AWS Now Supports Max Retry Attempts
Although the async config can be added as a custom resource I would argue that limiting the retry at...
33
amplify cli runtime parameter of nodejs8.10 is no longer supported
@asyschikov where can you find the build settings for amplify CLI that you are refering to ? For tho...
33
serverless FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
Had similar issues with largish typescript projects and serverless-webpack-plugin Try export NODE_OP...
30
amplify cli Connect a cognito user with an AppSync type
Any updates on this? Having to manually write a Lambda triggered by Cognito events to create an AppS...
29
serverless Access Outputs from CF template in serverless.yml
RE: cross-stack references.. Hi y'all Trying to transition to v1 and porting my CF template too I am...