Solvedakka Akka Http: Nginx proxy error when returning large json payloads

As mentioned in this user group post:!topic/akka-user/aDwaI8Xia74

I'm running into an error using akka http with nginx in front of it as a reverse proxy when I return large (MBs) json payloads. The error I see in the nginx log is:

upstream prematurely closed connection while reading upstream

I created a very simple route that can reproduce this issue. That code is as follows:

  pathPrefix("api" / "proxytest"){
    (get & parameters('[Int], '[Boolean])){ (aNum, chunk) =>  
      val bunchOfA = (for(i <- 1 to aNum) yield "a").mkString
      val json = s"""{"test": "$bunchOfA"}"""

      val entity = 
        if (chunk){
          val chunks = Source.single(json).mapConcat(_.grouped(1024*10).toList).map(HttpEntity.Chunk.apply(_))   
          HttpEntity.Chunked(ContentTypes.`application/json`, chunks)
          HttpEntity(ContentTypes.`application/json`, json )

      val response = HttpResponse(StatusCodes.OK, entity = entity)

If I test this route with a get request passing in numberOfA as something small, say 500, and chunk as false everything works when I go through nginx. But if I pass in something large, like 2100000 with chunk set to false, I get that nginx error. If I pass in chunk=true, everything works. Also, if I skip nginx and go right to my server, everything works regardless of chunking and size. I also noticed that I always get this dead letter warning when I hit the nginx error:

[INFO] [01/20/2016 09:42:02.626] [] [akka://default/system/IO-TCP/selectors/$a/1] Message [$ChannelWritable$] from Actor[akka://default/deadLetters] to Actor[akka://default/system/IO-TCP/selectors/$a/1#-229858999] was not delivered. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.

This seems to imply that maybe the connection is closed prematurely in akka http before nginx thinks it's done with it.

I have also attached my nginx config (version 1.6.2). Let me know if you need anything more from me.

22 Answers

โœ”๏ธAccepted Answer

OK, I believe that it is a bug in 'akka X'. As far as I can tell, if a client sends header Connection: close, akka pushes data to tcp channel and promptly closes the connection (without waiting for ACKs and such).

We've found a work-around using nginx. Basically, you have to force HTTP 1.1 with Connection: keep-alive. In order to do so, you need to:

  upstream your-service {
    keepalive 100;
    server akka-http-box:port;

# (...)

  location /your/location/ {
    proxy_pass       http://your-service/;
    proxy_http_version 1.1;
    proxy_set_header Connection ""; # important bit: without it nginx will send "Connection: close" anyway

Hope that helps someone out there...

Related Issues:

akka Akka Http: Nginx proxy error when returning large json payloads
OK I believe that it is a bug in 'akka X' As far as I can tell if a client sends header Connection: ...
spring cloud netflix Eureka client shutting down unexpectedly after registering with Eureka server
I got the same problem I resolved it by adding this dependency: My service app-name shuts down unexp...
rook ceph: Dashboard throws errors
the only workaround i've managed to come up with is to create a new copy of the built-in administrat...
singularity /bin/bash: warning: setlocale: LC_ALL: cannot change locale (en_US.UTF-8)
This worked for me: Enjoy. Hi The wrapper that starts my app tries to perform some setlocale command...
argo workflows Argo *namespace-install* raises failed to save outputs: Failed to establish pod watch: unknown (get pods)
This is a great answer man Checklist: [ X ] I've included the version [ X ] I've included reproducti...
rook docs: Clarify removing the cluster and finalizer
For those who still stumble on this and get the list element types are not identical: [[customresour...
spring cloud config Datasource Issue with Spring Boot 2.1.x
Spring Boot 2.1.x disables the overrides of this type of bean by default I have a very simple spring...
buildkit Cache manifest lists can't be exported to gcr
Any update on this? I can't --export-cache type=registry,ref= to GCR neither mode=min nor max and ne...
spring cloud netflix Suggest supporting POJO as request parameter
I use RequestInterceptor to solve this problem: From @wanghongfei on August 12 2016 4:14 I'm using F...
kong how to let kong start automatically
you can use systemd for Control kong service First create kong.service file : and then put this line...
buildkit Build fails when FROM image isn't cached
For those still having issues with this we configured a global env var: DOCKER_BUILDKIT=0 as tempora...
quarkus Kotlin Coroutines in Quarkus
@heubeck Actually it would be even better if suspendable functions would supported in RestEasy React...
spring cloud netflix zuul proxy file upload, file name is Chinese garbled
file upload is supported better through the zuul servlet directly rather than the spring mvc control...
spring cloud netflix Connection refused while trying to register eureka-clients
After days looking thousand times on the same config file I realized the issue At the end it was jus...
rook CephFS volumes handling via PVCs (dynamic provisioning)
So just to clear something up: rook/cephfs works just fine via PVCs and has done so for approximatel...
emissary Dev Portal attempts to find docs for every mapping by default
+1 for this ticket I would also like to disable this feature as it is generating large numbers of lo...
linkerd HTTP/2 support
I'm happy to report that we have a variant of finagle-http2 that is able to transmit gRPC messages! ...
spring cloud netflix Feign Logging Configuration Not Working As Documented
@fabritma @pluttrell @khannedy Here goes the sample Based on the Spring Cloud documentation for Feig...
spring cloud netflix Zuul, Ribbon and Hystrix timeout confusion
Take a look at this section in the documentation
milvus Support collection rename
Hi sorry for the late reply Here is my rough design Any suggestions or feedbacks are welcome ...
buildkit ssh agent only forwarded from keys on Windows
Same problem here with a slightly newer Windows and Git Bash version Please note that it works when ...
rook Very high CPU usage on Ceph OSDs (v1.0, v1.1)
Feb 4 2020: Fixed the description based on the comments in #4802 I got an answer from a Ceph kernel ...
rook How can I resize pv created by ""
Login to rook toolbox pod run rbd resize <pv_id> --size=<size f.e 1T> Then either go to a node where...
lens Not able to load information for one cluster
Exactly the same as @dq-vizy On mac I deleted this repo: ~/Library/Application Support/Lens and the ...
micrometer LoggerContext changes when using Spring Cloud
In the end this is what I managed to get working: Hi ...
quarkus CORS not working correctly
I'm seeing this error on 0.22 I can't upgrade right now but if you get here through Google ...
kong request-transformer plugin fails if multipart/form-data contains file
@arvileino that's really valuable Summary When request-transformer plugin enabled multipart/form-dat...
argo workflows Default Archive Location Error
Configuration:\nartifactRepository: {}\nfeatureFlags: {}\nmetricsConfig:\n disableLegacy: false\npod...
buildkit Support custom DNS from dockerd config
I'm also experiencing this issue in a corporate environment Hi i think that I stumbled upon a proble...
spring cloud sleuth Spring Cloud Gateway and Sleuth / Zipkin integration
We ran into the same issue with Greenwich.M2 Can also confirm on Greenwich.M3 We used
spring cloud sleuth Zipkin Auto-Configuration Throws TimeoutException
We've also been running into this issue I am using .. Spring Boot: 2.2.0.RELEASE Spring Cloud: Hoxto...
falco Failure to find a BPF probe on Container Optimized OS
@caquino If you run this on your GKE cluster then let me know that it solves your problem I believe ...
openebs zrepl crashing on arm64
I built my own cstor-pool images just now I couldn't run your normal compilation scripts (e.g build_...
rook rook-ceph-osd Init:CrashLoopBackOff after ungraceful K8s node restart
vgchange -ay on the broken k8s node lets the rook-ceph-osd-2 pod start properly and ceph status repo...
rook Dashboard 500. Rook v1.3.0, Ceph 15.2.0
The minimal health API of the Dashboard gets iSCSI services from orchestrator which eventually invok...
spring cloud netflix Document working with multiple zones in eureka
thanks i google and read the doc and can't find some detailed description or example for config eure...
bloc Close instances of dart.core.Sink.
Hi @zeucxb ๐Ÿ‘‹ Thanks for opening an issue! This is a false positive in the dart linter dart-lang/lin...
bloc Doc request: How to use bloc across pages?
If you want your Bloc to be accessible globally (across different routes) you need to make sure that...
swoole src Compile Error on Mac Big Sur with PHP 8
I can get it work by manually symlink the required file. Please answer these questions before submit...
swoole src Swoole's admin interface hot-loads code from a third-party server ?
At this stage all the member's releases have to use matyhtf's PECL account which is ridiculous for a...
bloc [Proposal] Replace mapEventToState with on<Event> in Bloc
What would an example that includes some basic async processing look like? It could look something l...
RxGo RxGo v2
Hey My opinion about what should be part of RxGo v2 General Iterable should be moved to an interface...
bloc [Question] How would you manage animation in the bloc builder?
@felangel @basketball-ico Hello I want to use Navigator too when AuthenticationState changed I solve...
bloc Bad state: Cannot add new events after calling close after the bloc is disposed.
Hello Felangel how are you? Suggest that in a next version make an adjustment in the class Bolc In t...
ava Allow tests files to have any extension (i.e. .jsx, .ts)
I think what @sindresorhus is getting at is the following With the latest release(0.13) and the abil...
ava Only exclude helpers directory when inside a test directory
I feel like we're talking past each other Any chance the exclude rule for helpers can be relaxed so ...
ava Flow type definition
Okay so I was planning on just playing around with it but you nerd sniped me and I ended up doing th...
bloc Bad state: Cannot add new events after calling close
@rrifafauzikomara you need to remove final bloc = MealsDetailBloc(); from your detail bloc and inste...
bloc RefreshIndicator with bloc example
I created a Flutter Weather app and used the RefreshIndicator to address your request You can check ...
vibe.d undefined reference to methods in ssl
Also try to install libssl1.0-dev - this solves same problem for me Good day I'm trying to generate ...