Solvedaiohttp ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2605)

The following very simple aiohttp client:

#!/usr/bin/env python3

import aiohttp
import asyncio

async def fetch(session, url):
    async with session.get(url) as response:
        print("%s launched" % url)
        return response

async def main():
    async with aiohttp.ClientSession() as session:
        python = await fetch(session, 'https://python.org')
        print("Python: %s" % python.status)
        
loop = asyncio.get_event_loop()
loop.run_until_complete(main())

produces the following exception:

https://python.org launched
Python: 200
SSL error in data received
protocol: <asyncio.sslproto.SSLProtocol object at 0x7fdec8d42208>
transport: <_SelectorSocketTransport fd=8 read=polling write=<idle, bufsize=0>>
Traceback (most recent call last):
  File "/usr/lib/python3.7/asyncio/sslproto.py", line 526, in data_received
    ssldata, appdata = self._sslpipe.feed_ssldata(data)
  File "/usr/lib/python3.7/asyncio/sslproto.py", line 207, in feed_ssldata
    self._sslobj.unwrap()
  File "/usr/lib/python3.7/ssl.py", line 767, in unwrap
    return self._sslobj.shutdown()
ssl.SSLError: [SSL: KRB5_S_INIT] application data after close notify (_ssl.c:2605)

I noticed bug #3477 but it is closed and the problem is still there (I have the latest pip version).

% python --version
Python 3.7.2

% pip show aiohttp
Name: aiohttp
Version: 3.5.4
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Location: /usr/lib/python3.7/site-packages
Requires: chardet, multidict, attrs, async-timeout, yarl
Required-by: 
33 Answers

✔️Accepted Answer

For those looking for a work-around to at least silence these exceptions: the traceback seen is output produced by the loop.default_exception_handler() function. You can set your own exception handler (with loop.set_exception_handler()) that just ignores this specific exception if specific conditions are met. I also record the current exception handler and forward the rest to that.

From my main co-routine for the loop, I call ignore_aiohttp_ssl_eror(asyncio.get_running_loop()), which is defined as:

import asyncio
import ssl
import sys

SSL_PROTOCOLS = (asyncio.sslproto.SSLProtocol,)
try:
    import uvloop.loop
except ImportError:
    pass
else:
    SSL_PROTOCOLS = (*SSL_PROTOCOLS, uvloop.loop.SSLProtocol)

def ignore_aiohttp_ssl_eror(loop):
    """Ignore aiohttp #3535 / cpython #13548 issue with SSL data after close

    There is an issue in Python 3.7 up to 3.7.3 that over-reports a
    ssl.SSLError fatal error (ssl.SSLError: [SSL: KRB5_S_INIT] application data
    after close notify (_ssl.c:2609)) after we are already done with the
    connection. See GitHub issues aio-libs/aiohttp#3535 and
    python/cpython#13548.

    Given a loop, this sets up an exception handler that ignores this specific
    exception, but passes everything else on to the previous exception handler
    this one replaces.

    Checks for fixed Python versions, disabling itself when running on 3.7.4+
    or 3.8.

    """
    if sys.version_info >= (3, 7, 4):
        return

    orig_handler = loop.get_exception_handler()

    def ignore_ssl_error(loop, context):
        if context.get("message") in {
            "SSL error in data received",
            "Fatal error on transport",
        }:
            # validate we have the right exception, transport and protocol
            exception = context.get('exception')
            protocol = context.get('protocol')
            if (
                isinstance(exception, ssl.SSLError)
                and exception.reason == 'KRB5_S_INIT'
                and isinstance(protocol, SSL_PROTOCOLS)
            ):
                if loop.get_debug():
                    asyncio.log.logger.debug('Ignoring asyncio SSL KRB5_S_INIT error')
                return
        if orig_handler is not None:
            orig_handler(loop, context)
        else:
            loop.default_exception_handler(context)

    loop.set_exception_handler(ignore_ssl_error)

(Edited to disable on Python 3.7.4 and to support uvloop)

Other Answers:

Fixed by upstream: python/cpython#13548

The way I've gotten around this is to ensure that I always read/consume the response before being done with the response. While it's not ideal if you don't actually need the response body, for my use case, it's not burdensome to call await resp.text().

I did some more digging into this because we were seeing this error in HTTPX. This error also seems to be impacting people in various other libraries.

@socketpair has linked most of the relevant things above already, but here is a summary of what I understand:

  • This error type is new to OpenSSL 1.1.1 (older versions won't see this)
  • I'm not sure of exactly which Python versions this impacts but can confirm 3.7.0 through to 3.8.0 (despite some comments saying otherwise).
  • This error can be caused by a race condition and so may only show up sometimes and how frequently that happens can depend on a number of factors (i.e. is effectively random in practice).
  • This error happens when closing a connection that uses SSL (I don't know the aiohttp internals, but an example in asyncio of where this can happen is when calling wait_closed() on a asyncio.StreamWriter instance).
  • The reason code of KRB5_S_INIT is wrong. This reason code doesn't even seem to exist in recent versions of OpenSSL. This seems to be an old error code mapping in the Python ssl library to the code 291. The actual reason should be something like APPLICATION_DATA_AFTER_CLOSE_NOTIFY. See the OpenSSL source here.

As annoying as it sounds, this error message is saying exactly what's going on. When you close an SSL connection, the SSL_shutdown call is made to OpenSSL.SSL_shutdown is a pretty complicated function--here are the docs. The first time it is called, this function sends a close_notify message to the peer to signal that it has closed the write half of the connection. The peer can then signal the shutdown of the read half of the connection by sending a close_notify of its own. SSL_shutdown can be called a second time to confirm the peer's close_notify is received.

This second call to SSL_shutdown is where OpenSSL 1.1.1 added a new error code that is returned and causes this error in Python. If we send close_notify but the peer/server sends something back that isn't a close_notify of its own, then this error happens. This is kind of a completely reasonable thing to happen if the server was busy sending data (or we were still busy receiving it) when we called SSL_shutdown. It's also much more likely to happen in async code where we're doing things concurrently and not necessarily reading/writing all the response/request data in single, synchronous calls.

What does this all mean? There may be places/cases in aiohttp (again, I don't know it well) where a connection is closed before all the outstanding data is read from it. As far as I understand--and I am not a security expert so take this with a grain of salt--close_notify calls aren't considered important when using a self-terminated protocol like HTTP. Here is a nice Stack Exchange answer about that. So for HTTP things like aiohttp, it may be fine to catch & ignore this exception.

Working with python 3.7.4 sys.version_info(major=3, minor=7, micro=4, releaselevel='final', serial=0) still generates the same problem, using python:3.7.4-alpine:3.9 as dockerfile image

More Issues: