Pierluigi Paganini
July 18, 2026
Okta’s Red Team disclosed a denial-of-service vulnerability in OpenSSL they named HollowByte, and the attack payload is exactly 11 bytes. A remote, unauthenticated attacker sends that payload and the server allocates up to 131 KB of memory before the TLS handshake even begins, then blocks a worker thread waiting for data that never arrives. No credentials required, no prior access, no exploit chain.
“When a rogue header lands, the state machine triggers an unvalidated allocation.” reads the advisory. “When the malicious 11-byte payload arrives, the TLS state machine reads the 4-byte handshake header and triggers an unvalidated pre-allocation based on the header’s 3-byte length declaration:
Read Header⟶grow_init_buf()⟶OPENSSL_clear_realloc()⟶malloc(attacker_size)
Because there is no payload validation at this early stage, malloc() allocates up to 131 KB based solely on the untrusted packet’s claims. The worker thread then blocks, waiting indefinitely for data that will never arrive.”
Older versions of OpenSSL trust the attacker’s declared message size and allocate memory for it before any data has actually arrived. The fix is conceptually simple, grow the buffer only as bytes actually land on the wire, but the vulnerable design has been in place long enough to affect a wide range of software.
The memory problem doesn’t end when the connection drops.
“When an attacking connection drops, OpenSSL frees the buffer. However, glibc does not immediately return small-to-medium allocations to the operating system; it keeps them for potential reuse.” continues the advisory. “By launching waves of connections with randomized claimed sizes, an attacker prevents the allocator from reusing those freed chunks. “
The heap fragments progressively, the server’s resident memory footprint climbs continuously, and the only way to reclaim it is to restart the process. The attacker can disconnect and the damage stays.
Okta tested the attack against unpatched and patched NGINX instances under realistic conditions.
“In a standard 1 GB RAM environment, the unpatched server was OOM-killed at 547 MB of frozen, fragmented memory.” states Okta. “In higher-spec testing (16 GB RAM), the attack successfully locked up 25% of the system’s total memory while staying safely under the connection ceiling, meaning standard connection-limiting defenses won’t stop it.”
Rate limiting and connection caps, the standard first-line defenses against this type of attack, don’t help here because the attacker doesn’t need many connections to cause meaningful damage.
The scope is broad.
“Because OpenSSL is widely used and embedded, this vulnerability affects a variety of software, including web servers (Apache, NGINX), language runtimes (Node.js, Python, Ruby, PHP), and databases (MySQL, PostgreSQL).” concludes the report.
The OpenSSL team fixed it through incremental buffer growth, the server now only allocates memory as data actually arrives, so an empty claim costs nothing, merged in PRs 30792, 30793, and 30794. The fix was included in OpenSSL v4.0.1 and silently backported to versions 3.6.3, 3.5.7, 3.4.6, and 3.0.21. No CVE was assigned; OpenSSL treated it as a hardening fix rather than a security advisory. Update your OpenSSL packages regardless.
Follow me on Twitter: @securityaffairs and Facebook and Mastodon
(SecurityAffairs – hacking, HollowByte)
