Your tests are green. They have been for years. Then someone sends an input none of them ever tried, and the code writes past the end of an array. Green on the outside, red on the inside?

That’s Java4Shell and it’s worth a closer look, because it shows two things. The first is what memory safety changes when it goes all the way down to the fast paths. The second is what mutation testing can do once it does.

A disclosure first: I built mutastic, a mutation testing tool for Java and Rust. I got interested in the topic one evening then hyper-fixated it into existence to support the two memory-safe languages I use the most. It appears below. It can’t run on the code where this bug lived, and I’ll be clear about why and where that line falls.

What happened

Java4Shell (CVE-2026-114938) was disclosed by Calif on 17 Nov 2026 (Calif’s write-up). Credit for the research goes to Quang Luong and Jenish Rakholiya. The fix in the OpenJDK repository came from Tobias Hartmann and Albert Mingkun Yang.

The bug isn’t in Java code. It’s in machine code that the JVM writes for itself. HotSpot has a fast Base64 decoder for x86-64 CPUs with AVX-512 VBMI.

AVX-512 is a set of vector instructions that work on 64 bytes at once, instead of one value at a time. VBMI (Vector Byte Manipulation Instructions) is an extension of it that can shuffle and look up individual bytes across a whole 64-byte register. That’s what Base64 needs: turn each character into its 6-bit value, then pack those values into bytes. Intel CPUs from Ice Lake onward and AMD CPUs from Zen 4 on have it, so the fast path runs on most recent x86 servers. It never runs on ARM.

C++ in HotSpot emits the decoder at startup, and Java trusts it. It decodes 64 bytes at a time, then handles whatever is left over: the tail.

Base64 turns every 3 bytes into 4 characters, so decoding turns every 4 characters back into 3 bytes. When the original data doesn’t fill the last group, the encoder pads it with =:

  • TWFu is Man
  • TWE= is Ma
  • TQ== is M.

Each character stands for 6 bits, and 4 × 6 = 24 bits = 3 bytes. = stands for nothing:

          T      W      F      u
TWFu      010011 010110 000101 101110     4 characters × 6 = 24 bits
          01001101 01100001 01101110      the same bits, 8 at a time
          M        a        n             3 bytes

          T      W      E      =
TWE=      010011 010110 000100 ------     3 characters × 6 = 18 bits
          01001101 01100001 00            2 bytes, 2 bits left over
          M        a

          T      Q      =      =
TQ==      010011 010000 ------ ------     2 characters × 6 = 12 bits
          01001101 0000                   1 byte, 4 bits left over
          M

Each = costs one byte of output. The leftover bits are filler, and must be zero.

So for the tail, the decoder works out how many bytes to write: the length divided by four (rounding down), times three, less one for each = of padding.

TailLength ÷ 4× 3− paddingBytes to write
TWE=13− 12 ✓
A=00− 1−1 ✗

Valid Base64 always comes in whole groups of four, so a padded tail shorter than four characters is malformed. It isn’t even one group, so the formula starts at zero bytes. The padding rule then takes one more away, and the size comes out at −1. The right answer is to reject the input. Before the fix, nothing checked, and the decoder carried on with −1 as its size.

The next step turns the size into a mask: one bit for each of the 64 bytes in the vector register, set for each byte to write. The bzhi instruction builds it by keeping the lowest size bits of a register full of ones. A size of 2 gives 0b11, so two bytes are written. But bzhi reads only the low eight bits of the size. For −1 those read as 255, and anything 64 or more means “keep everything”. So the mask comes out all ones, and the masked store writes all 64 bytes, past the end of the destination array. Java throws its exception only afterwards.

Here’s that path in HotSpot’s C++, which emits the instructions one call at a time. It’s trimmed, with my comments:

// Now have 1 to 63 bytes left to decode
__ movl(output_size, length);
__ shrl(output_size, 2);                 // length / 4
__ lea(output_size, Address(output_size, output_size, Address::times_2, 0));  // × 3
 
__ cmpb(Address(source, length, Address::times_1, -1), '=');
__ jcc(Assembler::equal, L_padding);
...
__ BIND(L_padding);
__ decrementq(output_size, 1);           // 0 becomes -1
...
__ movq(output_mask, -1);
__ bzhiq(output_mask, output_mask, output_size);  // -1: nothing cleared, all ones
...
__ kmovql(k1, output_mask);
__ evmovdqub(Address(dest, dp), k1, tmp, true, Assembler::AVX_512bit);  // 64 bytes

The fix (openjdk/jdk 5177a70) adds a length check before all of that: a tail shorter than four goes back to the careful, byte-at-a-time path.

// A padded Base64 tail shorter than one complete block is handled by Java.
__ cmpl(length, 4);
__ jcc(Assembler::less, L_bruteForce);

It’s four lines, and it adds no test.

A regression test did exist. PR #32781, opened in public on 9 Sep 2026, had the same four-line fix and a 110-line test for exactly this input shape. It was closed unmerged on 4 Nov, after a bot flagged it as inactive. Its author co-wrote the final fix. That isn’t a story about anyone getting it wrong. Open-source maintainers have finite attention, and that’s normal. The point is that the test was the missing piece, and somebody had already written it.

Low Level has a video on Java4Shell too.

The same arithmetic in safe Rust

I wrote a small Rust crate that models the tail step. It isn’t a port of the SIMD code. It copies the arithmetic and uses an ordinary slice copy where HotSpot uses the masked vector store. It runs on my Mac, which is ARM, so no AVX-512 is involved. Here’s the tail, unguarded, the way HotSpot had it:

let start = buf.len() / 64 * 64;         // blocks of 64 already decoded
let len = buf.len() - start;             // 1 to 63 left
let mut out = len / 4 * 3;
if buf[start + len - 1] == b'=' {
    out -= 1;                            // line 10
    if buf[start + len - 2] == b'=' {
        out -= 1;
    }
}
let decoded = [0u8; 64];
let mut dst = vec![0u8; 64];
dst[..out].copy_from_slice(&decoded[..out]);  // line 17: the store

I fed it a padded tail shorter than four.

In Rust, a size is a usize, and a usize can’t be −1.

In a debug or test build, Rust checks arithmetic for overflow. The program stops at the subtraction:

panicked at examples/unguarded.rs:10:9:
attempt to subtract with overflow

Release builds don’t check overflow by default, so the size wraps around to 18,446,744,073,709,551,615. But the store is a slice copy, and slices check their bounds. The program stops at the store, on the same line where HotSpot wrote past the array:

panicked at examples/unguarded.rs:17:8:
range end index 18446744073709551615 out of range for slice of length 64

Either way, the result is a crash. A crash is still a bug. Someone could use it to knock a service over, a denial of service. But it isn’t a write into the heap, and it isn’t remote code execution. Turning a silent memory corruption into a loud, local failure is what memory safety is for.

The test still has to exist

Here’s the catch. A debug build only panics if something sends it that input. The ordinary development cycle — write code, run the tests, ship — catches this bug only if a test tries a padded tail shorter than four. The suite that shipped with HotSpot evidently never did, on hardware that runs that path. Rust doesn’t change that on its own.

This is where mutation testing helps. A mutation tool makes small changes to your code, one at a time, and runs your tests against each. If the tests still pass after a change, they don’t check what that code does.

In C++ there was nothing to change. Before the fix, no instruction said “a tail shorter than four”, because the check didn’t exist. You can’t mutate code that isn’t there.

Rust is different. To keep the code and pass a short-tail test, you have to write down what happens when the size would go below zero. There are two natural ways to do that. One is a length check, like OpenJDK’s fix:

pub fn decode_tail(src: &[u8], dst: &mut [u8]) -> Result<usize, DecodeError> {
    // The check HotSpot's fix added: a tail shorter than one block goes the careful way.
    if src.len() < 4 {
        return decode_careful(src, dst);
    }
    let mut out = src.len() / 4 * 3;
    // ...

The other is checked arithmetic. checked_sub(1)? returns no size at all instead of wrapping, and the caller goes the careful way when there’s no size:

pub fn checked_output_size(src: &[u8]) -> Option<usize> {
    let mut out = src.len() / 4 * 3;
    if src.last() == Some(&b'=') {
        out = out.checked_sub(1)?;
        if src.len().checked_sub(2).and_then(|i| src.get(i)) == Some(&b'=') {
            out = out.checked_sub(1)?;
        }
    }
    Some(out)
}

Once that decision is code, a mutation tool can change it, and it will notice if no test cares.

I ran mutastic on the model crate twice. The first run used a suite shaped like the one that shipped: whole blocks, padded blocks and a bad character, but no padded tail shorter than four.

assert_eq!(decode("TWFu").unwrap(), b"Man");
assert_eq!(decode("TWE=").unwrap(), b"Ma");
assert_eq!(decode("TQ==").unwrap(), b"M");
assert_eq!(decode("TW*u"), Err(DecodeError::Invalid));

With the shipped-style suite, mutastic reported these two as missed:

mf39a63b7  `src.len() < 4` is always false
           no test needs src.len() < 4 to be true in decode_tail()

m079146d4  `?` becomes `.unwrap()`
           no test makes this ? return its error from checked_output_size()

The first one is worth reading twice. Making the length check always false gives you the code as it was before the fix. mutastic names the missing test, on the line of the fix: nothing in your suite needs a tail shorter than four. The second says the same thing about the checked-arithmetic version: no test ever makes the subtraction fail.

The second run added a test in the spirit of PR #32781: a padded tail shorter than four is rejected, and nothing is written.

#[test]
fn a_padded_tail_shorter_than_a_block_is_rejected() {
    for s in ["=", "A=", "AA="] {
        let mut dst = [0u8; 64];
        assert_eq!(decode_tail(s.as_bytes(), &mut dst), Err(DecodeError::Invalid), "{s}");
        assert_eq!(dst, [0u8; 64], "{s} wrote bytes");
    }
}

With the PR-style test added, both are caught. The first is caught by the debug-build panic: attempt to subtract with overflow.

Five other mutants on those lines survive both runs. They’re equivalent: no test could tell them apart from the real code. Three of them just send more input down the careful path, which gives the same answer more slowly. One of them makes a subtraction that can’t fail anyway. The last makes checked_output_size always return no size, which also means the careful path. I’d rather say that than show only the tidy rows.

To be clear about what this does and doesn’t show:

  • mutastic doesn’t find the input. It tells you no test needs that check to be true. A person, or an agent, turns that into a test. Here that test had already been written.
  • This is a model, not HotSpot. mutastic can’t run on HotSpot. Its Java edition mutates Java bytecode, and this bug lived in C++ that emits assembly. This is an argument for how to write that layer, not a claim about any Java project.

Why not a like-for-like port?

The obvious objection is that a real fast decoder in Rust wouldn’t use a slice copy. It would call the AVX-512 masked store directly, through unsafe, with a raw pointer. That’s true; if you port HotSpot like-for-like — raw masked store in unsafe, wrapping arithmetic to match the C++ — you get HotSpot’s bug back, both debug and in release. The size wraps, the mask is all ones, and the store writes 64 bytes wherever the pointer says.

It looks like this. It’s an illustration I haven’t compiled, since my Mac has no x86 toolchain (I’ll update once I get it running on my pc):

use std::arch::x86_64::*;
 
// Like-for-like: the same bug, in Rust.
#[target_feature(enable = "avx512bw,bmi2")]
unsafe fn store_tail(dst: *mut u8, decoded: __m512i, len: usize, pads: usize) {
    let out = (len / 4 * 3).wrapping_sub(pads);     // wraps, as the C++ does
    let mask = _bzhi_u64(u64::MAX, out as u32);       // low 8 bits 255: all ones
    _mm512_mask_storeu_epi8(dst as *mut i8, mask, decoded);  // 64 bytes, no bounds
}

The safe version gives the store the destination itself, not a pointer to it:

fn store_tail(dst: &mut [u8], decoded: &[u8; 64], out: usize) {
    dst[..out].copy_from_slice(&decoded[..out]);  // a wrong `out` panics here
}

So rewriting in Rust isn’t the point. Memory safety all the way down is: keep the fast path inside the guarantees, not just the code around it.

  • Store through a bounds-checked slice. Do the vector work into a scratch buffer, then copy dst[..out], as above. A wrong size becomes a panic at the store, not a heap write.
  • Check the arithmetic. checked_sub, checked_mul and friends make the overflow a value you have to handle, and that gives a mutation tool something to work on. If you can’t do that everywhere, turn overflow checks on in release builds.
  • Lint for it. Clippy’s arithmetic_side_effects lint flags arithmetic that could overflow, so wrapping maths has to be written down on purpose, not by default.

Both of those last two go in Cargo.toml:

[profile.release]
overflow-checks = true
 
[lints.clippy]
arithmetic_side_effects = "warn"

With the lint on, the plain size sum is flagged and the checked one isn’t:

warning: arithmetic operation that can potentially result in unexpected side-effects
 --> src/lib.rs:6:5
  |
6 |     len / 4 * 3 - pads
  |     ^^^^^^^^^^^^^^^^^^
  • Keep unsafe small and named. If the raw store has to stay, fence it into one small function and pass it the destination length. Then make the tests and the reviewers spend their time there.
  • Or use a masked store that checks bounds itself. Rust’s portable SIMD API has store_select, a safe masked store. Its documentation says that if an element “is disabled or out of bounds, no memory access to that location is made”. An all-ones mask with a short destination simply writes less. It’s still nightly-only (portable_simd), so it isn’t an option for production code yet. Also not compiled here:
#![feature(portable_simd)]
use std::simd::{Mask, Simd};
 
fn store_tail(dst: &mut [u8], decoded: Simd<u8, 64>) {
    // Even Java4Shell's all-ones mask writes only dst.len() bytes.
    decoded.store_select(dst, Mask::from_bitmask(u64::MAX));
}

What ARM does

ARM’s NEON vector unit works 16 bytes at a time, and its TBL and TBX lookups can do Base64’s character step much as VBMI does. Its newer SVE adds masked stores, as AVX-512 does. The masks are called predicates, and they’re usually built from a count. Feed one a size of −1 and one letter decides what happens:

// x0 holds the size: -1
whilelt p0.b, xzr, x0     // signed:   0 < -1 is false, so no lanes are on
whilelo p0.b, xzr, x0     // unsigned: -1 is huge, so every lane is on
st1b    {z0.b}, p0, [x1]  // stores the lanes that are on

With WHILELT, nothing is written. With WHILELO, you have Java4Shell again. That snippet assembles, but I haven’t run it: my M1 has no SVE.

HotSpot’s own ARM decoder takes neither route (stubGenerator_aarch64.cpp). It uses NEON, and it never works out a size from the padding. Trimmed, with my comments:

__ sub(length, send, soff);
__ bfm(length, zr, 0, 1);              // round down to whole groups of 4
...
__ BIND(Process4B);
__ ldrw(r14, __ post(src, 4));         // 4 characters
...                                    // look each one up; '=' maps to 255
__ orrw(r14, r10, r11);
__ orrw(r15, r12, r13);
__ orrw(r14, r14, r15);
__ tbnz(r14, 7, Exit);                 // any 255: stop, and Java finishes
...
__ strh(r14, __ post(dst, 2));
__ strb(r13, __ post(dst, 1));         // always exactly 3 bytes

There’s no size to go negative and no mask to build from one. That’s why ARM64 isn’t affected. The design does the work a bounds check would: nothing in it can ask for more bytes than the input provides.

None of this makes an assembly kernel safe. It moves the default. In HotSpot, a wrong size was silent. In this shape of Rust, a wrong size is a crash, and the code that guards against it is something your tests have to answer for.

What about performance?

Benchmark to follow

I’ll benchmark the bounds-checked store against the raw unsafe masked store on AVX-512 VBMI hardware (a Ryzen 9 9950X3D) and update this section. Until then, no numbers.

Reasoning only, to be confirmed or corrected by that benchmark: the bounds check runs once per tail, not once per byte. Each decode has at most one tail of 1 to 63 bytes, after the 64-byte blocks. So I’d expect the cost to be small next to the vector work. That’s a guess about the shape of the cost, not a measurement.

What to take from it

  • A green suite isn’t proof of anything. It shows which inputs someone tried. This one passed for years because nobody tried a short padded tail on the CPU where that path runs.
  • Memory safety turns this class of bug into a crash. But only where the safety goes all the way down, into the fast path. A port that keeps the raw store in unsafe keeps the bug.
  • Mutation testing needs code to work on. Safe Rust forces you to write the overflow decision down. Once it’s written down, a mutation tool can tell you no test checks it.
  • The test was already written. The cheapest fix in this story was a regression test sitting in a public pull request.

If you run a JVM on x86-64 with AVX-512 VBMI, upgrade. Calif’s write-up has the details and the workaround.

BLUESKY — START THE THREAD KO-FI / RSS