Native functions may cause a crash if GC happens

[xxDark]

Question

        FloatBuffer buffer = ByteBuffer.allocateDirect(64).order(ByteOrder.nativeOrder()).asFloatBuffer();
        GL11.glLoadMatrix(buffer);
        // buffer is out of scope
        // Reference.reachabilityFence(buffer);

I have not observed the crash in a real time scenario, but that might happen if a cleaner of a native resource deallocates it at the right moment.

[xxDark]

The question is whether the caller should ensure that the buffer or whatever resource remains reachable during a native call or LWJGL should be taking care of this. (with Reference::reachabilityFence for example)

[Spasi]

Hey @xxDark,

This is a legitimate concern and something I've been worrying about since the very first lwjgl3 commit. For reference, the issue is explained in detail here:

JVM Anatomy Quark #8: Local Variable Reachability

In the context of LWJGL, the problem may occur in code such as:

// in user code
ByteBuffer buffer = ByteBuffer.allocateDirect(...);
bindingMethod(..., buffer, ...);

// in LWJGL code
void bindingMethod(..., ByteBuffer buffer, ...) {
    long a = memAddress(buffer);
    nbindingMethod(..., a, ...);
}

LWJGL is being naughty here, because it reads the buffer's off-heap memory address, then proceeds to call the native method, which accepts the buffer address, not the buffer object itself. When this code gets JIT compiled (and everything gets inlined nicely), the buffer instance is considered unreachable after the memAddress call, which makes it fair game for GC. If that happens before the native method returns, the buffer's off-heap memory may be freed while still in use.

I've never been able to reproduce this and there have been no relevant bug reports in the 10 years since lwjgl3's release.

Before replying to this issue however, I figured it might be a good idea to retest with latest JVMs and GC algorithms. Turns out, it is indeed possible to reproduce, even on JDK 8. My mistake in previous attempts was using debug mode to identify buffer cleanup (with breakpoints inside JDK) but debug mode effectively disables GC of unreachable local variables (makes sense in hindsight). This time I used normal execution and Cleaner registration to detect cleanup. It indeed can happen and was able to reproduce actual crashes because of it.

So, yeah, bad news so far. The good news is that it's extremely unlikely to happen in actual programs, for the following reasons:

• LWJGL 3 promotes extensive use of stack-allocated memory (via MemoryStack) and manually-managed memory (via MemoryUtil.memAlloc*/memFree), applicable to the majority of buffer allocations (temporary buffers, or buffers with trivial/clear life-cycles). GC-managed buffers (via BufferUtils, i.e. ByteBuffer.allocateDirect) are rarely used outside of demos & toy programs.
• When GC-managed buffers are actually necessary (e.g. complicated/unclear life-cycle), there is almost always some kind of strong reference holding on to them, outside of any native calls.
• Even in legitimate scenarios when that is not true, it's still very hard for a premature free to happen. When a reference registered with a Cleaner (like GC-managed buffers are) becomes unreachable, it is marked as such (becomes a phantom reference) in the next GC cycle, but is not cleaned immediately. The cleanup action happens in the GC cycle after that. So, memory pressure has to be so high, that GC happens TWICE during the same native call.

We also cannot use Reference::reachabilityFence in LWJGL, because of Java 8 compatibility (it was added in JDK 9). Supposedly we could define a custom method and it would achieve the same effect, but I feel it's too late to bother with this issue in lwjgl3. As I said no one has reported this as an actual problem, it's unlikely to ever happen in real programs and there is a very simple workaround for those few buffers that may of legitimate concern: just hold a strong reference to them while in use. That is, leave the responsibility to the caller as you suggested.

Note that in lwjgl4 this is automatically handled for MemorySegment instances by the FFM API itself. Wherever LWJGL decides to use raw pointers (i.e. long values) internally, the new binding generator will produce reachabilityFence calls as needed. So this will never be an issue in lwjgl4.