Proposal Details

Hi folks, I'd like to explore the possibility for the runtime to 'adopt' externally-allocated memory by tracking pointers to the span and unmapping the underlying memory if there are no more references. This opens up many interesting use cases for folks that need to interact with C or other FFI memory, or maybe even for writing custom allocators.

API

// AddForeignCleanup attaches a cleanup function to a non-Go memory region described
// by ptr and size. It behaves similarly to [AddCleanup], but must be used on memory that
// was not created by Go.
//
// AddForeignCleanup may panic if the given memory region overlaps, but this is not
// guaranteed.
func AddForeignCleanup(ptr unsafe.Pointer, size int, cleanup func(ptr uintptr, size int))

To be used as:

ptr, _ := unix.MmapPtr(fd, 0, nil, size, unix.PROT_READ|unix.PROT_WRITE, unix.MAP_SHARED)

runtime.AddForeignCleanup(ptr, size, func(p uintptr, s int) {
_ = unix.MunmapPtr(p, s)
})

Implementation

Incorporating early feedback on this proposal, it would be ideal if this would support memory ranges of arbitrary sizes. As far as I can tell, the runtime doesn't have a facility for representing ranges smaller than a page, so @mknyszek suggested that a new data structure may need to be introduced.

The Linux kernel 'recently' switched its memory management over to the Maple Tree, a B-tree variant optimized for storing non-overlapping ranges, which happens to be exactly what we need as well. The kernel implementation is completely general-purpose and very complex, but we might be able to get away with a subset of it.

If the runtime has active manual memory mappings, the garbage collector would first try to find a pointer's mspan, falling through to querying the Maple Tree if that turns up nothing. During a cycle, Maple Tree nodes would be considered heap allocations, scheduling a cleanup if all references are gone.

I originally got this idea from https://pkg.go.dev/github.com/Jille/gcmmap, a package that mmaps over the Go heap using MAP_FIXED. It uses runtime.mallocgc() but allocating a byte slice works just as well. This approach works, but makes several hard assumptions:

• there's no moving garbage collector (although we have runtime.heapObjectsCanMove now)
• the heap is always mapped using PROT_READ|PROT_WRITE and MAP_ANON|MAP_PRIVATE
• there are no other protections like mseal()

Please let me know what you think. Thank you!