Reading runtime.MemStats.NumGC is common enough in dependency code that its
absence is a compile error for programs that never look at the value. Rather
than add the field as a constant zero, track it:
- gc_blocks: count completed cycles in runGC, so collections triggered by an
allocation are counted as well as explicit runtime.GC() calls. The counter is
read and written under gcLock, like the other counters beside it.
- gc_boehm: report bdwgc's own gc_no from the prof_stats struct.
- gc_leaking: always 0, since that collector never completes a cycle.
Using blocksPerStateByte instead of bytesPerBlock caused the old size to be underestimated for multi-block allocations, leading to data truncation on growing reallocations.
Match the Go runtime by terminating for deadlocks, stack overflows,
runtime and GC invariants, invalid lock operations, and platform
initialization failures instead of routing them through panic/recover.
Keep language-level runtime errors and unsupported user operations
recoverable. Add crash coverage that verifies fatal errors bypass deferred
recover calls.
Match the Go runtime by terminating instead of unwinding when an
allocator exhausts the heap. Recovering an OOM can leave allocator locks
held and cannot safely continue when the panic path itself needs memory.
Instead of referring to an unused global, use a constant value. This is
safe even when using `-gc=none` (since no actual memory gets allocated)
which wasn't the case before. It should also reduce binary size by a few
bytes for most programs.
This moves the objHeader from before an object body to after it.
On 32-bit systems with 16-byte alignment requirements (x86, ARM, RISC-V), we previously padded the header to a whole block.
This wastes up to 12 bytes, as on -gc=conservative the header is a single pointer.
With this change, no padding is required (beyond that from rounding the size up).
The "head" block in the metadata was moved to the end of the range to match the header location.
This changed the block loop directions throughout the GC logic.
The bit hacks used by sweep no longer work because there is no equivalent of addition that carries downwards.
However it is now possible to merge the sweep and free range list rebuild passes because their loop directions match.
There are two other places where we rebuilt the free ranges list: when initializing or growing the heap.
The former can be easily replaced with a single hardcoded range containing the entire heap.
In the latter case, I opted to only add the new space to the existing list.
These replacements allowed me to fully remove the buildFreeRanges function.
Instead of looping over each block, we can use bit hacks to operate on an entire state byte.
I deinterleaved the state bits in order to enable these tricks.
Sweep used to count free/freed allocations/blocks.
I managed to move/remove all of these counters:
- The free space is now calculated in buildFreeRanges by adding the range lengths.
- ReadMemStats counts freed objects by subtracting live objects from allocated objects.
- gcFreedBlocks was never necessary because MemStats.HeapAlloc is the same as MemStats.HeapInUse.
The allocator originally just looped through the blocks until it found a sufficiently-long range.
This is simple, but it fragments very easily and can degrade to a full heap scan for long requests.
Instead, we now maintain a sorted nested list of free ranges by size.
The allocator will select the shortest sufficient-length range, generally reducing fragmentation.
This data structure can find a range in time directly proportional to the requested length.
Loop over valid pointer locations in heap objects instead of checking if each location is valid.
The conservative scanning code is now shared between markRoots and the heap scan.
This also removes the ending alignment requirement from markRoots, since the new scan* functions do not require an aligned length.
This requirement was occasionally violated by the linux global marking code.
This saves some code space and has negligible impact on performance.
The blocks GC originally used a fixed-size stack to hold objects to scan.
When this stack overflowed, the GC would fully rescan all marked objects.
This could cause the GC to degrade to O(n^2) when scanning large linked data structures.
Instead of using a fixed-size stack, we now add a pointer field to the start of each object.
This pointer field is used to implement an unbounded linked stack.
This also consolidates the heap object scanning into one place, which simplifies the process.
This comes at the cost of introducing a pointer field to the start of the object, plus the cost of aligning the result.
This translates to:
- 16 bytes of overhead on x86/arm64 with the conservative collector
- 0 bytes of overhead on x86/arm64 with the precise collector (the layout field cost gets aligned up to 16 bytes anyway)
- 8 bytes of overhead on other 64-bit systems
- 4 bytes of overhead on 32-bit systems
- 2 bytes of overhead on AVR
This fixes a bug where runtime.alloc would not clear the padding from rounding the allocation up to a multiple of the block size.
The GC still has to scan this, and this could result in permanent false positives (and therefore memory leaks).
It also handles overflow in the size calculation.
This is not a scheduler in the runtime, instead every goroutine is
mapped to a single OS thread - meaning 1:1 scheduling.
While this may not perform well (or at all) for large numbers of
threads, it greatly simplifies many things in the runtime. For example,
blocking syscalls can be called directly instead of having to use epoll
or similar. Also, we don't need to do anything special to call C code -
the default stack is all we need.
Using a global lock may be slow, but it is certainly simple and safe.
If this global lock becomes a bottleneck, we can of course look into
making the GC truly support multithreading.
This adds support for the well-known Boehm GC. It's significantly faster
than our own naive GC and could be used as an alternative on bigger
systems.
In the future, this GC might also be supported on WebAssembly with some
extra work. Right now it's Linux only (though Windows/MacOS shouldn't be
too difficult to add).
For example, with -gc=none and -gc=leaking, no heap needs to be
allocated when initializing the runtime. And some GCs (like -gc=custom)
are responsible for allocating the heap themselves.
This moves all scheduler code into a separate file that is only compiled
when there's a scheduler in use (the tasks or asyncify scheduler, which
are both cooperative). The main goal of this change is to make it easier
to add a new "scheduler" based on OS threads.
It also fixes a few subtle issues with `-gc=none`:
- Gosched() panicked. This is now fixed to just return immediately
(the only logical thing to do when there's only one goroutine).
- Timers aren't supported without a scheduler, but the relevant code
was still present and would happily add a timer to the queue. It
just never ran. So now it exits with a runtime error, similar to any
blocking operation.
This is similar to https://github.com/tinygo-org/tinygo/pull/3899, but
smaller and hopefully just as efficient.
Thanks to @HattoriHanzo031 for starting this work, benchmarking, and for
improving the performance of the code even further.
Every time we overflow the stack, we have to do a full rescan of the heap. Making this larger
means fewer overflows and thus fewer secondary+ heap scans.
Instead of markGlobals calling markRoots unconditionally (which doesn't
make sense for -gc=none and -gc=leaking), provide markRoots as a
callback function.
This is in preparation for -gc=boehm, where the previous design is even
more awkward and a callback makes far more sense.
I've tested the size impact using `make smoketest XTENSA=0`. There is
none, except for two cases:
* One with `-opt=0` so const-propagation for the callback didn't take
place.
* One other on AVR, I don't know why but as it's only 16 bytes in a
very specific case I'm going to assume it's just a random change in
compiler output that caused a size difference.
This is not very useful in itself, but makes it possible to detect this
address in the output. See the next commit.
This adds around 50 bytes to each binary (except for AVR and wasm). This
is unfortunate, but I think this feature is quite useful still.
A future enhancement might be to create a build tag for extended panic
information that's not set by default.
We have an optimization for this specific pattern, but it's really just
a hack. With the addition of unsafe.Add in Go 1.17 we can directly
specify the intent instead and eventually remove this special case.
The code is also easier to read.
This implements the block-based GC as a partially precise GC. This means
that for most heap allocations it is known which words contain a pointer
and which don't. This should in theory make the GC faster (because it
can skip non-pointer object) and have fewer false positives in a GC
cycle. It does however use a bit more RAM to store the layout of each
object.
Right now this GC seems to be slower than the conservative GC, but
should be less likely to run out of memory as a result of false
positives.
Most of the code of the conservative GC can be reused for the precise
GC. So before adding precise GC support, this commit just moves code
around to make the next commit cleaner. It is a non-functional change.
Moses Narrow
Konstantin Sharlaimov
Jake Bailey
felipegenef
Ayke van Laethem
Nia Waldvogel
Damian Gryski
dkegel-fastly
Anuraag Agrawal