* compiler, runtime, reflect: generate type-specific hash/equal for composite map keys
For map keys that are not trivially binary-comparable, the compiler now
generates type-specific hash and equal functions as LLVM IR instead of
going through the interface+reflection path. This covers comparable
types: strings, floats, complex numbers, interfaces, channels, and
composites containing any mix of these.
Previously, maps with composite keys containing strings or floats
converted the key to interface{}, hashed via reflection, and compared
through interface equality. Now the compiler walks struct fields and
array elements directly, dispatching to the right runtime helper for
each field type and storing keys at their actual type.
Struct keys are always handled field-by-field so padding bytes do not
affect equality or hashing. Blank fields are ignored, matching Go
equality. Generated hash/equal function names use canonical underlying
type structure so structurally identical key types can share generated
functions. Padding zeroing before map operations is no longer needed
because structs no longer use the binary key path.
Also fix reflect map iteration for interface-keyed maps: MapIter.Key
returns an interface Value for map[interface{}] keys instead of
unpacking to the concrete key kind.
* compiler: generate loops for array map key hash/equal
Previously, array key hash and equal functions were unrolled at compile
time, generating one block of IR per element. For large arrays like
[1000]int inside a struct with non-binary fields, this caused code
explosion.
Now, binary-element arrays dispatch directly to hash32/memequal for the
whole array. Non-binary-element arrays generate an LLVM IR loop. The
equal loop short-circuits on the first mismatch.
Small arrays are still unrolled instead of looping, keeping the simple
cases compact.
* reflect: fix at-runtime map issues from review, and more found locally
Maps created through reflect.MakeMap need hash/equal behavior that
matches compiler-created maps. Add hashmapMakeReflect for composite key
types, using runtime closures that reconstruct interface{} values from
raw key bytes and delegate to the interface hash and equality paths.
Interface-keyed maps are already stored as interface values, so use the
existing interface hash/equal helpers directly for those. This keeps
reflect insert, lookup, delete, and compiled lookup paths consistent.
Also fix addressable small values used as interface map keys or
interface map values. loadSmallValue puts small indirect values back in
the pointer-sized interface data field the same way valueInterfaceUnsafe
does.
* compiler, interp, reflect: fix pointer map literals; remove interface fallback
Package-level map literals with pointer keys (both *T and
unsafe.Pointer) crash the compiler: the interp pass panics when trying
to hash pointer data as raw bytes, because pointer values in the interp
memory model are symbolic identities that do not fit in a byte.
Fix this by setting a recoverable error flag instead of panicking. The
interp detects the error after each instruction and defers the map
insert to runtime init code, where real addresses are available for
hashing. This matches how the interp already handles other operations
it cannot evaluate at compile time.
With this fix, unsafe.Pointer can also be classified as a binary map
key, which was the last type requiring the interface-based fallback.
Since all comparable types now use either the binary or the
compiler-generated hash/equal path, remove the interface fallback from
the compiler and reflect packages.
* compiler, transform: always pass hash/equal function pointers to hashmapMakeGeneric
The compiler now always resolves the hash and equal functions at compile
time and passes them directly to hashmapMakeGeneric, instead of passing
an algorithm enum to hashmapMake and resolving at runtime. For string
keys, the runtime hashmapStringPtrHash/hashmapStringEqual functions are
referenced directly. For binary keys, hash32/memequal are referenced.
The old hashmapMake with alg enum is retained for reflect, which still
needs runtime resolution when creating maps dynamically.
The OptimizeMaps transform pass is updated to handle both hashmapMake
and hashmapMakeGeneric, and to recognize hashmapGenericSet in addition
to hashmapBinarySet and hashmapStringSet. The now-unused
hashmapCanGenerateHashEqual helper is removed.
* runtime: store large map keys and values indirectly
When a map key or value exceeds 128 bytes, the bucket now stores a
pointer to separately allocated memory instead of the data inline. This
matches Go's MapMaxKeyBytes/MapMaxElemBytes threshold and prevents
bucket sizes from exploding for large key/value types.
For example, map[[256]byte]int previously used 2128 bytes per bucket
(16 header + 256*8 keys + 8*8 values); now it uses 144 bytes per bucket
(16 header + 8*8 pointers + 8*8 values).
The indirection is fully encapsulated in the runtime via helper
functions. Store the computed key and value slot sizes on the hashmap so
all runtime and reflect paths use the same bucket layout, including
non-indirect keys and values.
Add big-key golden coverage and benchmarks. Make the benchmark vary
enough key bytes to exercise hashing.
The compiler now implements the copy builtin directly instead of calling sliceCopy.
The length is calculated with the llvm.umax.* intrinsics, and the move is performed by llvm.memmove.*.
Both of these operations are easily understood by LLVM's optimization passes.
The type's alignment is also provided to llvm.memmove.*, which is useful when rewriting the move.
Interp no longer needs to reimplement sliceCopy.
Some edge case handling was implemented by sliceCopy but not llvm.memmove.*/llvm.memcpy.*.
I copied this over, so copies of external slices should work now.
Volatile moves/copies are now run at runtime by interp.
There is a 4-byte size increase due to some confusing length logic in sendUSBPacket.
I will look at sendUSBPacket in a future PR.
Writing the pointer of a buffer to memory-mapped I/O will normally cause
it to escape, which forces the compiler to heap-allocate the buffer. But
we do know how long the value stays alive, so we can tell the compiler
to keep it alive exactly until it is not needed anymore - and tell it to
not treat the pointer-to-uintptr cast as escaping.
Strings are readonly, but the compiler doesn't always know this. Marking
them as readonly in the frontend allows the compiler to optimize based
on this knowledge.
This provides some small code size benefits. I didn't measure running
speed.
Apparently this package imports `runtime.getRandomData` from the gojs
module. This is not yet implemented, but simply allowing this package to
do such imports gets crypto/sha256 to compile.
This mangles CGo identifier names to something like "_Cgo_foo" instead
of using literal identifiers like "C.foo". This works around
https://github.com/golang/go/issues/71777.
I don't like this solution, but I hope we'll find a better solution in
the future. In that case we can revert this commit.
older behavior for wasi modules to not return an exit code as if they were reactors.
See #4726 for some details on what this is intended to address.
Signed-off-by: deadprogram <ron@hybridgroup.com>
This only works on declarations, not definitions. This is intentional:
it follows the upstream Go implemetation.
However, we might want to loosen this requirement at some point: TinyGo
sometimes stores pointers in memory mapped I/O knowing they won't
actually escape, but the compiler doesn't know about this.
compiler: align with current wasm types proposal
https://github.com/golang/go/issues/66984
- Remove int and uint as allowed types in params, results, pointers, or struct fields
- Only allow small integers in pointers, arrays, or struct fields
- enforce structs.HostLayout usage per wasm types proposal
https://github.com/golang/go/issues/66984
- require go1.23 for structs.HostLayout
- use an interface to check if GoVersion() exists
This permits TinyGo to compile with Go 1.21.
- use goenv.Compare instead of WantGoVersion
- testdata/wasmexport: use int32 instead of int
- compiler/testdata: add structs.HostLayout
- compiler/testdata: improve tests for structs.HostLayout
This adds support for the `//go:wasmexport` pragma as proposed here:
https://github.com/golang/go/issues/65199
It is currently implemented only for wasip1 and wasm-unknown, but it is
certainly possible to extend it to other targets like GOOS=js and
wasip2.
This package can never be a full version as seen in upstream Go, because
TinyGo is very different. But it is necessary to define so that no code
can accidentally use this package (now or in the future).
It currently defines:
- NoEscape which is needed by strings.Builder since Go 1.23.
- FuncPCABI* which is needed by internal/syscall/unix on MacOS.
This is a big change: apart from removing LLVM 14 it also removes typed
pointer support (which was only fully supported in LLVM up to version
14). This removes about 200 lines of code, but more importantly removes
a ton of special cases for LLVM 14.
This commit adds support for LLVM 16 and switches to it by default. That
means three LLVM versions are supported at the same time: LLVM 14, 15,
and 16.
This commit includes work by QuLogic:
* Part of this work was based on a PR by QuLogic:
https://github.com/tinygo-org/tinygo/pull/3649
But I also had parts of this already implemented in an old branch I
already made for LLVM 16.
* QuLogic also provided a CGo fix here, which is also incorporated in
this commit:
https://github.com/tinygo-org/tinygo/pull/3869
The difference with the original PR by QuLogic is that this commit is
more complete:
* It switches to LLVM 16 by default.
* It updates some things to also make it work with a self-built LLVM.
* It fixes the CGo bug in a slightly different way, and also fixes
another one not included in the original PR.
* It does not keep compiler tests passing on older LLVM versions. I
have found this to be quite burdensome and therefore don't generally
do this - the smoke tests should hopefully catch most regressions.
When a function is exported using //export, but also had a
//go:wasm-module pragma, the //export name was ignored. The
//go:wasm-module doesn't actually do anything besides breaking the
export (exported functions don't have a module name).
I've refactored and cleaned up the code, and in the process removed this
weird edge case.
This is a small change that's not really important in itself, but it
avoids duplicate errors in a future commit that adds error messages to
//go:wasmimport.
This gives a small improvement now, and is needed to be able to use the
Heap2Stack transform that's available in the Attributor pass. This
Heap2Stack transform could replace our custom OptimizeAllocs pass.
Most of the changes are just IR that changed, the actual change is
relatively small.
To give an example of why this is useful, here is the code size before
this change:
$ tinygo build -o test -size=short ./testdata/stdlib.go
code data bss | flash ram
95620 1812 968 | 97432 2780
$ tinygo build -o test -size=short ./testdata/stdlib.go
code data bss | flash ram
95380 1812 968 | 97192 2780
That's a 0.25% reduction. Not a whole lot, but nice for such a small
patch.
Previously we were using a really weird calculation to determine the
alignment in bits - written by me (no idea what I was thinking at the
time, it's obviously incorrect). Just replace it with the much more
obviously correct multiplication by 8 to get bits from bytes.
Found while working on properly dealing with alignment in `-size=full`.
ThinLTO results in a small code size reduction, which is nice
(especially on these very small chips). It also brings us one step
closer to using ThinLTO everywhere.
--allow-undefined can be a problem: it allows compiling code that will
fail when loaded. This change makes sure that if some symbols are
undefined, they are reported as an error by the linker.
Previously, people could get away with importing a function that was not
defined, like this:
func add(int a, int b) int
func test() {
println(add(3, 5))
}
This was always unintended but mostly worked. With this change, it isn't
possible anymore. Now every function needs to be marked with //export
explicitly:
//export add
func add(int a, int b) int
func test() {
println(add(3, 5))
}
As before, functions will be placed in the `env` module with the name
set from the `//export` tag. This can be overridden with
`//go:import-module`:
//go:import-module math
//export add
func add(int a, int b) int
func test() {
println(add(3, 5))
}
For the syscall/js package, I needed to give a list of symbols that are
undefined. This list is based on the JavaScript functions defined in
targets/wasm_exec.js.
This removes the parentHandle argument from the internal calling convention.
It was formerly used to implment coroutines.
Now that coroutines have been removed, it is no longer necessary.
This patch adds //go: pragmas directly to declared functions and
globals found during CGo processing. This simplifies the logic in the
compiler: it no longer has to consider special "C." prefixed function
names. It also makes the cgo pass more flexible in the pragmas it emits
for functions and global variables.
This matches Clang, and with that, it adds support for inlining between
Go and C because LLVM only allows inlining if the "target-cpu" and
"target-features" string attributes match.
For example, take a look at the following code:
// int add(int a, int b) {
// return a + b;
// }
import "C"
func main() {
println(C.add(3, 5))
}
The 'add' function is not inlined into the main function before this
commit, but after it, it can be inlined and trivially be optimized to
`println(8)`.
Instead of doing everything in the interrupt lowering pass, generate
some more code in gen-device to declare interrupt handler functions and
do some work in the compiler so that interrupt lowering becomes a lot
simpler.
This has several benefits:
- Overall code is smaller, in particular the interrupt lowering pass.
- The code should be a bit less "magical" and instead a bit easier to
read. In particular, instead of having a magic
runtime.callInterruptHandler (that is fully written by the interrupt
lowering pass), the runtime calls a generated function like
device/sifive.InterruptHandler where this switch already exists in
code.
- Debug information is improved. This can be helpful during actual
debugging but is also useful for other uses of DWARF debug
information.
For an example on debug information improvement, this is what a
backtrace might look like before this commit:
Breakpoint 1, 0x00000b46 in UART0_IRQHandler ()
(gdb) bt
#0 0x00000b46 in UART0_IRQHandler ()
#1 <signal handler called>
[..etc]
Notice that the debugger doesn't see the source code location where it
has stopped.
After this commit, breaking at the same line might look like this:
Breakpoint 1, (*machine.UART).handleInterrupt (arg1=..., uart=<optimized out>) at /home/ayke/src/github.com/tinygo-org/tinygo/src/machine/machine_nrf.go:200
200 uart.Receive(byte(nrf.UART0.RXD.Get()))
(gdb) bt
#0 (*machine.UART).handleInterrupt (arg1=..., uart=<optimized out>) at /home/ayke/src/github.com/tinygo-org/tinygo/src/machine/machine_nrf.go:200
#1 UART0_IRQHandler () at /home/ayke/src/github.com/tinygo-org/tinygo/src/device/nrf/nrf51.go:176
#2 <signal handler called>
[..etc]
By now, the debugger sees an actual source location for UART0_IRQHandler
(in the generated file) and an inlined function.
Ayke van Laethem
Jake Bailey
Nia Waldvogel
Michael Smith
Damian Gryski
deadprogram
Randy Reddig
cui fliter
Kenneth Bell