compiler, runtime, reflect: generate type-specific hash/equal (#5359)

* compiler, runtime, reflect: generate type-specific hash/equal for composite map keys For map keys that are not trivially binary-comparable, the compiler now generates type-specific hash and equal functions as LLVM IR instead of going through the interface+reflection path. This covers comparable types: strings, floats, complex numbers, interfaces, channels, and composites containing any mix of these. Previously, maps with composite keys containing strings or floats converted the key to interface{}, hashed via reflection, and compared through interface equality. Now the compiler walks struct fields and array elements directly, dispatching to the right runtime helper for each field type and storing keys at their actual type. Struct keys are always handled field-by-field so padding bytes do not affect equality or hashing. Blank fields are ignored, matching Go equality. Generated hash/equal function names use canonical underlying type structure so structurally identical key types can share generated functions. Padding zeroing before map operations is no longer needed because structs no longer use the binary key path. Also fix reflect map iteration for interface-keyed maps: MapIter.Key returns an interface Value for map[interface{}] keys instead of unpacking to the concrete key kind. * compiler: generate loops for array map key hash/equal Previously, array key hash and equal functions were unrolled at compile time, generating one block of IR per element. For large arrays like [1000]int inside a struct with non-binary fields, this caused code explosion. Now, binary-element arrays dispatch directly to hash32/memequal for the whole array. Non-binary-element arrays generate an LLVM IR loop. The equal loop short-circuits on the first mismatch. Small arrays are still unrolled instead of looping, keeping the simple cases compact. * reflect: fix at-runtime map issues from review, and more found locally Maps created through reflect.MakeMap need hash/equal behavior that matches compiler-created maps. Add hashmapMakeReflect for composite key types, using runtime closures that reconstruct interface{} values from raw key bytes and delegate to the interface hash and equality paths. Interface-keyed maps are already stored as interface values, so use the existing interface hash/equal helpers directly for those. This keeps reflect insert, lookup, delete, and compiled lookup paths consistent. Also fix addressable small values used as interface map keys or interface map values. loadSmallValue puts small indirect values back in the pointer-sized interface data field the same way valueInterfaceUnsafe does. * compiler, interp, reflect: fix pointer map literals; remove interface fallback Package-level map literals with pointer keys (both *T and unsafe.Pointer) crash the compiler: the interp pass panics when trying to hash pointer data as raw bytes, because pointer values in the interp memory model are symbolic identities that do not fit in a byte. Fix this by setting a recoverable error flag instead of panicking. The interp detects the error after each instruction and defers the map insert to runtime init code, where real addresses are available for hashing. This matches how the interp already handles other operations it cannot evaluate at compile time. With this fix, unsafe.Pointer can also be classified as a binary map key, which was the last type requiring the interface-based fallback. Since all comparable types now use either the binary or the compiler-generated hash/equal path, remove the interface fallback from the compiler and reflect packages. * compiler, transform: always pass hash/equal function pointers to hashmapMakeGeneric The compiler now always resolves the hash and equal functions at compile time and passes them directly to hashmapMakeGeneric, instead of passing an algorithm enum to hashmapMake and resolving at runtime. For string keys, the runtime hashmapStringPtrHash/hashmapStringEqual functions are referenced directly. For binary keys, hash32/memequal are referenced. The old hashmapMake with alg enum is retained for reflect, which still needs runtime resolution when creating maps dynamically. The OptimizeMaps transform pass is updated to handle both hashmapMake and hashmapMakeGeneric, and to recognize hashmapGenericSet in addition to hashmapBinarySet and hashmapStringSet. The now-unused hashmapCanGenerateHashEqual helper is removed. * runtime: store large map keys and values indirectly When a map key or value exceeds 128 bytes, the bucket now stores a pointer to separately allocated memory instead of the data inline. This matches Go's MapMaxKeyBytes/MapMaxElemBytes threshold and prevents bucket sizes from exploding for large key/value types. For example, map[[256]byte]int previously used 2128 bytes per bucket (16 header + 256*8 keys + 8*8 values); now it uses 144 bytes per bucket (16 header + 8*8 pointers + 8*8 values). The indirection is fully encapsulated in the runtime via helper functions. Store the computed key and value slot sizes on the hashmap so all runtime and reflect paths use the same bucket layout, including non-indirect keys and values. Add big-key golden coverage and benchmarks. Make the benchmark vary enough key bytes to exercise hashing.
2026-08-06 20:13:40 +00:00 · 2026-05-18 04:31:27 -07:00
parent 89d9e33bca
commit 18033ebc36
19 changed files with 1303 additions and 358 deletions
@@ -13,16 +13,27 @@ import (

 // The underlying hashmap structure for Go.
 type hashmap struct {
-	buckets    unsafe.Pointer // pointer to array of buckets
-	seed       uintptr
-	count      uintptr
-	keySize    uintptr // maybe this can store the key type as well? E.g. keysize == 5 means string?
-	valueSize  uintptr
-	bucketBits uint8
-	keyEqual   func(x, y unsafe.Pointer, n uintptr) bool
-	keyHash    func(key unsafe.Pointer, size, seed uintptr) uint32
+	buckets       unsafe.Pointer // pointer to array of buckets
+	seed          uintptr
+	count         uintptr
+	keySize       uintptr
+	valueSize     uintptr
+	keySlotSize   uintptr // == keySize, or sizeof(ptr) if indirect
+	valueSlotSize uintptr // == valueSize, or sizeof(ptr) if indirect
+	bucketBits    uint8
+	flags         uint8
+	keyEqual      func(x, y unsafe.Pointer, n uintptr) bool
+	keyHash       func(key unsafe.Pointer, size, seed uintptr) uint32
 }

+const (
+	hashmapMaxKeySize   = 128
+	hashmapMaxValueSize = 128
+
+	hashmapFlagIndirectKey   = 1 << 0
+	hashmapFlagIndirectValue = 1 << 1
+)
+
 // A hashmap bucket. A bucket is a container of 8 key/value pairs: first the
 // following two entries, then the 8 keys, then the 8 values. This somewhat odd
 // ordering is to make sure the keys and values are well aligned when one of
@@ -34,6 +45,48 @@ type hashmapBucket struct {
 	// allocated but as they're of variable size they can't be shown here.
 }

+// hashmapBucketHeaderSize is the offset in bytes from the start of a bucket to
+// the first key, aligned to 8 bytes. This ensures that keys requiring 8-byte
+// alignment (float64, complex128, uint64 on strict-alignment architectures
+// like MIPS) are properly aligned in the bucket.
+const hashmapBucketHeaderSize = (unsafe.Sizeof(hashmapBucket{}) + 7) &^ 7
+
+// hashmapKeySlotSize returns the size of a key slot in the bucket. For indirect
+// keys, this is the pointer size; otherwise the actual key size.
+//
+//go:inline
+func hashmapKeySlotSize(m *hashmap) uintptr {
+	return m.keySlotSize
+}
+
+// hashmapValueSlotSize returns the size of a value slot in the bucket.
+//
+//go:inline
+func hashmapValueSlotSize(m *hashmap) uintptr {
+	return m.valueSlotSize
+}
+
+// hashmapSlotKeyData returns a pointer to the actual key data for a given slot.
+// For indirect keys, the slot contains a pointer that must be dereferenced.
+//
+//go:inline
+func hashmapSlotKeyData(m *hashmap, slotKey unsafe.Pointer) unsafe.Pointer {
+	if m.flags&hashmapFlagIndirectKey != 0 {
+		return *(*unsafe.Pointer)(slotKey)
+	}
+	return slotKey
+}
+
+// hashmapSlotValueData returns a pointer to the actual value data for a given slot.
+//
+//go:inline
+func hashmapSlotValueData(m *hashmap, slotValue unsafe.Pointer) unsafe.Pointer {
+	if m.flags&hashmapFlagIndirectValue != 0 {
+		return *(*unsafe.Pointer)(slotValue)
+	}
+	return slotValue
+}
+
 type hashmapIterator struct {
 	buckets      unsafe.Pointer // pointer to array of hashapBuckets
 	numBuckets   uintptr        // length of buckets array
@@ -66,20 +119,35 @@ func hashmapMake(keySize, valueSize uintptr, sizeHint uintptr, alg uint8) *hashm
 		bucketBits++
 	}

-	bucketBufSize := unsafe.Sizeof(hashmapBucket{}) + keySize*8 + valueSize*8
+	var flags uint8
+	keySlotSize := keySize
+	if keySize > hashmapMaxKeySize {
+		flags |= hashmapFlagIndirectKey
+		keySlotSize = unsafe.Sizeof(unsafe.Pointer(nil))
+	}
+	valueSlotSize := valueSize
+	if valueSize > hashmapMaxValueSize {
+		flags |= hashmapFlagIndirectValue
+		valueSlotSize = unsafe.Sizeof(unsafe.Pointer(nil))
+	}
+
+	bucketBufSize := hashmapBucketHeaderSize + keySlotSize*8 + valueSlotSize*8
 	buckets := alloc(bucketBufSize*(1<<bucketBits), nil)

 	keyHash := hashmapKeyHashAlg(tinygo.HashmapAlgorithm(alg))
 	keyEqual := hashmapKeyEqualAlg(tinygo.HashmapAlgorithm(alg))

 	return &hashmap{
-		buckets:    buckets,
-		seed:       uintptr(fastrand()),
-		keySize:    keySize,
-		valueSize:  valueSize,
-		bucketBits: bucketBits,
-		keyEqual:   keyEqual,
-		keyHash:    keyHash,
+		buckets:       buckets,
+		seed:          uintptr(fastrand()),
+		keySize:       keySize,
+		valueSize:     valueSize,
+		keySlotSize:   keySlotSize,
+		valueSlotSize: valueSlotSize,
+		bucketBits:    bucketBits,
+		flags:         flags,
+		keyEqual:      keyEqual,
+		keyHash:       keyHash,
 	}
 }

@@ -118,8 +186,6 @@ func hashmapKeyEqualAlg(alg tinygo.HashmapAlgorithm) func(x, y unsafe.Pointer, n
 		return memequal
 	case tinygo.HashmapAlgorithmString:
 		return hashmapStringEqual
-	case tinygo.HashmapAlgorithmInterface:
-		return hashmapInterfaceEqual
 	default:
 		// compiler bug :(
 		return nil
@@ -132,8 +198,6 @@ func hashmapKeyHashAlg(alg tinygo.HashmapAlgorithm) func(key unsafe.Pointer, n,
 		return hash32
 	case tinygo.HashmapAlgorithmString:
 		return hashmapStringPtrHash
-	case tinygo.HashmapAlgorithmInterface:
-		return hashmapInterfacePtrHash
 	default:
 		// compiler bug :(
 		return nil
@@ -172,7 +236,7 @@ func hashmapLen(m *hashmap) int {

 //go:inline
 func hashmapBucketSize(m *hashmap) uintptr {
-	return unsafe.Sizeof(hashmapBucket{}) + uintptr(m.keySize)*8 + uintptr(m.valueSize)*8
+	return hashmapBucketHeaderSize + hashmapKeySlotSize(m)*8 + hashmapValueSlotSize(m)*8
 }

 //go:inline
@@ -191,16 +255,14 @@ func hashmapBucketAddrForHash(m *hashmap, hash uint32) *hashmapBucket {

 //go:inline
 func hashmapSlotKey(m *hashmap, bucket *hashmapBucket, slot uint8) unsafe.Pointer {
-	slotKeyOffset := unsafe.Sizeof(hashmapBucket{}) + uintptr(m.keySize)*uintptr(slot)
-	slotKey := unsafe.Add(unsafe.Pointer(bucket), slotKeyOffset)
-	return slotKey
+	slotKeyOffset := hashmapBucketHeaderSize + hashmapKeySlotSize(m)*uintptr(slot)
+	return unsafe.Add(unsafe.Pointer(bucket), slotKeyOffset)
 }

 //go:inline
 func hashmapSlotValue(m *hashmap, bucket *hashmapBucket, slot uint8) unsafe.Pointer {
-	slotValueOffset := unsafe.Sizeof(hashmapBucket{}) + uintptr(m.keySize)*8 + uintptr(m.valueSize)*uintptr(slot)
-	slotValue := unsafe.Add(unsafe.Pointer(bucket), slotValueOffset)
-	return slotValue
+	slotValueOffset := hashmapBucketHeaderSize + hashmapKeySlotSize(m)*8 + hashmapValueSlotSize(m)*uintptr(slot)
+	return unsafe.Add(unsafe.Pointer(bucket), slotValueOffset)
 }

 // Set a specified key to a given value. Grow the map if necessary.
@@ -234,9 +296,9 @@ func hashmapSet(m *hashmap, key unsafe.Pointer, value unsafe.Pointer, hash uint3
 			}
 			if bucket.tophash[i] == tophash {
 				// Could be an existing key that's the same.
-				if m.keyEqual(key, slotKey, m.keySize) {
-					// found same key, replace it
-					memcpy(slotValue, value, m.valueSize)
+				if m.keyEqual(key, hashmapSlotKeyData(m, slotKey), m.keySize) {
+					// found same key, replace the value
+					hashmapStoreValue(m, slotValue, value)
 					return
 				}
 			}
@@ -251,11 +313,45 @@ func hashmapSet(m *hashmap, key unsafe.Pointer, value unsafe.Pointer, hash uint3
 		return
 	}
 	m.count++
-	memcpy(emptySlotKey, key, m.keySize)
-	memcpy(emptySlotValue, value, m.valueSize)
+	hashmapStoreKey(m, emptySlotKey, key)
+	hashmapStoreValue(m, emptySlotValue, value)
 	*emptySlotTophash = tophash
 }

+// hashmapStoreKey stores a key into a bucket slot, allocating backing storage
+// if the key is indirect (first insert) or copying into it (shouldn't happen
+// for keys, but handles it correctly).
+//
+//go:inline
+func hashmapStoreKey(m *hashmap, slotKey, key unsafe.Pointer) {
+	if m.flags&hashmapFlagIndirectKey != 0 {
+		p := alloc(m.keySize, nil)
+		memcpy(p, key, m.keySize)
+		*(*unsafe.Pointer)(slotKey) = p
+	} else {
+		memcpy(slotKey, key, m.keySize)
+	}
+}
+
+// hashmapStoreValue stores a value into a bucket slot. For indirect values,
+// it allocates backing storage on first insert or copies into the existing
+// backing on overwrite.
+//
+//go:inline
+func hashmapStoreValue(m *hashmap, slotValue, value unsafe.Pointer) {
+	if m.flags&hashmapFlagIndirectValue != 0 {
+		p := *(*unsafe.Pointer)(slotValue)
+		if p == nil {
+			// First insert: allocate backing storage.
+			p = alloc(m.valueSize, nil)
+			*(*unsafe.Pointer)(slotValue) = p
+		}
+		memcpy(p, value, m.valueSize)
+	} else {
+		memcpy(slotValue, value, m.valueSize)
+	}
+}
+
 // hashmapInsertIntoNewBucket creates a new bucket, inserts the given key and
 // value into the bucket, and returns a pointer to this bucket.
 func hashmapInsertIntoNewBucket(m *hashmap, key, value unsafe.Pointer, tophash uint8) *hashmapBucket {
@@ -267,8 +363,8 @@ func hashmapInsertIntoNewBucket(m *hashmap, key, value unsafe.Pointer, tophash u
 	slotKey := hashmapSlotKey(m, bucket, 0)
 	slotValue := hashmapSlotValue(m, bucket, 0)
 	m.count++
-	memcpy(slotKey, key, m.keySize)
-	memcpy(slotValue, value, m.valueSize)
+	hashmapStoreKey(m, slotKey, key)
+	hashmapStoreValue(m, slotValue, value)
 	bucket.tophash[0] = tophash
 	return bucket
 }
@@ -331,12 +427,12 @@ func hashmapGet(m *hashmap, key, value unsafe.Pointer, valueSize uintptr, hash u
 	for bucket != nil {
 		for i := uint8(0); i < 8; i++ {
 			slotKey := hashmapSlotKey(m, bucket, i)
-			slotValue := hashmapSlotValue(m, bucket, i)
 			if bucket.tophash[i] == tophash {
 				// This could be the key we're looking for.
-				if m.keyEqual(key, slotKey, m.keySize) {
+				if m.keyEqual(key, hashmapSlotKeyData(m, slotKey), m.keySize) {
 					// Found the key, copy it.
-					memcpy(value, slotValue, m.valueSize)
+					slotValue := hashmapSlotValue(m, bucket, i)
+					memcpy(value, hashmapSlotValueData(m, slotValue), m.valueSize)
 					return true
 				}
 			}
@@ -370,13 +466,15 @@ func hashmapDelete(m *hashmap, key unsafe.Pointer, hash uint32) {
 			slotKey := hashmapSlotKey(m, bucket, i)
 			if bucket.tophash[i] == tophash {
 				// This could be the key we're looking for.
-				if m.keyEqual(key, slotKey, m.keySize) {
+				if m.keyEqual(key, hashmapSlotKeyData(m, slotKey), m.keySize) {
 					// Found the key, delete it.
 					bucket.tophash[i] = 0
-					// Zero out the key and value so garbage collector doesn't pin the allocations.
-					memzero(slotKey, m.keySize)
+					// Zero out the slot so the GC won't pin the allocations.
+					keySlotSize := hashmapKeySlotSize(m)
+					memzero(slotKey, keySlotSize)
 					slotValue := hashmapSlotValue(m, bucket, i)
-					memzero(slotValue, m.valueSize)
+					valueSlotSize := hashmapValueSlotSize(m)
+					memzero(slotValue, valueSlotSize)
 					m.count--
 					return
 				}
@@ -438,13 +536,13 @@ func hashmapNext(m *hashmap, it *hashmapIterator, key, value unsafe.Pointer) boo

 		// Found a key.
 		slotKey := hashmapSlotKey(m, it.bucket, it.bucketIndex)
-		memcpy(key, slotKey, m.keySize)
+		memcpy(key, hashmapSlotKeyData(m, slotKey), m.keySize)

 		if it.buckets == m.buckets {
 			// Our view of the buckets is the same as the parent map.
 			// Just copy the value we have
 			slotValue := hashmapSlotValue(m, it.bucket, it.bucketIndex)
-			memcpy(value, slotValue, m.valueSize)
+			memcpy(value, hashmapSlotValueData(m, slotValue), m.valueSize)
 			it.bucketIndex++
 		} else {
 			it.bucketIndex++
@@ -491,6 +589,112 @@ func hashmapBinaryDelete(m *hashmap, key unsafe.Pointer) {
 	hashmapDelete(m, key, hash)
 }

+// Hashmap with compiler-generated key hash/equal functions.
+// Unlike the binary path (which uses hash32/memequal), these use the
+// type-specific keyHash and keyEqual function pointers stored in the hashmap
+// struct. This is used for composite key types (e.g. structs containing
+// strings) where the compiler generates specialized hash/equal functions.
+
+func hashmapGenericSet(m *hashmap, key, value unsafe.Pointer) {
+	if m == nil {
+		nilMapPanic()
+	}
+	hash := m.keyHash(key, m.keySize, m.seed)
+	hashmapSet(m, key, value, hash)
+}
+
+func hashmapGenericGet(m *hashmap, key, value unsafe.Pointer, valueSize uintptr) bool {
+	if m == nil {
+		memzero(value, uintptr(valueSize))
+		return false
+	}
+	hash := m.keyHash(key, m.keySize, m.seed)
+	return hashmapGet(m, key, value, valueSize, hash)
+}
+
+func hashmapGenericDelete(m *hashmap, key unsafe.Pointer) {
+	if m == nil {
+		return
+	}
+	hash := m.keyHash(key, m.keySize, m.seed)
+	hashmapDelete(m, key, hash)
+}
+
+// hashmapMakeGeneric creates a new hashmap with compiler-provided hash and
+// equal functions. This avoids the interface/reflection path for composite
+// key types like structs containing strings.
+func hashmapMakeGeneric(keySize, valueSize uintptr, sizeHint uintptr,
+	keyHash func(key unsafe.Pointer, size, seed uintptr) uint32,
+	keyEqual func(x, y unsafe.Pointer, n uintptr) bool) *hashmap {
+	bucketBits := uint8(0)
+	for hashmapHasSpaceToGrow(bucketBits) && hashmapOverLoadFactor(sizeHint, bucketBits) {
+		bucketBits++
+	}
+
+	var flags uint8
+	keySlotSize := keySize
+	if keySize > hashmapMaxKeySize {
+		flags |= hashmapFlagIndirectKey
+		keySlotSize = unsafe.Sizeof(unsafe.Pointer(nil))
+	}
+	valueSlotSize := valueSize
+	if valueSize > hashmapMaxValueSize {
+		flags |= hashmapFlagIndirectValue
+		valueSlotSize = unsafe.Sizeof(unsafe.Pointer(nil))
+	}
+
+	bucketBufSize := hashmapBucketHeaderSize + keySlotSize*8 + valueSlotSize*8
+	buckets := alloc(bucketBufSize*(1<<bucketBits), nil)
+
+	return &hashmap{
+		buckets:       buckets,
+		seed:          uintptr(fastrand()),
+		keySize:       keySize,
+		valueSize:     valueSize,
+		keySlotSize:   keySlotSize,
+		valueSlotSize: valueSlotSize,
+		bucketBits:    bucketBits,
+		flags:         flags,
+		keyEqual:      keyEqual,
+		keyHash:       keyHash,
+	}
+}
+
+// hashmapMakeReflect creates a hashmap for reflect.MakeMapWithSize using
+// closures that reconstruct interface{} values from raw key bytes,
+// delegating to hashmapInterfaceHash for hashing and == for equality.
+func hashmapMakeReflect(keySize, valueSize, sizeHint uintptr, keyType unsafe.Pointer) *hashmap {
+	t := (*reflectlite.RawType)(keyType)
+	if t.Kind() == reflectlite.Interface {
+		// Interface keys are already stored as interface values in the
+		// bucket; use the existing interface hash/equal directly.
+		return hashmapMakeGeneric(keySize, valueSize, sizeHint,
+			hashmapInterfacePtrHash, hashmapInterfaceEqual)
+	}
+	keyHash := func(key unsafe.Pointer, size, seed uintptr) uint32 {
+		return hashmapInterfaceHash(rawToInterface(t, key), seed)
+	}
+	keyEqual := func(x, y unsafe.Pointer, n uintptr) bool {
+		return rawToInterface(t, x) == rawToInterface(t, y)
+	}
+	return hashmapMakeGeneric(keySize, valueSize, sizeHint, keyHash, keyEqual)
+}
+
+// rawToInterface reconstructs an interface{} from raw bytes at ptr.
+func rawToInterface(t *reflectlite.RawType, ptr unsafe.Pointer) interface{} {
+	var val unsafe.Pointer
+	if t.Size() <= unsafe.Sizeof(uintptr(0)) {
+		val = reflectliteLoadSmallValue(ptr, t.Size())
+	} else {
+		val = ptr
+	}
+	i := composeInterface(unsafe.Pointer(t), val)
+	return *(*interface{})(unsafe.Pointer(&i))
+}
+
+//go:linkname reflectliteLoadSmallValue internal/reflectlite.loadSmallValue
+func reflectliteLoadSmallValue(ptr unsafe.Pointer, size uintptr) unsafe.Pointer
+
 // Hashmap with string keys (a common case).

 func hashmapStringEqual(x, y unsafe.Pointer, n uintptr) bool {
@@ -626,28 +830,3 @@ func hashmapInterfacePtrHash(iptr unsafe.Pointer, size uintptr, seed uintptr) ui
 func hashmapInterfaceEqual(x, y unsafe.Pointer, n uintptr) bool {
 	return *(*interface{})(x) == *(*interface{})(y)
 }
-
-func hashmapInterfaceSet(m *hashmap, key interface{}, value unsafe.Pointer) {
-	if m == nil {
-		nilMapPanic()
-	}
-	hash := hashmapInterfaceHash(key, m.seed)
-	hashmapSet(m, unsafe.Pointer(&key), value, hash)
-}
-
-func hashmapInterfaceGet(m *hashmap, key interface{}, value unsafe.Pointer, valueSize uintptr) bool {
-	if m == nil {
-		memzero(value, uintptr(valueSize))
-		return false
-	}
-	hash := hashmapInterfaceHash(key, m.seed)
-	return hashmapGet(m, unsafe.Pointer(&key), value, valueSize, hash)
-}
-
-func hashmapInterfaceDelete(m *hashmap, key interface{}) {
-	if m == nil {
-		return
-	}
-	hash := hashmapInterfaceHash(key, m.seed)
-	hashmapDelete(m, unsafe.Pointer(&key), hash)
-}