unique: optimize interning of string([]byte)

[martin-sucha]

Proposal Details

Currently the following function allocates a new string for the string(data) conversion even when the value already exists in the unique map.

func makeUnique(data []byte) unique.Handle[string] {
	return unique.Make(string(data))
}

The built-in map has an optimization to not allocate a string for map lookups, this optimization would be similar.

It seems that during the discussions in the original unique proposal the assumptions were that the string conversion will not allocate in this case, but such behavior was not documented in the final proposed API.

This proposal is to:

• Add an optimization to the compiler/unique package so that unique.Make(string(data)) allocates only when the value does not already exist in the unique map.
• Update the documentation of the unique package to mention this special case.
• Update https://go.dev/wiki/CompilerOptimizations to include the optimization.

Such optimization would be useful for parsers that handle byte slices.

[gabyhelp]

Related Issues

• unique: large string still referenced, after interning only a small substring #69370 (closed)

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[mateusz834]

This does not need to be a proposal.

[mateusz834]

Also it would be nice if this worked with structs containing string, then net/netip could take use of that:

go/src/net/netip/netip.go

Lines 1061 to 1063 in fba83cd

	case n > 16:
	*ip = AddrFrom16([16]byte(b[:16])).WithZone(string(b[16:]))
	return nil

go/src/net/netip/netip.go

Lines 489 to 502 in fba83cd

	// WithZone returns an IP that's the same as ip but with the provided
	// zone. If zone is empty, the zone is removed. If ip is an IPv4
	// address, WithZone is a no-op and returns ip unchanged.
	func (ip Addr) WithZone(zone string) Addr {
	if !ip.Is6() {
	return ip
	}
	if zone == "" {
	ip.z = z6noz
	return ip
	}
	ip.z = unique.Make(addrDetail{isV6: true, zoneV6: zone})
	return ip
	}

[martin-sucha]

Also it would be nice if this worked with structs containing string

If I understand correctly, in your example the allocation would happen already when calling WithZone() as zone is a string parameter. Couldn't net/ip call unique.Make for the zone too?

zone := unique.Make(string(b[16:]))
*ip = AddrFrom16([16]byte(b[:16])).WithZone(zone.Value())

[ianlancetaylor]

CC @mknyszek

[mknyszek]

You can safely do:

func makeUnique(data []byte) unique.Handle[string] {
	return unique.Make(unsafe.String(&data[0], len(data)))
}

We have a test for it, too.

I've been meaning to document that unique.Make clones strings passed to it (provided they're not stored in an interface) if necessary, making this optimization safe. It would be even safer, of course, if there was some help from the compiler. For now, I think this is an OK workaround.

[gopherbot]

Change https://go.dev/cl/671955 mentions this issue: cmd/compile: treat unique.Make[go.shape.string] as non-escaping

[gopherbot]

Change https://go.dev/cl/672135 mentions this issue: cmd/compile: eliminate string copies for calls to unique.Make