net: add UDPMsg, (*UDPConn).ReadUDPMsgs, (*UDPConn).WriteUDPMsgs

[bradfitz]

(co-written with @neild)

Linux has recvmmsg to read multiple UDP packets from the kernel at once.

There is no Recvmmsg wrapper func in golang.org/x/sys/unix. That's easy enough to add, but it's not ideal: it means all callers of it would be using a thread while blocked waiting for a packet.

There is, however, batch support in golang.org/x/net/ipv{4,6}: e.g. https://pkg.go.dev/golang.org/x/net/ipv4#PacketConn.ReadBatch (added around golang/net@b8b1343). But it has the same thread-blocking problem. And it has the additional problem of having separate packages for IPv4 vs IPv6.

It'd be nicer to integrate with the runtime's poller.

Adding API to do this in the net package would mean both:

1. we'd be able to integrate with the runtime poller (epoll, etc) and not waste a thread during a read
2. there'd be portable API to do this regardless of whether the platform/kernel version supports something like recvmmsg.

For writing, net.Buffers already exists, as does golang.org/x/net/ipv{4,6}'s PacketConn.WriteBatch, so is less important, but could be done for consistency.

As far as a potential API, https://pkg.go.dev/golang.org/x/net/ipv4#PacketConn.ReadBatch is close, but the platform-specific flags should probably not be included, at least as an int. While there's some precedent with https://golang.org/pkg/net/#UDPConn.ReadMsgUDP use of flags int, we could probably use a better type if we need flags for some reason.

Alternatively, if callers of x/sys/unix or x/net/ipv{4,6} could do this efficiently with the runtime poller, that'd also work (even if they'd need to use some build tags, which is probably tolerable for anybody who cares about this).

[ianlancetaylor]

What should the allocation strategy be here? The existing ReadMsgUDP method takes a pair of buffers and fills them in. Should this new method take a slice of buffers and somehow return how much data was read into each buffer?

[bradfitz]

At a high level, the caller should supply all the memory. The whole thing should be zero allocations (see also: #43451 and https://golang.org/cl/291509), otherwise the sort of people who'd want to use this probably wouldn't want to use it.

Probably pass a slice of messages similar to ipv4.PacketConn.ReadBatch but likely with a slightly different Message. I don't think the Message.Addr net.Addr field is amenable to the midstack inlining optimization from https://golang.org/cl/291509.

[neild]

A possible concrete API:

type UDPMessage struct {
  // recvmmsg accepts a per-message scatter/gather array, so this could be [][]byte instead.
  Buffer []byte
  OOB    []byte
  Addr   UDPAddr

  // The existing (*UDPConn).ReadMsgUDP method returns flags as an int,
  // but perhaps this should be a more abstract type.
  Flags  int
}

// ReadUDPBatch reads multiple messages from c
// It returns the number of messages read.
// It reads the payload into Buffer and associated out-of-band data into OOB,
// and sets the length of each slice to the extent of the read data.
// It sets the Addr and Flags fields to the source address and flags set on each message.
func (c *UDPConn) ReadUDPBatch(ms []UDPMessage) (int, error)

This API preserves the existing limitation that there is no way to provide recvmsg flags to *UDPConn methods. (I'm quite confused, by the way, since it looks like we never set the MSG_OOB flag on recvmsg calls even when reading OOB data. Is this flag not actually necessary?)

[ianlancetaylor]

UDP doesn't have out-of-band data. TCP does. I have no idea why UDPConn.ReadMsgUDP takes an oob argument or an oobn result. Maybe I'm missing something.

For that matter I'm not sure off hand how to read out-of-band data for a TCP socket using the net package.

[neild]

UDP doesn't have out-of-band data.

Oh, right. I mean "control" data--the msg_control field of a struct msghdr. The fact that ReadMsgUDP calls its parameter oob for some reason led me astray.

[ianlancetaylor]

Whoops, that led me astray also. But the end result is the same. As you say, the oob parameter to ReadMsgUDP, if not empty, can be filled with the ancillary data that the readmsg system call returns in the msg_control field. But UDP sockets never have any ancillary data. So it's still pointless.

[neild]

UDP sockets can have ancillary data: Setting the IP_PKTINFO sockopt will give you a control message containing the interface index the packet was received on among other info (useful for DHCP servers), and the IP_TOS sockopt will give you the ToS byte.

[ianlancetaylor]

Ah, OK, thanks.

[rsc]

We may want to wait on doing anything here until we figure out what to do with IP addresses, which still allocate.

[josharian]

It is important that the caller be able to specify whether they want to block for N packets, or receive up-to-N packets but only block until at least one is available. On linux that's accomplished through flags, but we might want a nicer, higher-level way to express that.

[ianlancetaylor]

@josharian Is it important to support both? For Read we only support "up-to-N", and make up for the lack via io.ReadFull.

[josharian]

Hmm. Yeah, I think always up-to-N seems fine, at least for my uses.