Reduce decoding allocations
The decoder operates using the io.Reader
abstraction, which means that if it e.g. wants to decode a length prefix, it must allocate 8 bytes, read into the byte slice, and decode the result. This is unavoidable if the underlying io.Reader
is an *os.File
or a net.Conn
. However, in the majority of cases, we are decoding a byte slice that has already been allocated, as in Unmarshal
:
func Unmarshal(b []byte, v interface{}) error {
r := bytes.NewReader(b)
return NewDecoder(r).Decode(v)
}
In this case, we should be able to skip the Read
call and work directly with the data present in b
.
To implement this, I added a type assertion that checks if the underlying io.Reader
is a bytes.Buffer
. If it is, we can call the Next
method to return the next n
bytes without allocating. Otherwise, we fallback to the allocation-based code.
Profiling suggests that this saves a few GB of allocations per rescan. At some point we should fully implement UnmarshalSia
for types.Block
so that it's fast, reflection-free, and keeps allocations to a minimum.