proposal: io/fs.Hash and Hasher

[earthboundkid]

Background:

• When serving an embed.FS as an http.FileSystem, it would be nice to have etag support.
• Many file stores (e.g. S3) provide a way to get the MD5 hash of file.

Proposal:

Add a func to io/fs to support getting the hash of file.

Perhaps this?

func Hash(fs FS, h hash.Hash, path string) ([]byte, error)

type Hasher interface {
   Hash(h hash.Hash, path string) ([]byte, error)
}

If there's a dependency import issue, maybe put it into package hash.

Ref. #41191 (comment)

[earthboundkid]

On further consideration, crypto.Hash is better than hash.Hash. But that would almost certainly require not putting it into io/fs.

[tv42]

hash.Hash is an interface (https://golang.org/pkg/hash/#Hash), is this supposed to be used like h := fs.Hash(fsys, fnv.New128a(), "index.html")? There's some cost to such a New call, more than passing a string identifier or such.

If you meant crypto.Hash (https://golang.org/pkg/crypto/#Hash), I'd like to protest against a central registry that doesn't cater to non-stdlib use.

Also, I think we should cater for a "just give me whatever hash you happen to have" use; e.g. ETags don't care.

[earthboundkid]

I was thinking there was a way to look at a hash.Hash and figure out that it was an MD5 hasher, but I think I was thinking of crypto.Hash. The fact that cypto.Hash can't be extended outside stdlib is bad, yes. I guess the question is whether just having any hash is good enough or users will care to be able to ensure that the hash is a particular one.

https://godoc.org/gocloud.dev/blob#Attributes is probably what should be used as prior art and copied:

    // CreateTime is the time the blob was created, if available. If not available,
    // CreateTime will be the zero time.
    CreateTime time.Time
    // ModTime is the time the blob was last modified.
    ModTime time.Time
    // Size is the size of the blob's content in bytes.
    Size int64
    // MD5 is an MD5 hash of the blob contents or nil if not available.
    MD5 []byte
    // ETag for the blob; see https://en.wikipedia.org/wiki/HTTP_ETag.
    ETag string

[tv42]

Some alternatives:

1. Identify hash algorithms by strings. Empty string means "anything".
   1a. Just let chaos reign, like database/sql.Register
   1b. stdlib gets dotless crypto.Hash.String names, hash/fnv gets "FNV-1", "FNV-1a", "FNV-1a-64" and such. Non-stdlib hash implementations like Blake3 are required to use their import paths. (edit: as prefixes, with variants allowed much like Go identifiers)
   1c. (edited to add) Even stdlib hashes are identified by their import paths.

2. Identify hash algorithms by an interface{} value exported by the relevant hash library, with the same rules about uniqueness (edit: and comparability) as context.WithValue. nil means "anything".

[earthboundkid]

So what if the proposal is instead http.ReadETagFS:

type ReadETagFS interface {
    fs.FS
    ReadETag(path string) (string, error)
}

and embed.FS gets a ReadETag method that just returns the MD5? ISTM that solves the major use case and is compatible with what you get from S3 and other blob storage.

[tv42]

I wouldn't tie fs.FS to specifically ETags or MD5.

type ShortHashFS interface {
    fs.FS

    // ShortHash returns a short hash digest summarizing the file.
    // The returned digest is expected to be different if the file content is different,
    // but the hash algorithm used is not required to be cryptographically strong
    // and the length of the digest may not be long enough to provide perfect
    // uniqueness.
    // Length of the digest must be at least 8 bytes.
    ShortHash(path string) (digest []byte, err error)
}

(And then error can be whatever ErrNotSupported convention wins.)

I'm returning []byte not string as I'd rather have encoding the digest be the caller's responsibility, if and when that is needed, as 1) it's not always needed, e.g. for use in protobuf 2) the allowed character set varies, and e.g. ETag has a bunch of extra rules on top.

I think the use case for asking for a specific hash is valid, but that can be a separate concern. Example: To upload to S3, I can ensure integrity by providing an MD5 as Content-MD5, but it's not mandatory. Except if I'm using signed urls (AWS sig v4), then I need SHA256 in x-amz-content-sha256 instead. In 2022, FooCloud might prefer Blake3.

[tv42]

Requirements for an ideal "do you have a shortcut for my custom hash" extension interface:

1. Return a hash.Hash so caller can e.g. say r := h.(*blake3.Hasher).XOF() https://godoc.org/lukechampine.com/blake3#Hasher.XOF for arbitrary-size streaming output instead of a predetermined []byte.

2. Take a "configurable hash algorithm request" as argument, to support keyed (personalized/salted) hashes and e.g. hashes where the result depends on parallelization. E.g. a content-addressed storage mechanism might not want to store a SHA256 that allows a confirmation-of-a-file attack, but use a hash that is meaningful only if you already know the key. The earlier idea of taking interface{} much like context.WithValue works here.

type PrehashedFS interface {
    fs.FS

    // Prehashed returns a hasher that has already digested the file contents,
    // if one has been precomputed.
    // Calling hash.Hash.Write on the returned hasher MAY return an error.
    // algo follows the same rules as context keys, and for parameterized
    // hashes can also be an instance of a configuration data type.
    // If algo is not recognized by the FS, or the digest is not already available,
    // returns ErrNotSupported.
    Prehashed(path string, algo interface{}) (hash.Hash, error)
}

// Prehashed ...
// Prehashed will not compute a hash digest if one was not already available.
func Prehashed(fsys FS, path string, algo interface{}) (hash.Hash, error) {
    ...
}

Note that there's no guarantee that the returned hash.Hash has the same underlying type as e.g. sha256.New(). Many hash implementations don't support that. To unlock special features like XOF, new hash implementations can support marshaling+unmarshaling their state, and then the caller can type assert the returned hash.Hash to something meaningful.

Note that the hash.Hash returned may have to break https://golang.org/pkg/hash/#Hash rules that say Write never returns an error. If all I have is a []byte result of an earlier SHA256, there's no way I can continue it. If this feels too ugly, then this needs its own interface defined that's a subset of hash.Hash, or something like that. That doesn't change the big idea: return something that can be type asserted for functionality beyond the basic.

Example simple use:

h, err := fs.Prehashed(fsys, "index.html", crypto.SHA256)
if err != nil { ... }
fmt.Printf("hash is %x\n", h.Sum(nil))

Example advanced use:

import "imaginary.example.com/blake4"

algo := &blake4.Config{
    // Hash is personalized with this key, see https://godoc.org/lukechampine.com/blake3#New
    Key: []byte("example.com/frobnicator 2020-12-08 blob hash"),
}
h, err := fs.Prehashed(fsys, "index.html", algo)
if err != nil { .... }
buf := make([]byte, something)
reader := h.(*blake4.Hasher).XOF()
_, err := io.ReadFull(reader, buf)

(That would need an extra non-stdlib API that https://godoc.org/lukechampine.com/blake3 doesn't provide, to construct a keyed hasher from its marshaled internal state.)

[tv42]

And the alternative for ReadETagFS or ShortHashFS using just PrehashedFS, without those special case interfaces but just nil arg to say "anything goes", would look like

h, err := fs.Prehashed(fsys, "index.html", nil)
if err != nil { ... }
buf := h.Sum(nil)

I'd still document some base assumptions, like minimum 8 byte length, in the PrehashedFS interface.

[Merovius]

Personally, I don't like the idea of identifying the hash algorithm via a string. It's too easy to mistype and there is no compiler-check. I think a better solution would be an exported constant, so you'd pass sha256.Hash (or something). Or just accept an interface { New() hash.Hash }, which a) can still be a symbolic constant implementing that interface and b) in particular, could be a crypto.Hash (but the fs package doesn't have to actually mention crypto).

However, really, I don't know if it even makes sense to pass a hash algorithm. From what I can tell, a fs.FS could only really provide useful support, by having hashes pre-computed - that is, //go:embed would precompute a sha256 hash and embed it and an S3-FS can retrieve an MD5 hash via the API. But in that case, the choice of hash algorithm is fixed by the filesystem, so it doesn't really make sense for the user of the fs.FS to pass a hash algo. Even in the "integrity-protect an S3 upload" use-case, there only really is a benefit if the FS you are uploading from already has that specific hash computed - so again, the filesystem itself chooses the Algorithm it supports.

So, personally, I don't really think it makes a lot of sense to pass a hash function of any sorts to the FS. At least not as part of a generalized filesystem API. For the ETag use-case, an optional interface like @tv42's ShortHashFS seems much more reasonable.