int64 IDs

[bradfitz]

On 32-bit platforms, a Go int is an int32.

I worry about the use of int as the type for Github IDs in the go-github package.

Is there documentation to suggest that Github IDs are limited to 2 billion items? Seems unlikely. But I can't find anything.

I propose we keep "Number" fields as int, but change all the ID fields to be int64.

Thoughts?

/cc @gmlewis @willnorris @shurcooL

[gmlewis]

I totally agree. Absolutely.

[dmitshur]

I've noticed that too.

By now, there are definitely some IDs that exceed the capacity of int32. I have the following example. Consider the endpoint that returns all public events:

https://api.github.com/events

The first event I got from a query just now was:

{
  "id": "5510765310",
  "type": "WatchEvent",
  "actor": {
    "id": 12230841,

  },
  "repo": {
    "id": 76620213,

  },
  "payload": {
    "action": "started"
  },
  "public": true,
  "created_at": "2017-03-17T01:19:24Z"
}

The ID of that event exceeds 2^31 - 1.

So far we've been using int IDs for consistency with existing fields. But it definitely makes sense to address all of them at once in a future change.

I propose we keep "Number" fields as int, but change all the ID fields to be int64.

SGTM.

[dmitshur]

Actually, that's not a great example because https://godoc.org/github.com/google/go-github/github#Event.ID type is string, not int.

[dmitshur]

The fact that both events and notifications (types of objects that are very likely to have IDs surpass 2 billion) encode their IDs as JSON strings rather than JSON numbers suggests that other IDs may be much smaller.

Is there documentation to suggest that Github IDs are limited to 2 billion items? Seems unlikely. But I can't find anything.

I've looked, and couldn't find anything either. But this is something that's important for the API docs to specify.

Perhaps it'd be useful to send GitHub support an email to inquire about this. If they can assure us that those IDs are guaranteed to fit into int32, we won't have to make a change now. If they can confirm that int32 may not be enough, we can feel better about making the change.

[bradfitz]

If they can assure us that those IDs are guaranteed to fit into int32

It's a global ID, right? It's not an ID scoped to just a single org or repo, right? If so, we don't need their confirmation. 2 billion is too small.

because https://godoc.org/github.com/google/go-github/github#Event.ID type is string

That is its own question. Why is it a string and not some integer type? Can it actually be non-numeric?

[dmitshur]

It's a global ID, right? It's not an ID scoped to just a single org or repo, right?

Which ID are you talking about? As far as I know, different types of objects have different ID namespaces.

There are IDs for users, repositories, events, notifications, gists (these are non-numbers), installations, etc.

I'm sure that at least the user and repo IDs are separate namespaces (so may share equal IDs; here is user with ID 1, and here's repo with ID 1), but not completely sure about the others.

Why is it a string and not some integer type?

The JSON object actually contains a JSON string, so it has to be decoded into a string.

We could consider the "string" option that encoding/json offers, then decoding into int64 is possible:

The "string" option signals that a field is stored as JSON inside a JSON-encoded string. It applies only to fields of string, floating point, integer, or boolean types. This extra level of encoding is sometimes used when communicating with JavaScript programs:

Int64String int64 `json:",string"`

But then we'd need to be very confident it would always work (not contain non-numeric characters, and not overflow int64). It's easier to be confident that string will always work.

Both Event.ID and Notification.ID were added in 2013-2014, and I didn't see any discussion about the decision to use string type for ID there (maybe the "string" option wasn't even in encoding/json back then).

Can it actually be non-numeric?

It doesn't seem that way. I find it somewhat hard to feel confident if GitHub API docs don't actually specify this.

On the other hand, Gist IDs are non-numeric, and it's clear from examples.

[bradfitz]

I didn't know (or don't know whether) there is an ID namespace per resource type. But that also doesn't change my point. That just gives us ~5x more headroom if so.

If we're concerned about quoted integers versus not, we could make our own numeric ID type:

type ID int64

... that implements https://golang.org/pkg/encoding/json/#Unmarshaler and deals with string-quoted integers or non-integers and not work about ",string" or changes to the encoding over time. The accessor methods could return int64 instead of github.ID.

For Gists or other hashes-as-IDs, I'd just keep it as "ID *string".

[dmitshur]

Relevant issue (in a theoretical discussion sense, not practical one):

golang/go#19623

[sahildua2305]

@shurcooL @gmlewis I see that there was agreement on changing to int64. What's the take on this at this point?

[dmitshur]

There is no agreement at this time. We're waiting on someone to run into a concrete issue before we can decide how to act on this. So far, there have not been any issue reports, so it's better to wait.

Go 2 might change int to be arbitrary precision (golang/go#19623), so it would be better not to change to int64 if that happens.

Also, see what I wrote in #733 (comment) relatively recently:

IMO it's unclear what the state of #597 is right now. The issue has been opened a while ago, but I haven't seen any reports of issues from production use. I'm not convinced that we should act right away, or how we should act. I think that we need to collect more data in that issue (or, wait until there's a concrete need to act).