Add support for hub.datalad.org

[jwodder]

For #47.

This PR includes updates to datalad-repos.json and the READMEs for hub.datalad.org. Fetching all of the repositories took 13 minutes.

[yarikoptic]

Let's also restrict query only to non-private repos. On https://hub.datalad.org/api/swagger#/repository/repoSearch I see

so it could potentially include private repos for the user providing the token.

[jwodder]

@yarikoptic Should this be done for GIN as well? If so, should anything be done to datalad-repos.json to remove private repositories?

[yarikoptic]

yes, if it is a supported option there -- I didn't check

[yarikoptic]

For now, since other work already builds on it, let's proceed with this and improve upon in subsequent PRs

Thank you @jwodder !!!

[jwodder]

@yarikoptic

• This API reference does not list such a query parameter, and this comment states that /repos/search on GIN only returns public repositories.

• Note that the repository objects do include a "private" field, so GIN does support private repositories.

• Using the following script, I was able to find the private repository sanook/sanook on page 34 of the /repos/search results. For whatever reason, this repository is not listed in datalad-repos.json.

  #!/usr/bin/env -S pipx run
  # /// script
  # requires-python = ">=3.11"
  # dependencies = ["python-dotenv ~= 1.0", "requests ~= 2.20"]
  # ///
  
  from __future__ import annotations
  import os
  from dotenv import load_dotenv
  import requests
  
  
  def main() -> None:
      load_dotenv()
      with requests.Session() as s:
          s.headers["Authorization"] = "token " + os.environ["GIN_TOKEN"]
          url = "https://gin.g-node.org/api/v1/repos/search"
          page = 1
          while True:
              print(".", end="", flush=True)
              r = s.get(url, params={"page": page})
              if r.status_code == 500:
                  page += 1
                  continue
              r.raise_for_status()
              if repos := r.json()["data"]:
                  for repo in repos:
                      if repo["private"]:
                          print()
                          print(f"Found private repo {repo['full_name']} on page {page}")
                          return
              else:
                  break
              page += 1
  
  
  if __name__ == "__main__":
      main()

• Adding private=false&is_private=false to an authenticated curl request for /repos/search?page=34 did not remove the private repository from the results.

[jwodder]

@yarikoptic Please create an issue about handling of private repositories on hub.datalad.org and GIN.

[yarikoptic]

done:

• Exclude private repositories #60