[clang/python] Add type annotation to libclang Python binding

[linux4life798]

The libclang Python binding needs type annotations. Type annotations are important for libraries because they enable the IDE to help users learn the API and steer them in the correct direction through code completions. Furthermore, they allow the IDEs + static checkers to identify a large range of misuse errors.

This topic has been discussed a bit in DeinAlptraum's conversation on Discourse.

---

There are a few issues that would need to be addressed before we can simply merge type annotations into the python files.

1. The straightforward way to implement type annotations requires Python 3, but the main cindex.py still makes mention of Python 2.
   • We should be dropping Python 2 support.
2. Type annotations have evolved a bit throughout Python 3 (ex. shift from using List[str] to list[str] OR Optional[str] to str | None). Using the newer syntax/style may make the library incompatible with earlier versions of Python 3. So, we need to figure out a reasonable Python 3 version cutoff and implement CI checks to ensure compatibility with the earliest version we want to support.
   • Python 3.7 might be a good cutoff. This aligns with Debian buster (previous previous stable).
   • Issue [clang/python] Add Python libclang unit tests to CI #76601 should add CI to check with older python versions.

Helpful Resources:

• https://discourse.llvm.org/t/type-annotations-for-libclang-python-bindings/70644/9
• 83768e6
• https://typing.readthedocs.io/en/latest/source/best_practices.html
• https://docs.python.org/3/whatsnew/3.7.html#pep-560-core-support-for-typing-module-and-generic-types
• https://peps.python.org/pep-0585
• https://stackoverflow.com/questions/63939138/is-there-a-way-to-use-python-3-9-type-hinting-in-its-previous-versions

[linux4life798]

As Endill pointed out in this discourse thread, it looks like we are enforcing Python 3.6 minimum in cmake, but RFC Upgrading LLVM’s minimum required Python version tried to raise the minimum to 3.8. It doesn't look like this has been enforced, yet.

• Since Python 3.3, it looks like collections.abc was available and the use of abc as part of the collections base module was removed in Python 3.8. So, for any version 3.3 and later, you should import from collections.abc, I believe.
• Since Python 3.5, you can express types using the typing module.
• Since Python 3.6, you can use f-strings.
• Since Python 3.6, you can use the os.fspath function.
• Starting with Python 3.7, you can use the new std generics list[str] style annotation if you from __future__ import annotations. See https://peps.python.org/pep-0585/#implementation.
• Starting in Python 3.8, you can express literal types, like Literal['r', 'rb', 'w', 'wb'].
  • Python 3.7 equivalent: I don't believe there is one. You just can't express a type that is always a literal value.
• Starting in Python 3.9, you can use the new std generic syntax (list[str]) without the special imports. See https://peps.python.org/pep-0585/#backwards-compatibility.
  • Python 3.7 equivalent: When you import annotations from __future__ you can use the nice std type generics.
• I do see Python in other parts of the LLVM source tree that are using the newer Python 3.9 syntax without future imports.

  • llvm-project/llvm/utils/lldbDataFormatters.py

    Line 421 in 90c397f

    key_buckets: dict[str, list[int]] = collections.defaultdict(list)

  • llvm-project/libcxx/utils/generate_extended_grapheme_cluster_table.py

    Line 241 in 90c397f

    ranges: list[PropertyRange], props: list[str]

  • llvm-project/libcxx/utils/generate_extended_grapheme_cluster_test.py

    Lines 23 to 28 in 90c397f

    	class BreakTestItem:
    	code_points: list[int] = field(default_factory=list)
    	encoded: str = ""
    	breaks_utf8: list[int] = field(default_factory=list)
    	breaks_utf16: list[int] = field(default_factory=list)
    	breaks_utf32: list[int] = field(default_factory=list)

• Starting in Python 3.10, you could start expressing union types, like Optional[str] as str | None. See https://peps.python.org/pep-0604.
  • Python 3.7 equivalent: You can't use the fancy | union type operator. You need to use the traditional classes in the typing module, like typing.Optional and typing.Union.
• Starting in Python 3.11, you could use the Self type within a class. See https://peps.python.org/pep-0673.
  • Python 3.7 equivalent: When you import annotations from __future__ you can use the class's name inside that class. See https://stackoverflow.com/a/33533514/940118.

[anujrmohite]

• @linux4life798 I've been reading through the thread on adding type annotations support to libclang. As I am new to LLVM, I was wondering if I could help out with parts of implementing this? I'd be happy to give it a try and appreciate any style guide to follow.

[JamesParrott]

This is a noble goal Craig, and would be a great improvement to the Python part of the code base.

It would be helpful for potential contributors to specify which directories are part of the libclang Python binding. I would go one further and as a first step only add annotations to functions and classes in the external API, as that seems to be what's requested by users.

Otherwise it's a huge task. There are 2381 .py files in the repo, 332,752 lines of Python code overall (475 files and 119,829 lines excluding tests).

I couldn't even search for all the files in File Explorer - I had to use Python itself. That alone makes me wonder what automated tooling is available.

>>> import pathlib
>>> llvm_repo = pathlib.Path('.')
>>> sum(1 for __ in llvm_repo.glob('**/*.py'))
2381
>>> sum(1 for py_file in llvm_repo.glob('**/*.py') for line in py_file.open('rt', encoding='utf8') if 'shtest-encoding' not in str(py_file))
332752
>>> sum(1 for py_file in llvm_repo.glob('**/*.py') for line in py_file.open('rt', encoding='utf8') if 'test' not in str(py_file))
119829

[linux4life798]

Thanks @anujrmohite! I already have a good chunk of annotations added in https://github.com/linux4life798/llvm-project/tree/libclang-python-type-annotations branch, but it still needs to be cleaned up a bit and tested. Feel free to comment on, test, and add to those changes!

I think we need the proper CI in place before we merge a bunch of changes. I'm trying to make progress on that in #76601.

[linux4life798]

Thanks @JamesParrott! Luckily the libclang Python binding is pretty standalone and isolated from the rest of the Python in the llvm-project. The Python binding for libclang is really just https://github.com/llvm/llvm-project/blob/main/clang/bindings/python/clang/cindex.py file.

The root directory https://github.com/llvm/llvm-project/tree/main/clang/bindings/python also holds some unittest and example scripts.