[Hinty] Add type hints to Scapy

[gpotter2]

Project "Hinty" aims at adding Type hints to Scapy. It will help discover bugs, improve the API, and make Scapy up-to-date with the high standards of Python libraries.

Implementation

We use mypy to ensure automatic testing of the work that has already been completed. PRs that fall under project Hinty will process one (or a few) files and register them into the checks. The file .config/mypy/mypy_enabled.txt lists the files in which mypy checks are enabled. This process has been added by #2162

Because we support Python 2.7, the format of the Type hints should be the following:
https://mypy.readthedocs.io/en/latest/cheat_sheet.html

Guide on how to contribute

0. 💿 Make sure you've "git clonned" the latest version of scapy and are working from within it !

1. ✏️ Pick a file you want to work on. If you're new on scapy, pick a file from within the layers/ or contrib/ folder that is small in size and complexity: at this point in time, the remaining layers in Scapy's core are often hard to type.

2. ✏️ Add that file to .config/mypy/mypy_enabled.txt. This will enable it in our test suite.

3. ➡️ pip install pyannotate cryptography tox mock (install tox, pyannotate and cryptography)

4. ➡️ Run the tests for your file: tox --sitepackages -- -x -K tcpdump -K manufdb -K wireshark -K tshark -K brotli -K zstd -K ci_only -K not_pyannotate -N
   You shall specify a -e py[38]-[linux]_non_root (fill the brackets with your OS infos) option before the --sitepackages option if you have multiple versions of python installed. To see all available tox tags (the whole py...non_root string), type tox -l. The -x option tells our test suite to use pyannotate.
   This will create test/pyannotate_results

   • ⚠️ ✏️ In many cases, pyannotate will glitch out (ParseError: Invalid type comment: (__main__:f.<locals>.A) -> None dropbox/pyannotate#92). You need to open pyannotate_results with your favorite editor and replace all occurrences of <locals> (with the <>) by blanks (or anything really). For instance on vim: :%s/<locals>//g

5. ➡️ pyannotate --type-info test/pyannotate_results -w [the file.py you are working on]

6. ✏️ The file has been automatically processed. Now, edit it to fix the mistakes, and check your work with tox -e mypy. This might take time

   • ℹ️ Remember that you can use cast(type, obj) from typing to help mypy without affecting the code.
   • ⚠️ Import any types you might need from scapy.compat instead of typing. It provides a fallback if typing isn't installed (python < 3.5). I know it's annoying.
   • ℹ️ If a class extends Field directly (very few do) it will require to be passed the internal and machine representation of the field. See this if you don't know what I'm talking about. For instance: class subField(Field[float, int]): is a field where i=float and m=int. In most cases, it should be [int, int] (a normal integer field).

7. ⚠️ If you get error: No library stub file for module ..., add an exception for it in .config/mypy/mypy.ini (follow the existing format)

8. ✅ Check that the files still pass PEP8: tox -e flake8

9. 🥇 When everything passes, submit a PR. 😄

Active hinty related issues

• Discuss strict equality in Mypy [Hinty] Strict equality #2823

Documentation for advanced typing (that I always lose the links for..)

• Generics
• Methods overloading

Hinty status

MyPy Support: 25.44%

Module	Typed code (lines)	Typed files
[core]	100.00%	33/33
arch	100.00%	13/13
layers	3.06%	2/92
contrib	14.28%	27/167
libs	23.30%	6/8
asn1	100.00%	4/4
tools	11.08%	1/9
modules	0.00%	0/8

[guedou]

It this compatible with type hints? https://docs.python.org/3/library/typing.html

I had a look at static typing for Scapy, and it could helps us catch some bugs and improve the API consistency.

[gpotter2]

It would be pretty amazing to get static typing, but it's not supported below Python 3.5.

Python 3.4 already had its EOL in March, so the only thing that's preventing us from doing that is the 2.7 support.

[guedou]

There is a type hints syntax that works with Python2: https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code

It works fine with our code base =)

[gpotter2]

Because we have great tests, it's really easy to generate automatic type hinting wigh pyannotate based on the values taken during execution.

I've setup the utilitary classes in #2162

[guedou]

Awesome! I tried pyannotate but I did not find time to do more than a quick PoC. I remind that mypy outputs a lot of errors that need to be fixed. I had to fix things (like explicitly adding members to Conf). Also, flake8 complains about unused imports and I did not think of a better way to fix this than using noqa. Do you know if we can exclude files (i.e. six.py & winpcapy.py) from a mypy run? I propose “hinty” for the project name.