Support for namespace packages (PEP 420)

[gdetrez]

PEP 420 introduced implicit namespace packages, which are packages without __init__.py. Those packages, and the modules under them, don't seem to appear in the dependency graph generated by pydeps.

Here's an example. Given the following directory structure:

.
├── main.py
├── pkg1
│   └── foo.py
└── pkg2
    ├── bar.py
    └── __init__.py

And main.py:

import pkg1.foo
import pkg2.bar

pydeps main.py generates the following graph:

I would have expected pkg1.foo to be included in the dependency graph. (On the other hand, I'm not sure there's a point in having pkg1 there since it's going to be empty.)

[thebjorn]

Hi @gdetrez and thank you for the bugreport.

modulefinder can't find pkg1 and reports it as "Missing" (internally it calls it a bad module ;-)

c:\srv\tmp\pep420>\Python34\python.exe -m modulefinder main.py

  Name                      File
  ----                      ----
m __main__                  main.py
P pkg2                      pkg2\__init__.py
m pkg2.bar                  pkg2\bar.py

Missing modules:
? pkg1.foo imported from __main__

pydeps --externals will list "bad modules":

c:\srv\tmp\pep420>pydeps --externals main.py
[
    "pkg1",
    "pkg2"
]

I suppose we could include all the "bad" modules in the graph as well, but with the current code they would be empty.

The best alternative would be to fix pydeps' copy of modulefinder (mf27.py) to also find PEP420 modules, but at this point I'm not sure if that's even possible...?

[gdetrez]

Hi @thebjorn. Sorry for the radio silence.

I see that if I use the --include-missing flag, I get a bit more complete picture. But it seems that I don't get the transitive dependencies of the namespaced package (i.e if main imports ns1.foo and ns1.foo import ns1.bar, the latter is not included in the graph).

While researching the same issue for mypy, I found python/mypy#5691 which implement PEP 420 support (it's behind a --namespace-packages flag). The module is also called modulefinder but it looks rather different than the one here so I'm not sure how much is reusable.

Otherwise, if you don't mind the extra dependency, maybe it'd be worth taking a look at modulegraph by Ronald Oussoren, which already supports namespaces:

❯❯❯ python3 -m modulegraph main.py

Class           Name                      File
-----           ----                      ----
Script          /home/gregoire/experiments/20190128-pydeps-namespace/test2/main.py /home/gregoire/experiments/20190128-pydeps-namespace/test2/main.py
NamespacePackage ns1                       -
SourceModule    ns1.bar                   /home/gregoire/experiments/20190128-pydeps-namespace/test2/ns1/bar.py
SourceModule    ns1.foo                   /home/gregoire/experiments/20190128-pydeps-namespace/test2/ns1/foo.py

[kinow]

Just tried pydeps after using it a long time ago on the project I work on. We recently moved to namespace packages too, and now instead of the nice graph, I get only three nodes in the graph, and only if I specify pydeps cylc/flow. It fails when running pydeps cylc (parent directory). Looks like it's an issue with modulefinder, so hopefully that'll be fixed soon there, so that I can use pydeps again 🤞

[ssbarnea]

That is a huge bug and --include-missing seems to avoid it but not without nasty side effects. Sadly I will not be able to adopt pydeps as part of the docs until is fixed.

[thebjorn]

@ssbarnea I'm wondering if you're using this feature to create namespace package, or if it's simply to avoid having an __init__.py file? I've never found a reason to use namespace packages that outweigh the mess it causes. Regardless, pydeps is dependent on modulefinder (a python standard module - vendorized with some bug-fixes) supporting the feature first. I'll happily merge a PR if you or anyone else wants to work on it though...

[thebjorn]

modulegraph seems to be built on the same foundational technology and does some work to support namespace packages (https://github.com/ronaldoussoren/modulegraph/blob/master/modulegraph/modulegraph.py#L63)

[dg-pb]

Thanks, will keep it in mind. Decided to go with findimports. It is fast and does the trick for the time being. Although it doesn't have good clustering / collapsing, my needs are still simple enough for it to not be a problem.

When I need something more extensive & customisable I will have a look at modulegraph, pydeps and findmodules in more depth.