FileNotFoundError: [Errno 2] No such file or directory: '/home/user/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info/METADATA'

[jwfu]

Environment

• pip version: 19.0.2
• Python version: 3.6
• OS: Linux Mint 19.1 (Linux 4.15.0-45-generic x86_64)
• conda 4.6.2

Description
When running Jupyter Notebook, following error prevented kernel starting:

ModuleNotFoundError: No module named 'prompt_toolkit.formatted_text'

Based on this, I tried to pip install -U jupyter_console. This failed, giving

FileNotFoundError: [Errno 2] No such file or directory: '/home/user/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info/METADATA'

Upon further investigation, the reason for this is that there was as subfolder created with another psutil-5.4.8.dist-info created. Thus, by moving the files out of this nested subfolder, everything worked again.

KO: /home/user/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info/psutil-5.4.8.dist-info/METADATA

OK: /home/user/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info/METADATA

I wasn't manually doing anything in these package folders; any idea what automatic process (suspect pip-related) could have caused this?

Expected behavior
Directory structure should be /home/user/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info/METADATA

How to Reproduce
pip install -U jupyter_console

Output

Error checking for conflicts.
Traceback (most recent call last):
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2897, in _dep_map
    return self.__dep_map
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2691, in __getattr__
    raise AttributeError(attr)
AttributeError: _DistInfoDistribution__dep_map

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2888, in _parsed_pkg_info
    return self._pkg_info
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2691, in __getattr__
    raise AttributeError(attr)
AttributeError: _pkg_info

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_internal/commands/install.py", line 503, in _warn_about_conflicts
    package_set, _dep_info = check_install_conflicts(to_install)
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 108, in check_install_conflicts
    package_set, _ = create_package_set_from_installed()
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_internal/operations/check.py", line 47, in create_package_set_from_installed
    package_set[name] = PackageDetails(dist.version, dist.requires())
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2635, in requires
    dm = self._dep_map
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2899, in _dep_map
    self.__dep_map = self._compute_dependencies()
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2908, in _compute_dependencies
    for req in self._parsed_pkg_info.get_all('Requires-Dist') or []:
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 2890, in _parsed_pkg_info
    metadata = self.get_metadata(self.PKG_INFO)
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1410, in get_metadata
    value = self._get(self._fn(self.egg_info, name))
  File "/home/jwfu/anaconda3/lib/python3.6/site-packages/pip/_vendor/pkg_resources/__init__.py", line 1522, in _get
    with open(path, 'rb') as stream:
FileNotFoundError: [Errno 2] No such file or directory: '/home/jwfu/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info/METADATA'

[cjerdonek]

/cc @zooba

[cjerdonek]

@jwfu Before you moved the inner (nested) psutil-5.4.8.dist-info folder out of the parent /home/user/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info folder, what were the contents of the parent folder? Just the inner psutil-5.4.8.dist-info, or were there other files?

[jwfu]

@cjerdonek The dir was empty.

[cjerdonek]

@jwfu I was asking what the contents of the parent folder (.../site-packages/psutil-5.4.8.dist-info) were before you moved the inner (nested) psutil-5.4.8.dist-info folder out of it. I thought it wasn't empty because it contained that inner nested folder. Can you clarify what you mean and which folder was empty?

[jwfu]

@cjerdonek Sorry, concretely:

~/anaconda3/lib/python3.6/site-packages/psutil-5.4.8.dist-info$ tree
.
└── psutil-5.4.8.dist-info
    ├── INSTALLER
    ├── LICENSE
    ├── METADATA
    ├── RECORD
    ├── top_level.txt
    └── WHEEL

[zooba]

Yeah, this is part of the bug I fixed. renames would move a directory into a directory if you specified it as the target, but the intent was to rename the directory. And then pkg_resources checks for a non-empty directory (which it is, because it contains another directory) and assumes that means it has valid METADATA (which it doesn't), raising errors that pip doesn't try to handle right now.

[cjerdonek]

@zooba So are you saying this is happening only because the directory structure was left over from before? Is there anything further that pip can or should be doing here to improve things more, or can this be closed?

[zooba]

pip could also handle more errors from pkg_resources when trying to find distributions (or encourage pkg_resources to convert ValueError and apparently FileNotFoundError into their own type, which pip already handles - didn't you already file a bug over there on this?).

I started adding the extra handling to one of my PRs and was told to take it out, but haven't gone back to it. Personally I see this as a boundary, which means handling all errors and continuing safely is okay (as an exception doesn't indicate corrupt state in pip itself - just like OSError is generally safely continuable but an internal TypeError is not). However, I know not everyone has that POV :)

[cjerdonek]

pip could also handle more errors from pkg_resources when trying to find distributions (or encourage pkg_resources to convert ValueError and apparently FileNotFoundError into their own type, which pip already handles - didn't you already file a bug over there on this?).

Okay, thanks. I did file pypa/setuptools#1664. It was just to include more information in the error though, rather than asking for a specific type that pip can handle.

I started adding the extra handling to one of my PRs and was told to take it out, but haven't gone back to it. Personally I see this as a boundary, which means handling all errors and continuing safely is okay (as an exception doesn't indicate corrupt state in pip itself - just like OSError is generally safely continuable but an internal TypeError is not). However, I know not everyone has that POV :)

If it was me that asked you to take it out (if I remember what you're referring to) :) , it was more to limit the scope of the PR to one thing. Re: the other point, generally speaking, my own view is for pip not to try to be too "smart" by trying to fix errors. (pip already struggles simply to do what it needs to do correctly.) However, I am really in favor of providing more useful info so that exceptions can become a lot easier to diagnose, which will make it easier for people to fix issues on their own, file bug reports to pip, or file bug reports to the proper upstream.

[pradyunsg]

@cjerdonek I updated your comment to refer to the correct repository. :)

[pradyunsg]

@cjerdonek Could you give this an appropriate "type" label?

[cjerdonek]

@pradyunsg I'm marking this "needs triage" for now. I think there's something we can be doing to make a better user experience in the case of a bad .dist-info directory found during an install, but I'm not quite sure yet. At the least, there should be a better error message telling the user how to rectify the situation. Also, there's a chance this is a duplicate issue as I'm pretty sure this has been reported before (it may be relatively common as far as install errors go).

[yuvirl]

I solved the problem in this way.

pip install -U pip
pip install psutil

[rodrigoruiz]

@yuvirl the pip install -U pip command gives me the error FileNotFoundError: [Errno 2] No such file or directory: 'c:\\users\\gamer\\anaconda3\\envs\\tensorflow\\lib\\site-packages\\chardet-3.0.4.dist-info\\METADATA'

[giacomoarienti]

I solved my problem in this way:
conda uninstall pip

Now pip works fine but I had to reinstall all my packages.

[denieboy]

conda install rsa

[pradyunsg]

We haven't received new reports around this in a while; as far as I can see so I'll close this out on the basis that the source of this issue has been fixed. My best guess right now is that this was weird interaction between Conda <-> pip; from Conda's end and it's been resolved now.