Deploying local file-backed distributions with pipenv fails

[ahonnecke]

This is a Bug Report

History

Reopening in this repo, original PR here:
serverless/serverless#5768

Description

pipenv allows users to install a local path into the pipenv created venv:

pipenv install -e './subs/panic'

Which results in an entry in the pipfile that points to the local directory:

[packages]
requests = "*"
boto3 = "*"
numpy = "*"
dad-panic = {editable = true, path = "./subs/panic"}

And (while in the venv) allows the user to import the install package

% pipenv run python3 -i                                                                                                                                                             
Python 3.7.1 (default, Nov  6 2018, 18:45:35)
[Clang 10.0.0 (clang-1000.11.45.5)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from dad.panic import trigger_incident
>>>exit()

However, serverless fails while attempting to deploy a service to aws lambda that contains a locally installed package as detailed above:

% export SLS_DEBUG=* && export AWS_SDK_LOAD_CONFIG=true && sls deploy --verbose
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generating requirements.txt from Pipfile...
Serverless: Parsed requirements.txt from Pipfile in /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.7
./subs/panic should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:262:11)
    at installRequirementsIfNeeded (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:451:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:530:29)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:409:17)
    at Deploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:117:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:107:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:422:17)
    at variables.populateService.then.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:694:18)
    at tryOnImmediate (timers.js:665:5)
    at processImmediate (timers.js:647:5)
    at process.topLevelDomainCallback (domain.js:121:23)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:44:28)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           10.11.0
     Serverless Version:     1.35.1

• What went wrong?
  sls deploy failed because (seemingly) pip or pipenv failed to find "./subs/panic" as a local path

• What did you expect should have happened?
  The project should have built in the container and deployed up to aws with dad.panic in the python namespace, specifically I expect that any packages that I can import and use by activating the virtual environment locally will:

1. Deploy to the service and 2) import and work once deployed to the service

• What was the config you used?

service: timescale-monitor-lambda # NOTE: update this with your service name

# You can pin your service to only deploy with a specific Serverless version
# Check out our docs for more details
# frameworkVersion: "=X.X.X"

provider:
  name: aws
  runtime: python3.7
  profile: web
  stage: dev

functions:
  run:
    handler: handler.run

plugins:
  - serverless-python-requirements

custom:
  pythonRequirements:
    dockerizePip: non-linux

• What stacktrace or error message from your provider did you see?

Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command requirements
Serverless: Load command requirements:clean
Serverless: Load command requirements:install
Serverless: Load command requirements:cleanCache
Serverless: Invoke deploy
Serverless: Invoke package
Serverless: Invoke aws:common:validate
Serverless: Invoke aws:common:cleanupTempDir
Serverless: Generating requirements.txt from Pipfile...
Serverless: Parsed requirements.txt from Pipfile in /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/.serverless/requirements.txt...
Serverless: Installing requirements from /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/.serverless/requirements/requirements.txt ...
Serverless: Docker Image: lambci/lambda:build-python3.7
./subs/panic should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

  Error --------------------------------------------------

  null

     For debugging logs, run again after setting the "SLS_DEBUG=*" environment variable.

  Stack Trace --------------------------------------------

Error: null
    at installRequirements (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:262:11)
    at installRequirementsIfNeeded (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:451:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:530:29)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:409:17)
    at Deploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:117:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:107:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:422:17)
    at variables.populateService.then.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:694:18)
    at tryOnImmediate (timers.js:665:5)
    at processImmediate (timers.js:647:5)
    at process.topLevelDomainCallback (domain.js:121:23)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:44:28)

  Get Support --------------------------------------------
     Docs:          docs.serverless.com
     Bugs:          github.com/serverless/serverless/issues
     Issues:        forum.serverless.com

  Your Environment Information -----------------------------
     OS:                     darwin
     Node Version:           10.11.0
     Serverless Version:     1.35.1

Similar issues (possibly):

• Plugin doesn't seem to support -r in requirements.txt #57

Additional Data

% cat /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/.serverless/requirements.txt
-i https://pypi.org/simple
-e ./subs/panic
certifi==2018.11.29
chardet==3.0.4
environs==4.1.0
idna==2.8
marshmallow==2.18.0
numpy==1.16.0
python-dotenv==0.10.1
requests-mock==1.5.2
requests==2.21.0
urllib3==1.24.1 ; python_version >= '3.4'

• Serverless Framework Version you're using:
  1.35.1

• Operating System:
  OSX 10.13.4

• Stack Trace:

    at installRequirements (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:262:11)
    at installRequirementsIfNeeded (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:451:3)
    at ServerlessPythonRequirements.installAllRequirements (/Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/node_modules/serverless-python-requirements/lib/pip.js:530:29)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.spawn (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:409:17)
    at Deploy.BbPromise.bind.then (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:117:50)
From previous event:
    at Object.before:deploy:deploy [as hook] (/usr/local/lib/node_modules/serverless/lib/plugins/deploy/deploy.js:107:10)
    at BbPromise.reduce (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:55)
From previous event:
    at PluginManager.invoke (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:391:22)
    at PluginManager.run (/usr/local/lib/node_modules/serverless/lib/classes/PluginManager.js:422:17)
    at variables.populateService.then.then (/usr/local/lib/node_modules/serverless/lib/Serverless.js:157:33)
    at runCallback (timers.js:694:18)
    at tryOnImmediate (timers.js:665:5)
    at processImmediate (timers.js:647:5)
    at process.topLevelDomainCallback (domain.js:121:23)
From previous event:
    at Serverless.run (/usr/local/lib/node_modules/serverless/lib/Serverless.js:144:8)
    at serverless.init.then (/usr/local/lib/node_modules/serverless/bin/serverless:44:28)

• Provider Error messages:

• Generated requirements.txt:

% cat /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/.serverless/requirements.txt
-i https://pypi.org/simple
-e ./subs/panic
certifi==2018.11.29
chardet==3.0.4
environs==4.1.0
idna==2.8
marshmallow==2.18.0
numpy==1.16.0
python-dotenv==0.10.1
requests-mock==1.5.2
requests==2.21.0
urllib3==1.24.1 ; python_version >= '3.4'

• Manual pip install:

ahonnecke ~/.../timescale-monitor-lambda/.serverless
% cat requirements.txt                                                                                                                                                              (submodule-install *!?)
-i https://pypi.org/simple
-e ./subs/panic
certifi==2018.11.29
chardet==3.0.4
environs==4.1.0
idna==2.8
marshmallow==2.18.0
numpy==1.16.0
python-dotenv==0.10.1
requests-mock==1.5.2
requests==2.21.0
urllib3==1.24.1 ; python_version >= '3.4'
ahonnecke ~/.../timescale-monitor-lambda/.serverless
% pip install -r requirements.txt                                                                                                                                                   (submodule-install *!?)
./subs/panic should either be a path to a local project or a VCS url beginning with svn+, git+, hg+, or bzr+

[ahonnecke]

This seems to work if I manually perform the install from the directory where the pipfile is packaged in:

cd /Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda

pip install -r .serverless/requirements.txt                                                                                                                                       (submodule-install *!?)
Obtaining file:///Users/ahonnecke/Code/repos/timescale-monitor-lambda/timescale-monitor-lambda/subs/panic (from -r .serverless/requirements.txt (line 2))
Collecting certifi==2018.11.29 (from -r .serverless/requirements.txt (line 3))
  Downloading https://files.pythonhosted.org/packages/9f/e0/accfc1b56b57e9750eba272e24c4dddeac86852c2bebd1236674d7887e8a/certifi-2018.11.29-py2.py3-none-any.whl (154kB)
    100% |████████████████████████████████| 163kB 840kB/s
Requirement already satisfied: chardet==3.0.4 in /usr/local/lib/python3.7/site-packages (from -r .serverless/requirements.txt (line 4)) (3.0.4)
Requirement already satisfied: environs==4.1.0 in /usr/local/lib/python3.7/site-packages (from -r .serverless/requirements.txt (line 5)) (4.1.0)
Collecting idna==2.8 (from -r .serverless/requirements.txt (line 6))
  Downloading https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl (58kB)
    100% |████████████████████████████████| 61kB 5.6MB/s
Requirement already satisfied: marshmallow==2.18.0 in /usr/local/lib/python3.7/site-packages (from -r .serverless/requirements.txt (line 7)) (2.18.0)
Collecting numpy==1.16.0 (from -r .serverless/requirements.txt (line 8))
  Downloading https://files.pythonhosted.org/packages/83/0d/1dd2f96eff7f5df22166066f7dbd213428d46f78f8ed9dea8345ca1a1f51/numpy-1.16.0-cp37-cp37m-macosx_10_6_intel.macosx_10_9_intel.macosx_10_9_x86_64.macosx_10_10_intel.macosx_10_10_x86_64.whl (13.9MB)
    100% |████████████████████████████████| 13.9MB 2.6MB/s
Requirement already satisfied: python-dotenv==0.10.1 in /usr/local/lib/python3.7/site-packages (from -r .serverless/requirements.txt (line 9)) (0.10.1)
Collecting requests-mock==1.5.2 (from -r .serverless/requirements.txt (line 10))
  Downloading https://files.pythonhosted.org/packages/f3/5b/7fc2615e6ae2d229214fd4d7617dd28475d728124c5f3d0653663113ac85/requests_mock-1.5.2-py2.py3-none-any.whl
Collecting requests==2.21.0 (from -r .serverless/requirements.txt (line 11))
  Downloading https://files.pythonhosted.org/packages/7d/e3/20f3d364d6c8e5d2353c72a67778eb189176f08e873c9900e10c0287b84b/requests-2.21.0-py2.py3-none-any.whl (57kB)
    100% |████████████████████████████████| 61kB 9.4MB/s
Collecting urllib3==1.24.1 (from -r .serverless/requirements.txt (line 12))
  Using cached https://files.pythonhosted.org/packages/62/00/ee1d7de624db8ba7090d1226aebefab96a2c71cd5cfa7629d6ad3f61b79e/urllib3-1.24.1-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python3.7/site-packages (from requests-mock==1.5.2->-r .serverless/requirements.txt (line 10)) (1.12.0)
boto3 1.7.79 has requirement botocore<1.11.0,>=1.10.79, but you'll have botocore 1.12.71 which is incompatible.
Installing collected packages: certifi, idna, numpy, urllib3, requests, requests-mock, dad-panic
  Found existing installation: certifi 2018.4.16
    Uninstalling certifi-2018.4.16:
      Successfully uninstalled certifi-2018.4.16
  Found existing installation: idna 2.7
    Uninstalling idna-2.7:
      Successfully uninstalled idna-2.7
  Found existing installation: urllib3 1.23
    Uninstalling urllib3-1.23:
      Successfully uninstalled urllib3-1.23
  Found existing installation: requests 2.19.1
    Uninstalling requests-2.19.1:
      Successfully uninstalled requests-2.19.1
  Running setup.py develop for dad-panic
Successfully installed certifi-2018.11.29 dad-panic idna-2.8 numpy-1.16.0 requests-2.21.0 requests-mock-1.5.2 urllib3-1.24.1

[tacincorporated]

I have a pull request to fix this.
#321