Kronk Have Many Package
I am currently working on two internal two Python packages: an end-user application called
openrelay, and its library dependency,
openarc. In addition to this internal dependency chain, both packages have numerous external dependencies, and I would like everything to be installed nicely when I type
pip install openrelay.
After struggling mightily with assorted hacks, I went searching for prior work. The common theme in all discussions was
devpi, so I gave it a shot.
The experience was surprisingly intuitive, and I thought I'd write about it for future reference.
devpi suite in using
pip-3.6 install devpi-server devpi-client --user
Initialize and start
devpi-server. This will start a python package index on
devpi-server --init --start
Check in on the status of the server:
ardentprayer% devpi-server --status 2019-03-20 07:03:53,744 INFO NOCTX Loading node info from /home/kchoudhu/.devpi/server/.nodeinfo 2019-03-20 07:03:53,745 INFO NOCTX wrote nodeinfo to: /home/kchoudhu/.devpi/server/.nodeinfo server is running with pid 109
Out of the box,
devpi creates a non-editable index
root/pypi which simply proxies requests to external PyPI index and caches the result for future use:
pip-3.6 install Flask --index http://localhost:3141/root/pypi --user Collecting Flask Downloading http://localhost:3141/root/pypi/+f/a08/0b744b7e345cc/Flask-1.0.2-py2.py3-none-any.whl (91kB) 100% |████████████████████████████████| 92kB 24.4MB/s
Create an internal-only index
A caching package proxy for external PyPI servers is cool and all, but our goal is to transparently distribute internal projects that are not available on external PyPI servers. To achieve this, we create a secondary index that inherits from
root/pypi, and upload our development work to that.
Some admin work first. Create a user who will be able to upload to our secondary index:
devpi user -c distuser password=123
Login using the new user:
devpi login distuser --password=123
And create a new
devpi index -c dev bases=root/pypi volatile=True
volatile keyword tells
devpi that users will be able to upload to this index.
Upload projects to the internal-only index
Back to our initial problem:
openarc, and their external dependencies.
Both packages have coherent
setup.py files, and I drop into their development directories and upload them to the
devpi-server (for brevity's sake, only the output for
openrelay is listed).
# cd ~/src/openrelay # devpi upload using workdir /tmp/devpi3 copied repo /usr/home/kchoudhu/src/openrelay/.git to /tmp/devpi3/upload/openrelay/.git pre-build: cleaning /usr/home/kchoudhu/src/openrelay/dist --> /tmp/devpi3/upload/openrelay$ /usr/local/bin/python setup.py sdist --formats gztar built: /usr/home/kchoudhu/src/openrelay/dist/openrelay-0.0.1.tar.gz [SDIST.TGZ] 15.535kb register openrelay-0.0.1 to http://localhost:3141/distuser/dev/ file_upload of openrelay-0.0.1.tar.gz to http://localhost:3141/distuser/dev/
With both packages now uploaded to
openrelay can be installed by issuing:
pip-3.6 install --index http://localhost:3141/distuser/dev openrelay --user
External dependencies are fetched and cached from PyPI, and
openrelay were pulled in from
devpi-server using one command.
pip to use
Telling pip to use the new index with every invocation is tiresome, remove the need to do so by editing
[global] index-url = http://localhost:3141/distuser/dev [search] index = http://localhost:3141/distuser/dev
Do not put trailing slashes on the URLs;
pip gets very testy indeed if you do.
I made developing multiple internal Python packages slightly little easier for myself, and heartily recommend that you follow in my footsteps. It is rare for a tool to improve productivity after, like, 6 commands, but here we are:
You should use it.