Python packaging

How can I reuse local python code in different projects?

I've collected a few useful utility functions to help me with advent of code 2021. So far it has been great, the folder lives in the same directory as my yearly advent of code folder like this

advent-of-code/
    2020/
    2021/
        data/
        day1.py
        ...
        toolbox/
            core.py
            ...
    LICENSE.txt
    README.md

For each puzzle I create a new python file with the first line always starting with from toolbox.core import ... to handle parsing data.

Everything works smoothly until I wanted to re-use my toolbox modules for 2020 problems. I don't want to duplicate my toolbox for every advent of code year I participate in. A logical place I thought, would be to place the toolbox package in the root of the project like so:

advent-of-code/
    2020/
    2021/
        data/
        day1.py
        ...
    toolbox/
        core.py
        ...
    LICENSE.txt
    README.md

This way I won't have to duplicate my code and could expose the toolbox for each year to use. Not quite.

The problem is that when running the python file inside the 2021/ directory, the python interpreter is unable to locate the toolbox/. The list of system paths tells python where to look for python code that it can import such as the inbuilt collections, functools and math packages, and pip-installed modules such as pytest, numpy and pandas. We could, for each runnable script, add the location of the toolbox to the system path (sys.path). However, this is cumbersome; there is a much better way.

Local modules

A simple fix is to create a local package using the setuptools package. setuptools can be installed using pip install setuptools.

If you want a minimal solution look no further, Here it is.

setup.py

from setuptools import setup

setup(
    name="local_toolbox",
    version="0.1.0",
    packages=["toolbox",],
    )

And to install the package locally on the command line.

$ pip install -e .

From pip install --help we see that the -e flag or --editable is to "install a project in editable mode (i.e. setuptools "develop mode") from a local project path ...".

Enjoy :).

For those who want to know a little more, let's continue.

setup.cfg and setup.py

There are two ways to specify setup config, dynamically as parameters of the setuptools.setup() function and in a setup.cfg config file.

The equivalent setup to the first example would be split across the executable and config file.

setup.py

from setuptools import setup

setup()

setup.cfg

[options]
packages = toolbox

[metadata]
name = local_toolbox
version = 0.1.0

As you can see, the name, version and packages attributes are now configured as options and metadata. There is no definitive way in using setuptools. I personally prefer to have all config in a config file. With additional optional fields such as a long_description, it is easy to edit a config file than source code. Even if it is python.

Sharing is Caring

Finally, if you're excited to share your python project as a package, it is easy to register it using the Python Package Index (PyPI). I recommend looking at the setuptools documentation and Packaging Python Authority (PyPA) to learn more about the standard way of building and distributing python projects.

In Conclusion

Reusing python code across local projects can be done in as little as a two liner python script and pip install command. Whilst there are different options in packaging a project to share on the Python Package Index, setuptools makes it easy to build a package and registering is also fairly simple.