Packaging for Jupyter in 2022
Packaging has been difficult and changing over time in the Python ecosystem, and packaging extensions for Jupyter has been especially troublesome. Jupyter extensions typically have a build step to handle JavaScript assets, and provide Jupyter configuration and data using the data_files capability in Setuptools. If you’ve ever developed your own Jupyter extension before, you understand the pain all too well.
We’ve tried to help over the years. We created the Jupyter Packaging library, with helper functions and Setuptools builder classes. We have also created cookiecutter repositories for JupyterLab, Widget, and Server extensions. Additionally, there have been a number of libraries and Python Enhancement Proposals produced by the Python Packaging Authority, aiming to standardize and improve the Python Packaging ecosystem. Even with all these improvements, packaging Jupyter libraries remained a challenge.
Then we discovered the Hatch project.
Hatch is a standards-based Python packaging backend that is flexible and extensible. Hatch came with many of the things we needed. It provides a supported replacement for Setuptool’s (deprecated) data_files called shared-data. We could easily extend Hatch to handle the JavaScript build step required for most of our applications and extensions with a new plugin called Hatch Jupyter Builder that is tailored for Jupyter extensions. The best news? No more complicated setup.py files in your package. Just a single, standards-driven pyproject.toml in your root directory describing all of your package details.
We believe this is a major improvement for Jupyter developers and extension authors.
For Jupyter extension authors that previously used one of our cookiecutter templates or Jupyter Packaging directly, we offer a migration script and guides to convert the Setuptools and Jupyter Packaging configuration to use Hatch and Hatch Jupyter Builder. We also provide a script that will compare the files in an sdist or wheel before and after migration to make sure you do not accidentally add or subtract files.
We have already been migrating many of the core Jupyter and IPython libraries to use the Hatch build backend, using Hatch Jupyter Builder where appropriate. We have upgraded several of our cookiecutter templates to use Hatch as well. The transition has been quite smooth!
We highly encourage you to try out Hatch and Hatch Jupyter Builder for existing or new Python libraries.
About the authors
Steven Silvester is a Senior Python Engineer at MongoDB working on the Database Experience Team. He is a Jupyter Steering Council Member and contributes to the maintenance of many of the Jupyter projects.
Zach Sailer is a Senior Software Engineer at Apple, where he is the Jupyter Open Source Champion. He is a Jupyter Distinguished Contributor and leader of the Jupyter Server subproject.
