Jupyter Blog - notebook format

nb-cli: A Command-Line Interface for AI Agents and Notebook Automation

2026-05-11T19:04:00+00:00

The rise of AI coding agents has transformed how we think about developer tools. Large language models like Claude, GPT, and others are remarkably effective at using command-line interfaces — they’ve been trained on billions of lines of CLI usage from documentation, Stack Overflow, and GitHub. But when it comes to working with Jupyter notebooks programmatically, there’s been a gap: existing tools focus on running agents within notebooks, but what about agents that need to work with notebooks as artifacts?

nb-cli, an experimental open-source command-line interface designed specifically for AI agents, automation scripts, and developers who need programmatic access to Jupyter notebooks. Built with Rust for performance and reliability, nb-cli provides a fast, composable way to read, write, execute, and manipulate notebooks through a clean command line interface that follows the nbformat specification.

The Problem: Notebooks as Black Boxes

While Jupyter notebooks are indispensable for interactive exploration, their underlying .ipynb JSON structure has long been a friction point for programmatic interaction, especially for shell scripts and Large Language Models (LLMs).

Traditional workflows often break down when automation or AI-driven analysis is required. Consider the following scenarios where standard notebook interfaces prove insufficient:

Autonomous Analysis: An AI agent tasked with auditing a data science workflow must programmatically inspect individual cells to map out the analysis pipeline.
Automated Validation: CI/CD systems require a reliable method to execute notebooks, validate outputs, and catch errors before deployment.
Documentation at Scale: Developers need tools to automatically transform notebook content into clean, accessible documentation.
Production Debugging: Teams need a way to troubleshoot notebook execution failures in headless production environments without manual intervention.
Notebooks as Data: Analysts may want to treat a notebook as a structured database to programmatically generate business reports, research summaries, or custom visualizations.

Historically, solving these problems required labor-intensive workarounds like manually navigating the JupyterLab UI, writing brittle Python scripts to parse complex JSON files, or using execution tools that lack real-time integration.

nb-cli bridges this gap by offering a CLI-first interface designed for the modern era of automation. By leveraging command-line patterns and Unix composability, it provides the structured output and predictable interface that AI agents and developers need to treat notebooks as first-class citizens in any software stack.

Key Features

Works With or Without a Jupyter Server

nb-cli doesn’t require a running Jupyter server. By default, it reads and writes .ipynb files directly and communicates with kernels over ZeroMQ for execution. This makes it well-suited for scripting, CI pipelines, and any workflow where launching a server is unnecessary overhead.

# Create a notebook — no server needed
nb create analysis.ipynb

# Add cells
nb cell add analysis.ipynb --source "import pandas as pd"
nb cell add analysis.ipynb --source "# Data Analysis" --type markdown

# Execute
nb execute analysis.ipynb

# Read back with outputs
nb read analysis.ipynb

Connecting to a server becomes valuable if multiple users and/or agents are editing the same notebook simultaneously within a JupyterLab session. Once connected, nb-cli uses Y.js, the same CRDT protocol JupyterLab uses internally for conflict-free real-time synchronization.

# Auto-detect and connect to local Jupyter server
nb connect

# Or, connect to a specific server
nb connect --server http://localhost:9999 --token abc

# Add a cell - it appears instantly in JupyterLab
nb cell add experiment.ipynb --source "df.head()"

# Execute via the remote kernel
nb execute experiment.ipynb --cell fe456

# Restart the kernel before execution for reproducibility checks
nb execute experiment.ipynb --restart-kernel

# Disconnect
nb disconnect

When connected to a Jupyter server, nb-cli detects whether a notebook is open in JupyterLab and uses server APIs for conflict-free collaborative editing. If the notebook isn’t open, it seamlessly falls back to file-based operations.

AI-Optimized Markdown Format

Language models don’t parse JSON, they predict tokens. This distinction matters more than you’d think when you’re building a tool that feeds notebook content into an LLM’s context window. Jupyter’s native notebook format is deeply nested JSON. Source code is stored as arrays of strings. Outputs carry base64-encoded blobs. Metadata nests several levels deep. This is fine for a JSON parser, but for a language model working within a fixed context window, 30–40% of those tokens are structural characters — braces, brackets, escaped newlines that carry no semantic value. Another common option is plain Markdown which is token-efficient and human-readable, but it’s ambiguous. A # could be a markdown heading or a Python comment. A fenced code block could be a notebook cell or an example inside a markdown cell’s documentation. When an LLM is asked to “fix the error in cell 7,” it needs to reliably locate that cell in the text and plain markdown gives it no structural markers to count on, just a sequence of code fences that all look the same.

So we designed a line-oriented sentinel format:

@@notebook {"format":"ai-notebook","metadata":{"kernelspec":{"name":"python3"}}}

@@cell {"index":0,"id":"f68t57","cell_type":"code","execution_count":1}
```python
df.head()
```
@@output {"output_type":"execute_result"}
```text
   col_a  col_b
0      1      a
```

The format makes a few deliberate tradeoffs. @@cell and @@output sentinels give the model unambiguous structural boundaries without counting braces or tracking nesting. Inline JSON metadata on each sentinel line places cell type, index, and execution count in the tokens immediately before the content — matching how attention mechanisms locate information. Code in fenced blocks with language hints activates the model’s syntax-level training. And because each cell block is self-contained, truncation degrades gracefully — unlike JSON, where a cut anywhere breaks the entire structure.

Designed for Composability

nb-cli follows Unix conventions — plain text output, stdin support, predictable exit codes — so it composes naturally with other CLI tools. For AI agents, this matters because a single shell command can replace what would otherwise be multiple tool calls with intermediate parsing.

Consider an agent asked to “add a summary section to the notebook and run it.” Without nb-cli, this requires separate API calls to read the notebook, parse the structure, insert a cell, write the file, find a kernel, execute, read back the output — each consuming tokens for the request and response. With nb-cli:

nb cell add analysis.ipynb --source "$(cat <<'EOF'
@@markdown
# Summary

@@code
print(f"Rows: {len(df)}, Columns: {len(df.columns)}")
df.describe()
EOF
)" && nb execute analysis.ipynb -i -2 -i -1 && nb read analysis.ipynb -i -1

Three operations — add cells, execute them, read the result — in a single shell invocation. The agent gets back only the output it needs without re-reading the entire notebook.

The same principle applies to debugging. An agent investigating a failed notebook doesn’t need to read every cell to find the problem.

# Find cells with errors — returns only the relevant cells
nb search analysis.ipynb --with-errors

One call, targeted output, no wasted tokens on cells that ran successfully.

Stable Cell Referencing

nb-cli supports two ways to reference cells.

Index-based:--cell-index 0 (supports negative indexing: -1 = last cell)
ID-based: --cell f68t57 (doesn’t change when cells move)

# Reference by position
nb cell update analysis.ipynb --cell-index 0 --source "x = 42"

# Reference by stable ID — safe even after cells are reordered
nb cell update analysis.ipynb --cell ce456 --source "print('Done')"

# Execute the last cell
nb execute analysis.ipynb --cell-index -1

Powerful Search Capabilities

nb-cli includes built-in search to quickly locate cells by content, type, or execution errors. By default, search matches against cell source code, but a scope filter extends it to execution outputs.

# Search for cells containing a pattern
nb search analysis.ipynb "import pandas"

# Find all cells with execution errors
nb search analysis.ipynb --with-errors

# Search within outputs instead of source
nb search analysis.ipynb "KeyError" --scope output

# Filter by cell type
nb search analysis.ipynb "TODO" --cell-type markdown

For AI agents, --with-errors is particularly useful — instead of reading an entire notebook to find what failed, the agent gets back only the cells that need attention. Combined with --scope output, it can search error tracebacks directly without parsing every cell’s results. The same capabilities are useful for humans auditing notebooks for deprecated APIs, locating specific functions across large notebooks, or extracting patterns before a refactor.

Multi-Cell Operations

One of the most common patterns when working with notebooks programmatically is adding a sequence of cells — a markdown header, then setup code, then analysis. Doing this one cell at a time means multiple round-trips and index bookkeeping. Instead, nb-cli accepts multiple cells in a single call using the sentinels format we saw earlier.

# Add a markdown header followed by a code cell in one command
nb cell add report.ipynb --source "$(cat <<'EOF'
@@markdown
# Results

@@code
import pandas as pd
df = pd.read_csv('results.csv')
df.head()
EOF
)"

For more control, sentinels also accept the full @@cell {"cell_type": "..."} JSON format.

nb cell add report.ipynb --source "$(cat <<'EOF'
@@cell {"cell_type": "markdown"}
# Analysis Header

@@cell {"cell_type": "code"}
print("hello")
EOF
)"

Both formats work with stdin, making it easy to compose cells from scripts or pipelines.

printf '@@markdown\n## Summary\n\n@@code\ndf.describe()\n' | nb cell add report.ipynb --source -

The same batching philosophy extends to execution and deletion:

# Execute cells 2 through 5
nb execute analysis.ipynb --start 2 --end 5

# Delete specific cells
nb cell delete analysis.ipynb -i 0 -i 2

# Delete a range of cells
nb cell delete analysis.ipynb --range 0:3

Environment-Aware Execution

--uv and --pixi flags are supported on nb connect, nb execute, and nb create, telling nb to discover Jupyter servers and kernels through the appropriate environment manager. nb status --python returns the command prefix needed to run Python in the same environment as the connected kernel — useful when agent-generated shell commands need to match the active notebook environment.

# Connect using a uv-managed environment
nb connect --uv

# Execute in a pixi-managed environment
nb execute analysis.ipynb --pixi

# Get the Python prefix for agent-generated shell commands
nb status --python
# Returns: "uv run", "pixi run", or empty for system Python

# Use in a pipeline
$(nb status --python) python -c "import pandas; print(pandas.__version__)"

Real World Use Cases

AI Agent Workflows

AI coding agents can now manipulate notebooks as a part of their analysis workflow.

# Surface all failing cells
nb search data_analysis.ipynb --with-errors

# Apply the fix
nb cell update data_analysis.ipynb --cell-index 3 --source "df = pd.read_csv('data.csv', encoding='utf-8')"

# Re-execute to verify
nb execute data_analysis.ipynb --cell-index 3

CI/CD Integration

Automated testing and validation of notebooks in continuous integration pipelines.

echo "Executing notebook..."
nb execute pipeline.ipynb --allow-errors

echo "Checking for errors..."
if nb search pipeline.ipynb --with-errors; then
  echo "Notebook execution failed"
  exit 1
fi

echo "Clearing outputs before commit..."
nb output clear pipeline.ipynb

echo "✓ All cells executed successfully"

Programmatic Notebook Generation

Generate documentation, reports, and analysis automatically.

# Create a report notebook
nb create report.ipynb

# Add title, introduction, and analysis in one multi-cell command
nb cell add report.ipynb --source "$(cat <<'EOF'
@@markdown
# Monthly Sales Report

@@markdown
Generated on $(date)

@@code
import pandas as pd
df = pd.read_csv('sales_data.csv')
df.describe()
EOF
)"

# Execute to populate outputs
nb execute report.ipynb

Debugging Production Notebooks

Quickly inspect and diagnose issues in deployed notebooks.

# Find all cells with errors
nb search failing_notebook.ipynb --with-errors

# Search for cells with deprecated API usage
nb search analysis.ipynb "pandas.np"

# Find cells with potential security issues
nb search notebook.ipynb "eval("

# Examine specific failing cell with full output
nb read failing_notebook.ipynb --cell-index 5

# Restart kernel and re-run for clean reproducibility check
nb execute failing_notebook.ipynb --restart-kernel

nb-cli in Action

To illustrate how AI agents use nb-cli naturally, here are some examples of agent interactions.

Example 1: Claude creating a RL for LLMs notebook

User Prompt: Help me learn about reinforcement learning for LLMs by creating a notebook and explaining at each cell how it all works. Cover the key concepts: policy model, reward model, KL divergence penalty, PPO, and GRPO. Use a tiny toy model (small vocab, GRU-based) so everything runs on CPU without any API keys.

Example 2: Codex fixing multiple bugs in a notebook

User Prompt: The file churn_analysis.ipynb is a broken research notebook that was last updated in 2023. Fix it so it runs cleanly end-to-end. Identify every cell that fails, fix each issue and verify the notebook executes successfully. After fixing, add a brief markdown note above each cell you changed explaining what was broken and why.

Codex fixed four bugs in churn_analysis.ipynb: a hardcoded file path, DataFrame.append() (removed in pandas 2.0), sklearn.cross_validation (removed in sklearn 0.20), and plot_confusion_matrix (removed in sklearn 1.2) and verified the notebook runs end-to-end after these chages.

Getting Started with nb-cli

Installation

Use the install script.

curl -fsSL https://raw.githubusercontent.com/jupyter-ai-contrib/nb-cli/main/install.sh | bash

If your platform is not supported, and you get an error during install, use cargo to install.

cargo install nb-cli

Or build from source.

git clone https://github.com/jupyter-ai-contrib/nb-cli.git
cd nb-cli
cargo build --release

The binary will be available at target/release/nb.

To enable your AI agents to use nb for all notebook operations, install the skill.

npx skills install jupyter-ai-contrib/nb-cli

About the developers

Andrii Ieroshenko is a Software Development Engineer at AWS. He is a long term contributor to project Jupyter working on JupyterLab, Jupyter AI and several other projects. He is also a member of the Jupyter Media Strategy Working Group.

Brian Granger is a Senior Principal Technologist at AWS. Brian is a cofounder of Project Jupyter, a board member of the Jupyter and PyTorch Foundations.

Piyush Jain is a Principal Engineer at AWS working on Jupyter and Agentic AI. He is a distinguished Jupyter contributor and a member of the Jupyter Server Council.

How can you help?

We’re just getting started with nb-cli. Please join us!.

Install and use the nb-cli. If you find any bugs or have suggestions, please create issues on GitHub.
Join the discussion about nb-cli, open issues or add to discussion in jupyter-ai-contrib.
Contribute: Your bug reports, feature requests, and pull requests will help improve this project for everyone.

The Jupytext Menu is back!

2024-01-31T20:29:00+00:00

A few weeks back, jupytext==1.16.0 went out. That release included multiple amazing contributions by Mahendra Paipuri, this post will go over the new features, and thank Mahendra for his great work.

As we will see below, Mahendra restored the Jupytext Menu, made the extension fully compatible with JupyterLab 4 and Jupyter Notebook 7, and added the option to create Text Notebooks directly from the launcher.

What is Jupytext

Jupytext is a Python package that lets you save Jupyter Notebooks as text notebooks. Multiple formats are supported, and the notebooks can be saved either as Markdown documents with a .md extension, or as scripts with e.g. a .py extension (assuming you use Python - otherwise multiple languages are supported).

A .py notebook in the percent format looks like this:

# %% [markdown]
# This is a markdown cell

# %%
def f(x):
  return 3*x+1

# %%
[f(i) for i in range(6)]

To open a text notebook as a notebook in Jupyter, right-click on the document and select “Notebook” (you can also change the default viewer to “Jupytext Notebook” if you wish: see below the section about the settings)

Right click to open a text notebook with the Notebook editor

Text notebooks are conveniently edited and executed in Jupyter as notebooks:

A Text Notebook in JupyterLab

You can also edit them with your favorite text editor. You will get the changes back in Jupyter by re-opening or by reloading the document: click e.g. on reload Python file from disk in the File menu.

Text notebooks only contain the notebook inputs. For that reason they use much less disk space than .ipynb notebooks. They are also better suited for version control as they only contain the content that was actually typed by the user. But even more useful are paired notebooks: text notebooks paired with an .ipynb notebook where the notebook outputs are preserved. When a paired notebook is saved, Jupytext writes the notebook to both files. When the notebook is read or reloaded in Jupyter, the notebook inputs are loaded from the most recent file, meaning that any edits on the text notebook will be reflected in Jupyter (and propagated to the .ipynb file the next time it is saved).

One of the most frequent operation when using Jupytext is to pair an .ipynb notebook with a text notebook in the format of your choice.

In earlier versions of Jupytext, pairing a notebook had to be done through the command palette.

In Jupytext v1.16, thanks to Mahendra’s work on the front-end extension, you can directly use the Jupytext Menu for this. And the menu is available for both JupyterLab 4 and Jupyter Notebook 7!

If you are new to Jupytext, we recommend that you pair your .ipynb notebooks with either a percent script or with a MyST Markdown file. The former format works well if you want to save and edit your notebook as a script, while the latter is well suited for writing documentation.

Once you have paired the notebook with a Jupytext format, save your notebook, and the paired files will be created or updated.

The Jupytext Menu

Creating new text notebooks

For some notebooks you might decide that you don’t need to save the outputs at all. In that case you can work with an (unpaired) text notebook.

To create a text notebook you can use the New Text Notebook sub-menu under File:

Create Text Notebooks directly from the menu

If, later on, you decide that you want to preserve the notebook outputs in a paired .ipynb notebooks, you will just have to pair the document to an .ipynb notebook using the Jupytext Menu documented at the previous paragraph.

Text Notebooks in the launcher

In Jupytext v1.16, thanks to Mahendra’s work on the front end, text notebooks are also available in a Jupytext section in the Jupyter launcher:

Text Notebooks are also available in the launcher

Jupytext Settings in JupyterLab

Since there are many possible formats for text notebooks, we have decided to expose only the most common ones, by default, in the launcher and in the New Text Notebook menu.

You can include more formats in the menu and in the launcher by changing the Jupytext settings in the Settings Editor (in the Settings menu):

The Jupytext Menu/Launcher settings

You might also want to open certain text documents as notebooks with a single click. You can achieve this by setting the default viewer for those documents to “Jupytext Notebook”. For instance, if you want to open .py and .md files as notebooks with a single click, you can configure the default viewers like this:

Changing the default viewers to open Text Notebooks with a single click

Alternatively, you can also list and set the default viewers with the jupytext-config utility (which was developed recently by Thierry Parmentelat), using e.g.

jupytext-config set-default-viewer python markdown

to set “Jupytext Notebook” as the default viewer for both Python and Markdown files.

Pairing notebooks globally

The pairing commands provided through the Jupytext Menu and commands act on individual notebooks, by settings a jupytext.formats metadata in the notebook.

It is also possible to pair all the notebooks within a certain folder using a jupytext.toml configuration file - see Jupytext’s documentation. Please note that:

The local metadata in the notebook takes precedence over the global configuration, and
Jupyter is aware of the global jupytext.toml file and will pair the notebooks accordingly, however the Jupytext Menu is not aware of the global configuration, so the paired formats selected through the global configuration will not be checked in the menu.

Installing Jupytext v1.16

The front end extension for JupyterLab shipped with Jupytext v1.16 requires JupyterLab 4, and/or Jupyter Notebook 7. Please upgrade Jupyter accordingly. Then, install Jupytext with

pip install 'jupytext>=1.16.0'

conda install 'jupytext>=1.16.0' -c conda-forge

and restart your Jupyter server with e.g. jupyter lab.

Acknowledgments

I would like to thank Mahendra Paipuri for his impressive work. I am pretty sure that the Jupytext Menu, and the option to create new text notebooks, will be much appreciated by Jupytext users!

While users will mostly notice Mahendra’s work on the front end extension, it was actually not his only contribution to this release! Mahendra also thoroughly revisited the packaging of Jupytext, and helped us transition the project to the src layout, and our old setup.py to an up-to-date pyproject.toml configuration that uses hatch to build Jupytext.

I also want to thank Cristian Le for his precious advice regarding the layout refactoring, and for helping us to tackle the CI reorganization. Thierry Parmentelat, who had previously ported the front-end extension to JupyterLab 4, contributed much testing and feedback on this new version of the front-end extension.

It is always a pleasure for me to maintain Jupytext, an amazing adventure that started five years ago already. However, I can get busy at times (Jupytext comes in addition to my day job, one cat, two bikes, three kids), so I am really thankful for receiving such contributions, especially when they are of such a great quality!

Jupyter Notebook format workshop outcomes

2023-04-12T17:40:00+00:00

Workshop Social Event challenging our senses

The Jupyter Community Workshop on the notebook file format took place at the Safran Campus near Paris from February 28th to March 2nd. It was a great opportunity to gather various Jupyter stakeholders from private and public affiliations to bootstrap new features for the Notebook file format.

Jupyter Community Workshops are a series of events designed to bring together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics.

Community discussions

The community has lots of ideas to improve the Notebook format. So we split in three smaller groups with the aim of drafting Jupyter Enhancement Proposal (JEP): the markdown group, the text-format group and the cell types group.

The markdown group focused on backward compatible enhancement for the Markdown cells. The discussions focused on two subjects:

The specification of the Markdown flavor (pre-proposal): the goals are to specify which Markdown flavor (e.g. GitHub, CommonMark, MyST,…) is used for the cell source, how to store rendered output for easier cross-compatibility and what is the default Markdown flavor.
The persistence of user expression (pre-proposal): in order to display inline expressions within Markdown cells, the results obtained from the kernel should be stored in the notebook. This proposal aims to define the schema modification to store such information.

The text-format group lays out a specification for an official Jupyter notebook textual format. The discussion went on after the meeting to prepare that proposal.

Finally the cell-types group took the hypothesis of starting from the blank page to create the best Jupyter notebook format building on top of 10-years of experience. The discussion will take time to settle down on a new specification. So if you are interested, join the weekly discussion (see the meeting notes for all the details). In addition to the fully new specification, two backward compatible enhancements have been proposed:

Adding $schema to the notebook format and deprecate the nbformat version keys (see proposal).
Adding extraSchema to the notebook format to optionally extend the schema to specify in particular metadata (see pre-proposal).

Follow-up

Six JEP’s are foreseen from the workshop discussions. But as mentioned earlier, the community has lots of great other ideas (like SQL cells, low-/no-code cells for inputs or visualization). So we would like to encourage anyone interested by any Jupyter enhancement to open issue on the Jupyter Enhancement Proposals repository (see the guidelines for more information).

With the new Jupyter governance in place, the new Software Steering Council is responsible for ensuring those proposals get reviewed and go through the approval process.

Acknowledgements

I deeply want to thank all participants to the workshop that took the time (some of them despite time zone difference) to bring very constructive and thoughtful discussion.

We are really grateful to Bloomberg and Amazon Web Services for their donations to the Jupyter Community Workshops program. This event would not have been possible without their generous support.

We are also grateful to Safran Group for hosting this workshop and the NumFOCUS foundation for helping and mentoring this workshop organization.

Jupyter Community Workshop: The notebook file format

2022-12-08T16:09:00+00:00

We are excited to announce the next in-person Jupyter Community Workshop! It will focus on the Notebook file format.

Jupyter Community Workshops are a series of events designed to bring together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics.

The Jupyter notebook file format has been around for 10 years. Its usage has grown in countless fields from teaching to data analysis in production pipelines. A great number of software applications and online services have added support for it.

We want this workshop to be an opportunity for various stakeholders to push forward the format while preserving its reusability in as many applications as possible. We could for example prototype a syntax for injecting variable values in Markdown cells, specify an alternative more textual format like RMarkdown, define the Markdown variant we support,… the boundary is our imagination. By the end of the workshop, we will submit those new specifications as Jupyter Enhancement Proposals to kick start the validation process for enhancing the official notebook format.

The workshop will last three days, with hands-on discussions, hacking sessions, and technical presentations. The goal of this event is to foster collaboration and the sharing of knowledge between maintainers of various platforms supporting the file format, downstream library authors and power users.

The workshop will be held at the Safran Campus in Paris suburb, France from February 28th to March 2nd, 2023. Travel funding assistance is available for attendees from academia and those from groups which are not well-represented within the Jupyter and wider tech community!

Application and all other details can be found in this form.

We are grateful to Safran Group for sponsoring this event. We are also grateful to the sponsors of the Jupyter Community Workshop series, Bloomberg and Amazon Web Services.

Ploomber: Maintainable and Collaborative Pipelines in Jupyter

2021-09-01T14:17:00+00:00

Ploomber is an open-source framework that allows teams to develop maintainable pipelines in Jupyter.

Jupyter is a fantastic tool for data exploration. The ability to transform our data interactively and get immediate visual feedback allows us to understand it quickly. However, when working on large projects, collaboration can become difficult. Features such as live collaboration are a gigantic leap forward for teamwork. Still, it has to complement an asynchronous workflow that allows team members to work at different times, especially in a remote-first workplace.

Back in 2020, I introduced Ploomber at JupyterCon to help practitioners build maintainable and reproducible data workflows. Fortunately, the community is growing. We’ve received great feedback from teams that use Ploomber to develop production-ready pipelines using Jupyter, debunking the notion that notebooks are only for prototyping.

As we’ve gathered more feedback from our community, we realized that while users had improved the reproducibility of their workflows, team dynamics didn’t change much. In many cases, collaborators worked in isolation, sharing processed data, which severely hindered reproducibility.

Teams often share processed data, which hinders reproducibility.

Since its first release, Ploomber aimed to promote software development best practices to produce more maintainable data projects. We’re now doubling our efforts to enable a collaborative and asynchronous workflow.

Enabling Code Reviews

We frequently hear from teams using Jupyter notebooks that it’s challenging to manage .ipynb files. For example, we heard from a data scientist that his company considered banning Jupyter notebooks because they couldn’t figure out how to follow software engineering best practices. Our fellow data scientist was highly frustrated by this situation since Jupyter turbocharges his ability to explore and understand data.

Managing .ipynb files is challenging for multiple reasons. Suppose I push a notebook to a git repository. Then, I add a comment and push it to the repository again. The difference between the two versions looks like this:

Diff view from a notebook with a new cell.

.ipynb files contain input and outputs in a single file, making them extremely useful when we want to share code and results with a colleague but complicates code reviews where we want to compare the previous version with the current one. However, we can fix this problem by changing the underlying file format.

Many practitioners don’t know that Jupyter is agnostic to the underlying file representation, allowing us to interact with different file formats as notebooks. Jupytext is a fantastic project that enables us to open various file formats such as .py and .md as notebooks.

Ploomber integrates with jupytext, allowing users to store their source code as .py files and explore data interactively with Jupyter. Since source code exists in .py files, this enables code reviews, file merging, and the flexibility to edit the code either in Jupyter or in any text editor, giving anyone on the team the freedom to use whatever tool they like the most.

The same .py file is displayed as a notebook in JupyterLab and as a script in VS Code.

Enabling Modularization

Often, teams develop significant parts of a project in a single notebook for convenience; however, this makes it hard to maintain the project in the long run. We know from decades of advancement in software engineering practice that modularizing our code produces more maintainable projects. Yet, we tend to work on single notebooks because breaking down analysis in multiple parts involves managing project structure, ensuring we route outputs correctly, and writing code to orchestrate all steps.

Ploomber allows users to concatenate multiple notebooks into a pipeline in two steps: list the notebooks in a YAML file and declare execution dependencies (e.g., download data, then clean it). Furthermore, Ploomber parses our execution dependencies and injects inputs into our notebook when opening it. Thus, there’s no need to hard-code any paths. The following image illustrates how to declare a pipeline in a pipeline.yaml file and the notebook’s code injection process:

Ploomber integrates with JupyterLab to auto-complete outputs from task dependencies.

Modularization has another benefit. It allows teams to collaborate on separate, well-defined streams of work. Data projects have a sequential nature, and by explicitly structuring them into small tasks, it is easy to assign parts to various team collaborators. For example, if a two-person team is working on a project that uses two sources of data, each member can take one dataset:

Modularization allows teams to work efficiently.

Enabling Testing

Modularization facilitates testing. A recommended practice when developing data pipelines is to test the output data from each task to ensure it has some minimum quality. Since we have clear boundaries among tasks, we can embed integration tests that check the output coming out of each step, ensuring that we know when something breaks.

Ploomber allows testing output artifacts from each task to check for data quality.

Examples of integration tests include: checking there are no NULL values in a specific column or verifying values fall into a certain range. In Ploomber, you can execute an arbitrary function after your notebook finishes to assert statements on your data:

Example of an integration test that ensures that a particular column does not have NAs.

Ensuring Reproducibility

Users can adopt a continuous integration workflow to test for reproducibility.

One of Jupyter notebook’s most recurring problems is hidden state; this happens when we execute code cells in an arbitrary order. However, when running cells in sequential order, the results do not match the recorded output.

Hidden state creates significant issues. For example, teams often execute notebooks locally, store them in a git repository, and don’t execute them again. For instance, I heard from a fellow data scientist that her team struggled to update a model in production because of a broken notebook that prevented her team from re-training the model: the notebook had been executed locally once but never tested for reproducibility.

The answer to hidden state is straightforward: test continuously. In software engineering, it’s common to run code and test it on each git push. However, such practice hasn’t found its way into the data world, primarily because running data processing code may take hours, making testing unfeasible.

Fortunately, most errors are detectable with small amounts of data, and Ploomber simplifies managing pipeline configurations. For example, a user can define a sample parameter to run the pipeline with a fraction of the input data:

An example parametrized pipeline that can execute with a sample of the data.

Then, a continuous integration script can run the pipeline with a sample by switching the sample parameter:

ploomber build --sample true

Furthermore, users can easily download remote artifacts to debug broken pipelines.

Enabling Pull Requests

Users can submit Pull Requests containing code (.py files) and results (executed notebooks).

Once a new feature is ready, a team member can open a pull request. Since the code is in .py files, the reviewer can easily compare code versions. However, metrics or charts are essential to evaluate data processing code. For this reason, Ploomber generates a .ipynb file for each .py file; such executed notebooks can be attached to a pull request to review code and output.

Pipeline execution generates executed notebook files that can be attached to a Pull Request.

A CI process can orchestrate pipeline execution with a single command: ploomber build. However, if working with large datasets, Ploomber can export pipelines to execute in AWS Batch, Airflow, or Kubernetes (Argo Workflows).

Enabling Fast Iterations

Ploomber skips execution of tasks whose source code has not changed since the last run.

Data projects are highly iterative and require us to run small experiments to evaluate our final results. For example, we may add a new feature and assess whether that improves model performance. The more experiments we try, the higher our chance of success.

Such experiments are small and often only touch a small portion of the pipeline. If most of our tasks are unaffected, re-running tasks is a waste of time since they’ll generate the same results. Given that a training pipeline may take hours to run, it is essential to speed things up as much as possible. To enable faster iterations, Ploomber builds pipelines incrementally, skipping tasks whose source code hasn’t changed. Incremental builds help in multiple scenarios, for example, when trying out local experiments or running the pipeline in the CI system. Furthermore, they enable crash recovery: if we execute our pipeline and it crashes, we can fix the failing task, submit it again, and execution will take off from the point of failure.

JupyterLab Is a Production-Ready Platform

It is common for teams to develop prototypes in Jupyter, then refactor them into Python modules; such an approach creates a lot of overhead and a tremendous burden for data scientists (who produce the models) and engineers (who have to refactor notebook-based prototypes). Instead, we believe data projects should start with production in mind by following best software development practices that allow teams to iterate quickly.

Providing such an experience is challenging. Nevertheless, we are working hard to achieve that goal; we want data scientists and engineers to collaborate to produce production-ready projects that instantly go from Jupyter to production.

The Future

There is still a long way to capture our vision of the future of data workflows. Furthermore, we want to keep building the project with the help of our users. So if you’re interested in building Ploomber with us, join our community, or follow us on Twitter! And if you believe in our mission, please show your support with a star on GitHub.

Thanks to Ido Michael and Filip Jankovic for providing feedback.