Jupyter Blog - debugging

Notebooks-on-Demand

2026-09-22T00:37:00+00:00

A tour of Notebooks-On-Demand

Nod: Notebooks-on-Demand is an open-source JupyterLab extension that lets you jump into a Jupyter notebook from anywhere in a Python codebase. Just call notebook() from anywhere in your .py file, run it with nod, and a notebook opens at that point in the program with all your variables already loaded into the notebook state. When you’re done making edits, send them back to your original Python file with a click.

Here’s what it looks like in practice. Say you have this Python program:

from nodpy import notebook

def f(a,b):
    x = 1
    notebook() #our program will pause here

def d(a):
  b = 2
  f(a,b)

d(4)

All you need to do is execute your program with nod at the start:

nod python -m myfile

and a new JupyterLab session will open which looks like this:

Left: The Nod Control Panel shows the call stack for our notebook() call, Center: The notebook created from the current Nod Session, Right: The JupyterLab Debugger shows the current variables in our kernel

You’ll notice a couple of things. First, on the right-hand pane, you’ll see that the current notebook state matches the variables available in the Python program at the place notebook() was called. Second, the body of the current function f is editable, and the surrounding code is displayed for context.

From here, you can use your on-demand notebook to write and debug the function f. When you’re done, you can send your new code back to the original source file with a button at the top of the Nod Panel on the left (see the repository for detailed instructions). Whatever you wrote in your on-demand notebook will replace the current body of f in the source file. The notebook cells will be converted to newlines, and markdown cells are converted to comments (thanks to Jupytext!). We can also re-run our program from the beginning with these new changes in our source file through the Nod panel.

Here’s a few scenarios that Nod might be helpful:

Scenario 1: You’re working in a large Python project and code is spread over countless files, classes, and function calls. You begin writing a new function, but the arguments are complex and questions abound: What exactly does that class method return? When I join these two tables, what does the output look like? How do I even construct the inputs to this function?

You think: This would be so much easier if I could just write this function in a notebook!

Instead of manually copying out code snippets, writing data structures to files and reading them into a notebook, and perhaps getting something to work eventually, you can just call notebook(), and a Jupyter notebook appears with just the body of your function and the function’s arguments already in your notebook state.

Scenario 2: You realize there’s a bug in a specific function when called with specific arguments. You could use a breakpoint() to inspect the program, but you don’t just want to look at the arguments, you want to see if different ways of changing the arguments remove the bug when you run the function code. Instead, you call notebook() when the function is invoked with the buggy arguments, opening a Jupyter notebook on the function definition with those arguments loaded in the notebook state. You can modify the arguments and rerun your code in the notebook to test different outcomes and implement your fix. When you’re done, you can send the changes back to your source file and restart (opening a notebook-on-demand for your new program), verifying that your program is now running as expected.

Scenario 3: You use an AI system to generate some code, but there’s a crucial part you want to really understand. Just call notebook() at that part, and now you have the full functionality of a notebook to walk line-by-line through the code, printing and visualizing the output at each step or experimenting with alternative decisions.

Nod’s on-demand notebooks are designed to be disposable. After you’ve sent your changes back to your source file, all the information to recreate the notebook is already in your Python file, so you can safely quit the JupyterLab session and continue programming in your text editor of choice. Your code can live in your text editor and Python source files, but Nod makes a quick trip to a notebook easy.

While I’ve found this workflow by itself to be really helpful in my own programming, Nod can do two other things that are pretty neat:

First, you don’t just have to stay in the function body where you called notebook(); you can also navigate up and down the call stack with the Nod panel on the left. For example, you can select the function d to see how f was called:

The Nod Panel on the Left navigates down the callstack, replacing the current notebook and the state to match.

Both our notebook and the current variables update to our new position in our program (x isn’t in our debugger pane on the right anymore, since it’s not available within function d).

Second, you can use nodLog() to collect a bunch of values from your program to poke and prod in an on-demand notebook. Take a look at the following file:

from nodpy import notebook, nodLog

def f(a, b):
    if a == 4:
        notebook()
    else:
        nodLog(a, b)

f(2, [1, 2, 3])
f(3, {"s": 1})
f(4, 1)

Running it will pop open a JupyterLab interface where, on the right, you’ll have access to a “Nod Log” pane, which lets you select assignments to a and b for the on-demand notebook session among the values you saved with nodLog():

nodLog() captures variables while your program runs and makes them available in the on-demand notebook session to experiment with.

When working with Python objects which can vary in unpredictable ways (like the results of a JSON API call or some other heavily-nested Python dictionary), I find nodLog() to be helpful in gauging the possible values I need to handle when writing a new function. It lets me inspect and switch between different inputs to my function easily.

Nod is free and open-source, so check it out here if you’re interested. Nod also works with JupyterHub with a slightly different workflow (see the repository for a guide!). My goal with Nod is to add to the situations where Jupyter Notebooks are a useful tool. If you have ideas about more ways Nod can be helpful or feedback to make it better, please leave an issue on the Github page and we can discuss!

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

Robotic Process Automation with JupyterLab

2021-01-18T10:36:00+00:00

Robotic Process Automation (RPA) differs from classical automation tools in that the actions to automate can be developed by observing a user perform a task in a graphical user interface, across multiple applications.

It is a means to lower the entry barrier of process automation and enable the use of tools that don’t provide a programmatic interface.

Most typically, RPA developers will use a mixed approach between textual programming and performing actions manually. The resulting programs are typically called software robots. Therefore, interactive computing tools like Jupyter are a natural environment for RPA, as the interactive nature of Jupyter allows for quick iterations and trial-and-errors when developing such robots.

While many RPA tools are commercial software, Robot Framework and the tooling developed by Robocorp provide an open-source RPA programming language, with a high-level syntax, extensible with Python plugins. It has a rich ecosystem of libraries and tools that are developed as separate projects.

Robot Framework and Project Jupyter

Today, we are happy to announce the first release of xeus-robot, a Jupyter kernel for Robot Framework based on xeus, supporting the new JupyterLab 3.0 visual debugger, auto-completion, and much more!

But before we dive into xeus-robot, we should recognize the earlier work by Nick Bollweg and Asko Soukka, who developed two Jupyter kernels for Robot Framework, irobotframework, and robotkernel, and gave an amazing talk together at RoboCon in 2019!

The motivation for us to reboot this effort with a kernel based on xeus was to enable the JupyterLab Visual Debugger for this kernel. This requires a different concurrency model than that of ipykernel, which underlies both irobotframework and robotkernel. In the end, we were able to provide the same features and more, including e.g. code completion in Python cells, debugging etc.

Diving into Xeus-robot

xeus-robot is a reboot of the already existing robotkernel, based on xeus.

Like most language kernels, xeus-robot supports code completion, inspection, error handling, etc. It also allows using Python cells to define custom robot keywords in Python, those Python cells support code completion as well.

Code Completion in Robot Framework

By using the libraries robotframework-seleniumlibrary and robotframework-seleniumscreenshots, you can even complete the selection of elements on the page you are currently testing!

Code completion with an action of the user in the UI.

The %%python cell magic makes it possible to extend Robotframework with Python modules in the notebook.

Extending RobotFramework with Python, and code completion in Python cells

Widget-based UIs are provided to test Robotframework “keywords”.

Testing Robot Framework “keywords” with Jupyter widgets

Last but not least, xeus-robot comes with full support for the JupyterLab Visual Debugger! After the xeus-python kernel, it is the second Jupyter kernel to support the Jupyter Debugger Protocol! We hope that many more will come.

You can set breakpoints, step in defined keywords (the equivalent of functions), inspect variables, and see the callstack, as shown below.

The JupyterLab Visual Debugger in action with Robot Framework

Installing xeus-robot

Xeus-robot is available for all platforms on conda-forge, and can be installed with conda or mamba.

mamba install xeus-robot -c conda-forge

The jupyterlab-robotmode package, which provides JupyterLab syntax highlighting for Robot Framework will also be installed as a dependency.

For your conda installation, we recommend starting from mambaforge or miniforge which are available for download here, and default to the conda-forge channel (making the -c conda-forge argument unnecessary).

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Acknowledgments

The work on xeus-robot by QuantStack was funded by Robocorp, and is part of their initiative to create an open-source RPA ecosystem. Beyond the xeus-robot kernel, Robocorp also funds the integration of Robot Framework with Visual Studio Code, RPA Framework, and other development to push the Robot Framework project forward.

The implementation of the debugger in xeus-robot relies on robotframework-lsp by Fabio Zadrozny and we want to thank him for his help in integrating it in xeus-robot.

About the authors

Martin Renou is a scientific software developer at QuantStack. He is the creator of the xeus-python and xeus-robot kernels.

Johan Mabille is a scientific software developer at QuantStack. He is a co-author of the xeus project, and developed the debugger extensions to xeus-python and xeus-robot. He also co-authored the front-end for the JupyterLab visual debugger.

A visual debugger for Jupyter

2020-03-25T15:53:00+00:00

Most of the progress made in software projects comes from incrementalism. The ability to quickly see the outcome of an execution and iterate has been one of the main reasons for the success of Jupyter, especially in scientific exploratory workflows.

Jupyter users like to experiment in the notebook, and to use the notebook as an interactive communication tool. However, for more classical software development tasks such as the refactoring of a large codebase, they often switch to general-purpose IDEs.

The JupyterLab environment.

The Jupyter project has made strides in the past few years towards filling that gap, notably with the JupyterLab project, which enables a richer UI including a file browser, text editors, consoles, notebooks, and a rich layout system.

However, a missing piece (which has remained one of the main reasons for users to switch to a different tool) is a visual debugger. This feature has long been requested by users, especially those accustomed to general-purpose development environments.

A debugger for Jupyter

Today, after several months of development, we are glad to announce the first public release of the Jupyter visual debugger!

This is just the first release, but we can already set breakpoints in notebook cells and source files, inspect variables, navigate the call stack and more.

Screencast of the JupyterLab visual debugger in action

Try the debugger on binder

You can also try the debugger online with binder. Just click on the binder link:

Click on the binder link to launch the demo

Installation

The debugger front-end can be installed as a JupyterLab extension.

jupyter labextension install @jupyterlab/debugger

The debugger front-end will be included in JupyterLab by default in a future release.

In the back-end, a kernel implementing the Jupyter Debug Protocol (which will be detailed in the next section) is required. The only kernel implementing this protocol, for now, is xeus-python a new Jupyter kernel for the Python programming language. (Support for the debugger protocol in ipykernel is also on the roadmap).

conda install xeus-python -c conda-forge

Once xeus-python and the debugger extension are installed, you should be all set to use the Jupyter visual debugger!

Note: Depending on the platform, PyPI wheels are available for xeus-python, but they are still experimental.

The Jupyter Debug Protocol

New message types for the Control and IOPub channels

Jupyter kernels (the part of the infrastructure that executes the user’s code) communicate with the rest of the infrastructure with a well-specified inter-process communication protocol.

Several communication channels exist, such as

the Shell channel, which is a request/reply channel for e.g. execution requests
the IOPub channel, which is a one-directional communication channel from the kernel to the client, and is used e.g. to forward the content of the standard output streams (stdout and stderr).

The Control channel is similar to Shell but operates on a separate socket so that messages are not queued behind execution requests, and have a higher priority. Control was already used for Interrupt and Shutdown requests, and we decided to use the same channel for the commands sent to the debugger.

Two message types were added to the protocol:

the debug_[request/reply] to request specific actions to be performed by the debugger such as adding a breakpoint or stepping into a code, which is sent to the Control channel.
the debug_event uni-directional message used by debugging kernels to send debugging events to the front-end. Debug events are sent over the IOPub channel.

Extending the Debug Adapter Protocol

A key principle to the Jupyter design is the agnosticism to the programming language. It is important for the Jupyter debug protocol to be adaptable to other kernel implementations.

A popular standard for debugging is Microsoft’s “Debug Adapter Protocol” (DAP) which is a JSON-based protocol underlying the debugger of Visual Studio Code and for which there already exist multiple language back-ends.

It was therefore natural for us to use the DAP messages over the debug_[request/reply] and debug_event messages that we just added.

However, it was not quite sufficient in the case of Jupyter. Indeed

In order to support page reloading, or a client connecting at a later stage, Jupyter kernels must store the state of the debugger (breakpoints, whether the debugger is currently stopped). The front-end can request that state over with a debug_request message.
In order to support the debugging of notebook cells and of Jupyter consoles, which are not based on source files, we also needed messages to submit code to the debugger to which breakpoints can be added.

Besides these two differences, the content of the debug requests and replies corresponds to the debug adapter protocol.

All these extensions to the Jupyter kernel protocol have been proposed for inclusion in the official specification. The JEP (Jupyter Enhancement Proposal) can be found here.

Xeus-python, the first Jupyter Kernel to support debugging

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel by itself but a library that helps kernel authoring. Xeus is useful when developing a kernel for a language that has a C or a C++API (like Python, Lua, or SQL). It takes the cumbersome task of implementing the Jupyter messaging protocol for the kernel author to focus on the core interpreter tasks: executing code, inspecting, etc.

Several kernels have been developed with xeus, including the popular xeus-cling kernel for the C++ programming language, based on the cling C++ interpreter from CERN. The xeus-python kernel is an alternative Python kernel to ipykernel, based on xeus. The first release of the xeus-python kernel was announced on this blog earlier this year: A new Python kernel for Jupyter

Xeus-python was an appropriate choice for this first implementation of the debugging protocol because

it has a pluggable concurrency model, which allowed running the processing of the Control channel in a different thread.
it has a lighter-weight codebase which made it a convenient sandbox to iterate upon. Implementing the first version of the protocol in ipykernel would have required more significant refactoring and consensus building at an early stage.

The roadmap of Xeus-python

The short-term roadmap for xeus-python includes

adding support for IPython magics in xeus-python, which is the main missing feature with respect to ipykernel.
improving the PyPI wheels of xeus-python.

What about other kernels?

The work in the front-end is valid for any kernel implementing the extended kernel protocol.

We will be working in 2020 to enable debugging with as many kernels as possible.

This will soon be the case for other xeus-based kernels which share a large part of the implementation with xeus-python, such as xeus-cling.

Diving into the debugger front-end architecture

The debugger extension for JupyterLab provides what users would typically expect from an IDE:

a sidebar with a variable explorer, a list of breakpoints, a source preview and the possibility to navigate the call stack
the ability to set breakpoints directly next to the code, namely in code cells and code consoles
visual markers to indicate where the current execution has stopped

When working with Jupyter notebooks, the state of the execution is kept in the kernel. But a cell can be executed and then deleted from the notebook. What should happen when a user wants to step in deleted code?

The extension supports that particular use case and enables retrieving a read-only view of the previously executed cell.

Stepping into a deleted cell

Consoles and files also have support for debugging.

Debugging code consoles in JupyterLab

Debugging files in JupyterLab

Debugging can be enabled on a notebook level, which lets users debug a notebook and work on a different one at the same time.

Debugging multiple notebooks simultaneously

Variables can be inspected using a tree viewer and a table viewer:

The variable explorer

The debugger extension for JupyterLab has been designed to work with any kernel that supports debugging.

By relying on the Debug Adapter Protocol, the debugger extension abstracts away language-specific features and provides a consistent debugging interface to the user.

The following diagram shows how the debug messages flow between the user, the JupyterLab extension and the kernel during a debugging session.

Using the Debug Adapter Protocol in the debugger extension (source)

Future developments

In 2020, we plan on making major improvements to the debugger experience:

Support for rich mime type rendering in the variable explorer.
Support for conditional breakpoints in the UI.
General improvements of the debugger user experience.
Enable the debugging of Voilà dashboards, from the JupyterLab Voilà preview extension.

Acknowledgements

The JupyterLab debugger is the result of the collaboration and coordination of developers from several institutions, including QuantStack, Two Sigma, and Bloomberg.

The work on both the front-end and the back-end at QuantStack by Jeremy Tuloup, Johan Mabille, Martin Renou, and Sylvain Corlay, was funded by Bloomberg.
The work on the Jupyter debugger by Borys Palka and Afshin Darian was made possible by Two Sigma.

About the developers

Jeremy Tuloup

Jeremy Tuloup is a Scientific Software developer at QuantStack. He authored a large part of the front-end of the JupyterLab debugger.

Borys Palka

Borys Palka is a software developer at Codete. He authored a large part of the front-end of the JupyterLab debugger.

Johan Mabille

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of xeus, and developed the debugger extension to xeus-python. He also authored a large part of the debugger front-end.

Martin Renou

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Afshin Darian

Afshin Darian is a software developer at Two Sigma, and one of the authors of JupyterLab.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.