Jupyter Blog - extensions

Notebooks-on-Demand

2026-09-22T00:37:00+00:00

A tour of Notebooks-On-Demand

Nod: Notebooks-on-Demand is an open-source JupyterLab extension that lets you jump into a Jupyter notebook from anywhere in a Python codebase. Just call notebook() from anywhere in your .py file, run it with nod, and a notebook opens at that point in the program with all your variables already loaded into the notebook state. When you’re done making edits, send them back to your original Python file with a click.

Here’s what it looks like in practice. Say you have this Python program:

from nodpy import notebook

def f(a,b):
    x = 1
    notebook() #our program will pause here

def d(a):
  b = 2
  f(a,b)

d(4)

All you need to do is execute your program with nod at the start:

nod python -m myfile

and a new JupyterLab session will open which looks like this:

Left: The Nod Control Panel shows the call stack for our notebook() call, Center: The notebook created from the current Nod Session, Right: The JupyterLab Debugger shows the current variables in our kernel

You’ll notice a couple of things. First, on the right-hand pane, you’ll see that the current notebook state matches the variables available in the Python program at the place notebook() was called. Second, the body of the current function f is editable, and the surrounding code is displayed for context.

From here, you can use your on-demand notebook to write and debug the function f. When you’re done, you can send your new code back to the original source file with a button at the top of the Nod Panel on the left (see the repository for detailed instructions). Whatever you wrote in your on-demand notebook will replace the current body of f in the source file. The notebook cells will be converted to newlines, and markdown cells are converted to comments (thanks to Jupytext!). We can also re-run our program from the beginning with these new changes in our source file through the Nod panel.

Here’s a few scenarios that Nod might be helpful:

Scenario 1: You’re working in a large Python project and code is spread over countless files, classes, and function calls. You begin writing a new function, but the arguments are complex and questions abound: What exactly does that class method return? When I join these two tables, what does the output look like? How do I even construct the inputs to this function?

You think: This would be so much easier if I could just write this function in a notebook!

Instead of manually copying out code snippets, writing data structures to files and reading them into a notebook, and perhaps getting something to work eventually, you can just call notebook(), and a Jupyter notebook appears with just the body of your function and the function’s arguments already in your notebook state.

Scenario 2: You realize there’s a bug in a specific function when called with specific arguments. You could use a breakpoint() to inspect the program, but you don’t just want to look at the arguments, you want to see if different ways of changing the arguments remove the bug when you run the function code. Instead, you call notebook() when the function is invoked with the buggy arguments, opening a Jupyter notebook on the function definition with those arguments loaded in the notebook state. You can modify the arguments and rerun your code in the notebook to test different outcomes and implement your fix. When you’re done, you can send the changes back to your source file and restart (opening a notebook-on-demand for your new program), verifying that your program is now running as expected.

Scenario 3: You use an AI system to generate some code, but there’s a crucial part you want to really understand. Just call notebook() at that part, and now you have the full functionality of a notebook to walk line-by-line through the code, printing and visualizing the output at each step or experimenting with alternative decisions.

Nod’s on-demand notebooks are designed to be disposable. After you’ve sent your changes back to your source file, all the information to recreate the notebook is already in your Python file, so you can safely quit the JupyterLab session and continue programming in your text editor of choice. Your code can live in your text editor and Python source files, but Nod makes a quick trip to a notebook easy.

While I’ve found this workflow by itself to be really helpful in my own programming, Nod can do two other things that are pretty neat:

First, you don’t just have to stay in the function body where you called notebook(); you can also navigate up and down the call stack with the Nod panel on the left. For example, you can select the function d to see how f was called:

The Nod Panel on the Left navigates down the callstack, replacing the current notebook and the state to match.

Both our notebook and the current variables update to our new position in our program (x isn’t in our debugger pane on the right anymore, since it’s not available within function d).

Second, you can use nodLog() to collect a bunch of values from your program to poke and prod in an on-demand notebook. Take a look at the following file:

from nodpy import notebook, nodLog

def f(a, b):
    if a == 4:
        notebook()
    else:
        nodLog(a, b)

f(2, [1, 2, 3])
f(3, {"s": 1})
f(4, 1)

Running it will pop open a JupyterLab interface where, on the right, you’ll have access to a “Nod Log” pane, which lets you select assignments to a and b for the on-demand notebook session among the values you saved with nodLog():

nodLog() captures variables while your program runs and makes them available in the on-demand notebook session to experiment with.

When working with Python objects which can vary in unpredictable ways (like the results of a JSON API call or some other heavily-nested Python dictionary), I find nodLog() to be helpful in gauging the possible values I need to handle when writing a new function. It lets me inspect and switch between different inputs to my function easily.

Nod is free and open-source, so check it out here if you’re interested. Nod also works with JupyterHub with a slightly different workflow (see the repository for a guide!). My goal with Nod is to add to the situations where Jupyter Notebooks are a useful tool. If you have ideas about more ways Nod can be helpful or feedback to make it better, please leave an issue on the Github page and we can discuss!

Announcing jupyter-builder: A Standalone Build System for JupyterLab Extensions

2026-06-19T13:11:00+00:00

We’re excited to announce the first stable release of jupyter-builder on PyPI and @jupyter/builder on npm. A standalone, configurable build system for JupyterLab and its extensions.

If you build or maintain a JupyterLab extension, this release is for you.

Motivation

For a long time, building a JupyterLab extension has meant installing all of JupyterLab. Take a look at any extension’s build dependencies and you’ll see jupyterlab listed there, not because the extension needs JupyterLab at runtime to be built, but because the build tooling lives inside the JupyterLab repository. The builder/ folder, the jlpm command, the helper scripts, they’re all coupled to the JupyterLab core.

This setup has caused real friction over the years:

Linux packagers have repeatedly raised concerns about the circular dependency where extensions like jupyterlab_pygments need jupyterlab to build, but jupyterlab ships with those extensions. It’s a chicken-and-egg problem that makes packaging painful.
Extension authors have to update their build system every time they update to a new JupyterLab version, which sometimes introduces breakages that have nothing to do with their extension’s actual code.
Users still occasionally run into installation issues that trace back to the incomplete migration from “source” extensions to prebuilt ones.

The need to separate the build system was identified back in jupyterlab#13456, and initial work began during GSoC 2024. With the support of the Jupyter Foundation’s first community-funded proposal, we’ve now brought that work to a stable release.

What `jupyter-builder` does

jupyter-builder extracts all the Node.js-based build tooling out of JupyterLab into a dedicated package. You install it on its own, and it builds your extension, no full JupyterLab installation required.

It ships in two parts:

jupyter-builder (PyPI) — the Python package that provides the jupyter-builder CLI.
@jupyter/builder (npm) — the underlying JavaScript package containing the build scripts. The Python CLI invokes build-labextension.js from this package, which then runs the rspack compiler with all the configuration needed to produce a JupyterLab extension.

Instead of jupyter labextension build ., you now run:

jupyter-builder build .

Build compatibility without JupyterLab installed

One interesting problem we had to solve: if JupyterLab isn’t installed, how does the builder verify that an extension is compatible with a given JupyterLab version?

The answer is @jupyterlab/core-meta, a small npm package that publishes JupyterLab’s core metadata on its own. @jupyter/builder declares a dependency on a specific @jupyterlab/core-meta version, so by default your extension is checked for compatibility against whatever JupyterLab version that pin corresponds to. In general, the latest @jupyter/builder will track the latest JupyterLab.

If you want to build against a different JupyterLab version, pass --core-version:

jupyter-builder build . --core-version 4.4.x

We’ll download that version’s metadata from @jupyterlab/core-meta on npm and check compatibility against it.

Migrating an existing extension

If you’re starting a new extension with the extension template, you’ll get all of this set up automatically. For existing extensions, there are two changes to make.

1. Update the Python build dependencies

In your pyproject.toml, swap jupyterlab for jupyter-builder:

# Before
requires = ["hatchling>=1.5.0", "jupyterlab>=4.0.0,<5", "hatch-nodejs-version>=0.3.2"]

# After
requires = ["hatchling>=1.5.0", "jupyter-builder>=1.0.0", "hatch-nodejs-version>=0.3.2"]

Then update the scripts section of your package.json to use the new CLI:

"build:labextension": "jupyter-builder build .",
"build:labextension:dev": "jupyter-builder build --development True .",
"watch:labextension": "jupyter-builder watch ."

Your build no longer pulls in all of JupyterLab.

2. Update the npm builder dependency

Replace @jupyterlab/builder with @jupyter/builder in your package.json:

"@jupyter/builder": "^1.0.0"

jupyter-builder is backwards-compatible with @jupyterlab/builder, so things will keep working if you don’t migrate immediately. That said, we strongly recommend switching: @jupyterlab/builder is no longer being actively developed, and all new work is happening in @jupyter/builder.

Note that as part of this transition JupyterLab has also moved from Webpack to Rspack. Most extensions won’t need any changes, but if yours uses a custom webpackConfig see the Rspack migration notes.

What’s next

This release is the first stable foundation. We’re continuing to:

Migrate more packages across the Jupyter organization to use jupyter-builder.
Make the builder more configurable, so it can eventually be used for projects beyond JupyterLab.

If you maintain an extension, please try the migration and let us know how it goes. Feedback from real extensions is the best way to make sure the migration path stays smooth.

Acknowledgements

This work was funded under the first round of Jupyter Community Funded Proposals. We want to thank the Jupyter Foundation and everyone involved in setting up this funding mechanism.

A huge thank you to Nicholas Bollweg for many thoughtful discussions on the design and packaging of jupyter-builder, and to Jeremy Tuloup for helping us resolve issues and for guiding the adoption of jupyter-builder in Jupyter Notebook.

This package was initially created during GSoC 2024 by Ronan Coutinho, mentored by Frédéric Collonval. Their work laid the foundation that made this release possible.

About the developers

Darshan Paudyal is an intern at OpenTeams and a member of the Jupyter Frontend Council. Darshan led the work to bring jupyter-builder to a stable release, completing the separation from JupyterLab core and shepherding the package through to its first published version.

Michał Krassowski is a Senior Software Engineer at OpenTeams. Mike provided guidance and technical direction throughout the development of jupyter-builder, helping shape its architecture and roadmap.

Useful links

📦 PyPI: jupyter-builder
📦 npm: @jupyter/builder
💻 Source: github.com/jupyterlab/jupyter-builder
📖 Migration guide: jupyter-builder migration section
💬 Issues & feedback: GitHub issues

Catching Jupyter-specific bugs before CI does: announcing jupyter eslint plugin

2026-06-11T15:13:00+00:00

We’re excited to announce the release of the first version of the Jupyter ESLint plugin @jupyter/eslint-plugin, now available on npm. It’s a custom ESLint plugin designed specifically for the official Jupyter frontends (JupyterLab, Notebook, JupyterLite) and TypeScript extensions built around them.

Motivation

If you’ve ever written a JupyterLab extension, you’ve probably hit something like this:

You wire up a new plugin, push a PR, walk away to make coffee, and 30 minutes later CI fails on a test you didn’t expect to be related. You scroll through logs, eventually trace it back, and realize the problem was a single misaligned argument in your activate function. TypeScript was happy. The editor showed no red squiggles. Nothing in your local toolchain said a word.

And the slow CI feedback loop is only half the story. Some of these mistakes don’t fail CI at all, a hardcoded English label, an enum in a settings schema, a malformed token id. They pass tests, ship to users, and only surface at runtime. They’re the kind of bugs you hear about from users months later, if at all.

JupyterLab, Jupyter Notebook, and the wider extension ecosystem follow a lot of project-specific patterns around plugins, commands, tokens, settings schemas, translations, and visual tests. These patterns aren’t enforced by TypeScript, because they live one level above the type system. They’re contracts about how the framework expects code to be written, not about whether the code type-checks.

A few examples of what generic linters and TypeScript can’t catch:

A JupyterFrontEndPlugin that requires [TokenA, TokenB] but whose activate function takes those arguments in the wrong order. Both are tokens, both type-check but the plugin fails at activation.
A command added with commands.addCommand(...) whose label is a hardcoded English string. The extension ships, gets translated by the community, and that one button silently stays in English.

These are real, recurring issues. They cost contributor time, they frustrate first-time contributors, and they pile onto reviewer load. They’re exactly the kind of knowledge that lives in the heads of long-time maintainers and exactly the kind of knowledge that a linter is great at encoding. So we built one.

What v1 catches

The plugin currently ships eight rules, grouped by what they protect:

Plugin structure

plugin-activation-args — Keeps activate function arguments aligned with requires and optional tokens.
plugin-description — Requires a non-empty description on every JupyterFrontEndPlugin.
token-format — Enforces the <package>:<TokenSymbol> convention for new Token(...) ids.

Commands and settings

command-described-by — Requires a describedBy schema on commands that accept args.
no-schema-enum — Disallows enum in settings JSON schemas in favor of translatable oneOf with const and title.

Internationalization

no-untranslated-string — Flags hardcoded user-facing strings in command labels, dialogs, accessibility attributes, JSX, and more.
no-translation-concatenation — Forbids dynamic concatenation inside trans.__() calls that translation extractors can’t pick up.

Testing

require-soft-assertions-before-snapshots — Requires expect.soft(...) for all but the last snapshot in a Playwright test, so a single failure doesn’t block the rest from updating.

Each rule has incorrect/correct examples, options, and rationale in the rules reference.

Getting started in 60 seconds

Install the plugin:

npm install --save-dev @jupyter/eslint-plugin

Add it to your ESLint config and turn on the recommended ruleset:

import jupyterPlugin from '@jupyter/eslint-plugin';

export default [
  {
    plugins: {
      jupyter: jupyterPlugin
    }
  },
  jupyterPlugin.configs.recommended
];

That’s it. Run jlpm eslint and you’ll start seeing Jupyter-specific issues alongside your existing lint output. If you’re still on legacy .eslintrc configs, there’s a recommended-legacy config too. However, we strongly recommend upgrading to ESLint v9 and using the new flat config format.

Full setup details, including type-aware configuration for plugin-activation-args and the JSON parser setup for no-schema-enum, are in the user guide.

If you maintain an extension, the most useful thing you can do today is install it and run it once on your codebase. Even if you don’t adopt it permanently, the first run is often informative most extensions surface a handful of issues they didn’t know they had.

Help us encode more knowledge

Eight rules is a starting point. The plugin is most valuable when it captures the recurring mistakes that maintainers across the ecosystem keep running into and a lot of that knowledge isn’t written down anywhere.

If you’ve ever been bitten by a Jupyter-specific issue that could have been a lint rule, something where you found yourself thinking “I wish my editor had warned me about this” please open an issue on the repository. Even a short description of the bug and a minimal example helps. We’d much rather encode that knowledge into an automated check once than have every new contributor learn it the hard way.

Acknowledgement

This work was funded under the first round of Jupyter Community Funded Proposals. We want to thank the Jupyter Foundation and everyone involved in setting up this funding mechanism.

We’re also grateful to Jeremy Tuloup and Nicolas Brichet for their help in adopting the plugin.

About the developers

Darshan Paudyal is an intern at OpenTeams and a member of the Jupyter Frontend Council. Darshan led the development of v1 of @jupyter/eslint-plugin, designing and implementing the rule set, the documentation, and the release.

Michał Krassowski is a Senior Software Engineer at OpenTeams. Mike provided guidance and direction for the development of the plugin.

Useful links

A User’s Journey with Plugin Playground: From First Idea to Installable JupyterLab Extension

2026-05-28T17:45:00+00:00

We are excited to announce the 1.0 release of JupyterLab Plugin Playground, allowing you to seamlessly experiment with the creation of Jupyter Notebook and JupyterLab plugins to add any functionality you may desire. To install:

pip install jupyterlab-plugin-playground

or use it from Binder or JupyterLite without installing anything by clicking one of these links: Binder (Lab), Binder (Notebook v7), JupyterLite (Lab), JupyterLite (Notebook v7).

Motivation

Building a JupyterLab extension has traditionally required setting up a complex local environment: installing dependencies from disjoint ecosystems (Node.js and Python), configuring tooling, and hunting down the right documentation. The goal of the Playground is to eliminate that setup overhead and streamline extension development. By bringing the runtime, documentation, and the context required for extension development into a single browser tab, you can go from idea to working extension without any local setup required.

Journey At a Glance

Start with a tiny extension idea that is easy to verify visually.
Choose a build path: Manual or AI-assisted.
Load and iterate inside JupyterLab until behavior is stable.
Build a second extension to prove the workflow is repeatable.
Share plugin files/packages for product and engineering review.
Export as a wheel(.whl) and validate installation in a clean Binder runtime.
Export as a zip and move the scaffold into a normal GitHub repository.

1.0: Pick Your Starting Path

At the beginning, choose the path that matches your confidence level and speed needs. If this is your first time with Plugin Playground, you can optionally run Take the Tour from the Launcher or Command Palette for quick orientation before starting.

Start from File If you already know the APIs you need and want direct control.
Build with AI If you know the intended behavior but want a first draft quickly.

Both paths converge to the same shipping workflow later.

2.0: Build Extension #1 Manually

For the first pass, we will create something small and observable.

Our Goal: add a command named Toggle Right Sidebar to the command palette.

2.1: Create the plugin file

Launch Start from File. This comes with a dummy template.

import {
  JupyterFrontEnd,
  JupyterFrontEndPlugin,
} from '@jupyterlab/application';

const plugin: JupyterFrontEndPlugin<void> = {
  id: 'hello-world:plugin',
  autoStart: true,
  activate: (app: JupyterFrontEnd) => {

  },
};

export default plugin;

Now build the plugin in order:

In Tokens section in the right sidebar, search ICommandPalette and click insert ( + icon ), so Plugin Playground adds token import and dependency wiring.

Add a simple command skeleton inside the activate function:

const commandID = 'my-first-playground-plugin:toggle-sidebar';
app.commands.addCommand(commandID, { label: 'Toggle Right Sidebar',
  execute: async () => {
      // place cursor here
  }
});

In Commands, search application:toggle-right-area, place your cursor inside execute of the added command ( see above ) and use Insert in selection This adds at the cursor position:
```
app.commands.execute('application:toggle-right-area');
```

commandPalette.addItem({ command: commandID, category: 'AAA' });

After those edits, your file should look like:

import { ICommandPalette } from '@jupyterlab/apputils';
import {
  JupyterFrontEnd,
  JupyterFrontEndPlugin,
} from '@jupyterlab/application';

const plugin: JupyterFrontEndPlugin<void> = {
  id: 'hello-world:plugin',
  autoStart: true,
  requires: [ICommandPalette],
  activate: (app: JupyterFrontEnd, commandPalette: ICommandPalette) => {
    const commandID = 'my-first-playground-plugin:toggle-sidebar'; 
    app.commands.addCommand(commandID, { label: 'Toggle Right Sidebar', 
      execute: async () => {
          app.commands.execute('application:toggle-right-area');
      } 
    });
      commandPalette.addItem({ command: commandID, category: 'AAA' });
  },
};

export default plugin;

Uses the typed plugin template (JupyterFrontEndPlugin<void>) and typed activate parameters. Imports ICommandPalette so the plugin can add entries to the Command Palette.
Registers a new command (my-first-playground-plugin:toggle-sidebar) and executes application:toggle-right-area when your command runs.
Adds your command to the palette under the category AAA.

2.3: Load it and verify behavior

Click Load Current File As Extension or click the run button in the toolbar, and open Command Palette.

When you run Load Current File As Extension, Plugin Playground compiles the active file and registers the plugin object into the live JupyterLab session. During iteration, if a plugin with the same id already exists, Plugin Playground attempts to deactivate and replace it so reload loops stay fast.

Expected result:

Toggle Right Sidebar appears in the command palette ( open command palette by going through view -> Activate command palette or Press Accel + shift + C).
Running it hides/shows the JupyterLab right sidebar.

3.0: Build with AI

If you want a faster draft, use AI assistance for the same extension workflow.

Goal: add the Show Active Notebook Cell Count command with a friendly no-notebook fallback.

3.1: Give AI a precise request

Use Build with AI and give a prompt:

for example, something like:-

Create a JupyterLab plugin named "active-notebook-cell-counter".
Add a command "Show Active Notebook Cell Count".
If no notebook is active, show a friendly dialog message.
If a notebook is active, show the current cell count in a dialog.
Register the command in the command palette under category "Playground Demo".
Export default plugin object.

What this block does:

This gives AI a concrete plugin goal, command name, and fallback behavior requirements.

3.2: Use AI + deterministic inserts together

In the Commands tab:

Insert in selection for predictable boilerplate insertion.
Prompt AI to insert when placement context is tricky.
{n} when you need argument shape clarity ( Inspect the command signature before insertion: it shows expected argument names/types and return details).

Use Insert in selection for predictable snippets, and switch to AI when insertion location or code adaptation depends on the surrounding context.

4.0: Iterate Faster Before Handoff

Once the extensions run, enable Run on save(from the toolbar or from the settings) while polishing labels, command categories, and small behaviour details.

With Run on save, each file save triggers a reload loop for that plugin file, which makes label tweaks, command grouping changes, and message-copy iteration much faster than manual reload cycles. And with enabling the global setting (Load as extension on save), this behaviour is present for all files by default.

Before packaging, you can ask a teammate to review behaviour or just share links for the extension for a quick look using:

Share a single file when sharing is about one plugin file.
Share a package when your logic spans several files.

This keeps review lightweight: collaborators can inspect and discuss behaviour before you commit to repository structure, CI, and release wiring.

6.0: Export as Wheel and Validate in Binder

Now test as an installable artifact in a separate environment.

6.1: Export `.whl`

From the export dropdown ( in toolbar ), choose Export as Python package (.whl).

6.2: Install in Binder

Open JupyterLab on Binder.
Upload the downloaded wheel.
Open a terminal and run:

python -m pip install --force-reinstall ./your_exported_package.whl

What this block does:

Installs the wheel you just exported from Plugin Playground into Binder. (Uses --force-reinstall so Binder replaces any previously installed copy with your newest build). Then refresh JupyterLab and verify commands.

7.0: Export as Zip and Create a GitHub Repository

After runtime validation, move to normal engineering workflows.

7.1: Export `.zip`

Choose Export as archive (.zip) from the toolbar export dropdown.

7.2: Bootstrap repository

Unzip locally, create an empty GitHub repo, then run:

git init
git add .
git commit -m "Initial extension scaffold from Plugin Playground"
git branch -M main
git remote add origin https://github.com/<your-org-or-user>/my-jlab-extension.git
git push -u origin main

What this block does:

Initializes a new Git repository from the exported scaffold and creates an initial commit that captures the generated extension baseline.

Now your prototype is in a standard repo flow with PRs, CI, and release automation.

What’s Next

Plugin Playground is still evolving. We’re actively working on the LSP integration for JupyterLite to bring code intelligence autocomplete and diagnostics directly into the editor, an “Ask AI” button on log errors that lets you send error context directly into the AI chat for instant debugging help. In the future, we would like to explore git integration to snapshots every successful load, paired with a built-in diff viewer to compare any snapshot against your current file. If you have any suggestions or encounter any problems, please let us know by opening an issue on GitHub.

Acknowledgements

We are grateful to the Jupyter Foundation and its members for sponsoring the development of Plugin Playground as part of Community Funding Proposals 2025.

We thank Jeremy Tuloup, Nick Bollweg, and Nicolas Brichet for providing inspiration, review, and advice across our work on the plugin and its dependencies. Finally, this work would not be possible without the authors of earlier iterations of Plugin Playground.

About the Developers

Anuj Singh is an OSS intern at OpenTeams. Anuj implemented Plugin Playground v1.0.0 and contributed to its dependencies during his internship at OpenTeams.

Smera Goel is a UI/UX designer at Quansight. Smera contributed to the user experience and design direction of Plugin Playground.

Michał Krassowski is a Senior Software Engineer at OpenTeams. Mike provided guidance and direction for the development of the Plugin Playground.

700 JupyterLab 4 Extensions!

2026-03-13T17:18:00+00:00

700 extensions for JupyterLab 4, and counting!

The JupyterLab extension ecosystem just crossed 700 extensions compatible with JupyterLab 4!

That’s 700 community-built plugins — from astronomical data viewers to reactive notebooks, from genome browsers to workflow managers — created by hundreds of developers, research labs, and companies around the world.

What Are JupyterLab Extensions?

Extensions are how JupyterLab becomes a Git client, a dashboard builder, a genomics viewer, or an AI workspace — without changing the core application. Install one with pip install, and it activates automatically.

Popular JupyterLab extensions

This is by design: JupyterLab itself is built as a collection of extensions — the file browser, the notebook editor, the terminal are all plugins. The same architecture that powers the core lets the community build what they need. For background, see 99 ways to extend the Jupyter ecosystem.

The Ecosystem at 700

700+ extensions compatible with JupyterLab 4
~960 total extensions published on PyPI
~9.8 million downloads/month
100M+ total downloads in the past year

By any measure, a substantial software layer has grown around JupyterLab.

How We Got Here

The ecosystem crossed 600 JL4-compatible extensions in late October 2025, days before JupyterCon in San Diego. At the conference, we ran a full-day Extension Development for Everyone tutorial with hands-on rapid prototyping. By early March 2026, we hit 700.

The ecosystem has been growing at a steady pace, averaging about 18 new extensions per month, with November 2025 setting an all-time monthly record of 33. Modern tooling is helping: better templates, documentation, and code generation tools have lowered the bar for what once required deep familiarity with TypeScript, Lumino, and JupyterLab internals.

Where the Extensions Are

Number of JupyterLab extensions by category

Monthly PyPI downloads by category

Development & Version Control dominates both in count (267) and downloads (5.4M/month). Visualization & Dashboards (2.7M/month) and System & Resource Management (602K/month) round out the top three most downloaded categories. But the fastest-growing categories point to where things are heading. Here’s what’s new in 2026:

JupyterLab’s AI Layer Starts Taking Shape

AI isn’t yet the biggest category in JupyterLab, but it may be the clearest signal of where new interaction patterns are emerging:

jupyter-ai-acp-client — Brings external AI agents into JupyterLab’s chat via the Agent Communication Protocol. Ships with Claude Code and Kiro personas.
nb-margin — Annotate cells with margin comments, and Claude Code edits them. A different paradigm from chat-based AI.
jupyterlite-ai-kernels — AI-powered kernels for JupyterLite, from Jeremy Tuloup. AI-assisted computation entirely in the browser.
jupyter-chat-components — Reusable chat UI components from Project Jupyter — building blocks for the next generation of AI tools.

These extensions reflect what the JupyterLab team identified as a 2026 priority: first-class integration with AI tooling.

Reproducibility Gets a Toolchain

calkit-python is the most downloaded new extension of 2026 (11,000+ monthly downloads). It gives notebooks project-scoped environments, graphical package management via Astral’s uv, and one-click notebook pipelines with freshness tracking. Think “Makefiles for notebooks” meets “Poetry for Jupyter.”

Calkit manages notebook pipelines with environment tracking and one-click reruns. The orange ‘run’ button signals stale outputs that need to be regenerated.

jupyter-projspec (from the fsspec contributors) takes a complementary approach — it brings projspec into JupyterLab, letting you scan and analyze project structures directly from the notebook environment.

jupyter-projspec integrates to system filebrowser to show the project metadata

Science

fitsview — Stream FITS astronomical data slices directly in JupyterLab without downloading full files.
jupyterlab-urdf-test — 3D robot model viewer/editor (URDF + Three.js), from jupyter-robotics.
climb-jupyter-igv— Integrative Genomics Viewer with S3 access for bioinformatics.
ggblab— GeoGebra interactive geometry with bidirectional Python communication. Second most downloaded new extension of 2026.

Accessibility

Accessibility has been a growing focus for JupyterLab core and extensions are starting to address it too:

jupyterlab-a11y-checker— From UC Berkeley’s DSEP infrastructure team, this extension scans notebooks for WCAG 2.1 AA issues: missing alt text, heading structure, table headers, color contrast, and link text. Guided fix interfaces, optional AI suggestions, and a CLI for CI pipelines. Over 11,000 total downloads and a documentation site.
jupyterlab-change-ui-font-size-fix — Fixes file browser icon misalignment when users change the UI font size — a small but real pain point for anyone who needs larger text.

27 Extensions, One Platform

Stellars is a JupyterLab-based data science platform — GPU support, MLflow, TensorBoard, Optuna — assembled from 27 custom extensions covering everything from branding and file icons to kernel management and diagram rendering, VS Code file icons, trash management, Mermaid-to-PNG conversion, and more.

JupyterLab is now flexible enough that one developer can assemble a domain-specific product entirely from extension building blocks.

Want to Build Your Own?

The JupyterCon tutorial is fully available: step-by-step materials and the complete YouTube recording. It covers scaffolding, plugin architecture, publishing to PyPI, and rapid prototyping techniques. The tools have never been more accessible.

How We Track This

The data behind this post comes from the JupyterLab Marketplace, a community project that tracks all published JupyterLab extensions using PyPI data. The marketplace refreshes automatically and provides download trends, category breakdowns, and discovery tools.

JupyterLab Marketplace

For more on the data and methodology, see our PyData Boston 2025 talk.

What’s Next

New interaction patterns are still being figured out — chat-based assistance, cell annotations, agent protocols. Probably all of them for different use cases.
Reproducibility tooling suggests the community is ready for opinionated workflow management built into the notebook experience.

Ensuring extensions keep working as JupyterLab evolves is critical — the team has been discussing extension compatibility testing at recent contributors calls.

For the Marketplace itself, we’re working on:

Deeper integration with JupyterLab Extension Manager — deep links and “Install in JupyterLab” buttons to go from discovery to installation in one click.
Expanding Trove classifiers to indicate Jupyter Notebook and JupyterLite support. All three use the same extension system, with important caveats: Notebook extensions need to target different UI elements, and JupyterLite extensions cannot have a server component.
Better contribution signals — surfacing commits, PRs, and issues to help users gauge how actively maintained an extension is.

At 700 extensions, the community now shapes JupyterLab as much as the core team does. If you’re building extensions, thank you! Every one of them makes Jupyter better for someone.

About the author

Konstantin Taletskiy is a Senior Software Engineer at Anaconda working on open-source Jupyter. He is a contributor to JupyterLab, maintainer of mamba-gator and jupyterlab-latex, and the creator of the JupyterLab Marketplace.

Instantly view Parquet files in JupyterLab with Arbalister

2026-01-29T16:18:00+00:00

A colleague sends you a SQLite file, a Parquet dataset, or an Avro snapshot.
You need to explore the content, but you’re not sure what’s inside or even how to open it. You shouldn’t have to write code, craft SQL queries, or recall the syntax for read_parquet in a Python library. Instead, you should be able to click the file once and instantly see the tables and metadata.

Despite being the de-facto standard for data science and exploration, JupyterLab lacks this basic capability out of the box. Of course, there are extremely well designed libraries to read, process, and transform tabular data — but opening and viewing a file with a table should be simpler. A workflow with libraries like Pandas or Polars require you to install relevant packages, create a notebook, and iterate on queries to get a full picture of the data.

import pandas as pd

pd.set_option("display.max_rows", 50)
pd.set_option("display.max_columns", 20)
pd.set_option("display.width", None)

df = pd.read_parquet("data/2025/PERFORMANCE_SNAPSHOT_2025_01_05.parquet")
df

Whether for a newcomer to data science, or someone that needs to investigate many different files with hard to remember schemas, this is a bit cumbersome. Comma separated value files (CSV) have better support: double click on the file to see it open (see figure below). That is a great way to understand the business logic and answer questions such as What type of information is in this file? What do values typically look like? Does it seem to contain all the data I need? What is the difference between customer_id and user_id?

The JupyterLab CSV viewer

Arbalister, our new JupyterLab extension, changes this. With Arbalister, you can double-click to instantly view a wide range of tabular data files: Parquet, CSV, Avro, ORC, SQLite, and more…

Arbalister viewer can open Parquet files and more

The extension will load data as it is displayed, making it possible to view it almost instantaneously even if the file does not fit in memory. A toolbar lets the user select some reading options, such as the delimiter for a CSV file, or selecting the table to display in the SQLite file!

Arrow over the wire

Fueling it all is the Apache Arrow ecosystem, an in-memory format for tabular data used in most data science libraries. Plenty of files types can already be read into an Arrow data structure so we exploit it heavily in Arbalister (with Apache Datafusion in our case).

The table displayed in the user’s browser reuses JupyterLab Lumino DataGrid with a custom data model. In the latter, we divide the whole table in chunks across both the rows and columns axes (e.g. 512 rows and 24 columns). When we need to display data from a chunk that is not already available, we make an HTTP request to a server-side extension that open the file, read the relevant portion into an Arrow table and return it to the client. The table is returned as Arrow IPC (Inter Process Communication, a binary format close to the original Arrow memory) in the response body. In the client, a lightweight Arrow implementation can read it back efficiently.

Arbalister client-server architecture

When nothing else is happening, the implementation will also pre-fetch some of the next chunks in the background so that scrolling past a chunk boundary is imperceptible.

Because it all works in Arrow, which is standard for tabular data, adding support for new types of files that can be read into the Arrow format is extremely easy. Vortex, Lance, or whichever other format becomes relevant. But that is not all…

The existing SQLite support is made possible through ADBC, an SQL connector that reads databases into Arrow format. SQLite databases fit into single files, which fit the abstraction used in Arbalister, but ADBC could also be used to add general database exploration (Postgres etc.) into Arbalister. With Datafusion, we could also add visualization for data lakehouses: partitioned datasets (e.g. DeltaLake or Apache Iceberg) over remote object storage (S3). With Intelligent predicate and projections pushdowns, we could visualize remote tera-byte table instantly by only downloading the relevant parts.

Looking further

There are multiple things that would make great improvements to Arbalister. We have already mentioned:

Optional support for newer file types: Vortex, Lance;
Object storage and data lakehouses support;
SQL databases support.

Another direction is support for JupyterLite, a WebAssembly distribution of JupyterLab running entirely in the browser. This setting is great for education because it does not require more than a static file server to host.
Being able to work directly with Parquet files in the browser is great for making dashboard and interactive demonstrations. Arrow and Datafusion already have some experimental support for WebAssembly so this is not so far-fetched.

Finally Arbalister could use a few features to investigate deeper questions about the data. For instance, beyond the What does the data look like? question, some column filters could help answer questions such as What does the data look like in November 2024 for John Doe? Perhaps even an SQL console for more advanced queries.

An extension point

The server extension being only a way to serve Arrow data, it can be reused by other libraries to use it in other ways. For instance, if we knew that our table contains time-series data, a second viewer could display it as a graph.
Similarly for geospatial data.

If you are interested to use or contribute to Arbalister, head to our GitHub!

About the Author

Antoine Prouvost is a senior scientific software engineer at QuantStack. His work on Arbalister was funded by Bloomberg.

Building AI Agents for JupyterLab using Notebook Intelligence

2025-02-24T18:03:00+00:00

Please note that this is not an official Jupyter subproject but an independent open-source tool for JupyterLab users who want to use GitHub Copilot as an AI coding assistant.

Notebook Intelligence (NBI) is an AI coding assistant and extensible AI framework for JupyterLab. (For an introduction to NBI see Introducing Notebook Intelligence and for basics of extending NBI see Building AI Extensions for JupyterLab blog posts.)

GitHub Copilot and other AI coding assistants are great at generating code and answering coding related questions. But they can do a lot more than generating text and code thanks to LLM features such as tool calling and AI agents. NBI provides an extensible AI framework to integrate tool calling and AI agents into JupyterLab Copilot Chat.

AI Agent extension example

What is tool calling and an AI Agent?

Tool calling is a feature of LLMs. It lets you introduce your own functions to LLM so that they can be called in response to chat prompts. LLM can convert natural language prompts to function calls with arguments. Tool calls are executed on the client side (i.e. Jupyter server) by your extension and only the function schema is provided to the LLM. Tool calling lets LLM interact with real time data, proprietary or external apps and services.

AI Agents are collections of tools that can run tasks on behalf of the user. Given a natural language prompt, LLMs can reason, create an execution plan and call multiple tools in a chain. NBI provides a framework to build these type of AI Agent integrations and handles the orchestration between LLMs and your tools.

AI Agent Extension Example

Let’s build an AI Agent for JupyterLab using Notebook Intelligence extension APIs. (The full source code for this extension is available here.) This will be an AI agent for map creation and notebook sharing. It will have the following capabilities:

Looking up geo-coordinates for an address
Showing maps centered at an address in the Copilot Chat UI
Creating notebooks that show maps centered at specified addresses
Sharing notebooks publicly using notebooksharing.space

The tasks above will be run by the AI Agent in response to natural language prompts by the user.

For this extension we will build four tools that will be integrated into JupyterLab Copilot Chat, for each of the tasks above. Tools are defined as classes derived from NBI Tool abstract class. A tool needs to implement the methods and properties defined in this base class.

Tool class provides the metadata information for the tool and implements the pre_invoke and handle_tool_call methods. pre_invoke method is called right before handle_tool_call with the tool arguments and it gives an opportunity for the tool to prompt for confirmation of the tool execution.

schema property of the Tool is the function schema based on OpenAI’s function calling schema. It lets you describe your function and its parameters as an object. A Tool is expected to return an object as response from the handle_tool_call method call.

Geo Coordinates Lookup Tool

This tool looks up geo-coordinates for an address using Nominatim library. pre_invoke method for this tool only shows a message in Chat UI before looking up for the geo-coordinates in handle_tool_call method.

class GeoCoordinateLookupTool(Tool):
    @property
    def name(self) -> str:
        return "geo_coordinate_lookup"

    @property
    def title(self) -> str:
        return "Get geo-coordinates from an address"
    
    @property
    def description(self) -> str:
        return "This is a tool that converts an address to a geo-coordinates"
    
    @property
    def schema(self) -> dict:
        return {
            "type": "function",
            "function": {
                "name": self.name,
                "description": self.description,
                "parameters": {
                    "type": "object",
                    "properties": {
                        "address": {
                            "type": "string",
                            "description": "Address to convert to geo-coordinates",
                        }
                    },
                    "required": ["address"],
                    "additionalProperties": False,
                },
            },
        }

    def pre_invoke(self, request: ChatRequest, tool_args: dict) -> Union[ToolPreInvokeResponse, None]:
        address = tool_args.get('address')
        return ToolPreInvokeResponse(
            message=f"Getting coordinates for '{address}'"
        )

    async def handle_tool_call(self, request: ChatRequest, response: ChatResponse, tool_context: dict, tool_args: dict) -> dict:
        address = tool_args.get('address')
        location = geolocator.geocode(address)
        return {"latitude": location.latitude, "longitude": location.longitude}

Get geo-coordinates tool

Map Response Generator Tool

This tool shows a map in Copilot Chat UI centered at geo-coordinates. In pre_invoke method this method only shows a notification message in Chat UI. In handle_tool_call method, this tool returns a HTMLFrame response that uses HTML to show a map centered at the requested location using Google Maps.

class MapResponseGeneratorTool(Tool):
    ...

    def pre_invoke(self, request: ChatRequest, tool_args: dict) -> Union[ToolPreInvokeResponse, None]:
        geo_coordinates = tool_args.get('geo_coordinates')
        latitude = geo_coordinates.get('latitude')
        longitude = geo_coordinates.get('longitude')
        return ToolPreInvokeResponse(
            message=f"Showing a map centered at latitude: {latitude} and longitude: {longitude}"
        )

    async def handle_tool_call(self, request: ChatRequest, response: ChatResponse, tool_context: dict, tool_args: dict) -> dict:
        geo_coordinates = tool_args.get('geo_coordinates')
        latitude = geo_coordinates.get('latitude')
        longitude = geo_coordinates.get('longitude')

        response.stream(HTMLFrameData(f"""<iframe width="100%" height="100%" frameborder="0" scrolling="no" marginheight="0" marginwidth="0" id="gmap_canvas" src="https://maps.google.com/maps?width=400&amp;height=400&amp;hl=en&amp;q={latitude},{longitude}&amp;t=&amp;z=11&amp;ie=UTF8&amp;iwloc=B&amp;output=embed"></iframe>""", height=400))
        response.finish()

        return {"result": "I showed the map"}

Below is an example prompt showing map centered at “Golden Gate Bridge, San Francisco”. Note that an address was provided to Copilot as the input but Map Response Generator Tool accepts only geo-coordinates as input. This is where LLM automatically decided that it needs to first call the Geo Coordinates Lookup Tool to get geo-coordinates for this address and then it called the Map Response Generator Tool with the geo-coordinates. LLM automatically chained multiple tools and NBI handled this chaining to get the correct response for the user’s prompt.

Map Response Generator Tool

Map Notebook Creator Tool

This tool creates a notebook centered at the specified geo-coordinates. In pre_invoke method this method only shows a notification message in Chat UI. In handle_tool_call method, the tool creates a notebook using nbformat library, saves it to disk and then opens the notebook in JupyterLab UI using the response.run_ui_command NBI method.

class MapNotebookCreatorTool(Tool):
    ...

    def pre_invoke(self, request: ChatRequest, tool_args: dict) -> Union[ToolPreInvokeResponse, None]:
        geo_coordinates = tool_args.get('geo_coordinates')
        latitude = geo_coordinates.get('latitude')
        longitude = geo_coordinates.get('longitude')
        return ToolPreInvokeResponse(
            message=f"Creating a map notebook for latitude: {latitude} and longitude: {longitude}"
        )

    async def handle_tool_call(self, request: ChatRequest, response: ChatResponse, tool_context: dict, tool_args: dict) -> dict:
        geo_coordinates = tool_args.get('geo_coordinates')
        latitude = geo_coordinates.get('latitude')
        longitude = geo_coordinates.get('longitude')

        now = dt.datetime.now()
        map_file_name = f"map_{now.strftime('%Y%m%d_%H%M%S')}.ipynb"

        nb = nbf.v4.new_notebook()
        header = """\
        ### This map notebook was created by an AI Agent using [Notebook Intelligence](https://github.com/notebook-intelligence)
        """

        install_code_cell = "%%capture\n%pip install folium"

        map_code_cell = f"""\
        import folium

        map = folium.Map(location=[{latitude}, {longitude}], zoom_start=13)
        map"""

        nb['cells'] = [
            nbf.v4.new_markdown_cell(header),
            nbf.v4.new_code_cell(install_code_cell),
            nbf.v4.new_code_cell(map_code_cell)
        ]
        nb.metadata["kernelspec"] = { "name": "python3"}
        nbf.write(nb, map_file_name)

        await response.run_ui_command("docmanager:open", {"path": map_file_name})

        return {"result": "I created and opened the map notebook"}

Map Notebook Creator Tool

This tool shares a notebook publicly by uploading it to notebooksharing.space and displays the link to the shared notebook.

In the pre_invoke method implementation, this tool asks for confirmation first as this operation is an undoable share of the notebook publicly. Only after the user confirms, handle_tool_call is executed. In handle_tool_call method the tool uploads the notebook at the notebook_file_path using nbss_upload library and then shows the link to the shared notebook on notebooksharing.space.

class NotebookShareTool(Tool):
    ...
    
    def pre_invoke(self, request: ChatRequest, tool_args: dict) -> Union[ToolPreInvokeResponse, None]:
        file_path = tool_args.get('notebook_file_path')
        file_name = path.basename(file_path)
        return ToolPreInvokeResponse(
            message=f"Sharing notebook '{file_name}'",
            confirmationTitle="Confirm sharing",
            confirmationMessage=f"Are you sure you want to share the notebook at '{file_path}'? This will upload the notebook to public internet and cannot be undone."
        )

    async def handle_tool_call(self, request: ChatRequest, response: ChatResponse, tool_context: dict, tool_args: dict) -> dict:
        file_path = tool_args.get('notebook_file_path')
        file_name = path.basename(file_path)
        share_url = nbss_upload.upload_notebook(file_path, False, False, 'https://notebooksharing.space')
        response.stream(AnchorData(share_url, f"Click here to view the shared notebook '{file_name}'"))

        return {"result": f"Notebook '{file_name}' has been shared at: {share_url}"}

Notebook Share Tool

Tool call schema definitions

It is important to define schemas of the tools clearly and disambiguate the tools as much as possible so that the LLM can invoke the proper tool based on the user prompt. LLM parses the user prompt, decides which tools to call and generates the input parameters for the call.

If current file or selection is made visible by the user, NBI can provide the file paths and content as context to the LLM. That way LLM can use those as additional context for a tool call. That is how Notebook Share Tool was able to access the current notebook file.

Tool chaining

After parsing the user prompt, LLM creates an execution plan and can call multiple tools in a chain. NBI handles this tool chaining for you. It is important to define your schemas with the chaining in mind. Consider defining matching tool outputs and inputs so that the output of a tool can be passed onto another one directly if needed.

Notice that in this extension example MapResponseGeneratorTool and MapNotebookCreatorTool both take in geo_coordinates (latitude, longitude) as input and GeoCoordinateLookupTool outputs geo_coordinates. This lets LLM to directly pass the output of GeoCoordinateLookupTool to MapResponseGeneratorTool and MapNotebookCreatorTool. It also lets a user to use an address to trigger MapResponseGeneratorTool and MapNotebookCreatorTool, because LLM knows that there is another tool it can call to generate input (geo_coordinates) from address for these tools.

Chat Participant

In NBI AI framework, AI Agents are defined as chat participants and tools are tied to specific chat participants. For our extension we create AIAgentChatParticipant as our participant (for more details on NBI extensions and chat participants see this blog). Our chat participant returns list of tools it defines from the tools property.

In handle_chat_request method our chat participant passes the request to the base ChatParticipant class to handle tool calling for us.

class AIAgentChatParticipant(ChatParticipant):
    @property
    def id(self) -> str:
        return "ai-agent"
    ...
    
    @property
    def tools(self) -> list[Tool]:
        return [GeoCoordinateLookupTool(), MapResponseGeneratorTool(), MapNotebookCreatorTool(), NotebookShareTool()]

    async def handle_chat_request(self, request: ChatRequest, response: ChatResponse, options: dict = {}) -> None:
        ...
        await self.handle_chat_request_with_tools(request, response, options)

NBI Extension

Finally we create our NBI extension class AIAgentExtension. This class basically registers our chat participant to NBI on extension activation.

class AIAgentExtension(NotebookIntelligenceExtension):
    ...

    def activate(self, host: Host) -> None:
        self.participant = AIAgentChatParticipant(host)
        host.register_chat_participant(self.participant)
        log.info("AI Agent example extension activated")

That is all there is to create an AI Agent for JupyterLab using Notebook Intelligence. The full source code for this example is available along with installation instructions for you to use as a reference and/or build on top.

I am looking forward to seeing the AI Agents built by the community. Please try the extension APIs and share your feedback using project’s GitHub issues! User feedback from the community will shape the project’s roadmap.

About the Author

Mehmet Bektas is a Senior Software Engineer at Netflix and a Jupyter Distinguished Contributor. He is the author of Notebook Intelligence, and contributes to JupyterLab, JupyterLab Desktop and several other projects in the Jupyter eco-system.

Introducing Notebook Intelligence!

2025-01-13T16:28:00+00:00

Please note that this is not an official Jupyter subproject but an independent open-source tool for JupyterLab users who want to use GitHub Copilot as an AI coding assistant.

I am thrilled to announce the release of Notebook Intelligence (NBI)! NBI is an AI coding assistant and extensible AI framework for JupyterLab. It uses GitHub Copilot under the hood and is inspired by its design principles. NBI greatly boosts the productivity of JupyterLab users with AI assistance powered by GitHub Copilot.

Generate code

Generate code, iterate on it

NBI integrates tightly with the notebook document. Using cell toolbar item “Generate code” or keyboard shortcut “Cmd + G” / “Ctrl + G”, you can launch the inline coding assistant popover to generate code cells.

Generate code popover

If the inline coding assistant is launched for a cell with existing code, then the generated code is shown in a diff view for approval.

Generate code with diff viewer

If you are not satisfied with the generated code, you can re-generate with an updated prompt. Diff viewer also lets you edit the generated code manually before accepting it.

Fix code in cell with NBI

Explain and fix code, troubleshoot errors reported

NBI adds a new sub menu to notebook cell context menu. Copilot can explain code in a cell or suggest fixes for any issues in it. Clicking these menu items opens Copilot Chat and generates a suggestion.

Copilot context menu

If the code cell has an output you can ask Copilot to explain it. If there are any errors reported, you can have Copilot to troubleshoot as well. These actions also take you to Copilot Chat interface.

Explain, fix, troubleshoot

Inline Completions

Notebook Intelligence integrates with JupyterLab’s inline completion APIs and provides code suggestions as you type in a code cell or a Python file.

Inline completions

Code suggestions are generated using GitHub Copilot. They are blazing fast and relevant to document you are working on. In addition to the code cell you are working on, the code in the surrounding cells are also used as context when generating suggestions.

Inline completions example

Copilot Chat

NBI provides a user friendly chat interface to chat with GitHub Copilot. You can ask questions related to coding.

Copilot Chat

If Copilot generates code snippets, they are rendered in a special format in a section with an action toolbar. Toolbar will have buttons to copy, insert, create new Python file and notebook from the snippet. Using these you can easily integrate the generated code into your project.

Copilot chat toolbar actions

Chat Commands

Chat interface also provides commands to generate new notebooks and Python code files based on your task described in the prompt. Commands start with “/” and a command auto-complete list is shown as you type. You can navigate between the commands using keyboard and choose a suggestion using “Enter” or “Tab” keys.

Chat command auto-complete

/newNotebook command

You can generate new notebooks from a prompt with the /newNotebook command. Notebook generation is shown interactively, a new empty notebook is created and opened, then code and markdown cells are added onto the notebook as they are generated by NBI and GitHub Copilot.

Generate notebook example

/newPythonFile command

You can also create new Python files from a prompt using the /newPythonFile command.

Generate Python file example

Getting Started with Notebook Intelligence

Notebook Intelligence is a JupyterLab extension published as a Python package. Simply install the package and restart JupyterLab. NBI will add a new sidebar item for Copilot Chat, a notebook context sub-menu, a cell toolbar item for “Generate code” and a status bar item for GitHub Copilot login status to JupyterLab UI. It will also be integrated with inline completion (AI suggestions for code completion).

pip install notebook-intelligence

Authentication with GitHub Copilot

Notebook Intelligence requires a GitHub Copilot subscription. NBI provides a user friendly interface to sign into your GitHub Copilot account from JupyterLab UI to activate access to your subscription.

GitHub Copilot authentication

Extensible Framework

Notebook Intelligence provides APIs to let developers extend its capabilities. You can add custom agents / chat participants, define tools (function calling) and add RAG capabilities to provide your own context to LLM for code / chat response generation. Stay tuned for my next blog post where I will walk you though extensibility features.

Notebook Intelligence is currently in beta and designed for Python (support for more languages coming soon). Please try it out and share your feedback and any feature requests using project’s GitHub issues! User feedback from the community will shape the project’s roadmap.

About the Author

The Jupytext Menu is back!

2024-01-31T20:29:00+00:00

A few weeks back, jupytext==1.16.0 went out. That release included multiple amazing contributions by Mahendra Paipuri, this post will go over the new features, and thank Mahendra for his great work.

As we will see below, Mahendra restored the Jupytext Menu, made the extension fully compatible with JupyterLab 4 and Jupyter Notebook 7, and added the option to create Text Notebooks directly from the launcher.

What is Jupytext

Jupytext is a Python package that lets you save Jupyter Notebooks as text notebooks. Multiple formats are supported, and the notebooks can be saved either as Markdown documents with a .md extension, or as scripts with e.g. a .py extension (assuming you use Python - otherwise multiple languages are supported).

A .py notebook in the percent format looks like this:

# %% [markdown]
# This is a markdown cell

# %%
def f(x):
  return 3*x+1

# %%
[f(i) for i in range(6)]

To open a text notebook as a notebook in Jupyter, right-click on the document and select “Notebook” (you can also change the default viewer to “Jupytext Notebook” if you wish: see below the section about the settings)

Right click to open a text notebook with the Notebook editor

Text notebooks are conveniently edited and executed in Jupyter as notebooks:

A Text Notebook in JupyterLab

You can also edit them with your favorite text editor. You will get the changes back in Jupyter by re-opening or by reloading the document: click e.g. on reload Python file from disk in the File menu.

Text notebooks only contain the notebook inputs. For that reason they use much less disk space than .ipynb notebooks. They are also better suited for version control as they only contain the content that was actually typed by the user. But even more useful are paired notebooks: text notebooks paired with an .ipynb notebook where the notebook outputs are preserved. When a paired notebook is saved, Jupytext writes the notebook to both files. When the notebook is read or reloaded in Jupyter, the notebook inputs are loaded from the most recent file, meaning that any edits on the text notebook will be reflected in Jupyter (and propagated to the .ipynb file the next time it is saved).

One of the most frequent operation when using Jupytext is to pair an .ipynb notebook with a text notebook in the format of your choice.

In earlier versions of Jupytext, pairing a notebook had to be done through the command palette.

In Jupytext v1.16, thanks to Mahendra’s work on the front-end extension, you can directly use the Jupytext Menu for this. And the menu is available for both JupyterLab 4 and Jupyter Notebook 7!

If you are new to Jupytext, we recommend that you pair your .ipynb notebooks with either a percent script or with a MyST Markdown file. The former format works well if you want to save and edit your notebook as a script, while the latter is well suited for writing documentation.

Once you have paired the notebook with a Jupytext format, save your notebook, and the paired files will be created or updated.

The Jupytext Menu

Creating new text notebooks

For some notebooks you might decide that you don’t need to save the outputs at all. In that case you can work with an (unpaired) text notebook.

To create a text notebook you can use the New Text Notebook sub-menu under File:

Create Text Notebooks directly from the menu

If, later on, you decide that you want to preserve the notebook outputs in a paired .ipynb notebooks, you will just have to pair the document to an .ipynb notebook using the Jupytext Menu documented at the previous paragraph.

Text Notebooks in the launcher

In Jupytext v1.16, thanks to Mahendra’s work on the front end, text notebooks are also available in a Jupytext section in the Jupyter launcher:

Text Notebooks are also available in the launcher

Jupytext Settings in JupyterLab

Since there are many possible formats for text notebooks, we have decided to expose only the most common ones, by default, in the launcher and in the New Text Notebook menu.

You can include more formats in the menu and in the launcher by changing the Jupytext settings in the Settings Editor (in the Settings menu):

The Jupytext Menu/Launcher settings

You might also want to open certain text documents as notebooks with a single click. You can achieve this by setting the default viewer for those documents to “Jupytext Notebook”. For instance, if you want to open .py and .md files as notebooks with a single click, you can configure the default viewers like this:

Changing the default viewers to open Text Notebooks with a single click

Alternatively, you can also list and set the default viewers with the jupytext-config utility (which was developed recently by Thierry Parmentelat), using e.g.

jupytext-config set-default-viewer python markdown

to set “Jupytext Notebook” as the default viewer for both Python and Markdown files.

Pairing notebooks globally

The pairing commands provided through the Jupytext Menu and commands act on individual notebooks, by settings a jupytext.formats metadata in the notebook.

It is also possible to pair all the notebooks within a certain folder using a jupytext.toml configuration file - see Jupytext’s documentation. Please note that:

The local metadata in the notebook takes precedence over the global configuration, and
Jupyter is aware of the global jupytext.toml file and will pair the notebooks accordingly, however the Jupytext Menu is not aware of the global configuration, so the paired formats selected through the global configuration will not be checked in the menu.

Installing Jupytext v1.16

The front end extension for JupyterLab shipped with Jupytext v1.16 requires JupyterLab 4, and/or Jupyter Notebook 7. Please upgrade Jupyter accordingly. Then, install Jupytext with

pip install 'jupytext>=1.16.0'

conda install 'jupytext>=1.16.0' -c conda-forge

and restart your Jupyter server with e.g. jupyter lab.

Acknowledgments

I would like to thank Mahendra Paipuri for his impressive work. I am pretty sure that the Jupytext Menu, and the option to create new text notebooks, will be much appreciated by Jupytext users!

While users will mostly notice Mahendra’s work on the front end extension, it was actually not his only contribution to this release! Mahendra also thoroughly revisited the packaging of Jupytext, and helped us transition the project to the src layout, and our old setup.py to an up-to-date pyproject.toml configuration that uses hatch to build Jupytext.

I also want to thank Cristian Le for his precious advice regarding the layout refactoring, and for helping us to tackle the CI reorganization. Thierry Parmentelat, who had previously ported the front-end extension to JupyterLab 4, contributed much testing and feedback on this new version of the front-end extension.

It is always a pleasure for me to maintain Jupytext, an amazing adventure that started five years ago already. However, I can get busy at times (Jupytext comes in addition to my day job, one cat, two bikes, three kids), so I am really thankful for receiving such contributions, especially when they are of such a great quality!

Generative AI in Jupyter

2023-08-02T16:15:00+00:00

Jupyter AI brings generative artificial intelligence to Jupyter notebooks, giving users the power to explain and generate code, fix errors, summarize content, ask questions about their local files, and generate entire notebooks from a natural language prompt. Using its powerful magic commands and chat interface, Jupyter AI connects Jupyter with large language models (LLM) from providers such as AI21, Anthropic, AWS, Cohere, and OpenAI. We use LangChain to support all popular LLMs and providers, giving you access to new models as they are released. LangChain will let Jupyter AI use local models as well. Jupyter AI version 1.0, for JupyterLab 3, and Jupyter AI 2.0, for JupyterLab 4, are now available as free and open source software.

Jupyter AI is designed with responsible AI and data privacy in mind. You can pick which LLM and embedding model best suit your needs. The underlying prompts, chains, and other software are open source, so you can see exactly how your data is being used. Jupyter AI saves metadata about model-generated content in each AI-generated code cell, so you and your collaborators can track where AI-generated code enters your workflow. Finally, Jupyter AI only contacts an LLM when you ask it to, directly; it does not read your data or transmit it to models without your explicit consent.

Jupyter AI is an official subproject of Project Jupyter and is available now as free, open source software. We designed it based on the same principles that underlie all of Project Jupyter: we made it simple, easy to use, modular, and extensible, and we prioritized ethical considerations and social responsibility. We’d love to hear from you about how we can improve it!

Getting started with Jupyter AI

Start using Jupyter AI by installing the appropriate version with pip:

pip install 'jupyter-ai>=1.0,<2.0' # If you use JupyterLab 3
pip install jupyter-ai             # If you use JupyterLab 4

Then, launch JupyterLab. Jupyter AI provides two different interfaces to interact with LLMs. In JupyterLab, you can converse with a chat UI to assist you with your code. Also, in any supported notebook or IPython environment, including JupyterLab, Notebook, IPython, Colab, and Visual Studio Code, you can invoke LLMs using the %%ai magic command. Jupyter AI can turn any Jupyter Notebook session into a generative AI playground with support for text and image models.

Project Jupyter is vendor-neutral, so Jupyter AI supports LLMs from AI21, Anthropic, AWS, Cohere, HuggingFace Hub, and OpenAI. More model providers will be added in the future. Please review a provider’s privacy policy and pricing model before you use it. We’re also working on support for locally-deployed models, for maximum privacy. Once you have installed Jupyter AI, before you can use magic commands, you will need to authenticate to each model provider that you wish to use. For most providers, this involves setting an environment variable. The user documentation has detailed instructions for configuring model providers.

The chat interface has its own configuration panel for choosing a language model and an embedding model, and for authenticating to each model’s provider. A language model responds to users’ messages in the chat panel. When you ask the chat interface to learn about local files, it uses an embedding model to parse these files and to assist when you ask questions about them.

You can find full details about how to configure and use Jupyter AI in the user documentation.

The chat interface, your AI assistant

The chat interface puts you in conversation with Jupyternaut, a conversational agent using a language model of your choice.

Jupyternaut communicates primarily through text, and it can also interact with files in JupyterLab. It can answer questions as a general-purpose AI assistant, include selections from your notebooks with your questions, insert AI-generated output into your notebooks, learn from and ask questions about your local files, and generate notebooks from a prompt. Jupyternaut can only see the information you send it by sending chat commands; it only reads your data when you specifically ask it to.

To get started, you can ask Jupyternaut a question:

You can also highlight part of your notebook and include it with your prompt.

Using prompts that include the selected code, you can ask Jupyternaut to explain your code in plain English (or in any other language it can speak), make modifications to it, and identify errors in it. If you want, Jupyternaut can even replace your selection with its response. Please review AI-generated code before you run it, as you would review code written by another person.

For example, you can ask Jupyternaut to rewrite code by adding comments to it:

Jupyternaut sends the code to your chosen language model, then replaces the selection with the language model’s response.

Generating a notebook from a text prompt

Jupyter AI’s chat interface can generate an entire notebook from a text prompt. To do this, run the /generate command and provide a text description. Jupyternaut will use its AI language model to name the workbook and fill it with markdown and code cells. This may take a few minutes. While Jupyternaut is working, you can continue to use the chat UI, and Jupyternaut will continue generating your notebook.

Once Jupyternaut has finished generating your notebook, it will send you a message with its filename, so that you can open it. Please review any AI-driven code before you run it.

Learning from and asking about local files

You can use the /learn command to teach Jupyternaut about local files, so that you can use the /ask command to ask questions about them. For example, using the /learn command, you can teach Jupyternaut about Jupyter AI’s documentation:

When you learn local files, Jupyternaut uses an embedding model to convert data, then stores the output in a local vector database. Please review the privacy policy for each model, and be aware of any restrictions about sharing your local data with third-party model providers. Once the learning process is complete, you can ask a question with the /ask command. Using retrieval-augmented generation (RAG), Jupyternaut will append relevant info to your question from its vector database, then it will use the AI language model you selected to answer your question.

Notebooks as generative AI playgrounds with magic commands

Jupyter AI also provides magic commands that you can run in notebook cells and in the IPython command-line interface. To get started, run %load_ext jupyter_ai_magics, which will load the magics extension. You can then use Jupyter AI with the %%ai magic command. You can run %ai help to learn about all the options and commands you can run using the %ai line magic and %%ai cell magic commands.

Each %%ai command requires a model, typically specified as provider-id:model-id. To use a particular provider, you’ll need to set its API key using an appropriate environment variable or Python module. See the model providers section of the user documentation for specific information. Once you’ve provided the key to your model provider, you can run a magic command by specifying the model on the first line and specifying your prompt on subsequent lines.

You can use the -f or --format parameter to customize the format of the output, including HTML, math, source code, and images.

You can interpolate a variable name or expression in a prompt by enclosing it in braces (curly brackets).

Interpolation also works with the special In and Out variables, which contain the inputs and outputs of code cells. Note that a cell output can contain both text and markdown values.

Jupyter AI adds a special Err variable, which stores the errors that occur while executing code. By interpolating this variable into a prompt, you can use an AI language model to explain and correct an error in your code.

About the developers

Jupyter AI is an officially supported Jupyter subproject. The following Jupyter contributors built Jupyter AI.

Brian Granger is a Senior Principal Technologist at AWS. Brian is a cofounder of Project Jupyter, a member of the Jupyter Executive Council, and a board member of the PyTorch Foundation.

Andrii Ieroshenko is a Software Development Engineer at AWS. Andrii is a Project Jupyter contributor and JupyterLab Council member.

Piyush Jain is a Software Development Engineer at AWS working on open source contributions to Project Jupyter. He is also a member of the Jupyter Server team, actively participating in maintenance and development of the Jupyter Server project.

David Qiu is a Software Development Engineer at AWS working on Project Jupyter. Previously, David was researching physical chemistry at the University of Illinois at Urbana-Champaign.

Jason Weill is a Senior Front-End Engineer at AWS; a contributor to JupyterLab, Jupyter Governance, and Jupyter Scheduler; and a member of the JupyterLab Council, the Jupyter Security Working Group, and the Jupyter Diversity, Equity, and Inclusion Standing Committee.

How can you help?

We’re just getting started with generative AI in Jupyter. Please join us! We have a list of issues that could use your help, such as adding support for locally hosted LLMs.

Install and use the Jupyter AI extension. If you find any bugs or have suggestions, please create issues on GitHub.
Join the discussion about generative AI in Jupyter in the “Generative AI in Jupyter” topic on Discourse.
Contribute: Your bug reports, feature requests, and pull requests will help improve this project for everyone.

Note: An earlier version of this story said that users can choose which vector database to use with Jupyter AI. As of Jupyter AI 2.1.0, only FAISS is available; users cannot choose another vector database. Sorry for the error.

A theme editor for JupyterLab

2023-06-19T08:11:00+00:00

JupyterLab is a comprehensive web user interface for scientific and technical computing providing tools such as notebooks, text editors, consoles, and terminals. Like many similar tools, it allows users to choose from light or dark color themes. However, users may want to further adjust the looks of the interface, just for fun or for specific personal preferences or needs.

The ability to fine-tune contrast, color palettes, and fonts can be very useful for accessibility, an essential requirement for software to be usable by the whole community. There is an ongoing project-wide effort to improve accessibility in Jupyter. Choices of color palettes impact color-blind users while font choices can have a significant impact on people affected by dyslexia, or who suffer from migraines.

This diversity of requirements and preferences shows that end users should have the means to adjust parameters and tweak existing themes. Subtle differences can improve comfort significantly.

JupyterLab themes

Theming in JupyterLab is enabled by the fact that most of the layout and colors of the UI are defined by a set of CSS variables. There is no need for a complete stylesheet: one can set values for the base parameters from which the appearance of the user interface is derived. This system ensures a consistent look and feel throughout the application.

Custom themes provide a set of values for the base CSS variables and package the resulting CSS file in JupyterLab extensions. A search on GitHub returns 29 repositories defining such custom themes for JupyterLab.

Creating a theme requires implementing values for dozens of base CSS variables. This is often achieved by theme authors tweaking the values given in the default dark or light themes.

Generating a consistent palette, testing, and packaging the outcome in the form of a JupyterLab extension requires development skills and is neither direct nor trivial.

A new tool for authoring JupyterLab themes

To lower the bar for customizing JupyterLab we created a new tool providing a simple interface for tuning the JupyterLab appearance interactively, allowing theme authors to:

use the selected colors in their application and “pin down” the result in the configuration,
export the outcome in a form amenable to packaging into a new JupyterLab theme.

The theme editor extension is a theme prototyping tool, a sandbox to test changes in colors, font family and font size, and a configuration tool for end users. It displays a reduced set of parameters one can play with, allowing users to select:

colors through color pickers,
numerical values from sliders (for font size, border radius and width),
font families from a predefined dropdown list.

A screenshot of JupyterLab with the theme editor panel is displayed in Figure 1, with a notebook opened in light theme. Figure 2 shows different screenshots with different custom themes.

Figure 1: The theme editor extension in JupyterLab.

Figure 2: Examples of dynamically editing the color and fonts scheme of JupyterLab with the theme editor.

Color palettes

For all the colors (accent, brand, border, error, info, layout, success, warn), the same logic is applied. The user picks a base color and a palette is automatically calculated using Microsoft’s fast-colors library.

Let’s take the specific example of the layout colors impacting the background of most elements of the interface. They are defined using --jp-layout-color[i] CSS variables from white to light grays (for the light theme).

--jp-layout-color0: white;
--jp-layout-color1: white;
--jp-layout-color2: var(--md-grey-200);
--jp-layout-color3: var(--md-grey-400);
--jp-layout-color4: var(--md-grey-600);

The number of colors in the palette fits the number of colors in the CSS set of variables (here the palette contains 5 colors). There are different ways to define colors (hexadecimal, RGBA, HSLA). The latter format has the advantage of giving direct access to luminance L, a measure of the lightness of a given color.

An inverse layout palette is computed for use with most text elements. The luminance of the inverse layout color L’ is calculated as 1-L plus a correction depending on how 1-L is close to 0.5. There isn’t a general CSS rule in JupyterLab concerning elements on top of a background but some PRs were proposed to reinforce the coupling between inverse layout elements and layout backgrounds with the same color index. The contrast still needs to be improved though, by using other palettes or better corrections when defining the inverse layout colors.

The full process just described from picking a layout color to resulting palettes and the corresponding interface is illustrated in Figure 3.

Figure 3: From a color to the layout, and inverse layout palettes with the resulting interface.

Font size and font family

As mentioned previously, font size and font family can be tuned too. Let’s take the example of the --jp-ui-font-size[i] CSS variables defining the text size of most of the text elements in the interface:

--jp-ui-font-scale-factor: 1.2;
--jp-ui-font-size0: 0.8333em;
--jp-ui-font-size1: 13px;  /* Base font size */
--jp-ui-font-size2: 1.2em;
--jp-ui-font-size3: 1.44em;

A slider lets the user control the base font size associated with --jp-ui-font-size1. The other font sizes are then calculated by applying a scale factor to the base font.

Concerning font families, a non-exhaustive list is proposed (i.e., default browser fonts or Google fonts). It includes both standard accessible fonts (Arial, Courier, Helvetica, Sans Serif, system-ui, Times New Roman, Verdana) and less usual ones to create artistic or special interfaces with cursive fonts ( for instance Dancing Script or Single Day). Only 2 fonts are proposed for code: Space Mono and monospace.

Storing the new theme

Once satisfied with a new theme, users can export it using a button at the top of the theme editor panel. This creates a variable.css file that can be packaged in a theme extension (see the extension template). The new parameters are also synchronously saved in the settings editor and can be restored for the next JupyterLab opening if the boolean useSettings is set to true. If not, the interface will look like in Figure 1: with light theme default CSS values. Resetting useSettings to true will restore the formerly tuned parameters.

Future development

The look and feel of the theme editor UI still needs to be improved. We will work on unifying the styling of all interfaces making use of react-jsonschema-form. It is already used for the JupyterLab settings editor, the notebook metadata editor of JupyterLab 4.0, and several extensions.

Moreover, the current implementation is bound to fast-colors palettes and we may want to use a different approach, like decoupling the layout and inverse layout palettes to reach better contrasts or give more freedom in the color choices.

Any help (e.g., filling issues for bugs or enhancement requests, opening pull requests) to improve the extension is welcome.

Try it out

You can install the PyPI package by running:

pip install jupyter_theme_editor

You can also try the theme editor on Binder.

Aknowledgement

Many thanks to Isabel Paredes for drawing the palette icon!

About the authors

Florence Haudin is a scientific software developer at QuantStack. She contributes to JupyterLab and to ipyleaflet.

Frédéric Collonval supervised this work as a technical director at QuantStack. He is a member of the JupyterLab core team and authored several JupyterLab extensions.

Introducing Jupyter Scheduler

2022-12-14T15:26:00+00:00

The Open Source Jupyter team at AWS is proud to announce the release of Jupyter Scheduler, a JupyterLab extension that can run your Jupyter notebooks in the background. Jupyter Scheduler was developed from the start as an open-source project under the umbrella of the Jupyter project and governance. You can set up your notebooks to run once or on a schedule. By default, Jupyter Scheduler runs notebooks on your local system where Jupyter Server is installed; you can also extend the scheduler to run your notebooks in the cloud, on an on-premise cluster, on JupyterHub, or even on supercomputers.

We hope you’ll find Jupyter Scheduler useful for running notebooks periodically, such as daily or weekly reports. You can install it in JupyterLab by running pip install jupyter_scheduler, then restarting JupyterLab. You can create jobs (single runs of an individual notebook) and job definitions (scheduled recurring notebook jobs) in two ways. You can right-click on a notebook in the file browser and choose “Create Notebook Job” from the context menu, or you can use the new toolbar button in an open notebook file.

When you right-click on a notebook file in the file browser, a “Create Notebook Job” option appears in the context menu.

In a notebook, the “Create Notebook Job” option appears on the right of the toolbar.

You can give your notebook job or job definition a name, choose an environment to run it in, select different output formats, and provide parameters that are set as local variables when your notebook gets executed. This parameterized execution is similar to Papermill; Jupyter Scheduler provides a UI for managing and accessing the outputs of your notebook jobs inside JupyterLab.

You can use the Create Job form to run a notebook once or to create a job definition that runs on a schedule.

In addition, if you select “Run on a schedule,” you can run your job on a recurring schedule. The scheduler provides a few useful shortcuts to, for example, run your notebook at a specified interval by the minute, hour, day or week. You can also specify a custom schedule in crontab format.

You can schedule a job definition using preset intervals, such as every weekday.

You can also select “custom schedule” and specify a cron expression for the notebook job definition’s schedule.

Once you’ve created a job or job schedule, you can use the “Notebook Jobs” icon on the launcher to view lists of jobs and job definitions. Each item’s name links to a detail view.

With the Jupyter Scheduler extension installed, JupyterLab’s launcher screen shows a “Notebook Jobs” button in the “Other” section. Click on it to view a list of jobs or job definitions.

The list view shows a list of notebook jobs and a list of notebook job definitions. Each one includes links to view details, to download files, and buttons to take quick actions on them.

The Job Detail view shows all available details for one job or one job definition.

When one of your jobs has completed, you can download its files from the list or detail view. When your notebook runs, the scheduler saves a copy of the input file, so that even if you later change the file, it captures the state of the notebook at the time your job ran. Jupyter Scheduler uses unique names for the input and output files so that rerunning the same notebook produces new files every time.

Jupyter Scheduler uses a REST API to create, update, delete and manage jobs and job definitions. You can also customize Jupyter Scheduler by plugging in your own scheduler class, for example, to replace the existing SQLite database or to run jobs in the cloud. AWS SageMaker Studio actually uses Jupyter Scheduler, with custom classes, as part of its automated notebooks feature, announced earlier this year. See the Jupyter Scheduler operator documentation for more information about how to write and use your own scheduler.

We’d love to hear what you think about Jupyter Scheduler! You can check out Jupyter Scheduler’s documentation on ReadTheDocs. Please open issues with ideas for enhancements and about any bugs you find. If you’d like to contribute code, see the Jupyter Scheduler contributors guide.

This article does not represent the official position of amazon.com or any of its subsidiaries.

About the authors

Andrii Ieroshenko is a Software Development Engineer at AWS. Previously, Andrii worked on the Notifications project as a part of the Jupyter Cal Poly student internship program.

David Qiu is a Software Development Engineer at AWS working on Project Jupyter. Previously, David was researching physical chemistry at the University of Illinois at Urbana-Champaign.

Jason Weill is a Senior Front-End Engineer at Amazon Web Services; a contributor to JupyterLab, Jupyter Governance, and Jupyter Scheduler; and a member of the JupyterLab Council, the Jupyter Security Working Group, and the Jupyter Diversity, Equity, and Inclusion Standing Committee.

Upgrading Nbgrader

2022-09-15T07:51:00+00:00

Project Jupyter provides a broad collection of open-source tools for interactive computing that has become ubiquitous in data science and scientific computing, and is very popular in educational contexts.

Nbgrader, started by Jessica Hamrick in 2014, is an extension for creating notebook-based assignments, online courses, and for auto-grading these assignments. It is central to many Jupyter-based services used by teaching institutions.

As a large part of the Jupyter user base has adopted the JupyterLab frontend, most of the popular extensions to the classic notebook UI now have an equivalent in the JupyterLab world. Unfortunately, the nbgrader extension was still outstanding as it had not yet been ported to the JupyterLab extension system. Resolving this has become critical as it was recently decided in a Jupyter Enhancement Proposal (JEP) that the next major version of the Notebook package (v7) will be based on JupyterLab components, and will rely on the JupyterLab extension mechanism. As part of this JEP, it was decided that the release of Notebook 7 should be contingent on porting certain popular classic notebook extensions to JupyterLab, including nbgrader.

Upgrading nbgrader is an important step for the advancement of the Jupyter project!

Nbgrader 0.8

The port of nbgrader to JupyterLab is a longstanding request of the community. A lot of work had already been done on this front by contributors to the ngshare project (Kevin Rong, Abigail Almanza, Lawrence Lee, and Eric Li) who developed a large part of the JupyterLab-based UI, but did not finalize the migration due to lack of time. Fortunately, they made their work available to the community for others to complete the upgrade. This includes the development of the remaining elements of the UI, the adaptation of the server-side extension, and adding tests on the JupyterLab extensions to improve the CI. The nbgrader 0.8 release also includes other significant work items such as the upgrade to nbconvert 6, the switch from the notebook server to jupyter-server, and the adoption of the jupyter releaser.

Today, we are pleased to announce the release of nbgrader 0.8.0, the first release of the extension to support the JupyterLab frontend.

Creating an assignment

Teachers can create assignments directly in the notebook UI by setting the proper information for each cell, such as the cell type (whether it is a locked cell, a manually graded answer, or an automatically graded answer) and the number of points the answer is worth.

The UI for creating an assignment in the notebook (teachers).

Managing assignments on the teacher side

Formgrader is the part of the nbgrader UI for the teachers to manage the assignments.

From this UI, teachers will be able to:

generate assignments from a Notebook file
share assignments with students
fetch assignments after student submission
grade assignments
provide feedback to students

The Formgrader UI to manage assignments (teachers).

Managing assignments on the student side

The assignments list UI is used by students to fetch assignments released by the teacher, submit their answers, and benefit from the feedback after the evaluation by teachers.

The assignments list UI to fetch and submit assignments (students).

What about the future?

Creating and grading assignments in nbgrader uses the metadata of individual notebook cells. The current release of nbgrader provides a widget to interact with the corresponding metadata.

As other extensions need a similar functionality, we are expanding the generic cell metadata editor to address this requirement in a more generic and pluggable way. The new metadata editor allows any extension to create a specific form to interact with the relevant part of the metadata, specified with a JSON schema and rendered with react-jsonschema-form.

Nbgrader could take advantage of this improvement in a future release.

Try it online

Installation

You can install the latest version of nbgrader using pip:

pip install nbgrader

The GitHub repository can be found here.

Acknowledgements

This work at QuantStack was partly funded by Université Paris Cité in the context of the PlasmaBio project.

We are grateful to Jessica Hamrick, Jeremy Tuloup, and Ian Stuart for their help and code reviews during the upgrade to nbgrader 0.8.0.

About the Author

Nicolas Brichet is a scientific software developer at QuantStack active in the Jupyter ecosystem. In the past few months, he has been working on the Nbgrader extension, as well as other developments in core JupyterLab such as an improved notebook metadata editor. Before joining QuantStack, Nicolas was a software engineer at INRAE.

Packaging for Jupyter in 2022

2022-09-06T15:15:00+00:00

Packaging has been difficult and changing over time in the Python ecosystem, and packaging extensions for Jupyter has been especially troublesome. Jupyter extensions typically have a build step to handle JavaScript assets, and provide Jupyter configuration and data using the data_files capability in Setuptools. If you’ve ever developed your own Jupyter extension before, you understand the pain all too well.

We’ve tried to help over the years. We created the Jupyter Packaging library, with helper functions and Setuptools builder classes. We have also created cookiecutter repositories for JupyterLab, Widget, and Server extensions. Additionally, there have been a number of libraries and Python Enhancement Proposals produced by the Python Packaging Authority, aiming to standardize and improve the Python Packaging ecosystem. Even with all these improvements, packaging Jupyter libraries remained a challenge.

Then we discovered the Hatch project.

“Hatch is a modern, extensible Python project manager.”

Hatch is a standards-based Python packaging backend that is flexible and extensible. Hatch came with many of the things we needed. It provides a supported replacement for Setuptool’s (deprecated) data_files called shared-data. We could easily extend Hatch to handle the JavaScript build step required for most of our applications and extensions with a new plugin called Hatch Jupyter Builder that is tailored for Jupyter extensions. The best news? No more complicated setup.py files in your package. Just a single, standards-driven pyproject.toml in your root directory describing all of your package details.

We believe this is a major improvement for Jupyter developers and extension authors.

For Jupyter extension authors that previously used one of our cookiecutter templates or Jupyter Packaging directly, we offer a migration script and guides to convert the Setuptools and Jupyter Packaging configuration to use Hatch and Hatch Jupyter Builder. We also provide a script that will compare the files in an sdist or wheel before and after migration to make sure you do not accidentally add or subtract files.

We have already been migrating many of the core Jupyter and IPython libraries to use the Hatch build backend, using Hatch Jupyter Builder where appropriate. We have upgraded several of our cookiecutter templates to use Hatch as well. The transition has been quite smooth!

We highly encourage you to try out Hatch and Hatch Jupyter Builder for existing or new Python libraries.

About the authors

Steven Silvester is a Senior Python Engineer at MongoDB working on the Database Experience Team. He is a Jupyter Steering Council Member and contributes to the maintenance of many of the Jupyter projects.

Zach Sailer is a Senior Software Engineer at Apple, where he is the Jupyter Open Source Champion. He is a Jupyter Distinguished Contributor and leader of the Jupyter Server subproject.

Visual programming in JupyterLab with Blockly

2022-07-29T13:12:00+00:00

Block-based programming has become ubiquitous in school curricula for early computer science education. It is an excellent means to teach basic programming concepts without having to deal with the specific syntax of a text-based programming language.

When moving from block-based programming to a more classical language like Python, students often have to switch to a completely new environment. In order to provide a smooth ramp of complexity for learners, we designed a JupyterLab extension for Blockly so that Jupyter can be used from the very first steps of their learning journey. We also used Blockly’s code generation feature to create a more integrated experience with Jupyter.

But what exactly is Blockly?

The JupyterLab-Blockly extension, running a simple loop, and the corresponding Python code in a cell.

Blockly is an open source library designed by Google to make coding easier and more accessible through block-based visual programming. It uses interlocking graphical blocks to represent coding concepts, while giving the user full creative freedom, it removes all language specific syntax requirements.

Getting started: installing the Blockly extension

To install the extension, you just need to execute the following command:

mamba install -c conda-forge jupyterlab-blockly

Once installed, you can easily open a Blockly editor directly from the launcher, and start experimenting!

Features

The UI is made primarily of the toolbox located on the left side and the canvas where you can drag and drop the blocks you want to use in your code. When you are ready, just click the run button and see how the syntactically correct converted code appears on the bottom part of your screen in the programming language of your choice along with the output!

Short demo for the JupyterLab-Blockly extension

The extension is also prepared to adapt to all your needs. The workspace is ready to generate the blocks using your selected JupyterLab kernel and modify its colors based on your individual theme, whether it’s dark, light or personalized.

JupyterLab-Blockly integrates smoothly with the rest of the Jupyter stack: by using Jupyter kernels to execute the generated code, by reusing the JupyterLab code cell component to display the generated code, etc.

In doing so, JupyterLab-Blockly remains consistent with the rest of the UI (consistent syntax highlighting, theming, and life cycle of Jupyter kernels).

Robotics education

On top of JupyterLab-Blockly, we also built another extension dedicated to operating robots with the help of visual programming — more specifically, we created a toolbox which includes the blocks necessary to control the Niryo One robot. Using ROS can be a time consuming task, but the pre-built visual blocks make it easier to experiment and faster to produce a prototype for any project you have in mind.

To install this extension you need to execute the following commands:

mamba install -c conda-forge jupyterlab-niryo-one

Then, all you need to do is switch from the ‘default’ toolbox to the ‘niryo’ one, and you will immediately gain access to forty-two new blocks, so if you have access to one of these robots, give it a try!

Short demo for the JupyterLab-NiryoOne extension

Both extensions can be found on GitHub. The JupyterLab-Blockly extension is ready to be used as a base for other projects; you can register new blocks, toolboxes and generators. Taking the JupyterLab-Niryo-One extension as an example, feel free to experiment on your own!

Controlling the Niryo-one robot with the JupyterLab-blockly extension.

You can also try out the JupyterLab-Blockly extension now, directly from your browser, using JupyterLite! Just click on the link below.

About the authors

Denisa Checiu is a Robotics working student at QuantStack. She is also currently pursuing her bachelor’s degree, majoring in Robotics and Intelligent Systems and minoring in Computer Science.

Carlos Herrero is a scientific software developer at QuantStack passionate about AI and its applications to robotics. His current work focuses on the JupyterLab stack and expanding the Jupyter ecosystem by developing new extensions to integrate ROS in JupyterLab and also contributing to Voilà.

Dashboarding with JupyterLab 3

2021-01-08T10:25:00+00:00

Project Jupyter offers a complete suite of open-source tools for the scientific computing community, reaching from the exploratory phase of a project to the presentation of the results. In this last but not least phase is where Voilà takes place, turning a technical document such as Notebook into an interactive presentation or a stand-alone web app ready to be shared with non-technical readers. You can learn more about Voilà in this blog post.

Voilà is an easy-to-use command-line tool that shines in its simplicity and flexibility. All its versatility comes from a custom template system built upon Jinja templates that allows everyone to create a custom UI to share Jupyter Notebooks. Examples of the versatility provided for the template system are voila-reveal and voila-gridstack.

Voila-gridstack is a Voilà template started by Bartosz Telenczuk to turn notebooks into dashboards following the specification introduced by the legacy jupyter-dashboards project. The idea behind is to be able to change the layout of the cells to re-configure your dashboards using drag-and-drop. Once you have your desired layout, its configuration stays in the metadata of the notebook. This makes it simple to carry around or share the notebook and its layout configuration.

With the release of JupyterLab 3.0, we wanted to update the voila-gridstack template by adding an editor for JupyterLab.

Configure your layout

The first way to open the voila-gridstack editor is by right-clicking on the notebook and selecting open with “Voilà Gridstack”. The editor is itself a JupyterLab Document Widget, which can be seen as another view of the same notebook document. The editor can also be launched from the classic notebook with a toolbar button.

Opening JupyterLab-Gridstack

Once the editor is open, every cell is decorated with a toolbar as a drag-and-drop handle and a button to remove the cell from the dashboard. When removing a cell, it is just marked as hidden in the metadata but it is not deleted from the notebook file. Besides, handles are provided for resizing cells in the layout.

Configuring your layout

When opening the editor for the first time in a new notebook, the metadata of the notebook is modified to follow jupyter-dashboards’ specification. By default, you will see an empty editor. To add new cells just drag them from the notebook view. It is not possible to add cells from another notebook.

Adding new cells

Finally, you can save the layout into the metadata of the notebook by clicking the “save” button or with the ctrl+s shortcut and then it’s time to hit the render button and Voilà! You will see your layout as a stand-alone web app.

Voilà! Your stand-alone web app

Try it online!

You can try the jupyterlab-gridstack extension online without installing anything by just clicking on the following image:

Installation

Voila-gridstack is available as a conda package, and you can easily install it by typing the following command in a conda environment:

mamba install -c conda-forge jupyterlab voila-gridstack

Acknowledgements

The development of Voilà and related packages at QuantStack is sponsored by Bloomberg.

The example notebook used as an example in this post was one of the example notebooks of the legacy Jupyter dashboards project.

About the authors

Carlos Herrero is a Computer Engineer passionate about AI and its applications to robotics. Currently working at QuantStack helping to develop Open Source projects.

Carlos Herrero

Jeremy Tuloup is a Scientific Software Developer at QuantStack, working on JupyterLab, Voilà, and various other projects within the Jupyter ecosystem.

Jeremy Tuloup

JupyterLab 3.0 is released!

2021-01-05T08:28:00+00:00

The 3.0 release of JupyterLab brings many new features to users and substantial improvements to the extension system.

Installation

To install JupyterLab with pip:

pip install jupyterlab==3

With mamba:

mamba install -c conda-forge jupyterlab=3

With conda:

conda install -c conda-forge jupyterlab=3

(Note that many third-party extensions are still in the process of updating to be compatible with JupyterLab 3.0 — please check the extensions you use and be patient, or even better, help them update! Also, it may help to uninstall JupyterLab 2 if you already have it installed.)

Here is an overview of the major user-facing changes in JupyterLab 3.0.

Debugger

Announced earlier this year, the visual debugger is now shipped by default with JupyterLab.

To use the visual debugger, you will need a kernel that supports debugging. An example of such kernel is xeus-python, the first Jupyter Kernel for Python with support for debugging.

Stepping into Python code in JupyterLab with the visual debugger

Check out the user documentation for more information, including a notebook based tutorial to interactively learn how to use the debugger.

The table of contents extension now ships with JupyterLab. This makes it easy to see and navigate the structure of a document.

Using the Table of Content in JupyterLab 3.0

Support for multiple display languages

JupyterLab now provides the ability to set the display language of the user interface.

Users will need to install the language pack as a separate Python package. Language packs are grouped in the language packs repository on GitHub, and can be installed with pip. For example, it is possible to install the language pack for Simplified Chinese using the following command:

pip install jupyterlab-language-pack-zh-CN

JupyterLab in Simplified Chinese

Check out the user documentation to learn more about the feature and how to add new language packs.

Improvements to the Simple Interface mode

The Simple Interface mode (previously known as the Single-Document Mode) has been significantly updated to have a more streamlined, document-oriented feel.

To make the Simple Interface mode more discoverable, there is now a switch to toggle it in the status bar. You can also toggle it from the View menu or the command palette, or use the default keyboard shortcut Ctrl/Cmd+Shift+D.

Enabling and disabling the Simple Interface Mode

Support for mobile has also greatly improved with a more compact layout. JupyterLab automatically switches to the Simple Interface when the window is resized down.

JupyterLab switches to the Simple Interface Mode on smaller screen

There is ongoing work to iterate on this concept and make the interface even more accessible on mobile devices.

Install new extensions with pip and conda / mamba

JupyterLab extensions can now be distributed as prebuilt extensions, which do not require a user to rebuild JupyterLab or have Node.js installed. Prebuilt extensions can be distributed as Python packages using familiar package managers such as pip, conda, and mamba. This makes installing and using extensions much faster and more convenient!

Installing a new extension with pip

Prebuilt extensions can be published to PyPI and conda-forge as separate packages, or bundled into existing packages with Jupyter Server extensions and Classic Notebook extensions. This helps make the ecosystem as a whole more coherent.

For example, install the new ipywidgets 7.6.0 with pip or conda to automatically enable ipywidgets in both the classic Jupyter Notebook and in JupyterLab 3.0 — no extra installation step or JupyterLab rebuild needed!

Installing ipywidgets works automagically in JupyterLab 3.0!

Improved development workflow for Extension Authors

The new prebuilt extensions are more convenient for extension authors to develop. The TypeScript extension cookiecutter has been updated to develop prebuilt extensions by default and provides all the necessary tooling to quickly create a new extension from scratch.

For more information, check out the Extension Developer Guide and the JupyterLab 2.x to 3.x Migration Guide.

Extension Examples

If you are looking for examples to learn how to make your own extension, make sure to check out the extension examples repository on GitHub. The examples have been updated to be compatible with JupyterLab 3.0 and provide a hands-on approach to developing extensions.

Changelog

This post was only an overview of what’s new in 3.0! If you are looking for a more complete list of the changes including bug fixes, be sure to have a look at the changelog.

Try it out on Binder

JupyterLab 3.0 can easily be tested on Binder using the following link:

Get Involved

There are many ways you can participate in the JupyterLab effort. We welcome contributions from all members of the Jupyter community:

Make your own JupyterLab extensions. Please add the jupyterlab-extension topic if your extension is hosted on GitHub. You can also help existing extensions update to JupyterLab 3.0. The changelog gives an overview of changes that may affect extension maintainers.
Contribute to the development, documentation, and design of JupyterLab on GitHub. To get started with development, please see the Contributing Guide and Code of Conduct. Many issues are ideal for new contributors and are tagged as “good first issue” or “help wanted”.
Connect with the community on GitHub or on the Gitter Channel. If you find a bug, have questions, or want to provide feedback, please join the conversation!

And don’t forget to participate in the user survey: https://www.surveymonkey.com/r/LCB7GBF

Thanks!

Reusable code snippets in JupyterLab

2020-11-18T18:08:00+00:00

Reusable code snippets in JupyterLab

In this blog post, we will introduce the Elyra code snippet extension, which enables us to reuse arbitrary snippets of code in your notebooks, source code, or markdown files in JupyterLab.

Let’s say you are working on a notebook and want to include a standard header or footer (like licensing text) or some code that you’ve previously created. This typically results in a hectic search. In which notebook did I use the code? Where’s the most current version of the disclaimer I need to add? While maintaining a “scratchpad” might help, native support for reusable code or documentation assets in JupyterLab holds a lot more appeal.

Code snippet ui

Installing the code snippet extension

The code snippet extension can be installed as part of Elyra or individually using pip or conda. In the context of this article, see below how to install the code snippet extension in your existing JupyterLab deployment.

Installing the extension using pip

$ pip install elyra-code-snippet-extension && jupyter lab build

Installing the extension using Anaconda

$ conda install -c conda-forge elyra-code-snippet-extension && jupyter lab build

Managing and using code snippets

Code snippets are accessed by opening the code snippets tab on the JupyterLab sidebar:

The code snippets UI is divided into two parts:

a quick search panel, and
the code snippet list, providing access to code snippet specific actions, such as copy, insert, edit, and delete.

Creating a snippet

To create a snippet click + above the search bar and provide the required information. You can optionally tag snippets to make them more easily discoverable.

Finding a snippet

To locate a code snippet, enter a search term and/or pick one of the listed tags.

Copying a snippet to the clipboard

You copy the snippet content to the clipboard by clicking the snippet’s copy icon.

Inserting a snippet

Click the snippet’s insert icon to paste the snippet content in the desired destination, such as a cell in a notebook or an open editor window. Drag-and-Drop of the snippet has also been incorporated in the latest Elyra release.

Do note that the editor does not keep track of where you’ve inserted a snippet. Also, note that snippets don’t act like macros and changes to a snippet are therefore not reflected in the locations where the snippet was embedded.

Editing a snippet

You edit a snippet by clicking the snippet’s pencil icon.

Deleting a code snippet

You delete a code snippet by clicking the snippet’s trash icon.

The code snippet extension currently doesn’t provide any import or export capabilities. However, you can access the relevant metadata files that contain the definitions.

The definitions are stored in the $JUPYTER_DATA_DIR/metadata/code-snippets directory, where $JUPYTER_DATA_DIR refers to the Jupyter data directory, which you can identify by running the following command in a terminal window:

$ jupyter --data-dir

In my environment, the code snippets are stored in /Users/ptitzler/Library/Jupyter/metadata/code-snippets/. To back up or share my snippets, I typically create an archive from the content of this directory.

Try Code Snippet from Binder

To experiment with the code snippet extension without installing it locally, just click on the binder link below and then look for the code-snippet icon on the JupyterLab side-bar :

Try the code snippet extension online on Binder

Acknowledgments

We’d like to thank Zach Sailer and Timothy George for all his UX contributions in conjunction with JaeWook (Jay) Ahn, Kiran Pinnipati, and Ai-Vy Dang the CalPoly interns that helped to implement these UX enhancements.

Closing thoughts

Elyra is constantly making Jupyter Notebooks better aiming to help data scientists, machine learning engineers, and AI developers through the model development life cycle complexities, making JupyterLab even better for AI practitioners.

Come and join our weekly dev meetings to learn more about the most recent updates to Elyra and to ask your questions to the community, including the project core developers.

We would also appreciate for you to get involved with the Elyra project. Read our contributing guidelines, create new issues if you have questions, suggestions for new features, or to report any bugs. We also welcome contributions via GitHub pull requests. And last but not least, don’t forget to star the project on Github.

JupyterLab, the Cloud Robotics Command Station

2020-08-11T13:19:00+00:00

JupyterLab-ROS with Voilà to quickly make a standalone web app

The ROS open-source project (Robot Operating System) has become a de-facto industry standard in the robotics community. Very much like with Project Jupyter, a large collection of software has been built upon well-specified communication protocols. ROS has become a rich ecosystem of standardized tools for building and distributing ROS-based software, exploratory robotics development, and much more.

With Industry 4.0, the number of robots and smart devices has been increasing exponentially. We are talking about fully automated factories with hundreds of robots and thousands of connected sensors, generating large amounts of raw data.

There is an opportunity in bridging the ROS and the open-source data science ecosystems and tools.

For this reason, we set ourselves to bring together these ecosystems,

by producing a collection of JupyterLab extensions for ROS, turning JupyterLab into a cloud robotics command station,
by integrating ROS packages with conda / mamba package management system, to enable their installation alongside other open-source data-science packages.

JupyterLab, the cloud robotics command station

While the traditional developer tools from the ROS stack such as RViz are desktop-based, we think that a robotics development environment running in a web browser would provide a lot of flexibility. Also, the ideal robotics developer platform should be plugin-based, and easily extensible to account for the diversity of use cases to enable, from simple interactive scripting to visualization or large datasets. Finally, we should build this environment upon broadly adopted foundations, with open governance, and not let a single entity hold the keys to the ROS ecosystem.

Jupyter is a natural candidate to serve as the foundation for a web-based robotics development environment.

JupyterLab is a web-based interactive development environment that has many of the much-needed features

In JupyterLab, everything is a plugin, including the core components to the application. Anyone can produce a “remix” of core and third-party JupyterLab extensions to their needs.
Very much in the spirit of ROS, one of the keys to Jupyter’s success is that the project was built upon well-documented and specified protocols and file formats that anyone could implement.
Jupyter is a multi-stakeholder project, not backed by a single corporation, but by a community of developers at a variety of companies, universities, as well as individual contributors.

Today, we are announcing the first beta of JupyterLab-ROS, a collection of JupyterLab plugins to integrate ROS with the JupyterLab platform. We combine the best of the data-science and robotics worlds to help developers build custom solutions for the industry. ROS can run in a high-performance server while JupyterLab is accessed in a web-browser (on Windows, OS X or Linux).

JupyterLab-ROS

JupyterLab-ROS automatically starts a rosmaster and rosbridge_server making all the painful setup of WebSockets connections completely transparent to users. A status bar widget allows you to start and stop the rosmaster server with a simple click and displays its state in real-time. Internally this widget runs a launch file which can be changed from the JupyterLab settings UI to launch additional nodes.

Master status widget

Another important component of the ROS stack is RViz, a desktop-based 3-D visualization tool. It’s web-based counterpart in JupyterLab is Zethus, started by Rapyuta Robotics. This plugin supports most of the display types supported by RViz. It also provides an info panel that displays the raw messages in realtime, and a web-based version of rqt_graph for visualizing the ROS node graph.

Zethus

A robotics station would be incomplete without a tool for debugging. For that purpose, ROS provides rosconsole, a package which allows developers to send messages to rosout and make them available on every node. The console gives access to these debugging messages in JupyterLab and provides additional features such as filtering by level and by node.

Log Console

Robotics Notebooks

A key component to the Jupyter stack is the Notebook, an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text. It’s frequently used to do data cleaning and transformation, numerical simulation, statistical modelling, data visualization, machine learning etc.

The notebook could be the perfect tool for robotics researchers and educators to make (and share) experiments in control algorithms, working in a dynamic and interactive development environment that allows quick prototyping and exploratory analysis while having access to the sophisticated mathematical libraries already well integrated with Jupyter. It can be tedious to have a different platform for interacting with robots and processing the data obtained from these, JupyterLab-ROS combines both toolsets and enables an integrated workflow.

At the same time, many projects are becoming more and more interdisciplinary. Currently, it is common to see projects mixing topics like machine learning and robotics. JupyterLab-ROS opens a new world of possibilities for machine learning researchers that are interested in applying their models to robots: now they have the possibility of connecting a robot to a development environment that they are very familiar with.

JupyROS is another powerful library that leverages ipywidgets and bqplot to integrate ROS in Jupyter Notebooks, being able to show real time plots from ROS messages and create intuitive forms to control robots in minutes.

Interactive Robotics in the Jupyter Notebook, with live plotting

ROS Package management

Modern scientific computing applications typically depend on a large number of libraries. Conda-forge is a community initiative to package scientific libraries for the conda package manager, and has become a de-facto standard source of packages for the scientific computing community, with tens of thousands of available packages, and thousands of contributors.

During the last year, we have worked hard to package more and more of ROS as conda packages. Conda-packages for ROS Melodic, including RViz are now available on all three operating systems! You can find the packages on the RoboStack channel (a more detailed blog post about this is going to follow soon).

Conda can be used to create virtual environments for ROS projects, which makes it possible to install Melodic, Kinect and Dashing side-by-side on the same Linux system, and have exact control over package versions.

We are also working on Mamba, a conda-compatible package manager implemented in C++. Together with Boa this effort will help to continuously build and release ROS packages on conda.

Try it online!

You can try the JupyterLab-ROS extension online without installing anything by just clicking on the following binder link:

JupyterLab-ROS is under active development. We are working on new features such as a rosbag widget to play and record bag files and improving existing ones like the roslaunch widget. If you are interested in contributing to JupyterLab-ROS you can find the code at this repository in the RoboStack organization. Contributions are always welcome!

Installation

You can install JupyterLab-ROS using Mamba, the first step to create a new environment with JupyterLab and the ROS packages necessary, then you can install the extension.

mamba create -n test -c conda-forge -c robostack python=3.6 nodejs=12 jupyterlab ros-melodic-ros-core ros-melodic-rosauth ros-melodic-rospy ros-melodic-rosbridge-suite ros-melodic-rosbag ros-melodic-tf2-web-republisher

conda activate test

pip install jupyter-ros-server

If you have any questions don’t hesitate to contact us on the RoboStack or QuantStack chats.

This work has many collaborators around the world, and we are especially grateful to Deep Chaitanya & Ashish Chaudhary from Rapyuta Robotics, Sean Yen from Microsoft for his amazing work on vinca that automagically generates the conda-recipes, Tobias Fischer for many patches and pushing us to bring OS X support to the conda packages as well. Many thanks to Jeremy Tuloup for his help with the JupyterLab extension system.

About the authors

Wolf Vollprecht works as a scientific and robotics software developer for QuantStack in Paris and Berlin.

Carlos Herrero is a Computer Engineer passionate about AI and his applications on robotics. Currently working at QuantStack helping to develop Open Source projects.

Elyra reaches 1.0.0

2020-08-10T15:55:00+00:00

Building on a Jupyter Notebooks foundation, the de facto tool for data scientists, machine learning engineers and AI developers, Elyra is an open-source project that provides a set of AI-centric extensions to JupyterLab aiming to help users through the model development life cycle complexities, making JupyterLab even better for AI practitioners.

Elyra is proud to announce its 1.0.0 Release. This release brings usability enhancements and bug fixes for existing features, such as enhanced inline user documentation and validation capabilities for the Pipeline Editor, improved performance for pipeline submission to Kubeflow Pipelines runtime. It also provides new capabilities such as a new reusable Code Snippets extension and the ability to configure runtimes directly on the JupyterLab user interface.

On the platform side, Elyra 1.0.0 is now based on JupyterLab 2.x and can easily be integrated into a JupyterHub deployment or other containerized environments such as Kubernetes and OpenShift or even be run in standalone mode using local docker deployment.

With Elyra 1.0.0 we also provide some helpful use-case scenarios leveraging Elyra features to build AI pipelines in order to demonstrate in-depth capabilities exposed by the project. Use cases include:

If this is the first time you are hearing about Elyra, check out our announcement blog for more details about the project.

The 1.0.0 release of Elyra includes:

Notebook Pipelines visual editor
Ability to run notebooks as batch jobs
Reusable Code Snippets (new)
Hybrid runtime support (based on Jupyter Enterprise Gateway)
Python script execution capabilities within the editor
Python script navigation using auto-generated Table of Contents
Notebook navigation using auto-generated Table of Contents
Notebook versioning based on Git integration
Reusable configuration and editor for runtimes (new)
Support for JupyterLab 2.x (new)
JupyterHub Support (new)
Ability to try Elyra from Binder (new)
Support for JupyterLab Dark Theme

Elyra 1.0 extensions to JupyterLab

Notebook Pipelines visual editor

Elyra’s Notebook Pipeline Editor simplifies the conversion of multiple notebooks into batch jobs or workflows. By leveraging cloud-based resources to run their experiments faster, the data scientists, machine learning engineers, and AI developers are then more productive, and therefore able to spend more of their time focusing on their technical skills.

Based on the great feedback from the Elyra user base, this release brings lots of bug fixes usability enhancements such as:

Enhanced inline user documentation
Validation capabilities to Pipeline Editor that notifies users of missing or invalid configuration values
Optimized dependency handling providing a much faster submission of pipelines
Easier access to previously submitted experiments from the Pipeline Editor
Support for “bring your own image” to be used as the environment to execute Notebooks on the external runtime

Below you can see some of the Pipeline Editor UI enhancements introduced in Elyra 1.0:

Elyra Pipeline visual editor UI enhancements

Reusable Code Snippets

Code snippets give you the ability to save time and reuse task-oriented blocks of code. Elyra’s new code snippets extension enables easy discovery, creation, and insertion of reusable snippets of code into your Notebooks, Python Scripts, or even Markdown files used for documentation directly from the JupyterLab workspace. This makes the process of writing code more efficient and accessible.

The list of available code snippets is found in the left side pane and includes a preview of each snippet, and an option to either copy a snippet or insert it directly inline.

Elyra code snippets pannel

Code snippets can also be conveniently created and edited from within JupyterLab.

Elyra code snippets editor

Leveraging Table of Contents for Notebooks and Python Scripts

Navigating large files to find specific sections in Notebooks or function definitions in Python Scripts can be difficult tasks. The Table of Contents extension, which was enhanced to support navigating Python Scripts provides an easy outline of your contents, enabling easy navigation.

Below is an example of an auto-generated table of contents for a Notebook based on its markdown sections:

Auto-generated Notebook Table of Contents

To allow for streamlined python development, Elyra’s python editor is now accompanied by an auto-generated Table of Contents, which allows for efficient navigation within large python scripts.

Auto-generated Python Script outline using Table Of Contents

Reusable configuration and editor for runtimes

Elyra introduced a ‘shared configuration service’ that simplifies workspace configuration management, enabling things like information around accessing external runtimes to be configured once and shared across multiple components.

With Elyra 1.0, this service is now used by multiple components and has been enhanced with schema-based validation capabilities and a full set of REST APIs. With this release of Elyra, users can also easily browse, create, and edit these configurations from within the JupyterLab user interface.

Runtime configuration editor

JupyterHub support

With Elyra 1.0.0 we have also created a docker image and provided necessary configuration steps to integrate Elyra with JupyterHub.

Elyra & JupyterHub deployment diagram

Using Elyra in real Analytics and AI scenarios

While building Elyra, we work very closely with data scientists, machine learning engineers, and AI developers, and we have been building a few scenarios to validate the user experience when developing models and other applications using Elyra.

Analyzing COVID-19 time-series data

One of the examples creates a pipeline to analyze COVID-19 time series data sets from both the USA and Europe which is available as open-source in the covid-notebook GitHub repository.

COVID-19 notebook pipeline

Analyzing NOAA weather time series data set and explore forecasting

Another example utilizes DAX — Data Asset Exchange NOAA dataset and produces a pipeline that consumes and applies ETL into the dataset, and then goes about analyzing and experimenting with different forecasting capabilities.

NOAA weather time-series notebook pipeline

Elyra community adoption

The Elyra community is working very hard to promote adoption and create a healthy community around the project. In the past few months, we are starting to see some momentum, and below are some details :

Github Stars: Elyra main repository is reaching close to 500 as of August 2020, please continue to show your support for the project by spreading the word about the project and giving us more stars.

Projects depending on Elyra: Other then the two examples scenarios mentioned above, we are starting to see other communities adopting Elyra into their projects, CalPoly has been using the Elyra code snippets extension in their summer-intern projects and other communities are also experimenting with Elyra.

GitHub projects depending on Elyra

Downloads: We have also seen an increase of downloads of the Elyra npm packages in the past several weeks, and we are floating between 60k — 70k weekly downloads for the past month:

Elyra packages download stats

Elyra in the Enterprise: Components of Elyra are integrated and available in IBM Cloud Pak for Data and Watson Studio offerings.

Try Elyra from Binder

To experiment with Elyra without installing it locally, just click on the binder link below:

Try Elyra online on Binder

What is next for Elyra

Come and join our weekly dev meetings to learn more about the most recent updates to Elyra and to ask your questions to the community, including the project core developers.

Jupyter Blog - extensions

Notebooks-on-Demand

Announcing jupyter-builder: A Standalone Build System for JupyterLab Extensions

Motivation

What jupyter-builder does

Build compatibility without JupyterLab installed

Migrating an existing extension

1. Update the Python build dependencies

2. Update the npm builder dependency

What’s next

Acknowledgements

About the developers

Useful links

Catching Jupyter-specific bugs before CI does: announcing jupyter eslint plugin

Motivation

What v1 catches

Plugin structure

Commands and settings

Internationalization

Testing

Getting started in 60 seconds

Help us encode more knowledge

Acknowledgement

About the developers

Useful links

A User’s Journey with Plugin Playground: From First Idea to Installable JupyterLab Extension

Motivation

Journey At a Glance

1.0: Pick Your Starting Path

2.0: Build Extension #1 Manually

2.1: Create the plugin file

2.2: Fill the missing pieces using the right sidebar

2.3: Load it and verify behavior

3.0: Build with AI

3.1: Give AI a precise request

3.2: Use AI + deterministic inserts together

4.0: Iterate Faster Before Handoff

5.0: Share for Review

6.0: Export as Wheel and Validate in Binder

6.1: Export .whl

6.2: Install in Binder

7.0: Export as Zip and Create a GitHub Repository

7.1: Export .zip

7.2: Bootstrap repository

What’s Next

Acknowledgements

About the Developers

Further reading

700 JupyterLab 4 Extensions!

What Are JupyterLab Extensions?

The Ecosystem at 700

How We Got Here

Where the Extensions Are

JupyterLab’s AI Layer Starts Taking Shape

Reproducibility Gets a Toolchain

Science

Accessibility

27 Extensions, One Platform

Want to Build Your Own?

How We Track This

What’s Next

About the author

Instantly view Parquet files in JupyterLab with Arbalister

Arrow over the wire

Looking further

An extension point

About the Author

Building AI Agents for JupyterLab using Notebook Intelligence

What is tool calling and an AI Agent?

AI Agent Extension Example

Geo Coordinates Lookup Tool

Map Response Generator Tool

Map Notebook Creator Tool

Notebook Share Tool

Tool call schema definitions

Tool chaining

Chat Participant

NBI Extension

Try it out and share your feedback!

About the Author

What `jupyter-builder` does

6.1: Export `.whl`

7.1: Export `.zip`