Jupyter Blog - JavaScript

Catching Jupyter-specific bugs before CI does: announcing jupyter eslint plugin

2026-06-11T15:13:00+00:00

We’re excited to announce the release of the first version of the Jupyter ESLint plugin @jupyter/eslint-plugin, now available on npm. It’s a custom ESLint plugin designed specifically for the official Jupyter frontends (JupyterLab, Notebook, JupyterLite) and TypeScript extensions built around them.

Motivation

If you’ve ever written a JupyterLab extension, you’ve probably hit something like this:

You wire up a new plugin, push a PR, walk away to make coffee, and 30 minutes later CI fails on a test you didn’t expect to be related. You scroll through logs, eventually trace it back, and realize the problem was a single misaligned argument in your activate function. TypeScript was happy. The editor showed no red squiggles. Nothing in your local toolchain said a word.

And the slow CI feedback loop is only half the story. Some of these mistakes don’t fail CI at all, a hardcoded English label, an enum in a settings schema, a malformed token id. They pass tests, ship to users, and only surface at runtime. They’re the kind of bugs you hear about from users months later, if at all.

JupyterLab, Jupyter Notebook, and the wider extension ecosystem follow a lot of project-specific patterns around plugins, commands, tokens, settings schemas, translations, and visual tests. These patterns aren’t enforced by TypeScript, because they live one level above the type system. They’re contracts about how the framework expects code to be written, not about whether the code type-checks.

A few examples of what generic linters and TypeScript can’t catch:

A JupyterFrontEndPlugin that requires [TokenA, TokenB] but whose activate function takes those arguments in the wrong order. Both are tokens, both type-check but the plugin fails at activation.
A command added with commands.addCommand(...) whose label is a hardcoded English string. The extension ships, gets translated by the community, and that one button silently stays in English.

These are real, recurring issues. They cost contributor time, they frustrate first-time contributors, and they pile onto reviewer load. They’re exactly the kind of knowledge that lives in the heads of long-time maintainers and exactly the kind of knowledge that a linter is great at encoding. So we built one.

What v1 catches

The plugin currently ships eight rules, grouped by what they protect:

Plugin structure

plugin-activation-args — Keeps activate function arguments aligned with requires and optional tokens.
plugin-description — Requires a non-empty description on every JupyterFrontEndPlugin.
token-format — Enforces the <package>:<TokenSymbol> convention for new Token(...) ids.

Commands and settings

command-described-by — Requires a describedBy schema on commands that accept args.
no-schema-enum — Disallows enum in settings JSON schemas in favor of translatable oneOf with const and title.

Internationalization

no-untranslated-string — Flags hardcoded user-facing strings in command labels, dialogs, accessibility attributes, JSX, and more.
no-translation-concatenation — Forbids dynamic concatenation inside trans.__() calls that translation extractors can’t pick up.

Testing

require-soft-assertions-before-snapshots — Requires expect.soft(...) for all but the last snapshot in a Playwright test, so a single failure doesn’t block the rest from updating.

Each rule has incorrect/correct examples, options, and rationale in the rules reference.

Getting started in 60 seconds

Install the plugin:

npm install --save-dev @jupyter/eslint-plugin

Add it to your ESLint config and turn on the recommended ruleset:

import jupyterPlugin from '@jupyter/eslint-plugin';

export default [
  {
    plugins: {
      jupyter: jupyterPlugin
    }
  },
  jupyterPlugin.configs.recommended
];

That’s it. Run jlpm eslint and you’ll start seeing Jupyter-specific issues alongside your existing lint output. If you’re still on legacy .eslintrc configs, there’s a recommended-legacy config too. However, we strongly recommend upgrading to ESLint v9 and using the new flat config format.

Full setup details, including type-aware configuration for plugin-activation-args and the JSON parser setup for no-schema-enum, are in the user guide.

If you maintain an extension, the most useful thing you can do today is install it and run it once on your codebase. Even if you don’t adopt it permanently, the first run is often informative most extensions surface a handful of issues they didn’t know they had.

Help us encode more knowledge

Eight rules is a starting point. The plugin is most valuable when it captures the recurring mistakes that maintainers across the ecosystem keep running into and a lot of that knowledge isn’t written down anywhere.

If you’ve ever been bitten by a Jupyter-specific issue that could have been a lint rule, something where you found yourself thinking “I wish my editor had warned me about this” please open an issue on the repository. Even a short description of the bug and a minimal example helps. We’d much rather encode that knowledge into an automated check once than have every new contributor learn it the hard way.

Acknowledgement

This work was funded under the first round of Jupyter Community Funded Proposals. We want to thank the Jupyter Foundation and everyone involved in setting up this funding mechanism.

We’re also grateful to Jeremy Tuloup and Nicolas Brichet for their help in adopting the plugin.

About the developers

Darshan Paudyal is an intern at OpenTeams and a member of the Jupyter Frontend Council. Darshan led the development of v1 of @jupyter/eslint-plugin, designing and implementing the rule set, the documentation, and the release.

Michał Krassowski is a Senior Software Engineer at OpenTeams. Mike provided guidance and direction for the development of the plugin.

Useful links

Exploring Open Access Art Using the Deno Jupyter Kernel

2025-04-17T15:27:00+00:00

Jupyter notebooks are the de facto standard for interactive computing and data analysis, combining code, prose, and visualizations in a single document.

In fact, this post was written in a notebook!

Deno is a simplified and secure runtime for JavaScript and TypeScript, with built-in tooling and modern defaults. Since v1.37, Deno ships with a built-in Jupyter kernel, making it easy to use TypeScript for data science and interactive computing.

Having worked extensively with computational notebooks (mainly in Python), I find this exciting for these reasons:

Simple setup — The kernel is built into the Deno CLI, so there’s no need for additional installation. Just install Deno.
Improved dependency management — Notebooks run like standalone scripts, and Deno’s support for importing dependencies in the code itself improves dependency management and reproducibility.
A unified ecosystem for interactive data analysis — Jupyter supports rich HTML/CSS/JS outputs, and the JavaScript ecosystem is ideal for interactive UIs. Deno connects the kernel and frontend, making notebooks more powerful and flexible.

In this post, we’ll explore the National Gallery of Art’s Open Access dataset in Jupyter with Deno. We’ll clean and analyze the data, looking at public domain works, artists, origins, and patterns. Along the way, we’ll introduce Deno-compatible tools like npm:nodejs-polars and npm:@observablehq/plot, and wrap up by adding widgets and custom displays for a more interactive experience.

The dataset

The National Gallery of Art (NGA) Open Data Program provides access to over 130,000 artworks and their creators, available on GitHub. This dataset includes valuable metadata such as titles, dates, artists, and classifications, all under a Creative Commons 0 (CC0) license, meaning it’s free to use and share.

The collection spans a wide variety of artworks, from sculptures to paintings, by famous artists like Mary Cassatt, M.C. Escher, Vincent van Gogh, Pablo Picasso, and Georgia O’Keeffe.

However, navigating this resource is challenging. The National Gallery’s website is not very user-friendly, making it difficult to make sense of what is actually in the dataset. It’s nearly impossible to get any high-level insights, like the number of paintings, which time periods are most represented, or which artists are most prevalent–let alone make specific queries.

Fortunately, the dataset is available on GitHub as a collection of related tables, exported as CSV files. For our analysis, we’ll focus on three key tables:

objects.csv — Metadata about artworks, including titles, dates, materials, and classifications.
constituents.csv — Artist details such as names, nationalities, and lifespans.
published_images.csv — Links to artwork images via the NGA’s IIIF API.

We’ll clean and join these tables to create a unified dataset.

But wait!

While these NGA tables are free to use, they only include metadata. The actual artwork images have separate licensing, with only about half being CC0-licensed. We’ll gather this info separately to identify freely available images.

Our goal is to unify the data to identify public domain images, explore subsets, and view available artworks. Maybe you’ll even find an upgrade for that default OS wallpaper.

Loading and cleaning the data

Open Access dataset

To get started, we first need to download the data. If you’ve worked with CSVs in JavaScript before, this typically looks like a “fetch” request followed by some parsing. Here, we’re using jsr:@std/csv to stream the data, which gives us an array of objects.

import * as csv from "jsr:@std/csv";

let baseUrl = new URL(
  "https://github.com/NationalGalleryOfArt/opendata/raw/refs/heads/main/data/",
);
let response = await fetch(new URL("objects.csv", baseUrl));

let objects = await Array.fromAsync(
  response.body.pipeThrough(new TextDecoderStream()).pipeThrough(
    new csv.CsvParseStream({ skipFirstRow: true }),
  ),
  (row) => ({
    objectid: +row.objectid,
    title: row.title,
    year: +row.beginyear,
    medium: row.medium,
    type: row.visualbrowserclassification,
  }),
);

objects.slice(0, 3);

[
  {
    "objectid": 0,
    "title": "Saint James Major",
    "year": 1310,
    "medium": "tempera on panel",
    "type": "painting"
  },
  {
    "objectid": 1,
    "title": "Saint Paul and a Group of Worshippers",
    "year": 1333,
    "medium": "tempera on panel",
    "type": "painting"
  },
  {
    "objectid": 4,
    "title": "Saint Bernard and Saint Catherine of Alexandria with the Virgin of the Annunciation [right panel]",
    "year": 1387,
    "medium": "tempera on poplar panel",
    "type": "painting"
  }
]

If we were just plotting the data, this would work fine. But an array of objects isn’t ideal for building our dataset. The NGA data is relational and needs merging, cleaning, and restructuring. A DataFrame is better suited for our use case, offering higher-level, efficient APIs that eliminate the need to write manipulation functions by hand.

Let’s see how we can load the same dataset using a DataFrame from npm:nodejs-polars:

import * as pl from "npm:nodejs-polars";

let response = await fetch(new URL("objects.csv", baseUrl));

let objects: pl.DataFrame = pl.readCSV(await response.text(), {
  quoteChar: '"',
});

objects = objects.select(
  pl.col("objectid"),
  pl.col("title"),
  pl.col("beginyear").as("year"),
  pl.col("medium"),
  pl.col("visualbrowserclassification").as("type"),
);

objects.head()

Here, we fetch the dataset as before, but instead of parsing the CSV ourselves, we read the text response directly with Polars to create a pl.DataFrame.

We then chain a .select expression to choose and rename the columns we want. Notice how, rather than operating on each row in a loop, with Polars we work directly with columns. This API allows us to express complex, efficient operations (in Rust) without materializing JavaScript objects.

Let’s load our other two datasets as pl.DataFrames as well:

The constituents.csv table contains information about any person or entity associated with a piece of art, such as artists, curators, or collectors.

let response = await fetch(new URL("constituents.csv", baseUrl));
let constituents = pl.readCSV(await response.text(), { quoteChar: '"' })
  .select(
    pl.col("constituentid"),
    pl.col("forwarddisplayname").alias("name"),
    pl.col("visualbrowsernationality").alias("nationality"),
  );

constituents.head();

The published_images.csv table contains additional information about the artwork’s image, including a URL for the thumbnail and mapping the objectid to the IIIF (International Image Interoperability Framework) for the image.

let response = await fetch(new URL("published_images.csv", baseUrl));
let publishedImages = pl.readCSV(await response.text(), { quoteChar: '"' })
  .select(
    pl.col("depictstmsobjectid").alias("objectid"),
    pl.col("uuid"),
    pl.col("iiifthumburl").alias("thumburl"),
  );
publishedImages.head();

The final table, objects_constituents.csv, represents a many-to-many relationship between artworks and the people or entities associated with them. Each artwork may have multiple people linked to it.

For our analysis, we’re interested in selecting the “primary” constituent (i.e., the artist). We define the primary artist as the one with the highest displayorder for each object.

let response = await fetch(new URL("objects_constituents.csv", baseUrl));
let objectToArtist = pl.readCSV(await response.text(), { quoteChar: '"' })
  .filter(pl.col("role").eq(pl.lit("artist")))
  .sort({ by: "displayorder" })
  .groupBy("objectid")
  .first()
  .select("objectid", "constituentid");

objectToArtist.head();

This is a much more advanced usage of Polars, so let’s break down what’s going on:

Filter the table to select only rows where the role is “artist”.
Sort the filtered data by displayorder to ensure all entries are at the top.
Group the rows by objectid, ensuring one row per artwork.
Get the first in each group (i.e., artist with highest display order).
Select just the “objectid” and “constituentid” for joining our tables.

Writing this logic by hand in JavaScript would be tedius, but with Polars it’s declarative and reads like a sentence. This type of chaining is common when working with relational data.

Determining whether art is in public domain

We’ve loaded the relevant tables from the Open Access dataset, but they do not indicate whether images associated with an artwork are in the public domain. While CC0 images can be freely downloaded, shared, and repurposed, not all art with the CC0 license are legally in the public domain everywhere due to local copyright laws and edge cases.

Let’s find a way to add public domain information to our data set.

That information is available on the NGA website, but only through the search user interface. Extracting it manually would be impractical, so I reverse-engineered an API call to retrieve the IDs of artworks with public domain images.

We only need the IDs, not the full data, but the query is slow. Since this isn’t an official API, I’ve saved the results separately. There are about 50k IDs identifying public domain images in our dataset.

// This is not an official API, so I’ve cached the results to avoid 
// repeatedly querying the server.
// 
// let response = await fetch("https://www.nga.gov/bin/ngaweb/collection-search-result/search.pageSize__100000.pageNumber__1.lastFacet__artobj_downloadable.json?artobj_downloadable=Image_download_available"); 
// let data = await response.json();
// Deno.writeTextFileSync("public-domain-ids.txt", data.results.map((object) => object.id).join("\n"));

let publicDomainIds = Deno.readTextFileSync("public-domain-ids.txt")
  .split("\n").map((d) => +d);

“Join” to single unified table

Now that we have all our data cleaned and loaded, we’ll perform a large join to combine all these tables into a single unified table.

Again, this challenging to do manually, but with Polars, we can nicely express these complex join operations. Finally, we use a .isIn expression to derive a new column that indicates whether a piece of art is public domain.

let df = objects
  .join(objectToArtist, { on: "objectid" })
  .join(constituents, { on: "constituentid" })
  .join(publishedImages, { on: "objectid" })
  .select(pl.exclude("constituentid"))
  .withColumns(pl.col("objectid").isIn(publicDomainIds).alias("public"))
  .sort({ by: "public" })
  .sort({ by: "year", descending: true, nullsLast: true });

df.head();

With our unified dataset finally loaded, let’s move on to some plotting.

Exploring art, who created it, when, and more with plots

The NGA portal allows some exploration of these data but focuses on specific artworks. We’re more interested in broader insights — where most art comes from, who created it, when it was made, and whether it’s public domain.

As a JavaScript/TypeScript runtime, Deno gives us access to many visualization libraries to ask these kinds of questions. We’ll use npm:@observablehq/plot (Observable Plot) to create plots, spot patterns, and better understand the collection.

Note this post is not a tutorial on Observable Plot, so don’t worry if some of the code below is not immediately clear. The goal is to demonstrate how to use Observable Plot in Jupyter with Deno to extract insights from our data. I’ll highlight any Deno-specific details or useful tips along the way.

Let’s start by examining the distribution of artworks across different “types” in the NGA collection.

import * as Plot from "npm:@observablehq/plot";
import { document } from "jsr:@manzt/jupyter-helper";

// Convert our DataFrame to Array<Object>
let records = df.toRecords();

Plot.plot({
  width: 900,
  marginLeft: 50,
  color: { legend: true },
  marks: [
    Plot.barY(
      records,
      Plot.groupX({ y: "count" }, {
        x: "type",
        sort: { x: "-y" },
        fill: (d) => d.public ? "Public domain" : "Copyrighted",
      }),
    ),
  ],
  // Provide a custom `document`
  document,
});

For the code above, the key things to keep in mind are:

Custom document — Observable Plot relies on the browser DOM, but since we’re in Deno, we provide a custom document from jsr:@manzt/jupyter-helper to enable rendering.
Converting our Polars DataFrame to records — Observable Plot works best with arrays of objects, so we use df.toRecords() to make the data easier to work with.

For specifics on the Plot API, see the documentation and examples available.

Right away, we can see that prints, drawings, and photographs make up most of the collection. Each category varies in how many images are public domain — images of prints, drawings, sculptures, and paintings are mostly public domain, while images of photographs and portfolios are largely copyrighted.

We can get more granular in our plot very easily with Observable Plot, which is composable and expressive. With just a few tweaks, you can completely change how the data is represented. For example, starting from the plot above, we can modify marks and some encoding fields and produce an entirely different chart, allowing for quick exploration and iteration — like this “waffle” bar chart.

Plot.plot({
  width: 900,
  marginLeft: 50,
  color: { legend: true },
  marks: [
    Plot.waffleY(
      /* changed, Plot.barY */
      records,
      Plot.groupZ(
        /* changed, Plot.groupX */
        { y: "count" },
        {
          fx: "type",
          fill: (d) => d.public ? "Public Domain" : "Copyrighted",
          sort: { fx: "-y" }, /* changed, sort: { x: "-y"  } */
          unit: 300, /* new */
        },
      ),
    ),
    Plot.ruleY([0]),
  ],
  document,
});

With both Polars and Plot, we can ask more specific questions about the data, such as “Which artists have the most artwork in the collection?” Here, we first use Polars to group the data by artist (“name”) and whether the artwork is public domain (“public”), counting the number of works for each. Then, we use Plot to visualize the top 25 artists.

const artworkTotals = df
  .groupBy("name", "public")
  .len()
  .sort("name_count", true)
  .head(25)
  .toRecords();

Plot.plot({
  marginLeft: 200,
  color: { legend: true },
  marks: [
    Plot.barX(artworkTotals, {
      x: "name_count",
      y: "name",
      sort: { y: "-x" },
      fill: (d) => d.public ? "Public domain" : "Copyrighted",
    }),
  ],
  document,
});

Again, we can draw some high-level takeaways from this plot: The photographer Robert Frank has nearly twice as many works as the next highest artist. Some entries, like “American 20th Century” and “German 15th Century,” represent groups rather than individuals. Additionally, an artist’s work is typically either entirely public domain or not at all — likely because public domain status is determined at the collection level, with institutions clearing entire groups of works at once rather than evaluating individual pieces separately.

With our tidy and clean dataset, we can quickly iterate in Deno to ask both broad questions (“What even is in this collection?”) and very specific ones (“Who created this artwork in a given year?”).

Deeper exploration with interactivity

So far, we’ve used static plots to explore our data, but Jupyter notebooks support rich, interactive outputs using HTML, CSS, and JavaScript.

With Deno, we can add more interactivity to our data exploration by either using off-the-shelf interactive anywidgets by quickly building custom components for visualizing subsets in ways prebuilt tools don’t support. Deno provides direct access to the full web ecosystem, making custom visualizations seamless.

In this section, we’ll look at:

Prebuilt interactive table anywidgets for exploring data
Create custom <Gallery /> component to visualize subsets of the dataset

Exploring data with DataFrame viewer anywidgets

In Jupyter, “widgets” extend notebook outputs with interactive views and controls for objects in the kernel (i.e., backend). Unlike standard outputs, they consist of both kernel-side and front-end code, communicating directly via custom messages. While the Jupyter Widget ecosystem is largely Python-centric, the anywidget project provides a kernel-agnostic way to create and share these components.

With anywidget, widgets can be published to JSR and used in the Deno Jupyter kernel. For example, jsr:@manzt/jupyter-helper provides some off-the-shelf anywidgets for interactively viewing Polars data frames.

import { agGrid } from "jsr:@manzt/jupyter-helper";

agGrid(df.head(200)); // just look at the first 200 items

The agGrid export renders the pl.DataFrame using the popular AG Grid library, allowing for interactive sorting, filtering, and pagination. This default view is minimal but can be easily extended with AG Grid’s many features.

import { quak } from "jsr:@manzt/jupyter-helper";

const paintings = df.filter(pl.col("type").eq(pl.lit("painting")));
quak(paintings);

The quak export renders the pl.DataFrame with the quak data table viewer. Like agGrid, it also includes summary visualizations above each column, showing 1D distributions, and can scale to much more data (e.g., the entire collection). These 1D summary visualizations are interactive, enabling cross-filtering across columns.

Together, the programmatic and interactive layers enable unique hybrid workflows working with data. Some queries are easier to express in code, while others are more intuitive through direct manipulation. In notebooks, the two approaches come together naturally.

For example, Polars can be used to subset the data to just public domain paintings (~2k out of ~115k artworks), and then a date range can be selected interactively by brushing in the UI to explore variation across other columns.

Creating a custom `<Gallery/>` component

Deno gives us direct access to the web ecosystem, making it easy to build small, useful UIs for exploring our data in ways that off-the-shelf tools don’t support.

For example, it would be helpful to actually see a set of artwork in a “gallery view” to provide intuition about what’s in that subset. We can do that since the dataset contains image URLs.

Let’s create a custom JSX component that server-side renders (SSR) a gallery view for any dataset subset using the render export from jsr:@manzt/jupyter-helper. This helper simply converts JSX to an HTML string, which we display using the Deno.jupyter.Displayable interface.

import * as React from "npm:react";
import { render } from "jsr:@manzt/jupyter-helper";

function Gallery({ objects, size = 100 }) {
  return (
    <div
      style={{
        display: "grid",
        gridTemplateColumns: `repeat(auto-fill, minmax(${size}px, 1fr))`,
        gap: "4px",
      }}
    >
      {objects.select("objectid", "thumburl", "title", "public")
        .map(([objectid, thumburl, title, publicDomain]) => (
          <div
            key={objectid}
            style={{ position: "relative", textAlign: "center" }}
          >
            <a
              href={`https://www.nga.gov/collection/art-object-page.${objectid}.html`}
              style={{
                display: "block",
                width: `${size}px`,
                height: `${size}px`,
                position: "relative",
              }}
            >
              <img
                src={thumburl}
                alt={title}
                style={{
                  width: "100%",
                  height: "100%",
                  objectFit: "cover",
                  borderRadius: "5px",
                }}
              />{" "}
              {publicDomain && (
                <img
                  src="https://mirrors.creativecommons.org/presskit/icons/zero.svg"
                  alt="Public Domain"
                  style={{
                    position: "absolute",
                    bottom: "3px",
                    right: "3px",
                    width: "20px",
                    height: "20px",
                    opacity: 0.60,
                  }}
                />
              )}
            </a>
          </div>
        ))}
    </div>
  );
}

Gallery is a custom component that displays a grid of artwork thumbnails from a given dataset. Each image links to its NGA collection page, and public domain works are marked with a CC0 icon.

We can SSR Gallery with our data to better understand a smaller subset, making it easier to explore specific artists, time periods, or categories visually. For example, a random sample from all the collection:

render(<Gallery objects={df.sample(20)} />);

Or sample of just paintings:

render(<Gallery objects={paintings.sample(20)} />);

It’s worth taking a moment to appreciate that in just a few lines of code, we’ve built a domain-specific utility that makes our data more intuitive to explore. This kind of flexibility is powerful and unique to Deno, letting us quickly craft custom utilities that provide deeper insights.

With our toolkit, let’s take a closer look at the NGA dataset:

Drilling down on when the art was created

So far, our data exploration has focused on high-level summaries of the NGA collection by artist and artwork type using static plots. Now, let’s drill down into something potentially more interesting using both our complete toolkit: when these works were created.

We’ll start with a stacked histogram to see the overall distribution of artwork types over time, then facet by public domain status to uncover any patterns.

Plot.plot({
  y: { grid: true },
  color: { legend: true },
  marks: [
    Plot.rectY(
      records.filter((r) => r.year > 1401), // 15th century or later
      Plot.binX({ y: "count" }, {
        x: (d) => new Date(d.year, 0, 1),
        fill: "type",
        fy: (d) => d.public ? "Public Domain" : "Copyrighted",
      }),
    ),
    Plot.ruleY([0]),
  ],
  marginLeft: 100,
  marginRight: 100,
  width: 1000,
  height: 400,
  document,
});

The distribution of artworks with copyrighted and public domain images are noticeably different. Nearly all copyrighted works appear after 1850, while public domain artworks are more evenly spread over time — except for a sharp spike in public domain drawings from the 1940s, with over 15k artworks added.

This is a striking anomaly. Are they all from the same artist? Why only drawings? Let’s take a closer look.

We’ll filter down the public domain data to this span.

let notablePeriod = df.filter("public") // just public domain
  .filter(pl.col("year").gt(pl.lit(1925))) // between 1925 - 1955
  .filter(pl.col("year").lt(pl.lit(1955))).sort({ by: "year" });

Plot.plot({
  y: { grid: true },
  color: { legend: true },
  marks: [Plot.rectY(
    notablePeriod.toRecords(),
    Plot.binX({ y: "count" }, {
      x: (d) => new Date(d.year, 0, 1),
      fill: "type",
    }),
  )],
  document,
});

Zooming into the 1925–1955 range, we see that almost all the data falls within a much narrower window. Interestingly, the spike is highly concentrated between 1935 and 1942, with very little artwork outside this period.

Narrowing in closer, we can see that in this window there are ~18k individual drawings:

const totalNumberOfDrawings = notablePeriod
  .filter(pl.col("year").gt(pl.lit(1934)))
  .filter(pl.col("year").lt(pl.lit(1943)))
  .shape
  .height;

totalNumberOfDrawings; // 18096

made by over 1000 individuals:

const numberOfArtists = notablePeriod
  .filter(pl.col("year").gt(pl.lit(1934)))
  .filter(pl.col("year").lt(pl.lit(1943)))
  .groupBy("name")
  .len()
  .shape
  .height;

numberOfArtists; // 1034

This gets even more interesting! The artwork is not from just one specific source — there was a surge of public works between 1935 and 1942.

Let’s use our custom <Gallery /> view to explore a subset of these works — maybe seeing them will give us some insight.

render(<Gallery objects={notablePeriod.sample(100)} />);

So all these artworks, despite being from different individuals, have a very similar style and medium. That’s something worth digging into — what might explain this pattern?

Thanks to our custom <Gallery /> component, we can click on artworks and view their metadata. Sampling a few, we see that they all belong to the “Index of American Design” collection.

Conceived as an effort to document and preserve American folk and decorative arts, the Index of American Design consists of 18,257 watercolor renderings created between 1935 and 1942 as part of a Federal Art Project (FAP) work-relief program. Around 400 artists meticulously recreated textiles, woodcarvings, weathervanes, and other objects from across the U.S., aiming to establish a distinct American visual lineage. The project was ultimately housed at the National Gallery of Art, becoming a widely exhibited visual archive.

That’s it! We found a large collection of public works funded by a federal program during the Great Depression — clear in both the data and historical context.

Conclusion

Using Deno’s Jupyter kernel, we explored the NGA Open Access dataset, combining static plots, interactive widgets, and custom JSX components to uncover patterns in the data. This led us to a historical insight — the Index of American Design, a public art project from the 1930s.

With Deno, JSX, and anywidget, we combine data analysis with interactive web-based visualization, making it easy to build domain-specific tools for deeper exploration.

Bringing Modern JavaScript to the Jupyter Notebook

2023-09-24T03:34:00+00:00

Visualizing Data via Deno, TypeScript, and VegaLite in JupyterLab

I never expected this.

using modern JS, TS, and npm in jupyter notebooks

👇

— Deno (@deno_land), September 21, 2023

Deno brings TypeScript, JavaScript, npm, and ES Modules to Jupyter with an easy to install kernel.

The Deno Kernel is the first language runtime with a builtin jupyter kernel.

There’s no better time to get started with Deno than now. Once deno is installed, run the deno jupyter kernel installation:

deno jupyter --unstable --install

Enabling Deno in VS Code

Since Deno supports web standards like fetch out of the box, loading data is simple and easy.

That’s so fetch

Not only is this a fast & secure runtime, it’s fun. The timing is great too — DataFrames aren’t just for Julia, Python, R, and Rust. Polars provides DataFrames for JavaScript.

Manipulating DataFrames in JavaScript

What’s surprising me is how easily I can use D3 with DataFrames.

D3 and Polars playing well together in a notebook

We can draw plots directly to a canvas.

as well as use any library that emits SVG

This enables a broad range of JavaScript developers and data professionals to iterate in a notebook writing documentation that runs.

If you want to explore and visualize data with JavaScript/TypeScript, check out Deno and their docs on getting started with Jupyter.

If you know Rust, come hack on Deno. If you don’t know rust, like me, come hack on Deno anyway. You can learn. The Deno team is a friendly group of people who believe deeply in the power of TypeScript, Jupyter, and standards.

You can also contribute by filing bugs, writing notebooks, and sharing the Deno kernel with others.

Jupyter Denos

Happy Notebooking!

anywidget: Jupyter Widgets Made Easy

2023-07-11T23:09:00+00:00

anywidget is a Python library that makes it simple and enjoyable to create custom Jupyter Widgets that run in classic Jupyter notebooks, JupyterLite, JupyterLab, Google Colab, VS Code, and more. It focuses on:

Prototyping: Create custom widgets right within your notebook.
Portability: Share widgets as Python scripts or pip-installable packages.
Productivity: Enjoy a modern front-end developer experience that supports real-time updates and removes manual project setup and other boilerplate.

anywidget enables real-time widget development entirely from within JupyterLab

Skip ahead to see an example.

Why making widgets is hard

Jupyter Widgets enrich notebooks with interactive JavaScript-based views and controls for Python objects in the Jupyter kernel. They enable a wide range of users, from students to professionals, to tailor their programming environment with custom or ready-made tools to interact with their programs and explore data. Consider a machine learning researcher adjusting model parameters with a slider or a computational biologist navigating a genome browser programmatically.

Jupyter Widgets link JavaScript-based views and controls with objects in the Jupyter kernel

A useful feature of widgets is that they may be packaged and distributed as pip-installable modules, offering reusable components to address both general and specialized needs. For example, the ipywidgets project provides basic elements like form controls and layout containers, while numerous community projects offer custom widgets for domain-specific tasks (e.g., bqplot, ipyvolume, ipyleaflet).

Comparing two massive genomic contract matrices within Jupyter using a custom widget from higlass-python

However, the growing number of Jupyter environments supporting .ipynb files complicates custom widget creation and sharing:

Developers must individually adapt front-end code to meet subtle, often undocumented, requirements of each notebook platform.
Basic prototyping involves setting up a local Python package and manually installing extensions.
Proper widget distribution requires deep understanding of both Python and JavaScript packaging.

I’ve expanded on these challenges previously, but to put it simply: traditional widget development has a steep learning curve and maintenance is both error-prone and tedious.

anywidget introduces a fresh approach for creating and sharing custom widgets. It lets you avoid traditional development complexities, simplifying the process and making it easier than ever to start building widgets.

Development without anywidget

anywidget is not a new framework, but rather a compatibility layer around traditional Jupyter Widgets. It utilizes the standard module system now found in web browsers to let widget developers write front-end code that executes universally. Think of anywidget as an adapter that runs your widget’s JavaScript across various notebook environments.

Development with anywidget

With anywidget, a custom widget consists of two components: a Python class and an ECMAScript module (ESM) — or web-standard JavaScript. You just write ESM and anywidget handles the platform-specifics quirks.

It takes less than 20 lines of code to recreate a simple “Hello World” widget.

import anywidget
import traitlets

class ExampleWidget(anywidget.AnyWidget):
    _esm = """
    export function render({ model, el }) {
        el.classList.add("custom-widget");
        function valueChanged() {
            el.textContent = model.get("value");
        }
        valueChanged();
        model.on("change:value", valueChanged);
    }
    """
    _css = """
    .custom-widget {
        background-color: lightseagreen;
        padding: 0px 2px;
    }
    """
    value = traitlets.Unicode("Hello World").tag(sync=True)

ExampleWidget()

You can copy and paste this code directly into Jupyter notebooks, JupyterLite, JupyterLab, Google Colab, or VS Code and it just works.

No installation, build configuration, or bundlers.

By comparison, creating an identical ExampleWidget the traditional way involves forking a Python repo template (including ~50 files), building JavaScript source code with Node.js and webpack, and manually installing the local extensions in classic notebooks or JupyterLab (not compatible with Google Colab or VS Code).

A realistic example

This tutorial demonstrates using anywidget to address a long-standing issue in the Altair: retrieving data from a brush selection back into Python. You can either follow along here or execute the notebook yourself in Colab.

We’ll begin with an interactive scatterplot example from the Altair docs:

import altair as alt
from vega_datasets import data

source = data.cars()
brush = alt.selection_interval()

points = alt.Chart(source).mark_point().encode(
    x="Horsepower",
    y="Miles_per_Gallon",
    color=alt.condition(brush, "Origin", alt.value("lightgray"))
).add_params(
    brush
)

bars = alt.Chart(source).mark_bar().encode(
    y="Origin",
    color="Origin",
    x="count(Origin)"
).transform_filter(
    brush
)

points & bars

Notice how the brush selection on the scatter plot filters the data in the linked bar chart. Neat, but unfortunately we are unable inspect the selected points because the selection is processed in JavaScript, and Altair lacks a mechanism to communicate back to Python.

Sounds like a job for a widget!

Behind the scenes, Altair produces JSON that follows the Vega-Lite visualization grammar. With anywidget, we can create a custom widget to render this validated JSON independently and additionally relay the JavaScript-based selections back to Python.

import anywidget
import traitlets

class ChartWidget(anywidget.AnyWidget):
    _esm = """
    import embed from "https://cdn.jsdelivr.net/npm/vega-embed@6/+esm";
    
    export async function render({ model, el }) {
        let spec = JSON.parse(model.get("spec"));
        let api = await embed(el, spec);
        api.view.addSignalListener(spec.params[0].name, (_, update) => {
            console.log(update);
            model.set("selection", update);
            model.save_changes();
        });
    }
    """
    spec = traitlets.Unicode().tag(sync=True)
    selection = traitlets.Dict().tag(sync=True)

What’s happening here? Our custom ChartWidget is defined by subclassing anywidget.AnyWidget:

_esm (line 6) defines the ECMAScript module for the widget. It imports necessary rendering dependencies (vega-embed) and exports render: a function to displays the Vega-Lite chart and (importantly) links the JavaScript-based selection to Python.
spec and selection (lines 19–20) define stateful properties via traitlets that are accessible by both client JavaScript and Python. These represent the Vega-Lite JSON specification and the JavaScript-based brush selection.

Now our ESM takes care of rendering instead of Altair.

chart_widget = ChartWidget(spec=(points & bars).to_json())

# Prints updates log console (JupyterLab: View > Show Log Console)
chart_widget.observe(lambda selection: print(selection.new), names=["selection"])

chart_widget

The original cross-filtering behavior stays the same, but now we have access to the JavaScript selection in Python via chart_widget.selection. The Python callback (line 4) prints the synchronized selection in the JupyterLab log console any time it changes.

Finally, we can present this data more effectively using a second widget that displays our selection as pd.DataFrame.

import ipywidgets

output = ipywidgets.Output()

@output.capture(clear_output=True)
def on_change(change):
    df = source
    selection = change.new
    for field, (lower, upper) in selection.items():
        df = df[(df[field] > lower) & (df[field] < upper)]
    display(df)

chart_widget.observe(on_change, names=["selection"])
ipywidgets.VBox([chart_widget, output])

The on_change callback (line 6) is invoked whenever the chart_widget.selection changes (line 13). It filters the original data based on the selection bounds and displays the given subset as a table within output.

With just a few lines of code, we enhanced Altair with new functionality using anywidget.

The ChartWidget can be shared in its current state. However, if we add more features, we could transition the JavaScript code from inline strings to separate files, gradually evolving the widget into a fully-fledged Python package. This incremental development is a feature of anywidget, allowing prototypes to grow into robust tools over time.

Modern web development meets Jupyter

anywidget further embraces modern JavaScript to reduce friction and make developing widgets more accessible and fun.

Because anywidget takes care of all the necessary plumbing, you no longer need to setup up a local Python package or manually install extensions to start building. Instead, you can prototype and share widget ideas directly from notebooks — just like regular Python scripts.

Since v0.2, anywidget allows you to use a file path to define your widget’s front-end code (i.e., the _esm and _css attributes). During development, anywidget will monitor for changes and immediately refresh the UI without requiring a full page reload or resetting widget model state.

import anywidget
import traitlets

class ExampleWidget(anywidget.AnyWidget):
    _esm = "index.js"
    _css = "styles.css"
    value = traitlets.Unicode("Hello World").tag(sync=True)

This feature has been popularized by modern web frameworks, but anywidget introduces it for the first time to Jupyter. See this real-time development workflow it in action or try it out yourself!

Real-time widget development entirely from within JupyterLab

Try it out!

anywidget is available on GitHub and PyPI and may be installed via pip:

pip install "anywidget[dev]"

I hope using anywidget is simple and enjoyable. I have found it valuable in my work as a biomedical visualization researcher, and it’s been exciting to see the positive reception from the wider Jupyter community.

Since its release a few months ago, anywidget already been adopted by several notable projects:

Mosaic: an extensible framework for linking interactive views to databases for scalable data processing.
jupyter-scatter: interactive 2D scatter plots that scale to millions of points and support view linking.
Mapwidget: interactive 2D/3D maps using popular JavaScript libraries with bidirectional communication, such as Cesium, Mapbox, MapLibre, Leaflet, and OpenLayers
pyobsplot: a Python interface for Observable Plot that supports Pandas and Polars dataframes.
ipyreact: a Python library for authoring Jupyter Widgets with React components.

… demonstrating its current viability as an alternative to traditional widget development. I am committed to maintaining the simplicity that defines anywidget while exploring ideas to further modernize widgets like:

A simple decorator-based API
Using builtin dataclasses or libraries like Pydantic or msgspec to define widget models and serialization logic
Handling TypeScript and JSX for front-end code

If you’re curious about custom widgets or facing difficulties in widget development, please give anywidget a try and share your experience. Happy coding!

GitHub - anywidget: custom jupyter widgets made easy

I’d like to extend my gratitude to the numerous contributors to our documentation and specifically Talley Lambert for his work towards modernizing widgets from a Python perspective. He’s played a key role in developing our experimental API, which allows more flexible communication between JavaScript and Python without requiring ipywidgets.

Build a Jupyter Widget with React and TypeScript

2021-07-30T19:02:00+00:00

Photo by Adi Goldstein on Unsplash

When using a Jupyter Notebook to work on data science projects, I’ve found small user interface abstractions to be very useful. Scrubbing a slider or uploading a file can be more intuitive than editing a script and running it manually. This is especially true when a notebook is shared with others, since most people already have a basic knowledge about user interface elements.

Jupyter Widgets are fantastic tools for simplifying Jupyter Notebook workflows with custom user interfaces. While widgets are versatile and composable, sometimes the default implementations provided by Jupyter don’t have the exact functionality we are looking for. Luckily, we are able to create Custom Widgets to suit our needs using web technologies. To get an idea of how Custom Widgets work, we are going to build a sleek color picker for JupyterLab using React.

Setup

Before we can start coding, there is some basic boilerplate to download. Start by installing cookiecutter.

pip install cookiecutter

Then download the React widget boilerplate code.

cookiecutter https://github.com/Waidhoferj/jupyter-widget-react-cookiecutter.git

The cookiecutter template will guide us through some setup questions. The author_name, author_email, and github_project_name fields are the most important. We can leave the others blank or go with the default values.

Next, create a development environment using Anaconda:

conda create -n jupyter-react-widget -c conda-forge nodejs yarn python jupyterlab

conda activate jupyter-react-widget

Every package we install for this project will now be contained in a neat bundle.

Inside the project directory created by cookiecutter, run the following script to install the dependencies that we’ll use to build the widget. Then connect our widget environment to JupyterLab:

pip install -e ".[test, examples]"

jupyter labextension develop --overwrite .

yarn run build

Thats it for the setup! Let’s see what the default widget can do.

To test out the widget, run jlpm watch in the project directory and open up JupyterLab with jupyter lab in another terminal window. In JuypterLab, open the notebook file located at <project>/examples/introduction.ipynb. Run all of the notebook cells to examine the state of the widget. In its current form, the widget displays a greeting alongside an input text field. Typing into the input field will update the target of the greeting. The value of the text box is automatically synced with w.value, allowing us to access the input’s contents as a Python string. With the default setup working, we can start designing our own custom widget.

Building a Color Picker

Let’s take a look at the architecture of a Custom Widget. The Jupyter Widget system allows developers to send data between Python and TypeScript using a Model-View-Controller Architecture. The Python Widget Model represents the state of the widget. The TypeScript Controller edits and responds to changes in the Python model based on user input. These updates are rendered in the notebook using React.

Data is passed from the Python model to the TypeScript controller to the React view and back

We can start building our color picker in example.py by adding a color property to the model.

class ExampleWidget(DOMWidget):
    ...
    # Your widget state goes here. Make sure to update the corresponding
    # JavaScript widget state (defaultModelProperties) in widget.ts
    value = Unicode('Jupyter').tag(sync=True)
    color = Unicode('Jupyter').tag(sync=True)

The color property is set equal to the Unicode traitlet, which represents a string of characters. The .tag(sync=True) method syncs the value of the color property with the TypeScript widget state.

In widget.ts, we define the corresponding TypeScript representation of color in the defaultModelProperties object. This object mirrors the structure of the Python model, allowing us to access the model state in our frontend code.

const defaultModelProperties = {
  value: 'Hello World',
  color: '',
};

Add color to the default model properties object

Now that we have defined our model, the rest of the work can be done in React. If you’d like to flex your web development skills and design a custom color picker, go for it! Alternatively, we can get a great plug-and-play interface by installing react-colorful:

jlpm add react-colorful

Create a ColorPicker.tsx file in the src folder. We can build out the component in a few lines of code:

import React from 'react';
import { HexColorPicker } from 'react-colorful';
import { useModelState } from './hooks/widget-model';

export default function ColorPicker() {
  const [color, setColor] = useModelState('color');

  return <HexColorPicker color={color} onChange={setColor} />;
}

The ColorPicker React component

Let’s walk through what’s happening here. At the top of the file, we import React, the react-colorful component, and a hook called useModelState. The useModelState hook operates like the useState hook but instead of accepting a default value as a parameter, it takes the name of a property from our Python model. The hook will automatically update the React view when the referenced property updates in Python. Also, the React widget can directly update the Python model with the setColor function. By passing color and setColor to the HexColorPicker, the Python widget model will reflect the chosen color whenever the user makes a selection in the interface.

To display the color picker in our notebook, add the component to ReactWidget.tsx, which is the top level React file in the widget system. Import the ColorPicker component at the top of the file and make it the return value of ReactWidget. We can delete all of the boilerplate within the ReactWidget component in the process. Additionally, delete the useModelState import at the top of the file, since it is no longer needed.

import ColorPicker from './ColorPicker';
...
function ReactWidget(props: WidgetProps) {
  return <ColorPicker />;
}

Add ColorPicker to ReactWidget

To test this out, restart the python kernel in JupyterLab and refresh the browser window. We will now see a color picker where the input field used to be. Pick a color and print out w.color to get the hex code in Python. Congrats! You now know how to create Jupyter Widgets using React!

The principles that we covered here can be applied to create more complex applications. To add additional properties, define them in both example.py and widget.ts, then interact with the data via the useModelState hook in a React component. Go build some amazing widgets!

Resources

Authoring Custom Jupyter Widgets

2018-03-05T12:12:00+00:00

Jupyter interactive widgets enhance the notebook experience by allowing users to create graphical user interfaces. They enable richer interaction with the data and computing resources.

While the base ipywidgets library comes with a number of controls such as sliders, buttons, and dropdowns, it is in fact much more than a collection of basic controls: it is the foundation of a framework upon which one can build arbitrarily complex interactions.

Examples of custom widget libraries built upon the foundational package are

bqplot, a d3-Jupyter bridge, and a 2-D plotting library following the constructs of the Grammar of Graphics,
ipyleaflet, a leaflet-Jupyter bridge enabling maps visualization in the Jupyter notebook,
pythreejs, a 3-D visualization library bringing the functionalities of Three.js into the Jupyter notebook,
ipyvolume, a 3-D plotting library also based on Three.js enabling volume rendering, quiver plots and much more.

Jupyter Widgets, a Bridge Between Two Continents

Jupyter widgets provide a means to bridge the kernel and the rich ecosystem of JavaScript visualization libraries for the web browser. It is an amazing opportunity for scientific developers to use all these resources in their language of choice.

This article is meant to serve as a guide for developers interested in authoring a custom widget library and bridge the gap between the basic examples of the official documentation and fully-fledged visualization libraries like the ones listed above.

A number of resources are provided alongside the article, including the complete source code of the examples, notebooks, and Binder links.

In this post, we focus on the Python back-end, even though dozens of Jupyter kernels exist. The Python kernel is the reference implementation of the Jupyter protocol and remains the most featureful. We should also mention QuantStack’s Xeus, a native C++ implementation of Jupyter kernel protocol, which supports interactive widgets. Xeus is used as the foundation for the support of Jupyter widgets for the R and C++ kernels.

Xeus: C++ implementation of Jupyter kernel protocol

A Hands-On Guide

While the Jupyter community is thriving, and we start seeing a growth in the number of custom widget libraries as well. Although the learning curve from the examples of the official Jupyter documentation to authoring state-of-the-art libraries like the ones listed above is steep and may be intimidating.

We started on this path a few months ago and benefited from guidance from core Jupyter developers along the way. Now, we would like to share the lessons learned. We hope this will help turn this mountain trail into a new silk road!

Let us start with the ‘Hello World’ example widget from the ipywidgets documentation, before moving on to more advanced use cases.

1 — Improving on the Hello-World Example from the Documentation

This section is derived from the ipywidgets documentation. The code snippets are CC-0 licensed.

The idea behind Jupyter widgets is to enable a bi-directional communication channel between the kernel (back-end) and the JavaScript front-end. In Python, widgets are special objects which are automatically synchronized with a counterpart object in the JavaScript front-end. In the back-end, the change events are handled by the traitlets package, which implements the observer pattern, while on the JavaScript side, this is done with the Backbone.js library.

The front-end implementation follows the MVC (Model View Controller) pattern. This allows the rendering of the same widget in multiple cell outputs, where all views share the same model, analogous to printing out a string variable multiple times.

The official documentation includes a Hello-World example widget, which offers an example of synchronization from Python to JavaScript, but not the other way around. Our jupyter-widget-hello-world-binder example provides a slightly more advanced version of it which demonstrates the bi-directional communication between the Python kernel and the JavaScript front-end. You can experiment with this widget on Binder or simply check out the example notebook with nbviewer.

Let us briefly present the main aspects of the implementation.

The JavaScript front-end defines a custom view by extending the base DOMWidgetView class from the base package:

var HelloView = widgets.DOMWidgetView.extend({
        render: function() {
            this.value_changed();
            this.model.on('change:value', this.value_changed, this);
        },

        value_changed: function() {
            this.el.textContent = this.model.get('value');
        }
    });

The Python back-end extends the corresponding base class from the ipywidgets package:

class HelloWidget(widgets.DOMWidget):
    _view_name = Unicode('HelloView').tag(sync=True)
    _view_module = Unicode('hello').tag(sync=True)
    _view_module_version = Unicode('0.1.0').tag(sync=True)

    value = Unicode('Hello World!').tag(sync=True)

The value attribute get synchronized between the back-end and the front-end.

The value_changed callback, which changes the HTML representation of the widget is attached to changes of the value property with the line:

this.model.on('change:value', this.value_changed, this);

In the Hello-World example from the documentation, there is no way to change the JavaScript value from the notebook front-end. We have added this feature in order to illustrate the bi-directional synchronization between JavaScript and Python. The idea is to trigger an event in the browser which will update the JavaScript model and then automatically - that’s the magic of ipywidgets - the Python back-end. These additional lines trigger the model update:

// update the JavasScript model
this.model.set('value', formElement[0].value);

// sync with Python
this.touch();

Conversely, when changing the value from the Python kernel the corresponding JavaScript value gets updated, as explained above.

Check out the jupyter-widget-hello-world-binder repo for more information. The notebook includes additional details and comments.

2 — First Example Involving Bi-Directional Communication

In the previous section, widgets were entirely defined in the notebook. We now show how to move the implementation outside of the notebook document and produce a proper installable package.

To help you create your own custom widget, the Jupyter team provides a cookiecutter template, producing a custom Jupyter widget library containing all the boilerplate for packaging. The cookiecutter is initialized with the hello-world widget from the documentation.

We used the widget cookiecutter to put the hello-world widget presented in the previous section into a well-organized GitHub repository. It contains

the Python part in the first_widget folder
the JavaScript part in the js folder.

The README provides detailed instructions to install the package, together with information about how to enable the Jupyter extension, and tips for custom widget authors. It delves into the technical details a bit more than this overview blog post.

Now that the basics of two-way synchronization are covered, you can create arbitrarily complex widgets! The key is to identify the data you would like to synchronize between the front-end and the back-end. Then you can set up the events that will update this data when a change is detected using the building blocks above.

Now you can ‘widget-ify’ any JavaScript library!

2.2 — Barebones `setup.py`

The setup.py described in the official documentation tries to automate many of the build steps, at the cost of readability. Fortunately, packaging Jupyter widgets will work with the bare-bones setup.py we provide.

The gain is a clearer, lighter (~100 less lines) setup.py, giving you a better understanding of what is happening.
The drawback is that you need one extra step to install the widget from source.

# Example: d3-slider
$ git clone https://gitlab.com/oscar6echo/jupyter-widget-d3-slider.git
$ cd js
$ npm install
$ cd ..
$ pip install -e .

# Extra line `jupyter nbextension install`
$ jupyter nbextension install --py --symlink --sys-prefix jupyter_widget_d3_slider
$ jupyter nbextension enable --py --sys-prefix jupyter_widget_d3_slider

More importantly we thought that it was clearer to keep the build steps of the Python and JavaScript packages separated. Both build processes are well documented, making the steps easier to follow. Finally, the inclusion of compiled JavaScript bundles in the Python package, is more explicit.

2.3 — Side Note: Naming Conventions

Naming conventions for Python packages are covered by PEP8. They can be a bit tricky in the case of 2-words names like “first-widget”. Where to use underscore (_) or hyphen (-) ? Typically “-” is used in GitHub repositories, and URLs and JavaScript while any folder or file in a Python module can only contain “_”. In the case of a Jupyter widget there is an extra attention point: in the setup.py file, the data_files argument in the setup function is a list of tuples. For each, the first element (representing a path in the filesystem) contains “-” as it relates to JavaScript code while the paths in the second contain “_” as they represent paths in the Python package. For a full example see the first-widget repo and the detailed README.

3 — Increasingly Complex Widgets

We made the following three widgets, gradually adding complexity. The first two examples are meant as educational examples:

d3-slider, a d3.js-based slider,
drawing-pad, a 2-D drawing pad,

We hope that the last example will become more than a demonstration:

ipypivot, a visual Pivot Table UI

In each case, the GitHub repository includes a demo notebook and the required boilerplate for Binder.

3.1 — d3-slider

This custom d3-slider widget wraps a simple custom slider based on the fantastic d3.js library. You can run and try it on the Binder repo or watch it on nbviewer.

d3-slider widget

To install:

$ pip install jupyter_widget_d3_slider

3.2 — drawing-pad

This small drawing pad app, is inspired from this codepen. You can run and try it on the Binder repo or watch it on nbviewer.

drawing pad widget

To install:

$ pip install jupyter-drawing-pad

3.3 — ipypivot

The ipypivot widget, wraps the convenient PivotTable.js library. You can run and try it on the binder repo or watch it on nbviewer.

ipypivot widget

To install:

$ pip install ipypivot

$ conda install -c conda-forge ipypivot

NOTE: The PivotUI widget is a combo of a custom widget and core widgets. This modular approach is more flexible (and looks nicer) but the same features can be made in a single custom widget containing extra buttons and display fields. The alt branch of the repo contains this version.

3.4 — Including JavaScript Callbacks

The ipypivot widget is an example of a widget that transparently wraps a JavaScript library. Transparent in the sense that all parameters of the JavaScript API are exposed, including functions, which are exposed in the form of strings on the Python side. In Python, JavaScript functions can only be strings, so there is an eval() to convert them to actual JavaScript functions.

The benefit is that all the functionalities of the JS libraries are exposed to the Python users with a very thin API. Developers can ‘widget-ify’ a large array of interesting libraries, thereby boosting the productivity of a Jupyter notebook user.

The downside is naturally the security concerns of enabling arbitrary JavaScript code to be injected by the notebook users. It is less a concern in the context of notebooks being shared within a small team of coworkers.

We are currently exploring means to execute the user-provided arbitrary JavaScript function in a sandboxed fashion, for example using the iframe srcdoc field (Cf. this repo for an example, though not in a Jupyter widget context) and messages can be sent back and forth between the main page and an iframe with the Window.postMessage function (see this gist for a bare bones example).

3.5 — Enabling Jupyter Widgets By Default

The jupyter nbextension enable command, arguably cumbersome, has become unnecessary as Jupyter widgets can be enabled by default from notebook version 5.3 (included). See this PR.

In order to be future-proof, all the widgets in this article include the file which triggers this ‘automatic enable’, and require notebook >= 5.3. Thus the pip-installation of our widgets is a one-line command. However, in dev mode, you still need to install and enable the notebook example (see section 3.1 for an example).

If you are working with an older version of notebook (run jupyter notebook --version to check), you will have to run the following command after pip-installing a widget:

$ jupyter nbextension enable --py --sys-prefix name_of_the_widget

4 — Packaging and Publishing

4.1 — PyPI and npm

If you have followed the “best practices” so far, packaging shouldn’t be an issue. Indeed, the cookiecutter provides a template of an easily-packageable widget.

Once your package is ready, you can publish it on:

PyPI for the package to be pip-installable
npmjs for the JavaScript extension, which is necessary to use the widget as a standalone application (outside of the notebook), render it with nbviewer, and also in the JupyterLab context.

To do so, you can follow these instructions in the documentation of first-widget.

4.2 — conda-forge

Conda has several advantages over pip:

It is a general-purpose package manager, which allows for non-python dependencies.
Unlike pip, it also has a real dependency solver, which prevents breaking your environments when updating a single package.
It allows creating virtual environments to isolate your projects.
It allows for one-line installation of Jupyter extension, including the enabling of the extension as a “post-link” script (temporary advantage: see the previous section).

Conda packages are available on different channels. The default channel is administrated by Anaconda Inc. The usually recommended channel to upload open source projects is conda-forge, as this article from Anaconda announces. To add this channel to your conda configuration, run the following command:

conda config --add channels conda-forge

Several steps are needed to publish a package on conda forge, using a so-called ‘recipe’:

writing the recipe, which describes how to build the package along with the dependencies required for building and running it
testing the recipe
publishing the package on the conda-forge GitHub by forking their staged-recipes repo
maintaining the package

The conda doc and the conda-forge doc are very clear and give you much more detailed information about this subject. If you do not want to read the full doc, and jump straight to the necessary information for publishing a new package, you may want to have a look at our first-widget repo. The README contains a section describing the publishing process for conda-forge.

4.3 — Automatic Push Script

The sequence of steps to update the version of a Jupyter widget and do all the pushing to the various repositories is quite long.

run npm prepare to build the js in folder static/
Update version in __meta__.py
push package to pypi
compute package sha256
tag repo with version
push to GitHub / GitLab / Bitbucket including tag
push js to npmjs
push to conda-forge (which includes sha256 hash)

If you want to automate the process we advise to have a look at Maarten Breddels’ releash package (release with relish :-) ), and how it is used in the context of ipyvolume and ipysheet.

4.4 — Binder and nbviewer

nbviewer and Binder are two fantastic tools to share both static and live notebooks.

nbviewer requires a URL to a valid notebook JSON file — typically hosted on GitHub/GitLab. To make widgets render in nbviewer, you need (1) to make the JavaScript package for your widget available on npm, (2) to run the notebook with the corresponding JavaScript extension (with the same version), and (3) to save the notebook widget state (in the ‘Widgets’ tab) before pushing it to GitHub / GitLab.
Binder requires a URL to a GitHub repository containing notebooks and a manifest of the dependencies required to run this notebook, which is used to produce a Docker image including all the resources to run the notebook. Check out the mybinder.org documentation to know how exactly to make use of it. Another resource is the BinderHub documentation if you want to host your own deployment of Binder. Either way we also highly recommend the article Binder 2.0, a Tech Guide.

5 — Conclusion

Hopefully you will have learned something reading this article. We believe in the potential of Jupyter widgets and hope that this intermediate-level article will help getting more people involved in the development of the ecosystem.

Note: This article only covers the case of the classic Jupyter notebook. The integration with JupyterLab will be covered in a future article!

If you find bugs or are interested in improving the example widgets presented here, please do not hesitate contact the authors or open a pull request!

About the Authors

Alphabetical order:

Olivier Borderies, Société Générale
Olivier Coudray, Student at École Polytechnique
Pierre Marion, Student at École Polytechnique

The software presented in this post was built upon the work of a large number of people including the Jupyter team. We are especially grateful to Sylvain Corlay, Jason Grout, Paul Ivanov, and Steven Silvester from the Jupyter Steering Council, as well as Maarten Breddels and Pascal Bugnion.

Jupyter Blog - JavaScript

Catching Jupyter-specific bugs before CI does: announcing jupyter eslint plugin

Motivation

What v1 catches

Plugin structure

Commands and settings

Internationalization

Testing

Getting started in 60 seconds

Help us encode more knowledge

Acknowledgement

About the developers

Useful links

Exploring Open Access Art Using the Deno Jupyter Kernel

The dataset

But wait!

Loading and cleaning the data

Open Access dataset

Determining whether art is in public domain

“Join” to single unified table

Exploring art, who created it, when, and more with plots

Deeper exploration with interactivity

Exploring data with DataFrame viewer anywidgets

Creating a custom <Gallery/> component

Drilling down on when the art was created

Conclusion

Bringing Modern JavaScript to the Jupyter Notebook

anywidget: Jupyter Widgets Made Easy

Why making widgets is hard

A universal widget adapter

A realistic example

Modern web development meets Jupyter

Try it out!

Build a Jupyter Widget with React and TypeScript

Setup

Interacting with the Widget

Building a Color Picker

Resources

Authoring Custom Jupyter Widgets

Jupyter Widgets, a Bridge Between Two Continents

A Hands-On Guide

1 — Improving on the Hello-World Example from the Documentation

2 — First Example Involving Bi-Directional Communication

2.1 — First Widget

2.2 — Barebones setup.py

2.3 — Side Note: Naming Conventions

3 — Increasingly Complex Widgets

3.1 — d3-slider

3.2 — drawing-pad

3.3 — ipypivot

3.4 — Including JavaScript Callbacks

3.5 — Enabling Jupyter Widgets By Default

4 — Packaging and Publishing

4.1 — PyPI and npm

4.2 — conda-forge

4.3 — Automatic Push Script

4.4 — Binder and nbviewer

5 — Conclusion

About the Authors

Creating a custom `<Gallery/>` component

2.2 — Barebones `setup.py`