Jupyter Blog - kernels

From Punch Cards to the Browser: Fortran Comes to JupyterLite

2026-10-01T04:00:00+00:00

Punched card from a Fortran program Z(1) = Y + W(1), plus sorting information in the last 8 columns - Arnold Reinhold, Wikimedia Commons, CC BY-SA 2.5

A language for people who think in formulas

In the early 1950s, programming meant writing numerical machine instructions by hand, one row at a time. John Backus, who led the effort that changed this at IBM, described the experience as “hand-to-hand combat with the machine.”

In 1953 he proposed a different bargain: let scientists write programs the way they already wrote their mathematics — as formulas — and let a machine do the translating. The skeptics said it could not work. Memory was scarce, computer time was precious, and code produced from anything readable by humans could surely never rival code crafted by hand. Three years later, in April 1957, IBM delivered the first FORTRAN compiler for the IBM 704, and the skeptics were answered by the world’s first optimizing compiler, whose output ran nearly as fast as anything a hand-coder could produce.

The name was the whole idea: FORmula TRANslation. Fortran let scientists and engineers work in the vocabulary of their own fields — arrays, floating-point numbers, complex arithmetic — rather than in registers and opcodes. It became the first programming language to be standardized, in 1966, and it has been actively maintained to this day (latest standard is 2023, next one in progress, targeted for 2028). Much of the software that runs on the world’s fastest computers is written in Fortran, and the dense numerical libraries that other languages call for their heavy lifting, LAPACK most prominently, are written in it too. If you have solved a linear system today, in Python or R or Julia, there is a decent chance Fortran did the work.

The other half of scientific computing

But Fortran was born in the batch era. Running a program meant keypunching a deck of cards, handing it to an operator, and coming back hours later, or the next day, to find out whether it had run at all. Computing meant deciding everything in advance.

A second tradition grew out of the opposite frustration: computing as a conversation. Type an expression, look at the result, adjust, repeat. That impulse ran through time-sharing terminals, through Mathematica and MATLAB, and into IPython, first released in 2001, which grew into Project Jupyter. The notebook, with code, results, and prose in a single living document, is now the default home of exploratory data analysis and computational science, and Jupyter’s architecture is deliberately language-agnostic: any language can implement a kernel and join the conversation. Python, Julia, and R most visibly; C++, Lua, and others besides.

In notebooks, though, Fortran has been present but invisible: the compiled layer underneath NumPy and its cousins, reached through wrappers, glimpsed only when a profiler points at it. The language in which so much of numerical computing is actually written has rarely been the language you can sketch an idea in.

Interactive Fortran

That is the gap LFortran was built to close. LFortran is a modern, open-source (BSD-licensed) Fortran compiler built on LLVM, an optimizing compiler infrastructure several generations on from the one Backus’s team delivered for the IBM 704. The question that compiler settled, whether code written for humans could run as fast as code written for the machine, is one its descendants answer routinely; what they rarely offer is a conversation.

What LFortran adds is exactly that second mode: alongside compiling Fortran to fast binaries for production work, it can evaluate code interactively, statement by statement, much like Python, Julia, or MATLAB.

Once a language has an interpreter, a Jupyter kernel is a small step away: from the language’s point of view, the kernel’s job is only to receive a cell of code and return its results. LFortran has shipped with such a kernel for some time. It is built on xeus, a C++ implementation of the Jupyter kernel protocol that already powers a long list of kernels, including xeus-python, xeus-cling, and xeus-r. The protocol plumbing is taken care of, so the kernel can concentrate on Fortran itself, and a single conda install -c conda-forge lfortran gives you both the compiler and the kernel.

Taking the kernel to the browser

Today we are announcing JupyterLite support: the LFortran kernel now runs in JupyterLite, a JupyterLab distribution that runs entirely in the browser. JupyterLite is served as a static website, with no Jupyter server, no container, and nothing provisioned on any machine.

This matters because it changes not just what a notebook is, but how it can be shared. A notebook becomes something you can publish like any web page: static, lightweight, and infinitely scalable. Whether you’re hosting it on GitHub Pages, a university server, or a personal blog, you can serve thousands of users with minimal resources. There is no backend to manage, no accounts to create, no queues to wait in. The gap between “here is a notebook” and “here, try it” disappears, and with it, many of the barriers that have traditionally limited access to computational tools.

This makes JupyterLite ideal for education, where students can experiment with code without installing anything; for live documentation, where readers can tweak and run examples from a project’s docs; and for citizen science, where communities can collaborate on data analysis without specialized infrastructure. Fortran, a language built for heavy computation, is now as accessible as a webpage.

Building the kernel on xeus carried it most of the way to the browser. The same xeus kernel code that ships on conda-forge for the desktop is built for WebAssembly by emscripten-forge and loaded into JupyterLite by jupyterlite-xeus, and many of the JupyterLite kernels available today are xeus kernels: Python, R, C++, Lua. The desktop and browser kernels are the same code.

The last obstacle was the interesting one. In interactive mode, LFortran leans on LLVM’s just-in-time compilation: each cell is compiled as it arrives and executed on the spot. A browser will not do that. WebAssembly keeps code and data in separate address spaces, so a program cannot write machine code into its own memory and then jump to it, and the classic JIT move is unavailable. What the browser does allow is loading a module: new code enters a running WebAssembly program by being handed to the host and instantiated, never by being written into memory. The problem becomes one of routing the compiler’s output through that door.

This is the wall xeus-cpp hit when it brought the clang-repl C++ interpreter to WebAssembly, and the solution here is the one devised there. Instead of executing generated code in place, the kernel passes it to the WebAssembly toolchain, itself compiled to WebAssembly and running in the same tab, which links it into a side module (the browser’s equivalent of a shared library) and loads it into the running kernel. Parsing, compilation, and linking all happen in the browser; the notebook’s program grows one module at a time.

Try it

You can try it right now on notebook.link, with nothing but a browser. Just click HERE! The demo ships with a handful of notebooks to start from, including a Mandelbrot example and one showing rich display output — plots and images rendered straight from Fortran cells.

The kernel itself is available as a package on emscripten-forge, a software distribution for WebAssembly. That means Fortran is now just another entry in an environment.yml: any JupyterLite deployment built with jupyterlite-xeus can include it. The deployment is a static site you can host anywhere (GitHub Pages, a blog) with no server costs and no limits on the number of users. For a Fortran project, this could mean embedding live, executable examples directly in your documentation, letting users experiment with your code without ever leaving the browser.

For a regular Jupyter installation, the kernel is still available the classic way:

conda install -c conda-forge lfortran jupyterlab

Inside the LFortran compiler

Another reason why LFortran is a particularly good fit for the notebook workflow lies in its compiler pipeline: its Abstract Semantic Representation, or ASR. While the AST records the syntax the user wrote, ASR records what the program means, with types, symbols, procedure interfaces, and other semantic information made explicit. LFortran can therefore perform high-level, Fortran-aware transformations on a valid program before lowering it to LLVM IR, leaving LLVM to handle lower-level optimization and code generation.

Fortran source → AST → ASR → LLVM IR → native code / WebAssembly

For interactive use, each notebook cell has its own ASR translation unit. Its symbol table points to the translation unit from the preceding cell, forming a chain through the session. A cell can therefore use variables, modules, and procedures defined earlier, while redefining a name shadows the old definition instead of modifying code that has already been compiled. Cell-level symbols are given distinct generated names, allowing old and new definitions to remain alive at the same time. Once the current cell has been lowered to an LLVM module, it is handed to the native On Request Compilation (ORC) JIT or to the browser executor described above.

Epilogue

When Fortran arrived in 1957, using it meant punching cards, standing in line at the keypunch and the card reader, and waiting for the batch to come back around. Today it means opening a URL, typing a few lines, and pressing Shift+Enter. The formulas look much the same as they did sixty-nine years ago.

What changed is everything around them.

About the authors

Ondřej Čertík is the creator of LFortran, which he started in 2017 and has led ever since; he is also the original author of SymPy and SymEngine, a co-founder of the fortran-lang community, and a Principal Software Engineer at Microsoft.

Anutosh Bhat did the work of bringing this kernel to the browser: he ported both xeus-cpp and the Fortran kernel to WebAssembly, and iterated extensively on the kernel itself. Anutosh is a scientific software engineer at QuantStack, an LLVM maintainer, and a co-author of xeus-cpp; his contributions span clang-repl, LLVM’s WebAssembly backend, and their packaging for emscripten-forge.

Acknowledgements

The work by Anutosh on the porting the LFortran kernel to WebAssembly was funded by QuantStack, the team behind JupyterLite, xeus, and the emscripten-forge distribution.

LFortran itself is the product of many years of sustained work by more than 130 contributors, made possible by the support of the Sovereign Tech Agency; Los Alamos National Laboratory, which supported its early development; GSI Technology; FLOSS/fund; QuantStack; John D. Cook; NumFOCUS; and Google Summer of Code, through which many contributors first joined the project. We also thank the individuals and organizations who have sponsored LFortran through GitHub and OpenCollective.

GNU Octave Meets JupyterLite: Compute Anywhere, Anytime!

2025-10-16T15:04:00+00:00

We are thrilled to announce the newest member of our JupyterLite kernel ecosystem: Xeus-Octave. Xeus-Octave allows you to run GNU Octave code directly on your browser. GNU Octave is a free and open-source Scientific Programming Language that can be used to run Matlab scripts. In this article, we present the challenges encountered when targeting WebAssembly, the current state of the Xeus-Octave kernel, and the future plans for expanding the GNU Octave ecosystem.

Earlier this year, we introduced the JupyterLite kernel for R, Xeus-R-Lite. Much like R, cross-compiling GNU Octave to WebAssembly required the same custom toolchain to enable the compilation of Fortran code, combining LLVM Flang and Emscripten.

Similar to many other mathematically oriented language packages, GNU Octave requires a BLAS/LAPACK implementation. Fortunately, OpenBLAS and the Netlib implementations of BLAS/LAPACK had already been added to the emscripten-forge WebAssembly distribution. Initially, OpenBLAS was the preferred implementation, but for the successful compilation of Octave, Netlib LAPACK was selected as it presented fewer hurdles during the build process.

Cross-Compilation of GNU Octave

One of the complications of cross-compiling Octave to WebAssembly, which had not been encountered with the R source code, was the extensive use of Fortran common symbols blocks in the internal libraries of Octave such as odepack.

C Source: liboctave/external/odepack/slsode.f
C-----------------------------------------------------------------------
C The following internal Common block contains
C (a) variables which are local to any subroutine but whose values must
C     be preserved between calls to the routine ("own" variables), and
C (b) variables which are communicated between subroutines.
C The block SLS001 is declared in subroutines SLSODE, SINTDY, SSTODE,
C SPREPJ, and SSOLSY.
C Groups of variables are replaced by dummy arrays in the Common
C declarations in routines where those variables are not used.
C-----------------------------------------------------------------------
      COMMON /SLS001/ CONIT, CRATE, EL(13), ELCO(13,12),
     1   HOLD, RMAX, TESCO(3,12),
     1   CCMAX, EL0, H, HMIN, HMXI, HU, RC, TN, UROUND,
     2   INIT, MXSTEP, MXHNIL, NHNIL, NSLAST, NYH,
     3   IALTH, IPUP, LMAX, MEO, NQNYH, NSLP,
     3   ICF, IERPJ, IERSL, JCUR, JSTART, KFLAG, L,
     4   LYH, LEWT, LACOR, LSAVF, LWM, LIWM, METH, MITER,
     5   MAXORD, MAXCOR, MSBP, MXNCF, N, NQ, NST, NFE, NJE, NQU

Initially, it was not possible to cross-compile these common blocks to WebAssembly because the latest version of LLVM (v20 at the time of testing) did not support common symbol linkage.

// Source: llvm/lib/MC/MCWasmStreamer.cpp
void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
  llvm_unreachable("Common symbols are not yet implemented for Wasm");
}

As a temporary solution, LLVM was patched with the help of Serge Guelton to simulate common symbols as weak symbols.

void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
-  llvm_unreachable("Common symbols are not yet implemented for Wasm");
+  auto *Symbol = cast<MCSymbolWasm>(S);
+  getAssembler().registerSymbol(*Symbol);
+  Symbol->setWeak(true);
+  Symbol->setExternal(true);
}

A proper solution to enable support of common symbols is currently in progress and will likely be included in the next release of LLVM v22 (see llvm-project/pull/151478). For curious readers, the patched version of LLVM can be found here (linux only).

In addition to the patches for LLVM, GNU Octave required a few minor modifications to target WebAssembly; mainly this entailed disabling the GUI functionalities and consolidating the Fortran function signatures and calling conventions. A full list of patches can be found in the recipe directory on emscripten-forge.

Xeus-Octave

Once GNU Octave had been successfully packaged for WebAssembly, bringing Xeus-Octave to JupyterLite was a simple matter of adding a recipe to emscripten-forge!

Try Xeus-Octave in JupyterLite!

Future Work

For our next steps, the team is planning on expanding the Octave ecosystem by adding Octave packages to both conda-forge and emscripten-forge. The packaging work will require defining a process where Octave packages can be installed in predetermined conda environments, perhaps with some minor modifications to the existing pkg utility.

About the Author

Isabel Paredes, who led the charge on bringing GNU Octave to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the R programming language and the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This project synthesizes work from many open-source contributors.

Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes, Anutosh Bhat, Martin Renou, Ian Thomas, Wolf Vollprecht, and Johan Mabille.
JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Xeus-Octave, the Xeus-based Jupyter kernel for GNU Octave, was created by Giulio Girardi and Antoine Prouvost.

C++ in Jupyter — Interpreting C++ in the Web

2025-06-19T15:29:00+00:00

Scientists and engineers utilize programming languages not only to build software systems but also to drive interactive exploratory workflows. They leverage developer tools to explore and reason through problems effectively. This process involves executing code, examining visualizations, loading data, and running additional code. Rapid iteration is particularly crucial during the exploratory phase of any technical project.

For this reason, the most popular tools for exploratory work tend to be interpreted languages, such as Python or R. The “Read-Eval-Print-Loop” (REPL) places the user at the center of these iterative workflows. While compiled languages such as C++ are essential for performance-sensitive applications, engineers accustomed to interactive tools often struggle in such environments.

At CERN, the C++ programming language has been pivotal to their data analysis platform for particle physics, notably in the ROOT framework. ROOT is underpinned by the Cling C++ interpreter, which is built on the Clang and LLVM stack. The Cling interpreter, co-created by Vassil Vassilev and Axel Naumann in the 2010s, effectively turned C++ into an agile scientific scripting language!

These interactive workflows are where the Jupyter stack shines compared to traditional IDEs. Indeed, the Jupyter project was designed with exploratory computing in mind. Crucially, it was built from the ground up to be language-agnostic, accommodating multiple programming languages seamlessly. In 2017, the Xeus-Cling Jupyter kernel was announced. Developed by Sylvain Corlay, Johan Mabille and Loic Gouarin, it integrated Cling in the Jupyter ecosystem, taking advantage of the rich features of Jupyter, such as rich mime type display and interactive widgets.

An example C++ Jupyter notebook powered by the original Xeus-Cling kernel

Packaging the Cling project proved challenging as it required patching the code base of Clang, and was not released as frequently as the LLVM stack, causing difficulties with rolling software distributions like conda-forge.

In an effort led by Cling co-creator Vassil Vassilev, the Clang-Repl project was born, which consists in building the foundations for interactive C++ in the core of Clang. Clang-Repl has been undergoing active development for a few years now and a Jupyter kernel for Clang-Repl, Xeus-Cpp, has also been developed, providing a similar set of features to Xeus-Cling.

Building the new C++ kernel upon Xeus was crucial as it enabled the porting of the C++ kernel to the Web browser with JupyterLite.

Scaling Jupyter with JupyterLite

The standard approach for Jupyter-based computing relies on providing each user with computing resources in the backend. Serving a few thousand end users necessitates substantial resources and complex deployment strategies.

The JupyterLite project shifts this paradigm by running code directly in the user’s Web browser, by relying on a WebAssembly build of the kernel and required libraries. This eliminates the linear relationship between the number of users and the required resources.

For instance, the NumPy.org website features a JupyterLite code console on its main page, offering a computational environment to thousands of monthly visitors without incurring any cloud costs. Similarly, the Capytale deployment of Jupyter, used in French high schools, operates on the same model. It accounts for over half a million registered users and hosts more than 200,000 user sessions weekly.

In order to provide a broad range of packages and libraries for this new platform, the JupyterLite team at QuantStack developed the Emscripten-forge distribution, utilizing the conda package manager for creating WebAssembly environments. Emscripten-forge comprises the main packages of the Python scientific computing ecosystem, as well as the R programming language, and several command-line applications powering the JupyterLite terminal.

Today, we are proud to announce that Xeus-Cpp and LLVM are available in emscripten-forge, effectively enabling interpreted C++ in the browser.

The new C++ Jupyter kernel in action in JupyterLite.

Feel free to try it out in your browser by clicking the link below!

We now dive into the details on the internals of the WebAssembly build of Xeus-Cpp, and the key features of the project.

Walking up the stack

Emulated Just-in-Time Compilation and WebAssembly

A fundamental challenge inherent to the WebAssembly is that it operates in a sandboxed Harvard architecture: code and data reside in completely distinct memory spaces. This makes conventional JIT compilation unfeasible in the browser, as it prohibits dynamic modification of executable memory.

The WebAssembly backend for Clang-Repl was introduced during the LLVM 17 development cycle, with the introduction of a WASM-specific IncrementalExecutor, sidestepping the standard LLVM JIT approach used on native platforms. This new WasmIncrementalExecutor class handled the WebAssembly-specific execution model as follows:

The LLVM IR from the REPL’s Partial Translation Unit (PTU) is compiled into an object file.
This object file is passed to wasm-ld with flags to generate a standalone WebAssembly module.
The resulting .wasm module acts as a dynamically linked side module, designed to be loaded on top of a persistent main module at runtime.

These side modules, while similar to shared libraries, are treated as auxiliary WASM modules that are dynamically attached to the main application using Emscripten’s dlopen mechanism. They share memory with the main module and can resolve symbols defined in previous executions. As a result, each REPL input produces a corresponding .wasm module that is compiled, linked, and loaded at runtime, incrementally extending the current execution environment.

Following the initial version in LLVM 17, the support for WebAssembly was incrementally consolidated by a series of pull requests addressing various issues and corner cases with respect to symbol duplication, memory corruption, and dynamic linking failures. We believe the version available today in LLVM 20 is a solid foundation to build upon.

For a deeper understanding of the constraints of the WebAssembly platform for JIT compilation, refer to the GSOC report: Anubhab Ghosh — WASM Clang-Repl.

Porting Xeus-Cpp to Emscripten-forge

In addition to challenges with emulating JIT compilation in WebAssembly, packaging Xeus-Cpp for emscripten-forge presented another challenge: managing shared objects within the initial WebAssembly executable, which are typically built statically for other xeus kernels. This issue was addressed in the following PRS:

As a result of this work, required side modules are now specified as metadata in the Jupyter kernelspec kernel.json file, enabling JupyterLite to automatically detect and utilize them.

Features

Inline documentation

To fully integrate C++ as a first-class citizen within the Jupyter ecosystem, supporting inline documentation is crucial. Similar to the Python kernel, utilizing the ? magic command should allow users to inspect the following type, perform a lookup in the available documentation, and display the relevant information.

The C++ kernel comes equipped with a lookup file for the standard library, making it possible to execute queries like ?std::vector seamlessly.

Inline help feature in the C++ JupyterLite kernel

This can be enabled for third-party libraries. Check out the documentation to enable inline help for your package.

This feature, inherited from the Xeus-Cling project, was initially created by Loic Gouarin.

Rich display

One of the core benefits of integrating C++ into the Jupyter ecosystem is the ability to leverage Jupyter’s rich MIME display system. Rather than limiting output to plain text, Xeus-Cpp allows you to render rich content like images, HTML tables, LaTeX, or even custom visualizations, all directly from C++.

This is made possible by the function xcpp::display, which sends a MIME bundle to the frontend. Each bundle can include multiple representations of the same object, such as text/plain, image/png, or even custom mime types.

This is illustrated in the following example:

Rich mime type rendering feature of the Xeus-Cpp kernel

A custom renderer for any given type can be defined by defining the mime_bundle_repr function in the corresponding namespace, which is then picked up by Xeus-Cpp’s display system through Argument-Dependent Lookup (ADL). A large collection of mime types are supported by the JupyterLab frontend, which can also be extended with plugins to support custom mime types.

Advanced Graphics

Another compelling example leveraging the rich rendering capabilities, is to combine it with frameworks such as SDL for rich graphic content. Emscripten has built-in support for SDL (enabled with the -s USE_SDL=2 compilation flag), which allows us to leverage its power in C++ notebooks.

To illustrate this, we ported Kevin Beason’s legendary smallpt global illumination renderer (a path tracer in 99 lines of C++) into a C++ notebook. The scene is rendered using SDL onto an in-memory canvas, and the resulting image is captured and displayed with xcpp::display.

This setup showcases the seamless integration of SDL and the Jupyter display system:

Use SDL for real-time or offscreen rendering
Then pipe the result through xcpp::display(…) as a PNG or pixel buffer.

PR #299 added an example notebook supporting smallpt with SDL-based rendering. Feel free to access the notebook through our lite link. Since the number of samples per pixel (spp) directly correlates with how long the process runs (approximately 2 minutes in our case), we’ve included debug logs to keep users informed during rendering, so they aren’t left wondering about the progress.

Smallpt tiny raytracer in action in JupyterLite

This approach paves the way for interactive C++ graphics demos, ray tracing notebooks, and even the prototyping of browser-based video games from within WebAssembly C++ notebooks.

Loading Third-Party Libraries

For a user’s perspective, a key difference between a Jupyter kernel for a compiled language like C++ and e.g. Python concerns how third-party libraries can be utilized. In Python, one merely needs to import the relevant installed packages, while in the case of C++, the relevant compiled libraries must be loaded in addition to including the corresponding headers.

Thankfully, Clang-Repl provides the LoadDynamicLibrary to dynamically load shared objects, which was adapted to the WebAssembly based in LLVM PR #133037. Instead of relying directly on raw dlopen calls, this function ensures that memory layout, symbol resolution, and constructor invocation are handled correctly — even in a WebAssembly environment.

Building on top of this, the CppInterOp project made these features available via a high-level C++ API.

Showcasing the dynamic loading of shared objects with CppInterOp

Library authors interested in facilitating the use of their package with the Jupyter kernel can include the calls to CppInterop’s LoadLibrary can simply be included in user-facing headers.

This has been done for example, in the Symengine library.

Symbolic Computing with Symengine

A compelling example of the Jupyter rich display system’s advanced capabilities with Xeus-Cpp is the SymEngine package, a powerful C++ library for symbolic computing.

Originally designed as a backend for symbolic Python libraries like SymPy, SymEngine offers native support for expression trees, calculus and Latex/MathJax rendering of mathematical expressions. We have integrated SymEngine into the emscripten-forge distribution, making it readily available for use. Once installed, it works seamlessly with Xeus-Cpp, requiring no additional setup.

SymEngine expressions can be rendered via xcpp::display(), which produces LaTeX-formatted expressions and delegates their rendering to MathJax in the frontend.

Rich mime type rendering used with Symengine

Array-based Computing

Jupyter notebooks are particularly popular among data scientists, students, and practitioners of scientific computing. Array-based computing is at the center of the practice, be it in Python with NumPy, in R, GNU Octave, or Julia.

A framework for array-based computing is C++ is Xtensor. Xtensor’s syntax closely mirrors that of NumPy, supporting features like array broadcasting, lazy evaluation, and element-wise operations with an idiomatic C++ API. Using Xtensor in combination with the C++ Jupyter kernel provides a similar user experience to that of NumPy with the Python Jupyter kernel.

JupyterLite used in combination with Xtensor, utilizing rich display for tables.

Built on top of Xtensor, Xtensor-BLAS brings the high-level syntax of Xtensor with the performance of OpenBLAS. It provides convenient, NumPy-style access to BLAS and LAPACK routines making operations like matrix inversion or system solving easy to express and execute.

JupyterLite used in combination with Xtensor-BLAS

To use these libraries inside a JupyterLite notebook, all you need is to include Xtensor-BLAS in your environment (available via emscripten-forge). Under the hood, Xtensor-BLAS dynamically loads libopenblas, which is bundled automatically as a dependency.

SIMD Acceleration

We can take a step further with SIMD (Single Instruction, Multiple Data) acceleration.

Using raw WebAssembly intrinsics : WebAssembly includes 128-bit SIMD instructions, which are provided in Clang’s <wasm_simd128.h> header. They can be used to perform vectorized operations directly.

Xeus-Cpp kernel showcasing the use of SIMD instructions

Using Xsimd:

Rewriting specialized vectorized routines for WebAssembly can be cumbersome. This is where frameworks like Xsimd come in. It provides a high-level SIMD API in C++ that abstracts platform-specific SIMD instructions. With Xsimd, you can efficiently operate on batches of numbers using familiar arithmetic operators or mathematical functions, while benefiting from optimized performance under the hood.

Xsimd powers SIMD acceleration in Xtensor and is also used in many other projects, including Apache Arrow, Firefox, Velox, Pythran, and Krita. Here’s how the same example would look when when using Xsimd instead of raw intrinsics:

Xeus-Cpp kernel showcasing the Xsimd library

Magic Commands

In addition to interpreting standard C++ code, Xeus-Cpp also supports magic commands, special notebook commands prefixed with % or %% that allow for extended functionality not defined by the C++ language itself.

These magics let you interact with the runtime environment, manage files, or perform meta-operations, all from within a code cell. The following magics are available:

%file : The %file magic enables file creation and editing directly from a code cell. You can append to an existing file using the -a flag

Using the %file magics in Xeus-Cpp-lite
%timeit : A %timeit magic has also been prototyped via PR #289 by @kr-2003, enabling performance measurement for expressions in the notebook.

However, due to the lack of last value printing support in Clang-Repl, this feature currently relies on a workaround and hasn’t yet been merged. Once upstream support is in place, %timeit will be integrated cleanly and available out of the box.

Here’s a preview of the %timeit magic in action:

Using the %timeit magics in Xeus-Cpp-lite

Magic commands open the door to a more expressive and interactive notebook experience for C++. While support is currently limited to a few core magics, this is just the beginning — with more capabilities and enhancements expected in the near future.

Interactive Widgets

Beyond rich mime type rendering and inline documentation, the Jupyter widgets system allows users to create interactive components in the Jupyter notebook, leveraging by-directional communication with the kernel for more interactivity.

The xwidgets package is a C++ implementation of the Jupyter widget protocol, comprising an implementation of all the core Jupyter widgets, and which underlies other interactive visualization libraries like xleaflet or xcanvas.

Here are a couple examples demonstrating interactive widgets through xwidgets & xcanvas with Xeus-Cpp-lite.

Displaying a slider using xwidgets

Displaying a canvas through xcanvas

If you inspect the codebase of xwidgets or xcanvas, you will notice that it contains very little code. All that is required on the C++ end is a declaration of the attributes that should be synchronized with the frontend, and serialization methods for complex data types.

Most of the programmatic logic for Jupyter interactive widgets lies in the frontend implementation. The consequence for the C++ Jupyter kernel is that with seemingly little implementation effort, we could enable an entire ecosystem of widget libraries, covering a large range of use cases. Interactive widgets for 2-D plotting (bqplot, plotly) and 3-D plotting (ipyvolume), GIS (ipyleaflet), efficient rendering of meshes (ipygany), graph visualization (ipycytoscape), and many more could be almost automatically connected to the C++ world.

How to deploy JupyterLite with Xeus-Cpp

The GitHub repository https://github.com/jupyterlite/xeus-lite-demo is a template for creating a JupyterLite deployment on GitHub pages that includes the packages specified in a conda environment.

The process is as follows:

Create a new repository from the GitHub template.
Enable the deployment on GitHub pages from a GitHub action, as shown in the README.
Edit the environment file to include the desired packages.

For example, to deploy an C++ kernel with Symengine & Xtensor-blas installed, the environment.yml file would contain the following:

name: xeus-cpp
channels:
  - https://repo.prefix.dev/emscripten-forge-dev
  - conda-forge
dependencies:
  - xeus-cpp
  - symengine
  - xtensor-blas

The deployment linked at the beginning of this article was created with this template.

Future work

The journey of interactive C++ in Jupyter, both natively and in the browser, is still evolving. While Xeus-Cpp already offer powerful foundations, several exciting developments are underway that will further expand their capabilities across platforms

Debugger Support

Currently, only a few Jupyter kernels like Xeus-Python offer full integration with the Jupyter Debug Protocol to enable the Jupyterlab debugger. We’re working to bring that experience to C++ as well.

This summer, thanks to Google Summer of Code 2025, Abhinav Kumar will be implementing native debugging support in Xeus-Cpp.The approach is based on using LLDB and its Debug Adapter Protocol (lldb-dap). Modeled after Xeus-Python, it leverages LLDB’s Clang and JIT debugging capabilities to enable breakpoints, variable inspection, and step-through execution directly within the JupyterLab debugger interface.

We already have a proof-of-concept demonstrating that debugging C++ in Jupyter is possible. You can track this effort here: xeus-cpp issue #282.

In the longer term, we aim to explore extending debugging to the browser through Xeus-Cpp.

Cuda Support in Xeus-Cpp

CUDA support in Clang-Repl was broken by the time LLVM 20 was released, but that changed with this upstream patch:

LLVM PR: #136404 — Enable CUDA support in clang-repl

We’re now preparing to bring CUDA support directly to Xeus-Cpp, enabling GPU programming from a Jupyter notebook. Early planning is underway:

GitHub issue: #300 — CUDA REPL support in xeus-cpp

Last Value Printing

In cling, a widely appreciated feature is last value printing, simply writing a variable at the end of a cell automatically displays its value, without needing any manual printing. This behavior naturally carried over to xeus-cling, making interactive exploration even smoother.

This feature stands almost done in Clang-Repl:

LLVM PR: #84769 — Last Value Printing in clang-repl

Once merged, this will be integrated into Xeus-Cpp, enabling seamless REPL experience for C++ users.

How to contribute?

Whether you’re a developer, package maintainer, educator, or simply curious about C++ in the browser, contributions are always welcome!

Packaging for the Browser

Many of the capabilities in xeus-cpp-lite rely on packages built for the WebAssembly platform via emscripten-forge. If you’re interested in porting a library or creating a new recipe, please visit emscripten-forge.org for detailed guidelines and examples.

PRs and discussions are welcome at github.com/emscripten-forge, especially if you’re looking to bring more numerical, graphical, or scientific libraries into the browser.

Improving Xeus-Cpp, CppInterOp and Clang-Repl

We welcome contributions to the broader xeus project, which is part of the Project Jupyter ecosystem. Xeus-Cpp in particular is actively seeking contributors, whether to improve kernel behavior, expand feature support, or explore new magic commands and UI integrations.

On the LLVM side, the Clang-Repl component is still under active development and can greatly benefit from community contributions, particularly around WebAssembly backend improvements and incremental execution features.

Areas of Need & Opportunities for Collaboration

In addition to code contributions, we are seeking funding opportunities to advance the project. We’ve identified several areas where focused effort could significantly enhance the ecosystem, and we warmly welcome collaborators or potential funding partners interested in helping us move these efforts forward, including:

Integration testing: LLVM currently lacks tests for Clang-Repl + WebAssembly. Establishing a reliable test harness in this area is both challenging and critical.
Interactive plotting: A robust plotting library or plugin for Xeus-Cpp would make the kernel far more expressive for teaching, data exploration, and scientific computing.
Plugin system for magics: Further decoupling kernel extensions would simplify development and allow community-driven growth. A concrete description of this project is framed on CERN’s website.

If you or your organization shares the vision of making modern C++ more accessible, especially in-browser, we’d love to hear from you.

About the Authors

Anutosh Bhat is a scientific software engineer at QuantStack. His contributions range from the core of LLVM and Xeus, to their packaging for the Emscripten-forge software distribution. In the course of this endeavor, Anutosh was granted commit rights to the LLVM project.

Vassil Vassilev is a Research Software Consultant with Princeton at CERN. As the co-creator of Cling and the lead developer of the Clang-Repl and CppInterOp projects, he has spearheaded the development of interpreted C++ for over a decade.

Acknowledgements

The work by Vassil Vassilev on this project is done in the context of the Compiler Research group, funded by the National Science Foundation grant OAC-2311471.

The work by Anutosh Bhat on this project is funded by QuantStack, as part of a broader initiative to enable the data science ecosystem within the Web browser.

Work on prototyping the WebAssembly infrastructure for Clang-Repl was done by Anubhab Ghosh, supported through Google Summer of Code 2023 via the LLVM mentoring organization.

A significant portion of the Xeus-Cpp codebase was inherited from the Xeus-Cling project and adapted for the Clang-Repl interpreter. This includes work by Sylvain Corlay, Johan Mabille, Loic Gouarin, and Martin Renou.

We are grateful to Thorsten Beier, the creator and lead developer of the emscripten-forge software distribution, for his support in packaging LLVM and Xeus-Cpp.

We extend our gratitude to Martin Renou and Anastasiia Sliusar for devising the processing of conda packages in the frontend, which underlies the support of third-party libraries in WebAssembly environments used by Xeus-Cpp and xeus-r. We are grateful to Johan Mabille for creating xeus and for his code reviews within the xeus stack. We also thank Jeremy Tuloup for creating JupyterLite.

We are grateful to Matthew Barton, Tharun Anandh & Abhinav Kumar for their numerous and significant contributions to the Xeus-Cpp project.

Finally, we are immensely grateful to the authors and maintainers of Emscripten and LLVM for building such remarkable technologies and for their engagement and support during our numerous interactions on GitHub.

Exploring Open Access Art Using the Deno Jupyter Kernel

2025-04-17T15:27:00+00:00

Jupyter notebooks are the de facto standard for interactive computing and data analysis, combining code, prose, and visualizations in a single document.

In fact, this post was written in a notebook!

Deno is a simplified and secure runtime for JavaScript and TypeScript, with built-in tooling and modern defaults. Since v1.37, Deno ships with a built-in Jupyter kernel, making it easy to use TypeScript for data science and interactive computing.

Having worked extensively with computational notebooks (mainly in Python), I find this exciting for these reasons:

Simple setup — The kernel is built into the Deno CLI, so there’s no need for additional installation. Just install Deno.
Improved dependency management — Notebooks run like standalone scripts, and Deno’s support for importing dependencies in the code itself improves dependency management and reproducibility.
A unified ecosystem for interactive data analysis — Jupyter supports rich HTML/CSS/JS outputs, and the JavaScript ecosystem is ideal for interactive UIs. Deno connects the kernel and frontend, making notebooks more powerful and flexible.

In this post, we’ll explore the National Gallery of Art’s Open Access dataset in Jupyter with Deno. We’ll clean and analyze the data, looking at public domain works, artists, origins, and patterns. Along the way, we’ll introduce Deno-compatible tools like npm:nodejs-polars and npm:@observablehq/plot, and wrap up by adding widgets and custom displays for a more interactive experience.

The dataset

The National Gallery of Art (NGA) Open Data Program provides access to over 130,000 artworks and their creators, available on GitHub. This dataset includes valuable metadata such as titles, dates, artists, and classifications, all under a Creative Commons 0 (CC0) license, meaning it’s free to use and share.

The collection spans a wide variety of artworks, from sculptures to paintings, by famous artists like Mary Cassatt, M.C. Escher, Vincent van Gogh, Pablo Picasso, and Georgia O’Keeffe.

However, navigating this resource is challenging. The National Gallery’s website is not very user-friendly, making it difficult to make sense of what is actually in the dataset. It’s nearly impossible to get any high-level insights, like the number of paintings, which time periods are most represented, or which artists are most prevalent–let alone make specific queries.

Fortunately, the dataset is available on GitHub as a collection of related tables, exported as CSV files. For our analysis, we’ll focus on three key tables:

objects.csv — Metadata about artworks, including titles, dates, materials, and classifications.
constituents.csv — Artist details such as names, nationalities, and lifespans.
published_images.csv — Links to artwork images via the NGA’s IIIF API.

We’ll clean and join these tables to create a unified dataset.

But wait!

While these NGA tables are free to use, they only include metadata. The actual artwork images have separate licensing, with only about half being CC0-licensed. We’ll gather this info separately to identify freely available images.

Our goal is to unify the data to identify public domain images, explore subsets, and view available artworks. Maybe you’ll even find an upgrade for that default OS wallpaper.

Loading and cleaning the data

Open Access dataset

To get started, we first need to download the data. If you’ve worked with CSVs in JavaScript before, this typically looks like a “fetch” request followed by some parsing. Here, we’re using jsr:@std/csv to stream the data, which gives us an array of objects.

import * as csv from "jsr:@std/csv";

let baseUrl = new URL(
  "https://github.com/NationalGalleryOfArt/opendata/raw/refs/heads/main/data/",
);
let response = await fetch(new URL("objects.csv", baseUrl));

let objects = await Array.fromAsync(
  response.body.pipeThrough(new TextDecoderStream()).pipeThrough(
    new csv.CsvParseStream({ skipFirstRow: true }),
  ),
  (row) => ({
    objectid: +row.objectid,
    title: row.title,
    year: +row.beginyear,
    medium: row.medium,
    type: row.visualbrowserclassification,
  }),
);

objects.slice(0, 3);

[
  {
    "objectid": 0,
    "title": "Saint James Major",
    "year": 1310,
    "medium": "tempera on panel",
    "type": "painting"
  },
  {
    "objectid": 1,
    "title": "Saint Paul and a Group of Worshippers",
    "year": 1333,
    "medium": "tempera on panel",
    "type": "painting"
  },
  {
    "objectid": 4,
    "title": "Saint Bernard and Saint Catherine of Alexandria with the Virgin of the Annunciation [right panel]",
    "year": 1387,
    "medium": "tempera on poplar panel",
    "type": "painting"
  }
]

If we were just plotting the data, this would work fine. But an array of objects isn’t ideal for building our dataset. The NGA data is relational and needs merging, cleaning, and restructuring. A DataFrame is better suited for our use case, offering higher-level, efficient APIs that eliminate the need to write manipulation functions by hand.

Let’s see how we can load the same dataset using a DataFrame from npm:nodejs-polars:

import * as pl from "npm:nodejs-polars";

let response = await fetch(new URL("objects.csv", baseUrl));

let objects: pl.DataFrame = pl.readCSV(await response.text(), {
  quoteChar: '"',
});

objects = objects.select(
  pl.col("objectid"),
  pl.col("title"),
  pl.col("beginyear").as("year"),
  pl.col("medium"),
  pl.col("visualbrowserclassification").as("type"),
);

objects.head()

Here, we fetch the dataset as before, but instead of parsing the CSV ourselves, we read the text response directly with Polars to create a pl.DataFrame.

We then chain a .select expression to choose and rename the columns we want. Notice how, rather than operating on each row in a loop, with Polars we work directly with columns. This API allows us to express complex, efficient operations (in Rust) without materializing JavaScript objects.

Let’s load our other two datasets as pl.DataFrames as well:

The constituents.csv table contains information about any person or entity associated with a piece of art, such as artists, curators, or collectors.

let response = await fetch(new URL("constituents.csv", baseUrl));
let constituents = pl.readCSV(await response.text(), { quoteChar: '"' })
  .select(
    pl.col("constituentid"),
    pl.col("forwarddisplayname").alias("name"),
    pl.col("visualbrowsernationality").alias("nationality"),
  );

constituents.head();

The published_images.csv table contains additional information about the artwork’s image, including a URL for the thumbnail and mapping the objectid to the IIIF (International Image Interoperability Framework) for the image.

let response = await fetch(new URL("published_images.csv", baseUrl));
let publishedImages = pl.readCSV(await response.text(), { quoteChar: '"' })
  .select(
    pl.col("depictstmsobjectid").alias("objectid"),
    pl.col("uuid"),
    pl.col("iiifthumburl").alias("thumburl"),
  );
publishedImages.head();

The final table, objects_constituents.csv, represents a many-to-many relationship between artworks and the people or entities associated with them. Each artwork may have multiple people linked to it.

For our analysis, we’re interested in selecting the “primary” constituent (i.e., the artist). We define the primary artist as the one with the highest displayorder for each object.

let response = await fetch(new URL("objects_constituents.csv", baseUrl));
let objectToArtist = pl.readCSV(await response.text(), { quoteChar: '"' })
  .filter(pl.col("role").eq(pl.lit("artist")))
  .sort({ by: "displayorder" })
  .groupBy("objectid")
  .first()
  .select("objectid", "constituentid");

objectToArtist.head();

This is a much more advanced usage of Polars, so let’s break down what’s going on:

Filter the table to select only rows where the role is “artist”.
Sort the filtered data by displayorder to ensure all entries are at the top.
Group the rows by objectid, ensuring one row per artwork.
Get the first in each group (i.e., artist with highest display order).
Select just the “objectid” and “constituentid” for joining our tables.

Writing this logic by hand in JavaScript would be tedius, but with Polars it’s declarative and reads like a sentence. This type of chaining is common when working with relational data.

Determining whether art is in public domain

We’ve loaded the relevant tables from the Open Access dataset, but they do not indicate whether images associated with an artwork are in the public domain. While CC0 images can be freely downloaded, shared, and repurposed, not all art with the CC0 license are legally in the public domain everywhere due to local copyright laws and edge cases.

Let’s find a way to add public domain information to our data set.

That information is available on the NGA website, but only through the search user interface. Extracting it manually would be impractical, so I reverse-engineered an API call to retrieve the IDs of artworks with public domain images.

We only need the IDs, not the full data, but the query is slow. Since this isn’t an official API, I’ve saved the results separately. There are about 50k IDs identifying public domain images in our dataset.

// This is not an official API, so I’ve cached the results to avoid 
// repeatedly querying the server.
// 
// let response = await fetch("https://www.nga.gov/bin/ngaweb/collection-search-result/search.pageSize__100000.pageNumber__1.lastFacet__artobj_downloadable.json?artobj_downloadable=Image_download_available"); 
// let data = await response.json();
// Deno.writeTextFileSync("public-domain-ids.txt", data.results.map((object) => object.id).join("\n"));

let publicDomainIds = Deno.readTextFileSync("public-domain-ids.txt")
  .split("\n").map((d) => +d);

“Join” to single unified table

Now that we have all our data cleaned and loaded, we’ll perform a large join to combine all these tables into a single unified table.

Again, this challenging to do manually, but with Polars, we can nicely express these complex join operations. Finally, we use a .isIn expression to derive a new column that indicates whether a piece of art is public domain.

let df = objects
  .join(objectToArtist, { on: "objectid" })
  .join(constituents, { on: "constituentid" })
  .join(publishedImages, { on: "objectid" })
  .select(pl.exclude("constituentid"))
  .withColumns(pl.col("objectid").isIn(publicDomainIds).alias("public"))
  .sort({ by: "public" })
  .sort({ by: "year", descending: true, nullsLast: true });

df.head();

With our unified dataset finally loaded, let’s move on to some plotting.

Exploring art, who created it, when, and more with plots

The NGA portal allows some exploration of these data but focuses on specific artworks. We’re more interested in broader insights — where most art comes from, who created it, when it was made, and whether it’s public domain.

As a JavaScript/TypeScript runtime, Deno gives us access to many visualization libraries to ask these kinds of questions. We’ll use npm:@observablehq/plot (Observable Plot) to create plots, spot patterns, and better understand the collection.

Note this post is not a tutorial on Observable Plot, so don’t worry if some of the code below is not immediately clear. The goal is to demonstrate how to use Observable Plot in Jupyter with Deno to extract insights from our data. I’ll highlight any Deno-specific details or useful tips along the way.

Let’s start by examining the distribution of artworks across different “types” in the NGA collection.

import * as Plot from "npm:@observablehq/plot";
import { document } from "jsr:@manzt/jupyter-helper";

// Convert our DataFrame to Array<Object>
let records = df.toRecords();

Plot.plot({
  width: 900,
  marginLeft: 50,
  color: { legend: true },
  marks: [
    Plot.barY(
      records,
      Plot.groupX({ y: "count" }, {
        x: "type",
        sort: { x: "-y" },
        fill: (d) => d.public ? "Public domain" : "Copyrighted",
      }),
    ),
  ],
  // Provide a custom `document`
  document,
});

For the code above, the key things to keep in mind are:

Custom document — Observable Plot relies on the browser DOM, but since we’re in Deno, we provide a custom document from jsr:@manzt/jupyter-helper to enable rendering.
Converting our Polars DataFrame to records — Observable Plot works best with arrays of objects, so we use df.toRecords() to make the data easier to work with.

For specifics on the Plot API, see the documentation and examples available.

Right away, we can see that prints, drawings, and photographs make up most of the collection. Each category varies in how many images are public domain — images of prints, drawings, sculptures, and paintings are mostly public domain, while images of photographs and portfolios are largely copyrighted.

We can get more granular in our plot very easily with Observable Plot, which is composable and expressive. With just a few tweaks, you can completely change how the data is represented. For example, starting from the plot above, we can modify marks and some encoding fields and produce an entirely different chart, allowing for quick exploration and iteration — like this “waffle” bar chart.

Plot.plot({
  width: 900,
  marginLeft: 50,
  color: { legend: true },
  marks: [
    Plot.waffleY(
      /* changed, Plot.barY */
      records,
      Plot.groupZ(
        /* changed, Plot.groupX */
        { y: "count" },
        {
          fx: "type",
          fill: (d) => d.public ? "Public Domain" : "Copyrighted",
          sort: { fx: "-y" }, /* changed, sort: { x: "-y"  } */
          unit: 300, /* new */
        },
      ),
    ),
    Plot.ruleY([0]),
  ],
  document,
});

With both Polars and Plot, we can ask more specific questions about the data, such as “Which artists have the most artwork in the collection?” Here, we first use Polars to group the data by artist (“name”) and whether the artwork is public domain (“public”), counting the number of works for each. Then, we use Plot to visualize the top 25 artists.

const artworkTotals = df
  .groupBy("name", "public")
  .len()
  .sort("name_count", true)
  .head(25)
  .toRecords();

Plot.plot({
  marginLeft: 200,
  color: { legend: true },
  marks: [
    Plot.barX(artworkTotals, {
      x: "name_count",
      y: "name",
      sort: { y: "-x" },
      fill: (d) => d.public ? "Public domain" : "Copyrighted",
    }),
  ],
  document,
});

Again, we can draw some high-level takeaways from this plot: The photographer Robert Frank has nearly twice as many works as the next highest artist. Some entries, like “American 20th Century” and “German 15th Century,” represent groups rather than individuals. Additionally, an artist’s work is typically either entirely public domain or not at all — likely because public domain status is determined at the collection level, with institutions clearing entire groups of works at once rather than evaluating individual pieces separately.

With our tidy and clean dataset, we can quickly iterate in Deno to ask both broad questions (“What even is in this collection?”) and very specific ones (“Who created this artwork in a given year?”).

Deeper exploration with interactivity

So far, we’ve used static plots to explore our data, but Jupyter notebooks support rich, interactive outputs using HTML, CSS, and JavaScript.

With Deno, we can add more interactivity to our data exploration by either using off-the-shelf interactive anywidgets by quickly building custom components for visualizing subsets in ways prebuilt tools don’t support. Deno provides direct access to the full web ecosystem, making custom visualizations seamless.

In this section, we’ll look at:

Prebuilt interactive table anywidgets for exploring data
Create custom <Gallery /> component to visualize subsets of the dataset

Exploring data with DataFrame viewer anywidgets

In Jupyter, “widgets” extend notebook outputs with interactive views and controls for objects in the kernel (i.e., backend). Unlike standard outputs, they consist of both kernel-side and front-end code, communicating directly via custom messages. While the Jupyter Widget ecosystem is largely Python-centric, the anywidget project provides a kernel-agnostic way to create and share these components.

With anywidget, widgets can be published to JSR and used in the Deno Jupyter kernel. For example, jsr:@manzt/jupyter-helper provides some off-the-shelf anywidgets for interactively viewing Polars data frames.

import { agGrid } from "jsr:@manzt/jupyter-helper";

agGrid(df.head(200)); // just look at the first 200 items

The agGrid export renders the pl.DataFrame using the popular AG Grid library, allowing for interactive sorting, filtering, and pagination. This default view is minimal but can be easily extended with AG Grid’s many features.

import { quak } from "jsr:@manzt/jupyter-helper";

const paintings = df.filter(pl.col("type").eq(pl.lit("painting")));
quak(paintings);

The quak export renders the pl.DataFrame with the quak data table viewer. Like agGrid, it also includes summary visualizations above each column, showing 1D distributions, and can scale to much more data (e.g., the entire collection). These 1D summary visualizations are interactive, enabling cross-filtering across columns.

Together, the programmatic and interactive layers enable unique hybrid workflows working with data. Some queries are easier to express in code, while others are more intuitive through direct manipulation. In notebooks, the two approaches come together naturally.

For example, Polars can be used to subset the data to just public domain paintings (~2k out of ~115k artworks), and then a date range can be selected interactively by brushing in the UI to explore variation across other columns.

Creating a custom `<Gallery/>` component

Deno gives us direct access to the web ecosystem, making it easy to build small, useful UIs for exploring our data in ways that off-the-shelf tools don’t support.

For example, it would be helpful to actually see a set of artwork in a “gallery view” to provide intuition about what’s in that subset. We can do that since the dataset contains image URLs.

Let’s create a custom JSX component that server-side renders (SSR) a gallery view for any dataset subset using the render export from jsr:@manzt/jupyter-helper. This helper simply converts JSX to an HTML string, which we display using the Deno.jupyter.Displayable interface.

import * as React from "npm:react";
import { render } from "jsr:@manzt/jupyter-helper";

function Gallery({ objects, size = 100 }) {
  return (
    <div
      style={{
        display: "grid",
        gridTemplateColumns: `repeat(auto-fill, minmax(${size}px, 1fr))`,
        gap: "4px",
      }}
    >
      {objects.select("objectid", "thumburl", "title", "public")
        .map(([objectid, thumburl, title, publicDomain]) => (
          <div
            key={objectid}
            style={{ position: "relative", textAlign: "center" }}
          >
            <a
              href={`https://www.nga.gov/collection/art-object-page.${objectid}.html`}
              style={{
                display: "block",
                width: `${size}px`,
                height: `${size}px`,
                position: "relative",
              }}
            >
              <img
                src={thumburl}
                alt={title}
                style={{
                  width: "100%",
                  height: "100%",
                  objectFit: "cover",
                  borderRadius: "5px",
                }}
              />{" "}
              {publicDomain && (
                <img
                  src="https://mirrors.creativecommons.org/presskit/icons/zero.svg"
                  alt="Public Domain"
                  style={{
                    position: "absolute",
                    bottom: "3px",
                    right: "3px",
                    width: "20px",
                    height: "20px",
                    opacity: 0.60,
                  }}
                />
              )}
            </a>
          </div>
        ))}
    </div>
  );
}

Gallery is a custom component that displays a grid of artwork thumbnails from a given dataset. Each image links to its NGA collection page, and public domain works are marked with a CC0 icon.

We can SSR Gallery with our data to better understand a smaller subset, making it easier to explore specific artists, time periods, or categories visually. For example, a random sample from all the collection:

render(<Gallery objects={df.sample(20)} />);

Or sample of just paintings:

render(<Gallery objects={paintings.sample(20)} />);

It’s worth taking a moment to appreciate that in just a few lines of code, we’ve built a domain-specific utility that makes our data more intuitive to explore. This kind of flexibility is powerful and unique to Deno, letting us quickly craft custom utilities that provide deeper insights.

With our toolkit, let’s take a closer look at the NGA dataset:

Drilling down on when the art was created

So far, our data exploration has focused on high-level summaries of the NGA collection by artist and artwork type using static plots. Now, let’s drill down into something potentially more interesting using both our complete toolkit: when these works were created.

We’ll start with a stacked histogram to see the overall distribution of artwork types over time, then facet by public domain status to uncover any patterns.

Plot.plot({
  y: { grid: true },
  color: { legend: true },
  marks: [
    Plot.rectY(
      records.filter((r) => r.year > 1401), // 15th century or later
      Plot.binX({ y: "count" }, {
        x: (d) => new Date(d.year, 0, 1),
        fill: "type",
        fy: (d) => d.public ? "Public Domain" : "Copyrighted",
      }),
    ),
    Plot.ruleY([0]),
  ],
  marginLeft: 100,
  marginRight: 100,
  width: 1000,
  height: 400,
  document,
});

The distribution of artworks with copyrighted and public domain images are noticeably different. Nearly all copyrighted works appear after 1850, while public domain artworks are more evenly spread over time — except for a sharp spike in public domain drawings from the 1940s, with over 15k artworks added.

This is a striking anomaly. Are they all from the same artist? Why only drawings? Let’s take a closer look.

We’ll filter down the public domain data to this span.

let notablePeriod = df.filter("public") // just public domain
  .filter(pl.col("year").gt(pl.lit(1925))) // between 1925 - 1955
  .filter(pl.col("year").lt(pl.lit(1955))).sort({ by: "year" });

Plot.plot({
  y: { grid: true },
  color: { legend: true },
  marks: [Plot.rectY(
    notablePeriod.toRecords(),
    Plot.binX({ y: "count" }, {
      x: (d) => new Date(d.year, 0, 1),
      fill: "type",
    }),
  )],
  document,
});

Zooming into the 1925–1955 range, we see that almost all the data falls within a much narrower window. Interestingly, the spike is highly concentrated between 1935 and 1942, with very little artwork outside this period.

Narrowing in closer, we can see that in this window there are ~18k individual drawings:

const totalNumberOfDrawings = notablePeriod
  .filter(pl.col("year").gt(pl.lit(1934)))
  .filter(pl.col("year").lt(pl.lit(1943)))
  .shape
  .height;

totalNumberOfDrawings; // 18096

made by over 1000 individuals:

const numberOfArtists = notablePeriod
  .filter(pl.col("year").gt(pl.lit(1934)))
  .filter(pl.col("year").lt(pl.lit(1943)))
  .groupBy("name")
  .len()
  .shape
  .height;

numberOfArtists; // 1034

This gets even more interesting! The artwork is not from just one specific source — there was a surge of public works between 1935 and 1942.

Let’s use our custom <Gallery /> view to explore a subset of these works — maybe seeing them will give us some insight.

render(<Gallery objects={notablePeriod.sample(100)} />);

So all these artworks, despite being from different individuals, have a very similar style and medium. That’s something worth digging into — what might explain this pattern?

Thanks to our custom <Gallery /> component, we can click on artworks and view their metadata. Sampling a few, we see that they all belong to the “Index of American Design” collection.

Conceived as an effort to document and preserve American folk and decorative arts, the Index of American Design consists of 18,257 watercolor renderings created between 1935 and 1942 as part of a Federal Art Project (FAP) work-relief program. Around 400 artists meticulously recreated textiles, woodcarvings, weathervanes, and other objects from across the U.S., aiming to establish a distinct American visual lineage. The project was ultimately housed at the National Gallery of Art, becoming a widely exhibited visual archive.

That’s it! We found a large collection of public works funded by a federal program during the Great Depression — clear in both the data and historical context.

Conclusion

Using Deno’s Jupyter kernel, we explored the NGA Open Access dataset, combining static plots, interactive widgets, and custom JSX components to uncover patterns in the data. This led us to a historical insight — the Index of American Design, a public art project from the 1930s.

With Deno, JSX, and anywidget, we combine data analysis with interactive web-based visualization, making it easy to build domain-specific tools for deeper exploration.

R in the Browser: Announcing Our WebAssembly Distribution

2025-02-28T08:31:00+00:00

R is now available in emscripten-forge, enabling the Xeus-R kernel in JupyterLite.

In the past few months, QuantStack has received funding from the Bill & Melinda Gates Foundation through a grant to CourseKata to develop support for the R programming language in JupyterLite and the Emscripten-forge distribution for WebAssembly.

Today, we are thrilled to announce the availability of the R programming language in the Emscripten-Forge distribution for WebAssembly, which enables R workflows in JupyterLite through the Xeus-R Jupyter kernel.

JupyterLite with an R Jupyter notebook in use

To experience R in JupyterLite, simply click on the link below:

In this article, we present the details of our efforts to enable R workflows in JupyterLite. We cover the challenges encountered in packaging R and its dependencies for this platform, the solutions we implemented, and the process of handling R mamba packages on the frontend.

Why WebAssembly? Scaling scientific computing workflows in Jupyter to millions of users

The main challenge in scaling Jupyter deployments lies in providing a live computing environment on the server side for each user session.

While Kubernetes-based JupyterHub deployments have been successfully implemented at a university scale (such as the UC Berkeley Data 8 class, or the JupyterHub deployment of Université Paris Saclay), they demand substantial computing resources and a skilled DevOps team to achieve and maintain this level of scalability.

In contrast, JupyterLite represents a fundamentally different paradigm. As a browser-based distribution of Jupyter, it runs language kernels directly in the browser, enabling significantly greater scalability and ease of deployment. Notable JupyterLite-based applications include:

A JupyterLite console integrated into NumPy’s front page, providing immediate access to NumPy functionality without requiring installation.
A similar console application on SymPy’s project page, fully hosted statically.
The official Jupyter website^[1].
The examples in Scikit-Learn’s documentation.

These examples illustrate how WebAssembly can be transformative by providing ready-to-use computing environments for scientific workflows, in documentation, blog posts, and scientific communications. The WebAssembly environments are made available to the hundreds of thousands of monthly visitors of these websites without requiring any computing resources on the backend.

JupyterLite deployment displayed on the NumPy website

JupyterLite deployment displayed on the SymPy website

JupyterLite deployment displayed on the Jupyter website

Beyond Python, enabling R in the browser, and more

While the initial focus of the JupyterLite team has been on the Python programming language, the JupyterLite infrastructure was designed to be language-agnostic from the outset. The same principle applies to the package management stack we have been contributing to over the past few years, namely, mamba and conda-forge.

When we began targeting the WebAssembly platform, we chose to build our efforts on the mamba and conda-forge ecosystem. This decision allowed us to provide customizable environments for WebAssembly and expand beyond Python. This enabled applications such as the JupyterLite terminal, created by Ian Thomas, which comprises a bash-like shell written in TypeScript called Cockle, and WebAssembly builds of native shell commands such as cat, grep, ls, and vim. You can read the earlier announcement introducing the JupyterLite terminal.

Today, we are excited to reveal the availability of R in emscripten-forge, enabling Jupyter kernels such as Xeus-R to be used in JupyterLite.

With the addition of R, emscripten-forge now provides a unified packaging solution for Python, R, and terminal applications for WebAssembly. This covers the scopes of Pyodide (a Python distribution for WebAssembly), WebR (an R distribution for WebAssembly), and Sandbox.bio (a shell emulator with WebAssembly commands).

Building R for Emscripten-forge

Emscripten-forge is a GitHub organization containing recipes to build conda packages for the emscripten-wasm32 platform; it is also the first distribution of conda packages targeting this platform. It is built upon a modernized stack, leveraging rattler-build (the successor to the conda-build package builder) and mamba. The emscripten-forge project, which was created and is still led by Thorsten Beier, is now a team effort covering a broad range of packages.

In order to build R with emscripten-forge, we first required a toolchain capable of converting Fortran and C/C++ code into WebAssembly. Since R relies on BLAS and LAPACK (which are written in Fortran) and many essential R packages wrap native libraries that use Fortran, this conversion capability was crucial.

Fortran

By default, the emscripten-forge toolchain can cross-compile any C/C++ packages into WebAssembly; however, the first challenge in building the R stack was expanding the toolchain to also compile Fortran code. The two compilers that we considered for this task are LFortran and Flang.

LFortran is a compiler built on top of LLVM and designed from the ground up as a cross-compiler. Despite its potential, LFortran is still in its early stages compared to Flang and does not yet fully support all Fortran features.
LLVM Flang is the Fortran compiler of the LLVM project. Unfortunately, it does not support cross-compilation.

Working with Serge Guelton and Kerim Birgi, we initially experimented with the LFortran compiler and contributed to addressing issues encountered when building Netlib LAPACK. The BLAS implementation from this package served as a valuable testing ground since BLAS is one of the core dependencies for R.

After careful consideration, we found that Flang, with its ability to compile a wider variety of Fortran projects, brought us closer to building the Fortran components of R to WebAssembly. George Stagg, the main developer of the WebR project, has written a detailed article on using Flang to produce WebAssembly, which was adapted to the (then) latest version of LLVM (v18). Inspired by George Stagg’s endeavors, Serge posted upstream contributions to the Flang project (#99465, #99822, #101242, #105589), resolving issues with 32-bit platforms that were impacting our effort to support WebAssembly. All of these changes will be included in LLVM/Flang 20.0.

The remaining patches which cannot be merged upstream at this point involve:

(1) enabling code generation that targets WebAssembly (198c0e1),
(2) encoding target-specific size definitions for various data types given that the host and target platforms have different architectures (1da66cd), and
(3) disabling support for 128-bit floating-point types (f20c7c0).

A version of (2) that could be contributed upstream is in the works, which would be one step closer to making WebAssembly an official target for Flang. Additionally, (3) will no longer be needed starting with Flang v20.

These patches were integrated by Axel Obermeier into the Flang build recipe for the conda-forge distribution, making the modified build accessible to everyone. It can be installed on Linux with mamba or micromamba by executing:

micromamba install conda-forge/label/emscripten::flang libllvm19 --no-channel-priority

The final ingredient for our toolchain involved creating a compatible Fortran runtime library. Since Flang implements its runtime library in C/C++, compiling this library with Emscripten proved to be straightforward. We added a recipe for libflang to emscripten-forge, which packages the runtime library and includes all the aforementioned patches.

In the following example, we invoke Flang on a simple Fortran script, and execute the resulting JavaScript and WebAssembly with NodeJS.

> ls
hello.f90  libFortranRuntime.a
> cat hello.f90
File: hello.f90
program hello
 print *, "Hello, Fortran!"
end program hello
> source ~/emsdk/emsdk_env.sh
> flang-new --target=wasm32-unknown-emscripten -c hello.f90 -o hello.o
> emcc hello.o libFortranRuntime.a -o hello.js
> ls
hello.f90  hello.js  hello.o  hello.wasm  libFortranRuntime.a
> node hello.js
Hello, Fortran!

With a working Fortran-to-WebAssembly toolchain, we managed to compile BLAS and LAPACK (See the LAPACK recipe). But before we could get started with building R, we needed to tackle the remaining R dependencies.

Walking up the stack

In R (v4.4), there are a handful of dependencies that are required to provide basic functionality; these are: libiconv, zlib, bzip2, xz, pcre2, and libcurl. Thanks to the packaging efforts led by Thorsten Beier and Wolf Vollprecht, these dependencies are available on emscripten-forge with the exception of libcurl. Because of its socket-based networking architecture and reliance on system-level resource access, libcurl faces fundamental incompatibilities with WebAssembly’s sandboxed environment constraints. Therefore, libcurl had to be disabled in the R source code. As a consequence, downstream R packages which depend on libcurl cannot be ported to WebAssembly until a suitable libcurl replacement is available.

Apart from the core dependencies, R also requires a BLAS implementation. Although external BLAS libraries can be used with R, the R code base contains linear algebra routines from Netlib BLAS and a subset of LAPACK where both libraries are written in C and Fortran. To simplify configuration, we opted for these internal implementations instead of using external BLAS and LAPACK libraries.

Furthermore, several key optional dependencies were cross-compiled to WebAssembly to enhance R’s graphical capabilities; these include libpng, libtiff, cairo, and pango, along with their respective subdependencies. One particularly challenging aspect of packaging these libraries was the cross-compilation of glib, which both cairo and pango depend on. Compiling glib was only possible thanks to the efforts of Kleis Auke Wolthuizen whose patches made glib compatible with Emscripten, and Johan Mabille who disabled multi-threading.

The R dependencies are summarized in the following table:

Table of R dependencies

Cross-compilation of R

Although R typically configures and builds seamlessly on most unix platforms, cross-compiling is not straightforward because R uses a bootstrap approach; first it creates a minimal version of R which is then used to run several R scripts to complete the build process. Bootstrapping is a challenge because the built binaries for the target platform (WebAssembly) are not executable (at least not directly) from the host platform. Therefore, the r-base package necessitates two build phases. First, we compile R for the host platform (Linux) with GCC and Flang; this generates the R and Rscript executables needed for the second phase. And then, we cross-compile to WebAssembly with Emscripten and Flang.

One of the major challenges of this setup was generating the R Data Base (.rdb) and R Data Index (.rdx) files for the internal packages of r-base (base, compiler, grDevices, graphics, grid, methods, parallel, splines, stats, stats4, tools, and utils). To accomplish this, it was necessary to dynamically load some of the internal shared libraries (tools, grDevices, graphics, utils, stats, and methods) which provide essential functionality to generate the data and index files. However, the R executables we created during the first phase of the build are incompatible with the WebAssembly shared libraries. In order to work around this, we temporarily replaced the WebAssembly libraries of the internal packages with the linux shared libraries we built during the first phase (See cross_libraries.sh). This makeshift solution is also used to cross-compile all other R packages; the setup is carried out by the activation script of the cross-r-base package (See activate-cross-r-base.sh).

Additionally, the R source code required a few modifications to enable cross-compilation to WebAssembly.

The Emscripten platform was added to select the correct configuration for the platform.
The dependency on libcurl was removed (as mentioned in the previous section) and as a consequence, the internet package which relies on libcurl was disabled.
System calls which are not compatible with our WebAssembly environment were disabled.
Cairo replaced Xlib as the default bitmap type to enable graphics.

The complete list of patches applied to R is available here: r-base patches.

The R package ecosystem

Building R itself was just the beginning. The next challenge was to provide a wide array of R packages to create a robust development environment. Fortunately, many packages are written in pure R and are available as “noarch” packages on conda-forge, making them directly compatible with emscripten-forge environments. However, for the R packages that require compilation, a package recipe needs to be created and added to emscripten-forge. Thanks to the help of Anutosh Bhat, the following packages have been compiled and are available to use.

r-askpass     r-digest    r-haven      r-magrittr  r-purrr    r-tidyr
r-base64enc   r-dplyr     r-hexbin     r-mass      r-rcpp     r-tzdb
r-bit         r-ellipsis  r-htmltools  r-matrix    r-readr    r-utf8
r-bit64       r-fansi     r-isoband    r-mgcv      r-rlang    r-vctrs
r-cachem      r-farver    r-jsonlite   r-mime      r-sp       r-vroom
r-cli         r-fastmap   r-later      r-nlme      r-stringi  r-xfun
r-colorspace  r-ggrepel   r-lattice    r-plyr      r-sys      r-yaml
r-data.table  r-glue      r-lazyeval   r-promises  r-tibble

Most of the time, cross-compiling an R package is very simple because the bulk of the work is handled by r-base. Building a package typically boils down to a single line:

$R CMD INSTALL $R_ARGS .

Occasionally, minor patches are applied to the package source code to enable the WebAssembly target platform or to simplify cross-compilation. Some R packages, such as r-nlme, contain Fortran code; for such packages, the custom flang compiler needs to be installed before the package can be cross-compiled. Should any package be needed for your use case, feel free to contribute them to emscripten-forge!

Xeus-R

Once a sufficient set of packages is available for the platform, the next step is to provide a development environment for end users. Our chosen tool for this purpose is Jupyter.

The most popular Jupyter kernel for R is IRkernel, which has been developed over several years. It includes complete ZeroMQ bindings for the R language and comprehensive coverage of the Jupyter protocol. However, since IRkernel is tied to ZeroMQ, we opted to work with the Xeus-R project instead, which was announced last year. Xeus-R was developed by Romain François in collaboration with our team and with JupyterLite in mind. It is built upon the Xeus library, which provides a native implementation of the Jupyter protocol and is adapted to work with the JupyterLite frontend.

Fortunately, the components of IRkernel responsible for the rich representation of R objects and rich display in Jupyter frontends were split into two separate R packages: IRdisplay and repr. These packages can be reused in Xeus-R. As a result, Xeus-R and IRkernel offer the same rich display functionality, ensuring that Jupyter notebooks created with one kernel will work with the other. However, Xeus-R provides a different implementation of the communication layer through jupyterlite-xeus, which enables integration with the JupyterLite stack. This integration was previously implemented for Xeus-Python and Xeus-Lua.

One key aspect of this integration is the populating of the in-memory file system used by JupyterLite kernels with the required runtime dependencies, particularly shared libraries. This process is done upon bootstrapping the kernel. For this purpose, Anastiasiia Sliusar and Martin Renou developed mambajs, which, in combination with jupyterlite-xeus, processes conda packages in the frontend and installs them in the kernel filesystem at startup. In WebAssembly, shared libraries need to be handled with special care in order to function as expected. Drawing inspiration from the Python community, we adapted the pioneering work of Pyodide to properly load shared libraries into the filesystem.

How to make a deployment with custom packages

The GitHub repository https://github.com/jupyterlite/xeus-lite-demo is a template for creating a JupyterLite deployment on GitHub pages that includes the packages specified in a conda environment.

The process is as follows:

Create a new repository from the GitHub template.
Enable the deployment on GitHub pages from a GitHub action, as shown in the README.
Edit the environment file to include the desired packages.

For example, to deploy an R kernel with coursekata installed, the environment.yml file would contain the following:

name: xeus-r
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-r
  - r-coursekata

The deployment linked at the beginning of this article was created with this template.

How to contribute

Contributors are always welcome! New R packages can be requested by opening an issue on emscripten-forge, or added directly by opening a pull request with the desired package recipe. For instructions on how to contribute, please visit https://emscripten-forge.org/.

What is in the works

Currently, we are developing the infrastructure to dynamically install emscripten-forge and conda-forge noarch packages in JupyterLite kernel environments. This will apply to all Xeus kernels that make use of these packages, such as Xeus-Python, Xeus-R, and Xeus-Lua. In the case of R, this may be directly bound to install.packages.

Additionally, we are working on more advanced features for the R kernel, including support for Jupyter interactive widgets and the split of the pure R parts of xeus-r into a separate package, which would facilitate a possible convergence with IRkernel.

Finally, other language kernels based on the same package management stack are also in the works. Stay tuned for future announcements!

About the Author

Isabel Paredes, who led the charge on bringing R to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This effort brought together several endeavors from many open-source developers.

JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes (for the R ecosystem), but also Kerim Birgi, Anutosh Bhat, Martin Renou, Wolf Vollprecht, and Johan Mabille.
Xeus-R, the Xeus-based Jupyter kernel for R, was created by Romain François.
IRDisplay and repr are pure R packages providing rich display and rich mime type rendering of many core R types, which are used in xeus-r. These two packages stem from the IRkernel project, a Jupyter kernel for R written in R, which was started by Thomas Kluyver, and has been maintained over the years by Philipp Angerer and Jan Katins.
MambaJS, the library enabling the processing of conda packages in the frontend, was developed by Anastasiia Sliusar and Martin Renou.
Conda-forge, the main distribution of conda packages, includes a large number of “noarch” recipes, including for R packages, and is developed by a broad community of contributors worldwide.
WebR, an R distribution for WebAssembly, was created by George Stagg. Although not directly used in this project, it documented numerous patches and changes to the Flang compiler that were adapted for this effort.

Funding

The work by Isabel Paredes and other QuantStack team members on bringing R and Xeus-R to Emscripten-Forge was supported by the Bill & Melinda Gates Foundation through a grant to CourseKata. CourseKata is a nonprofit project dedicated to improving statistics and data science education through interactive, research-backed curricula that integrate professional tools like R and Jupyter notebooks. This grant was part of CourseKata’s broader mission to scale its innovative curriculum, equipping students with the skills and confidence to engage with statistics, data science, and coding in meaningful ways.

Even though it powers the deployment of Jupyter on the main Jupyter website, JupyterLite is not an officially sanctioned Jupyter project. ↩︎

Meet Xeus-R: a future-proof Jupyter kernel for R

2024-01-22T17:20:00+00:00

JupyterLab session with a notebook powered by the Xeus-R kernel for R

The Jupyter ecosystem provides a versatile and user-friendly platform for data analysis and scientific computing, it has democratised access to powerful tools for researchers, scientists, educators, and data professionals.

The name Jupyter, a blend of Julia, Python, and R has always hinted at its relevance in multiple scientific programming languages and has since conquered new horizons thanks to its flexible and extensible design.

Underneath the interactive surface of Jupyter lie the Jupyter kernels, a set of language specific engines orchestrating the dynamic execution and exploration of code. Kernels implement the Jupyter protocol, a shared specification of the communication process between notebooks and kernels, defining how messages, commands and outputs are exchanged and processed.

The Xeus framework is a central piece of the puzzle, offering a reference to facilitate Jupyter kernel implementations. The speed, extensibility, and alignment with the official protocol make Xeus a perfect technology to implement new kernels.

Xeus-R

Today, we, a collaborative team led by Romain François and supported by QuantStack, are thrilled to announce the initial release of Xeus-R, a future-proof Jupyter kernel for R. Xeus-R builds upon the components of Xeus to facilitate the use of R in Jupyter notebooks and already supports typical capabilities: R code execution, inspection, completion, and contextual help.

The recent advancements in support for WebAssembly in the R world led by the WebR project, the flexible design of Xeus, and the rise of JupyterLite all contribute to our strong belief that Xeus-R has the potential to deliver Jupyter in-browser experience for data scientists using R.

Xeus-R is currently at the stage of a proof of concept, but we wanted to communicate about it early to broadcast our excitement about its potential, share some of our thoughts, and initiate a conversation.

Relationship with IRkernel

While the IRkernel project paved the way and is the current reference implementation of a Jupyter kernel for R, it does not take advantage of the Xeus infrastructure and includes a complete implementation of the Jupyter protocol, down to the composition of all of the JSON messages and most importantly how they are transported through ZeroMQ sockets.

In contrast, Xeus-R fully embraces and leverages the capabilities of Xeus, while borrowing some higher level concepts from IRkernel and reusing features from its dependent packages IRdisplay and repr.

We believe this path will benefit future development as it gives us the freedom to work on innovations that are specific to the needs of R users, and let Xeus absorb evolutions of the Jupyter protocol.

Key features

Xeus-R implements the basic building blocks of what you would expect from a Jupyter kernel. The results of a code cell, i.e. the value of its last expression are displayed in the output, taking advantage of ANSI escape codes for colours as were made popular by the crayon and later cli R packages.

Traditional and ggplot2-powered graphics are captured and displayed as images. The aspect ratio of plots can be controlled by setting the repr.plot.height and repr.plot.width options either globally, or locally via the cell_options() function.

Thanks to the scaffolding in place in the IRdisplay and repr packages, Xeus-R is also able to emit richer displays through mime bundles.

In contrast with IRkernel, the rich displays need to be explicitly requested with the display() function. There are cases however, such as HTML widgets, where automatic rich display makes the most sense because a text/plain representation is not valuable.

Xeus-R also supports contextual help and includes a basic completion system, based on the internal utils::.getTokenFromLine() system.

Installation and Getting Started

At the time of writing this article, Xeus-R is only available and tested to work on Unix-based systems. We consciously decided to focus initial efforts on features rather than platform comprehensiveness, but we should be able to fix this soon enough.

The easiest way to get up and running with Xeus-R is to install it from conda-forge:

mamba install xeus-r

Instructions to build from source are available in the github repository for Xeus-R: https://github.com/jupyter-xeus/xeus-r

Looking ahead: JupyterLite Integration

Another benefit of a Xeus-based approach is that it has been designed in a way that supports in-browser experience to power JupyterLite. Together with the recent developments around WebR, we can envision R becoming a first-class citizen in the JupyterLite world.

This is an exciting vision and will serve as our North Star ⭐️ for future developments of Xeus-R.

Conclusion

Join us in welcoming Xeus-R, a Jupyter kernel for R that leverages the robust foundation of Xeus architecture. R is a language of paramount importance for Data Science and Scientific Computing and it rightfully deserves its place as a first-class citizen within the Jupyter ecosystem.

We firmly believe that Xeus-R has the potential to become a reference implementation for an R kernel, well positioned to tackle the challenges of the ever-evolving landscape of in-browser Scientific Computing, a promising future led by the JupyterLite and WebR projects.

Acknowledgements

The work by Romain François on Xeus-R was funded by QuantStack.

Author Bio

Romain François is a prominent figure in the R community, recognised for his longstanding contributions to R programming and Data Analysis. With a profound passion for open-source software and commitment to advancing the capabilities of R, he has earned a well-deserved reputation as a respected expert.

Romain’s expertise encompasses a wide spectrum of R-related fields, including package development, data manipulation, and interoperability with other programming languages. He is widely recognized for his work on packages like dplyr, which has become a cornerstone for efficient data manipulation in R, and for his contributions to Rcpp, a library that bridges the power of C++ with R for optimizing R packages and enhancing performance.

Plug your application into the Jupyter world

2023-10-23T06:09:00+00:00

Kernels are a simple but powerful abstraction in the Jupyter architecture. They encapsulate language interpreters and make them accessible through a standardized interface. This is the key to Jupyter’s remarkable versatility, with over 100 supported languages. Embedding a kernel in your custom application can seamlessly expose it to the Jupyter ecosystem. Not only can you attach JupyterLab to your program for e.g. state inspection, as you would do with a debugger, but you can truly extend your application with all the power of the Jupyter ecosystem.

Applications: GUI and API

Several applications such as FreeCAD, Blender, or QGIS provide a Python API to access their functionalities programmatically, and include an embedded Python console which allows using this API from within the application. Users can both use the graphical user interface and operate on the application in a scripted fashion depending on their needs.

Python consoles in FreeCAD, Blender, and QGIS (from left to right).

If you think about it, these are the kinds of workflows that Jupyter has always been about, but Jupyter has evolved in the opposite direction. Instead of adding scripting to GUIs, Jupyter added GUIs to scripting. At first, Jupyter was IPython, a console for interactive computing. Then it went to the Web and became IPython Notebook, renamed to Jupyter Notebook. With the browser it became possible to display richer content and create small GUIs, using Jupyter Widgets. Then JupyterLab was created and allowed to create fully-fledged desktop-like applications in the browser, like 3-D CAD modelers.

JupyterLab showing notebooks using JupyterCAD and ipyleaflet.

Jupyter put a lot of attention on building a good experience interacting with the Python REPL, which is not the main focus of domain-specific desktop applications. The Python console embedded in desktop applications is often quite raw. Allowing users to connect from Jupyter can enable a much better experience.

What if there was a bridge that allowed you to connect both worlds, your application on one side and Jupyter on the other side? What is the common denominator allowing code to interact with an application, while also being shared with Jupyter? In the Jupyter infrastructure, we call it a kernel.

Jupyter kernels

Kernels provide a standardized way to expose language interpreters to applications, in a language-agnostic fashion. You can think of it as a server that accepts requests such as “execute this piece of code” and that replies with a result. The result is actually a data representation, it can be as simple as “3” for an input of “1+2”, or a complex interactive plot. A kernel usually lives in a separate process on your machine, but it can also run in the same process, or even remotely on another machine. Someone wanting to connect to this kernel will need the connection information such as the IP address and the ports of the different communication “channels”. It is stored in what is called a connection file.

The Jupyter kernel protocol allows clients to interact with different languages.

Anyone who has access to this connection file can then execute code in the kernel. This is “shared code”, meaning that every client to this kernel eventually runs code in the same interpreter. If one client sets a variable “a=1”, other clients will see this variable. They are all running the “same program”. If the kernel runs in the same process as the application, then the kernel can interact directly with it. This way of sharing the execution engine serves as a bridge between the application and Jupyter.

Now it is just a matter of exposing the connection file so that e.g. JupyterLab can see it. The application launching the kernel can save the connection file to a specific directory, and give the path to JupyterLab. And that’s it, JupyterLab can run a notebook using this kernel, allowing it to interact with the application.

Getting started

The xeus-qt-python repository is a great place to start and see how to embed a Python kernel inside a Qt application. Under the hood, it uses xeus-qt, an integration of the xeus kernel architecture in the Qt event loop. The Qt application actually runs JupyterLab in a Qt WebEngine window. Users can run code in a notebook and interact with the application, which in turn can interact with the notebook. Here, we create a Qt button from a notebook. When clicking on the button, a callback prints a message in the notebook. This demonstrates a two-way binding between the Qt application and JupyterLab.

Example of a Qt application embedding JupyterLab.

SlicerJupyter from Kitware was one of the first applications to take advantage of this approach. You can read more in this blog post. We hope that this technology will unlock a new brand of use-cases and allow applications to reach a new audience through the Jupyter ecosystem. Please reach out to us if you need help!

About the author

David Brochart is a technical director at QuantStack. He mostly works on the Jupyter backend stack. He contributed to the development of real-time collaboration in JupyterLab, and created the Jupyverse project, a next-generation implementation of the Jupyter server.

Acknowledgments

This work was made possible thanks to several contributors. Mariana Meireles was the initial author of xeus-qt. Thorsten Beier is a scientific software developer at QuantStack and contributes to the xeus kernel ecosystem. He is the initial creator of the xeus-qt-python demonstration. Johan Mabille is a technical director at QuantStack and he is the main author of xeus. Afshin Darian is a technical director at QuantStack and worked on the UI in JupyterLab 4.0 allowing to use an external kernel.

Bringing Modern JavaScript to the Jupyter Notebook

2023-09-24T03:34:00+00:00

Visualizing Data via Deno, TypeScript, and VegaLite in JupyterLab

I never expected this.

using modern JS, TS, and npm in jupyter notebooks

👇

— Deno (@deno_land), September 21, 2023

Deno brings TypeScript, JavaScript, npm, and ES Modules to Jupyter with an easy to install kernel.

The Deno Kernel is the first language runtime with a builtin jupyter kernel.

There’s no better time to get started with Deno than now. Once deno is installed, run the deno jupyter kernel installation:

deno jupyter --unstable --install

Enabling Deno in VS Code

Since Deno supports web standards like fetch out of the box, loading data is simple and easy.

That’s so fetch

Not only is this a fast & secure runtime, it’s fun. The timing is great too — DataFrames aren’t just for Julia, Python, R, and Rust. Polars provides DataFrames for JavaScript.

Manipulating DataFrames in JavaScript

What’s surprising me is how easily I can use D3 with DataFrames.

D3 and Polars playing well together in a notebook

We can draw plots directly to a canvas.

as well as use any library that emits SVG

This enables a broad range of JavaScript developers and data professionals to iterate in a notebook writing documentation that runs.

If you want to explore and visualize data with JavaScript/TypeScript, check out Deno and their docs on getting started with Jupyter.

If you know Rust, come hack on Deno. If you don’t know rust, like me, come hack on Deno anyway. You can learn. The Deno team is a friendly group of people who believe deeply in the power of TypeScript, Jupyter, and standards.

You can also contribute by filing bugs, writing notebooks, and sharing the Deno kernel with others.

Jupyter Denos

Happy Notebooking!

A Jupyter kernel for GNU Octave

2023-01-11T17:33:00+00:00

GNU Octave is a long-standing member of the scientific computing ecosystem, featuring a mathematics-oriented syntax with built-in 2D/3D plotting and visualization tools, and compatible with the Matlab syntax. It is a natural candidate for high-quality integration in the Jupyter ecosystem.

Jupyter has been devised with language agnosticism in mind since it was spun off from the IPython project. The Kernel (the part of the infrastructure responsible for executing the code input by the user) is a key extension point to Jupyter, and dozens of language kernels exist in the ecosystem.

Prior art on Jupyter / GNU Octave integration includes the Calysto/Octave_kernel project by Steven Silvester (who is also a core Jupyter maintainer and the co-creator of JupyterLab). This existing kernel is part of the family of kernels built on top of the ipykernel reference implementation of the Jupyter protocol, in Python.

Today, we are happy to announce the xeus-octave project, a Jupyter kernel for GNU Octave. Xeus-octave was created by Giulio Girardi, recently joined by Antoine Prouvost — and has been incorporated into the Project Jupyter governance.

Xeus-octave is built upon the Xeus library, a C++ implementation of the Jupyter kernel protocol that enables fast development of new kernels using programming languages’ native APIs, unlocking new possibilities without parsing standard outputs or requiring a Python interpreter at runtime. After all, a Jupyter kernel is merely an executable providing a well-defined communication protocol. It is not bound to Python APIs.

A fully-featured kernel

We strove to make the xeus-octave kernel as complete as possible for this first iteration.

Multiple graphical toolkits

Notebook

A notebook Octave graphical toolkit can be used to render plots natively, and present the figure as an image in the notebook.

The classic Octave sombrero plot rendered as an image inside the notebook

Plotly

An experimental plotly Octave graphical toolkit can also use Plotly to render code in the browser, with dynamic views, zooms, and hover tooltips.

Multiple plots rendered in a single plotly figure

Rich display

The kernel binds with the C++ interface of GNU Octave to provide a complete Octave experience. Having access to the exact internal type of the objects, xeus-octave can present data with rich output.

Matrices can be presented as HTML or LaTeX tables.

Symbolic expressions can be presented as LaTeX formulas.

Structured data can be visualized as interactive tree views

Interactive help

Formatted help, extracted from the GNU Octave reference manual, can be queried using the ? syntax.

Trying it online

You can try the new Jupyter kernel for GNU Octave by clicking on the link below:

The Binder linked above includes examples of notebooks in the fields of Electronics, demonstrating the usability of xeus-octave in real-life complex scenarios.

This interactive demo is also featured on the Jupyter website at https://jupyter.org/try#kernels.

Installing xeus-octave

Xeus-octave has been packaged on conda-forge and can be installed with mamba or conda.

$ mamba install xeus-octave -c conda-forge

Future Work

In future iterations, we will work towards consolidating the xeus-octave experience, by adding cell magics and improving the Plotly integration.

A major ongoing development is the implementation of an Octave backend for Jupyter interactive widgets, based on the xwidgets project. Combined with Voilà, it will be possible to use GNU Octave to create standalone web dashboards.

Another ongoing endeavor is to port a growing portion of the scientific computing stack to WebAssembly (using the emscripten-forge project). Producing a Wasm build of GNU Octave will enable its use with Jupyterlite, enabling large-scale deployments on websites, blogs, and arbitrary web pages without any need for scalable cloud infrastructure.

About the authors

Antoine Prouvost is a Scientific Software Engineer at QuantStack and a xeus-octave maintainer. Previously, Antoine was researching Machine Learning and Combinatorial Optimization at the DS4DM research chair and Mila.

Giulio Girardi is a Master’s degree student in Electronic Engineering at the UNIPD, currently working as an Electronic Engineer at Protech Engineering. Giulio is the creator and a maintainer of the xeus-octave project.

Xeus-Lite

2021-10-25T09:40:00+00:00

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. Xeus is C++ library for writing Jupyter kernels. In this blogpost we show how these two can work together.

JupyterLite

With vanilla Jupyter, kernels are running in separate processes. The communication between the kernels and the client is achieved with ZeroMQ. In contrast to this, JupyterLite runs entirely in the browser, including the language kernels. The following kernels are already implemented in the JupyterLite framework:

JavaScript Kernel
P5 Kernel: Since P5 is a JavaScript library, it is easy to provide a P5 kernel running in the browser.
A Python kernel based on pyodide called pyolite: While the two JavaScript based kernels are trivial to implement as in-browser kernels, pyodide is a Python interpreter compiled to WebAssembly via Emscripten. Not only Python itself has been compiled to WebAssembly, but also the whole Scientific Stack including packages like: Numpy, Scipy, Pandas and many more. An overview of packages can be found here. Furthermore any pure Python package can be installed from within pyolite.

Since JupyterLite runs entirely in the browser, it is very easy to try it out!

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing.
With the help of Xeus, it is very easy to write Jupyter kernels . In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-lua is a Xeus-based Lua kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework.
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

Since it is very easy to write Xeus based kernels for Jupyter, we also want to leverage Xeus for writing JupyterLite kernels. This has been the main motivation for Xeus 2.0.

Xeus Architecture

Xeus is has three extension points as depicted below

The Server is the middleware component responsible for receiving and sending messages to the Jupyter client. The default server it is built upon ZeroMQ and handles the concurrency model of the application.
As we will see later, a custom server is needed to connect Xeus with JupyterLite.
The Interpreter provides the interface that kernel authors must implement.
The Debugger provides the interface that can to be implemented when the kernel shall have debug capabilities, but we do not yet support this for Xeus-JupyterLite-Kernels.

JupyterLite Xeus-Kernels

Traditionally, the Jupyter kernel and the Client to the Jupyter kernel run in separate processes as depicted below:

The communication between the kernels and the client (ie the Jupyter-server) are done with inter-process communication via ZeroMQ. But with JupyterLite the kernel runs directly in the browser. In that case, ZeroMQ is not required anymore to communicate with the frontend. This was the motivation for Xeus 2.0, where it is possible to build Xeus without ZeroMQ. To implement JupyterLite Kernels with Xeus, we need to implement a custom Server which handles the Emscripten/Embind-based communication layer

The overall architecture is best described with the picture below: The C++ Xeus-Kernel is compiled to WebAssembly with Emscripten / Embind. The WebAssembly compiled code runs in a dedicated WebWorker thread. The WebWorker communicates via Message Passing with a thin layer of TypeScript which implements the IKernel API of JupyterLite. The communication between JupyterLite and the JupyterLite k ernel is done entirely in JavaScript.

The C++ code needed to compile an existing xeus-kernel to WebAssembly is very minimal:

Furthermore one needs a thin layer of TypeScript / JavaScript to implement the JupyterLite IKernel API. A good example of this code is given in the JupyterLite Xeus-Lua repository. While most of the repositories code is boilerplate, one needs to provide a simple Dockerfile which compiles the Xeus-Kernel to WebAssembly.

Xeus-Lua

We recently released a Xeus based Lua kernel. Even though its worthwhile to have Xeus-Lua kernel for vanilla Jupyter, the main purpose of Xeus-Lua was to experiment with Xeus-based JupyterLite kernels.
We choose Lua as the first language to try out Xeus-based JupyterLite kernels since Lua is easy to compile to WebAssembly.
A JupyterLite instance with Xeus-Lua included can be found here

Xeus-Lua has built in support for interactive widgets

Xeus-Wren

Wren is relatively new “Classy Little scripting Language” created by the author of the great book “Crafting Interpreters”. Wren can be described with the following:

Think Smalltalk in a Lua-sized package with a dash of Erlang and wrapped up in a familiar, modern syntax.

We choose Wren as a second language to experiment with Xeus + JupyterLite since it is small, simple to compile to WebAssembly and easy to embed in C++. A JupyterLite instance with Xeus-Wren included can be found here.

Xeus-Wren has built in support for displaying data of various MIME-types

Outlook

There are many languages which can be compiled to WebAssembly like:

Julia: https://github.com/Keno/julia-wasm
Octave: https://news.ycombinator.com/item?id=22926120
C++: https://github.com/binji/wasm-clang
Python: https://github.com/pyodide/pyodide

Also databases as sqlite can be compiled to WebAssembly:

Sqlite: https://github.com/sql-js/sql.js/

Our goal is to provide Xeus-based JupyterLite kernels for these projects to make the JupyterLite ecosystem even more powerful.

Acknowledgements

We like to thank

Jeremy Tuloup, Madhur Tandon, Martin Renou, and Nicholas Bollweg for their work on JupyterLite.
Johan Mabille for Xeus and his help implementing the Xeus-Emscripten server.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty and vigra in C++ to inferno, kipoi and ilastik in Python.

Xeus 2.0

2021-09-28T09:41:00+00:00

We have just released Xeus 2. This is a major release of the library. While it includes backward-incompatible changes, they are very limited and upgrading your kernels should be relatively easy.

How to upgrade Xeus-based kernels

Let’s illustrate that with the code of a simple kernel before and after upgrading to Version 2. Most typically, only the kernel’s main file will have to be updated:

diff --git a/example/src/main.cpp b/example/src/main.cpp
index 29de437..eb8fdba 100644
--- a/example/src/main.cpp
+++ b/example/src/main.cpp
@@ -21,12 +21,14 @@ int main(int argc, char* argv[])
     std::string file_name = (argc == 1) ? "connection.json" : argv[2];
     xeus::xconfiguration config = xeus::load_configuration(file_name);

+    auto context = xeus::make_context<zmq::context_t>();
+
     // Create interpreter instance
     using interpreter_ptr = std::unique_ptr<custom::custom_interpreter>;
     interpreter_ptr interpreter = interpreter_ptr(new custom::custom_interpreter());

     // Create kernel instance and start it
-    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(interpreter), xeus::make_xserver_zmq);
+    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(context), std::move(interpreter), xeus::make_xserver_zmq);
     kernel.start();

     return 0;

The first change is that you now have to instantiate and pass an additional argument to the kernel constructor: the context. Fortunately, Xeus provides a function to build such a context object, making the update straightforward. We will detail the reason for this change in the next section.

The second change is that the kernel constructor now requires a function to build the server while a default value was provided for it in Xeus 1.x. Again, the motivations for this change will be detailed in the next section. Finally the functions for building the servers have moved to different headers.

To summarize, here is the list of the breaking changes:

The kernel’s constructor now requires an additional argument, the context. Xeus provides a function to easily instantiate it.
The order of kernel’s constructor arguments has changed, and the function for building the server is now required.
The functions for building the servers have moved to the headers where the corresponding servers are defined.

Motivation for these changes

When we created Xeus, kernels were expected to run in separate processes. The communication between the kernels and the client was achieved with the ZeroMQ middleware framework, and it was natural to heavily rely on it in Xeus.

Recent developments have pushed in a different direction, where the kernel can run directly in the browser. In that case, a middleware library is not required anymore to communicate with the frontend. The architecture of Xeus was not suitable for this new kind of kernels. An update was required so that it is possible to build Xeus without the dependency on ZeroMQ.

About the developers

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of Xeus, and developed the debugger extension to xeus-python.

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored Xeus and xeus-python.

From Jupyter to the Moon

2021-09-24T12:24:00+00:00

Jupyter architecture in a nutshell

A key principle in Jupyter’s design is language agnosticism, and one of the main extension points of the Jupyter ecosystem is the kernel, the part of the architecture responsible for executing the user’s code.

In the end, a Jupyter kernel is merely an executable implementing a well-defined inter-process communication protocol.

A reference implementation of the kernel is ipykernel, which provides the default Python kernel, and utilities to build other language kernels. Another reference implementation of the protocol is Xeus.

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing. There is a long list of scripting languages which can be embedded in C/C++. With the help of Xeus, it is very easy to write kernels for such languages. In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework.
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

xwidgets

The xwidget library is a C++ implementation of the Jupyter Interactive Widget protocol. Xwidgets were initially developed for xeus-cling, the Xeus C++ kernel, but any Xeus-based kernel can reuse these components.

A xeus-based Lua kernel

Today, I am pleased to announce the release of xeus-lua, a new Jupyter kernel for Lua based on Xeus.

Lua

Lua, which means “Moon” in Portuguese, is a lightweight embeddable scripting language written in C. While Python is one of the leading scripting languages for scientific application, Lua is the leading scripting language used in games / game engines. The list of video games using Lua is long and includes triple-a games like Crisys and Farcry. Not only games rely on Lua, but also applications like Adobe Photoshop Lightroom.

Sol2

With the Sol2 library, Lua can be embedded into C++ in a very elegant way.

While Lua itself can be embedded in a C++ application quite easy, it is even easier when using Sol2. Sol2 is a C++ library that simplifies the usage of Lua from C++. Furthermore Sol2 can be used to expose C++ classes to Lua, in a similar fashion as pybind11 can be used to expose C++ classes to Python.

Xeus-lua

The compulsory “Hello, World!” from xeus-lua. This blog-post would be incomplete without it.

I am pleased to announce xeus-lua, a new Jupyter kernel for the Lua language based on Xeus. Using Sol2 and Xeus, it was very easy to implement a feature rich kernel using modern C++. With the help of Sol2, it was trivial to expose the xwidgets classes to Lua such that we can use the widgets within xeus-lua notebooks.

But why a Lua kernel?

While Lua might not be the hippest language in 2021, there are multiple reasons while I choose to implement such a Lua kernel, and why such a kernel can be useful:

Writing a Xeus-based kernel is the best way to learn and familiarize oneself with the xeus ecosystem. And since Lua is so easy to build and embed, Lua and Xeus are perfect match. Implementing this kernel gave me great insights into Xeus and Jupyter itself.
Learning Lua with xeus-lua is fun! Before writing xeus-lua, I had zero Lua experience. While implementing xeus-lua I learned Lua from within Jupyter by playing around with xwidgets from within Lua — this was a very joyful experience.
The xeus-lua implementation might serve as reference for future kernel authors. In particular the integration of xwidgets in xeus-lua might be useful for future implementations.
Since Lua is so easy to build, xeus-lua can be built for WebAssembly and will be integrated in JupyterLite — stay tuned!

Features

Obviously xeus-lua supports the execution of code.

Simple code execution

Streams are redirected to the notebooks output

Redirection of streams

Errors messages are shown in the notebooks output

Error Handling

Input streams are redirected to the notebooks input system

User Input

Statements can be auto-completed with a press of tab

Code Completion

Data of different sources as Latex, JSON and html can be displayed.

Display Data

Interactive widgets can be added to the notebook!

Widgets

We can feed the widgets with data!

Binary buffers for widgets

Try It Out

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Other Lua kernels

There are several Lua kernels available:

lupyter: A Python based implementation of a Lua Jupyter kernel. The heavy lifting is done in a C-extension.
ILua: A feature rich implementation. Ilua is implemented in Python and does the communication with Lua via IPC.
IPyLua: IPyLua is a pure Lua implementation.

While these implementations are feature rich and work well, none of the above supports interactive widgets. Since xeus-lua is based on Xeus, we can reuse components from the Xeus ecosystem like xwidgets with minimal effort.

Acknowledgments

We are grateful to Juliette Taka for the illustration.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty, and vigra in C++ to inferno, kipoi and ilastik in Python.

Enabling the JupyterLab debugger with ipykernel

2021-05-13T11:45:00+00:00

Support for the Jupyter Debugger Protocol just landed in ipykernel

JupyterLab 3.0 includes a visual debugger that allows to interactively set breakpoints, step into functions, and inspect variables with any Jupyter kernel that implements the Jupyter debugger protocol.

The first two language kernels to implement the new protocol were xeus-python (a Python kernel) and xeus-robot (a kernel for Robot Framework). Unfortunately, the reference Python kernel, ipykernel, did not support debugging yet, until now!

Today, we are pleased to announce that debugging support landed in ipykernel, and will be available in the next major release, ipykernel 6.0. Pre-releases including the ipykernel debugger are available.

Debugging with ipykernel

What will change with ipykernel 6.0?

Enabling support for debugging in ipykernel required important changes in the code base regarding the concurrency model of the kernel. The main change is that the processing of messages on the “control channel” now happens in a different thread, allowing for the processing to happen while user code is running.

Ipykernel 6.0 includes several other updates. Tornado coroutines were dropped in favor of native coroutines. The Matplotlib inline backend was split into a separate package, and ipykernel depends on debugpy, an implementation of the Debug Adapter Protocol for Python.

If you are interested in testing out the new features, check out the beta release!

pip install ipykernel --pre

Try it now!

Thanks to MyBinder, you can also try it out without the need of installing anything on your computer. Just follow this link:

What about the future?

A lot of new features are in the works with the JupyterLab debugger.

JupyterLab 3.1 will include several usability improvements to the debugger.
It will also add the ability to submit code for execution when stopped at a breakpoint.
We are working on a richer variable explorer, using Jupyter’s rich display system to enable the rich-rendering of variables in the explorer, to e.g. render dataframes as tables.

Finally, we also plan on adding debugging support to other language kernels.

Acknowledgements

The work of Johan and Sylvain at QuantStack on the debugger support in ipykernel was funded by Two Sigma. We are grateful to Min Ragan Kelley and Matthias Bussonnier, who reviewed the pull requests on debugger support.

About the authors

Johan Mabille is a scientific software developer at QuantStack.

Johan is very active in the Jupyter ecosystem, as the creator of xeus, a C++ implementation of the Jupyter protocol, and several language kernels, such as xeus-python, xeus-cling, and xeus-robot. Johan also made contributions to the Jupyter widgets ecosystem and to JupyterLab.

Sylvain Corlay is the founder and CEO of QuantStack.

As an open-source developer, Sylvain is very active in the Jupyter project with contributions in several components of the stack, including widgets, kernels, nbconvert, and others. He is also a steering committee member of the project.

Sylvain also does volunteer work for the community, as member of board of directors of NumFOCUS, co-organizer of the PyData Paris Meetup, and vice-chair of JupyterCon 2020.

Abracadabra! Bringing the magics to xeus-python

2021-02-18T13:30:00+00:00

Last year, we set ourselves to implement a visual debugger for JupyterLab. This endeavor required major developments in the JupyterLab front-end, in core-Jupyter protocols, and on the kernel side (the part of the Jupyter infrastructure responsible for executing the code).

For kernel-side development, we decided to start with the xeus-python kernel, a lightweight implementation of a Jupyter kernel for the Python programming language. Based on xeus, xeus-python has a flexible concurrency model that was required to e.g. add breakpoints while executing code.

Unfortunately, xeus-python did not provide all ipykernel features (magics, Matplotlib support, etc.). Furthermore, many notebooks depend on IPython, as they import it explicitely or make use of the IPython configuration system.

Today, we are proud to announce that xeus-python supports 100% of the IPython magics! This was achieved by leveraging the core IPython package. This is getting us closer to feature parity with ipykernel.

Magics

xeus-python now supports all magics that IPython provides and even user-defined magics!

Support for builtin IPython magics and user-defined magics

Matplotlib

Most Matplotlib back-ends are now supported by xeus-python, allowing you to show static plots with the inline back-end or interactive plots with ipympl in your Notebook:

Matplotlib inline back-end and ipympl interactive back-end

Extending

IPython has a rich configuration system that you can tweak in order to get an enhanced Notebook experience. Xeus-python now supports any configuration you setup for IPython:

Automatically import NumPy in the IPython settings

Visual Debugging

The latest JupyterLab version introduced a visual debugger in its interface. xeus-python was the first Jupyter kernel to add support for it!

Stepping into Python code in JupyterLab with the visual debugger

xeus-python is not the only Jupyter kernel that supports debugging anymore: xeus-robot is a xeus-based Jupyter kernel for the RobotFramework language that supports the JupyterLab visual debugger.

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Installation

You can install the latest xeus-python version using mamba or conda:

mamba install xeus-python -c conda-forge

conda install xeus-python -c conda-forge

You can also compile it yourself, instructions can be found in the repository:
https://github.com/jupyter-xeus/xeus-python

Acknowledgments

My work on xeus and xeus-python at QuantStack was funded by Bloomberg.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and JavaScript/TypeScript.

An SQL solution for Jupyter

2021-02-03T12:03:00+00:00

A few months ago we released xeus-sqlite, a Jupyter kernel that allows users to make SQLite queries directly from the notebook. With the needs of the Jupyter data science community in mind we decided to expand on this project.

RDBMSes supported by xeus-sql

Today, we’re happy to announce the next step of this endeavour, bringing you xeus-sql a general-purpose database access tool for Jupyter, which allows you to make SQL queries on many different databases:

MySQL
PostgreSQL
SQLite3
DB2
Oracle
Firebird
and any database with an ODBC driver.

To offer all these integrations we rely on the SOCI library as the backbone of the project. SOCI abstracts all the different database connections and query details behind a unified C++ API. xeus-sql uses SOCI and xeus to expose SQL functionalities to Jupyter.

Xeus-sql, as well as the MySQL, PostgreSQL and SQLite3 backends of SOCI are available on conda-forge, meaning you can install them easily with conda or mamba:

# Installing xeus-sql and the mysql backend
mamba install xeus-sql soci-mysql -c conda-forge

# Installing xeus-sql and the postgresql backend
mamba install xeus-sql soci-postgresql -c conda-forge

# Installing xeus-sql and the sqlite backend
mamba install xeus-sql soci-sqlite -c conda-forge

There are idiosyncrasies between these different SQL backends which we detail in the xeus-sql documentation and examples.

Visualizing your queries

HTML and text table visualizations for different Jupyter interfaces

You’re probably familiar with visualizing your SQL tables and query results in the form of… tables, of course. Jupyter’s rich display system offers the option of showing them in rich-text display or plain-text display depending on which application you’re using.

Beyond displaying queries with tables, you can now easily create Vega-Lite graphs from your query results directly in the notebook:

Running a series of examples on a notebook

To support this feature xeus-sql relies on xvega, a C++ backend to vega, and custom Jupyter magics for plotting through a mini-language implemented in the xvega-bindings utility library. Besides using the mini-language you can directly provide the JSON specification for your visualization, thanks to contributions by Wang Fenjin.

Vega-Lite is a powerful library and there are many different visualizations you can create from your relational data using xeus-sql. You can see some other examples below:

Examples of different visualizations generated with xeus-sql and xvega-lite by Wang Fenjin

We’re looking forward to exchange with users and learn more about their use cases for xeus-sql and xvega, so if you have any questions please drop them on the public QuantStack chat, or open an issue at the project’s GitHub page. As always, contributions are welcome!

Try it online!

You can try the stable version on Binder without the need of installing anything on your computer just by clicking on the image below:

Acknowledgements

We are grateful to Wang Fenjin for his contributions with respect to the use of vega specs in xeus-sql. The plotting mini-language was built upon the xvega library by Madhur Tandon. We are also grateful to the SOCI contributors for their awesome package that made this work possible.

About the author

My name is Mariana Meireles and I’m a software developer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.

Robotic Process Automation with JupyterLab

2021-01-18T10:36:00+00:00

Robotic Process Automation (RPA) differs from classical automation tools in that the actions to automate can be developed by observing a user perform a task in a graphical user interface, across multiple applications.

It is a means to lower the entry barrier of process automation and enable the use of tools that don’t provide a programmatic interface.

Most typically, RPA developers will use a mixed approach between textual programming and performing actions manually. The resulting programs are typically called software robots. Therefore, interactive computing tools like Jupyter are a natural environment for RPA, as the interactive nature of Jupyter allows for quick iterations and trial-and-errors when developing such robots.

While many RPA tools are commercial software, Robot Framework and the tooling developed by Robocorp provide an open-source RPA programming language, with a high-level syntax, extensible with Python plugins. It has a rich ecosystem of libraries and tools that are developed as separate projects.

Robot Framework and Project Jupyter

Today, we are happy to announce the first release of xeus-robot, a Jupyter kernel for Robot Framework based on xeus, supporting the new JupyterLab 3.0 visual debugger, auto-completion, and much more!

But before we dive into xeus-robot, we should recognize the earlier work by Nick Bollweg and Asko Soukka, who developed two Jupyter kernels for Robot Framework, irobotframework, and robotkernel, and gave an amazing talk together at RoboCon in 2019!

The motivation for us to reboot this effort with a kernel based on xeus was to enable the JupyterLab Visual Debugger for this kernel. This requires a different concurrency model than that of ipykernel, which underlies both irobotframework and robotkernel. In the end, we were able to provide the same features and more, including e.g. code completion in Python cells, debugging etc.

Diving into Xeus-robot

xeus-robot is a reboot of the already existing robotkernel, based on xeus.

Like most language kernels, xeus-robot supports code completion, inspection, error handling, etc. It also allows using Python cells to define custom robot keywords in Python, those Python cells support code completion as well.

Code Completion in Robot Framework

By using the libraries robotframework-seleniumlibrary and robotframework-seleniumscreenshots, you can even complete the selection of elements on the page you are currently testing!

Code completion with an action of the user in the UI.

The %%python cell magic makes it possible to extend Robotframework with Python modules in the notebook.

Extending RobotFramework with Python, and code completion in Python cells

Widget-based UIs are provided to test Robotframework “keywords”.

Testing Robot Framework “keywords” with Jupyter widgets

Last but not least, xeus-robot comes with full support for the JupyterLab Visual Debugger! After the xeus-python kernel, it is the second Jupyter kernel to support the Jupyter Debugger Protocol! We hope that many more will come.

You can set breakpoints, step in defined keywords (the equivalent of functions), inspect variables, and see the callstack, as shown below.

The JupyterLab Visual Debugger in action with Robot Framework

Installing xeus-robot

Xeus-robot is available for all platforms on conda-forge, and can be installed with conda or mamba.

mamba install xeus-robot -c conda-forge

The jupyterlab-robotmode package, which provides JupyterLab syntax highlighting for Robot Framework will also be installed as a dependency.

For your conda installation, we recommend starting from mambaforge or miniforge which are available for download here, and default to the conda-forge channel (making the -c conda-forge argument unnecessary).

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Acknowledgments

The work on xeus-robot by QuantStack was funded by Robocorp, and is part of their initiative to create an open-source RPA ecosystem. Beyond the xeus-robot kernel, Robocorp also funds the integration of Robot Framework with Visual Studio Code, RPA Framework, and other development to push the Robot Framework project forward.

The implementation of the debugger in xeus-robot relies on robotframework-lsp by Fabio Zadrozny and we want to thank him for his help in integrating it in xeus-robot.

About the authors

Martin Renou is a scientific software developer at QuantStack. He is the creator of the xeus-python and xeus-robot kernels.

Johan Mabille is a scientific software developer at QuantStack. He is a co-author of the xeus project, and developed the debugger extensions to xeus-python and xeus-robot. He also co-authored the front-end for the JupyterLab visual debugger.

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

Use Jupyter and 3D Slicer kernel to implement biomedical data processing workflows in a notebook.

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan

A Jupyter kernel for SQLite

2020-06-11T08:42:00+00:00

While it is well known in the Python scientific computing community, Jupyter is in fact a language-agnostic development environment. High-quality language kernels exist for the main languages of data sciences, such as Python, C++, R or Julia.
But another important tool for data science is the SQL family of programming languages. Today, we announce the release of a Jupyter kernel for SQLite.

This new kernel allows the user to use the complete SQLite syntax as well as some extra operations such as opening or closing a database file, or visualizing the data in different ways using Jupyter magics.

HTML and Text table visualizations for different Jupyter interfaces

To make the task of implementing a new Kernel for Jupyter easier, we make use of Xeus, a library providing a solid implementation of the Jupyter kernel protocol, so that we can focus on implementing the language-specific parts of the kernel.

As a connection between SQLite’s native C implementation and C++ we’re using SRombauts’ library SQLiteCpp.

Current state

xeus-SQLite is still under active development but it offers a fully functional SQLite interface and magics to perform higher-level operations that are outside of the scope of the SQL syntax, such as creating, opening, or closing SQLite databases. Some of the things you can do with xeus-SQLite are creating a new database, loading it, backing it up or deleting it. You can also check if a table exists, set and reset keys of a database and get information about it. You can find more about the use of these methods in SQLite’s documentation.

xeus-SQLite provides rich HTML display of tables in the Jupyter Notebook and Jupyter Lab. For the Jupyter Console we make use of the tabulate library for textual display.

Different commands being executed on xeus-SQLite

The future

For the future of xeus-SQLite we want to create an intuitive form of visualizing data: creating plots, graphs, charts, maps and much more from your SQLite query results, all in the same notebook. Currently we’re exploring the use of Vega in xeus-SQLite, a declarative language for creating interactive visualization designs and can do bar plots using jupyter magics:

SQLite query result with bar and table outputs for same query

This feature is still in very early stages and being developed in this branch.

We also plan on producing a static build of xeus-SQLite bundling xeus and the SQLite library into a single executable that can be easily distributed.

Try it online!

You can try the stable version without the need of installing anything on your computer just by clicking on the image below:

This won’t grant you access to the experimental branch where we’re testing, if you’re interested in trying it please check the installation guide and make sure you’re in the right branch.

Installation

Note that you first need to have Jupyter installed on your computer. You can install xeus-sqlite using mamba:

mamba install xeus-sqlite -c conda-forge

Or conda:

conda install xeus-sqlite -c conda-forge

About the author

Prior to QuantStack I worked as a developer on the PySide team at the Qt Company and as a web performance developer at Mozilla.

A visual debugger for Jupyter

2020-03-25T15:53:00+00:00

Most of the progress made in software projects comes from incrementalism. The ability to quickly see the outcome of an execution and iterate has been one of the main reasons for the success of Jupyter, especially in scientific exploratory workflows.

Jupyter users like to experiment in the notebook, and to use the notebook as an interactive communication tool. However, for more classical software development tasks such as the refactoring of a large codebase, they often switch to general-purpose IDEs.

The JupyterLab environment.

The Jupyter project has made strides in the past few years towards filling that gap, notably with the JupyterLab project, which enables a richer UI including a file browser, text editors, consoles, notebooks, and a rich layout system.

However, a missing piece (which has remained one of the main reasons for users to switch to a different tool) is a visual debugger. This feature has long been requested by users, especially those accustomed to general-purpose development environments.

A debugger for Jupyter

Today, after several months of development, we are glad to announce the first public release of the Jupyter visual debugger!

This is just the first release, but we can already set breakpoints in notebook cells and source files, inspect variables, navigate the call stack and more.

Screencast of the JupyterLab visual debugger in action

Try the debugger on binder

You can also try the debugger online with binder. Just click on the binder link:

Click on the binder link to launch the demo

Installation

The debugger front-end can be installed as a JupyterLab extension.

jupyter labextension install @jupyterlab/debugger

The debugger front-end will be included in JupyterLab by default in a future release.

In the back-end, a kernel implementing the Jupyter Debug Protocol (which will be detailed in the next section) is required. The only kernel implementing this protocol, for now, is xeus-python a new Jupyter kernel for the Python programming language. (Support for the debugger protocol in ipykernel is also on the roadmap).

conda install xeus-python -c conda-forge

Once xeus-python and the debugger extension are installed, you should be all set to use the Jupyter visual debugger!

Note: Depending on the platform, PyPI wheels are available for xeus-python, but they are still experimental.

The Jupyter Debug Protocol

New message types for the Control and IOPub channels

Jupyter kernels (the part of the infrastructure that executes the user’s code) communicate with the rest of the infrastructure with a well-specified inter-process communication protocol.

Several communication channels exist, such as

the Shell channel, which is a request/reply channel for e.g. execution requests
the IOPub channel, which is a one-directional communication channel from the kernel to the client, and is used e.g. to forward the content of the standard output streams (stdout and stderr).

The Control channel is similar to Shell but operates on a separate socket so that messages are not queued behind execution requests, and have a higher priority. Control was already used for Interrupt and Shutdown requests, and we decided to use the same channel for the commands sent to the debugger.

Two message types were added to the protocol:

the debug_[request/reply] to request specific actions to be performed by the debugger such as adding a breakpoint or stepping into a code, which is sent to the Control channel.
the debug_event uni-directional message used by debugging kernels to send debugging events to the front-end. Debug events are sent over the IOPub channel.

Extending the Debug Adapter Protocol

A key principle to the Jupyter design is the agnosticism to the programming language. It is important for the Jupyter debug protocol to be adaptable to other kernel implementations.

A popular standard for debugging is Microsoft’s “Debug Adapter Protocol” (DAP) which is a JSON-based protocol underlying the debugger of Visual Studio Code and for which there already exist multiple language back-ends.

It was therefore natural for us to use the DAP messages over the debug_[request/reply] and debug_event messages that we just added.

However, it was not quite sufficient in the case of Jupyter. Indeed

In order to support page reloading, or a client connecting at a later stage, Jupyter kernels must store the state of the debugger (breakpoints, whether the debugger is currently stopped). The front-end can request that state over with a debug_request message.
In order to support the debugging of notebook cells and of Jupyter consoles, which are not based on source files, we also needed messages to submit code to the debugger to which breakpoints can be added.

Besides these two differences, the content of the debug requests and replies corresponds to the debug adapter protocol.

All these extensions to the Jupyter kernel protocol have been proposed for inclusion in the official specification. The JEP (Jupyter Enhancement Proposal) can be found here.

Xeus-python, the first Jupyter Kernel to support debugging

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel by itself but a library that helps kernel authoring. Xeus is useful when developing a kernel for a language that has a C or a C++API (like Python, Lua, or SQL). It takes the cumbersome task of implementing the Jupyter messaging protocol for the kernel author to focus on the core interpreter tasks: executing code, inspecting, etc.

Several kernels have been developed with xeus, including the popular xeus-cling kernel for the C++ programming language, based on the cling C++ interpreter from CERN. The xeus-python kernel is an alternative Python kernel to ipykernel, based on xeus. The first release of the xeus-python kernel was announced on this blog earlier this year: A new Python kernel for Jupyter

Xeus-python was an appropriate choice for this first implementation of the debugging protocol because

it has a pluggable concurrency model, which allowed running the processing of the Control channel in a different thread.
it has a lighter-weight codebase which made it a convenient sandbox to iterate upon. Implementing the first version of the protocol in ipykernel would have required more significant refactoring and consensus building at an early stage.

The roadmap of Xeus-python

The short-term roadmap for xeus-python includes

adding support for IPython magics in xeus-python, which is the main missing feature with respect to ipykernel.
improving the PyPI wheels of xeus-python.

What about other kernels?

The work in the front-end is valid for any kernel implementing the extended kernel protocol.

We will be working in 2020 to enable debugging with as many kernels as possible.

This will soon be the case for other xeus-based kernels which share a large part of the implementation with xeus-python, such as xeus-cling.

Diving into the debugger front-end architecture

The debugger extension for JupyterLab provides what users would typically expect from an IDE:

a sidebar with a variable explorer, a list of breakpoints, a source preview and the possibility to navigate the call stack
the ability to set breakpoints directly next to the code, namely in code cells and code consoles
visual markers to indicate where the current execution has stopped

When working with Jupyter notebooks, the state of the execution is kept in the kernel. But a cell can be executed and then deleted from the notebook. What should happen when a user wants to step in deleted code?

The extension supports that particular use case and enables retrieving a read-only view of the previously executed cell.

Stepping into a deleted cell

Consoles and files also have support for debugging.

Debugging code consoles in JupyterLab

Debugging files in JupyterLab

Debugging can be enabled on a notebook level, which lets users debug a notebook and work on a different one at the same time.

Debugging multiple notebooks simultaneously

Variables can be inspected using a tree viewer and a table viewer:

The variable explorer

The debugger extension for JupyterLab has been designed to work with any kernel that supports debugging.

By relying on the Debug Adapter Protocol, the debugger extension abstracts away language-specific features and provides a consistent debugging interface to the user.

The following diagram shows how the debug messages flow between the user, the JupyterLab extension and the kernel during a debugging session.

Using the Debug Adapter Protocol in the debugger extension (source)

Future developments

In 2020, we plan on making major improvements to the debugger experience:

Support for rich mime type rendering in the variable explorer.
Support for conditional breakpoints in the UI.
General improvements of the debugger user experience.
Enable the debugging of Voilà dashboards, from the JupyterLab Voilà preview extension.

Acknowledgements

The JupyterLab debugger is the result of the collaboration and coordination of developers from several institutions, including QuantStack, Two Sigma, and Bloomberg.

The work on both the front-end and the back-end at QuantStack by Jeremy Tuloup, Johan Mabille, Martin Renou, and Sylvain Corlay, was funded by Bloomberg.
The work on the Jupyter debugger by Borys Palka and Afshin Darian was made possible by Two Sigma.

About the developers

Jeremy Tuloup

Jeremy Tuloup is a Scientific Software developer at QuantStack. He authored a large part of the front-end of the JupyterLab debugger.

Borys Palka

Borys Palka is a software developer at Codete. He authored a large part of the front-end of the JupyterLab debugger.

Johan Mabille

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of xeus, and developed the debugger extension to xeus-python. He also authored a large part of the debugger front-end.

Martin Renou

Afshin Darian

Afshin Darian is a software developer at Two Sigma, and one of the authors of JupyterLab.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Xeus is now a Jupyter subproject

2020-02-04T12:08:00+00:00

It is a great pleasure to announce that the Xeus project has been incorporated as a Jupyter subproject. Xeus will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Xeus incorporation is available here.

What is Xeus?

The Xeus project is a C++ implementation of the Jupyter kernel protocol. Xeus is not a kernel, but a library meant to facilitate the authoring of kernels.
Several Jupyter kernels have been created with Xeus:

xeus-cling, a kernel for the C++ programming language, based on the Cling C++ interpreter. The cling project comes from CERN and is at the foundation of the ROOT project.

The xeus-cling Jupyter kernel for the C++ programming language.
xeus-python, a kernel for the Python programming language, embedding the Python interpreter.

The xeus-python Jupyter kernel for the Python programming language
xeus-calc, a calculator kernel, meant as an educational example on how to make Jupyter kernels with Xeus.

Beyond these three kernels built on top of Xeus by the Xeus maintainers, third-parties have developed other Jupyter kernels with Xeus:

JuniperKernel, a kernel for the R programming language by Spencer Aiello.
xeus-fift, a kernel for the fift programming language by Michael Zaikin. The fift programming language was developed by Telegram to create TON blockchain contracts.
SlicerJupyter, a kernel for the Python programming language by Kitware which integrates into the Qt event loop of the Kitware “Slicer” project.

Finally, the xeus-python kernel includes a first implementation of the Jupyter debugger protocol used by the Jupyter debugger project. xeus-python enables the Debug Adapter Protocol over the Control channel through new debug request/reply and debug event messages.

Why moving Xeus under the Jupyter governance?

While Xeus started as a side project for QuantStack engineers, the project now has several stakeholders who depend on it. We think that moving the project to an open governance organization may be a better way to reflect this situation.

Acknowledgements

Xeus was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Xeus contributors work in many institutions, including Université Paris-Saclay and École Polytechnique.

About the Author

Johan Mabille is a Scientific Software Developer at QuantStack, specializing in high-performance computing in C++. He holds master’s degree in computer science from Centrale-Supelec.

As an open source developer, Johan coauthored xtensor, xeus, and xsimd. He also made major contributions to the JupyterLab debugger project and bqplot.

Jupyter Blog - kernels

From Punch Cards to the Browser: Fortran Comes to JupyterLite

A language for people who think in formulas

The other half of scientific computing

Interactive Fortran

Taking the kernel to the browser

Try it

Inside the LFortran compiler

Epilogue

About the authors

Acknowledgements

GNU Octave Meets JupyterLite: Compute Anywhere, Anytime!

Cross-Compilation of GNU Octave

Xeus-Octave

Future Work

About the Author

Acknowledgments

C++ in Jupyter — Interpreting C++ in the Web

Scaling Jupyter with JupyterLite

Walking up the stack

Emulated Just-in-Time Compilation and WebAssembly

Porting Xeus-Cpp to Emscripten-forge

Features

Inline documentation

Rich display

Advanced Graphics

Loading Third-Party Libraries

Symbolic Computing with Symengine

Array-based Computing

SIMD Acceleration

Magic Commands

Interactive Widgets

How to deploy JupyterLite with Xeus-Cpp

Future work

Debugger Support

Cuda Support in Xeus-Cpp

Last Value Printing

How to contribute?

Packaging for the Browser

Improving Xeus-Cpp, CppInterOp and Clang-Repl

Areas of Need & Opportunities for Collaboration

About the Authors

Acknowledgements

Exploring Open Access Art Using the Deno Jupyter Kernel

The dataset

But wait!

Loading and cleaning the data

Open Access dataset

Determining whether art is in public domain

“Join” to single unified table

Exploring art, who created it, when, and more with plots

Deeper exploration with interactivity

Exploring data with DataFrame viewer anywidgets

Creating a custom <Gallery/> component

Drilling down on when the art was created

Conclusion

R in the Browser: Announcing Our WebAssembly Distribution

Why WebAssembly? Scaling scientific computing workflows in Jupyter to millions of users

Beyond Python, enabling R in the browser, and more

Building R for Emscripten-forge

Fortran

Walking up the stack

Cross-compilation of R

The R package ecosystem

Xeus-R

How to make a deployment with custom packages

How to contribute

What is in the works

About the Author

Acknowledgments

Funding

Meet Xeus-R: a future-proof Jupyter kernel for R

Xeus-R

Relationship with IRkernel

Key features

Installation and Getting Started

Looking ahead: JupyterLite Integration

Conclusion

Acknowledgements

Author Bio

Creating a custom `<Gallery/>` component