Jupyter Blog - xeus

From Punch Cards to the Browser: Fortran Comes to JupyterLite

2026-10-01T04:00:00+00:00

Punched card from a Fortran program Z(1) = Y + W(1), plus sorting information in the last 8 columns - Arnold Reinhold, Wikimedia Commons, CC BY-SA 2.5

A language for people who think in formulas

In the early 1950s, programming meant writing numerical machine instructions by hand, one row at a time. John Backus, who led the effort that changed this at IBM, described the experience as “hand-to-hand combat with the machine.”

In 1953 he proposed a different bargain: let scientists write programs the way they already wrote their mathematics — as formulas — and let a machine do the translating. The skeptics said it could not work. Memory was scarce, computer time was precious, and code produced from anything readable by humans could surely never rival code crafted by hand. Three years later, in April 1957, IBM delivered the first FORTRAN compiler for the IBM 704, and the skeptics were answered by the world’s first optimizing compiler, whose output ran nearly as fast as anything a hand-coder could produce.

The name was the whole idea: FORmula TRANslation. Fortran let scientists and engineers work in the vocabulary of their own fields — arrays, floating-point numbers, complex arithmetic — rather than in registers and opcodes. It became the first programming language to be standardized, in 1966, and it has been actively maintained to this day (latest standard is 2023, next one in progress, targeted for 2028). Much of the software that runs on the world’s fastest computers is written in Fortran, and the dense numerical libraries that other languages call for their heavy lifting, LAPACK most prominently, are written in it too. If you have solved a linear system today, in Python or R or Julia, there is a decent chance Fortran did the work.

The other half of scientific computing

But Fortran was born in the batch era. Running a program meant keypunching a deck of cards, handing it to an operator, and coming back hours later, or the next day, to find out whether it had run at all. Computing meant deciding everything in advance.

A second tradition grew out of the opposite frustration: computing as a conversation. Type an expression, look at the result, adjust, repeat. That impulse ran through time-sharing terminals, through Mathematica and MATLAB, and into IPython, first released in 2001, which grew into Project Jupyter. The notebook, with code, results, and prose in a single living document, is now the default home of exploratory data analysis and computational science, and Jupyter’s architecture is deliberately language-agnostic: any language can implement a kernel and join the conversation. Python, Julia, and R most visibly; C++, Lua, and others besides.

In notebooks, though, Fortran has been present but invisible: the compiled layer underneath NumPy and its cousins, reached through wrappers, glimpsed only when a profiler points at it. The language in which so much of numerical computing is actually written has rarely been the language you can sketch an idea in.

Interactive Fortran

That is the gap LFortran was built to close. LFortran is a modern, open-source (BSD-licensed) Fortran compiler built on LLVM, an optimizing compiler infrastructure several generations on from the one Backus’s team delivered for the IBM 704. The question that compiler settled, whether code written for humans could run as fast as code written for the machine, is one its descendants answer routinely; what they rarely offer is a conversation.

What LFortran adds is exactly that second mode: alongside compiling Fortran to fast binaries for production work, it can evaluate code interactively, statement by statement, much like Python, Julia, or MATLAB.

Once a language has an interpreter, a Jupyter kernel is a small step away: from the language’s point of view, the kernel’s job is only to receive a cell of code and return its results. LFortran has shipped with such a kernel for some time. It is built on xeus, a C++ implementation of the Jupyter kernel protocol that already powers a long list of kernels, including xeus-python, xeus-cling, and xeus-r. The protocol plumbing is taken care of, so the kernel can concentrate on Fortran itself, and a single conda install -c conda-forge lfortran gives you both the compiler and the kernel.

Taking the kernel to the browser

Today we are announcing JupyterLite support: the LFortran kernel now runs in JupyterLite, a JupyterLab distribution that runs entirely in the browser. JupyterLite is served as a static website, with no Jupyter server, no container, and nothing provisioned on any machine.

This matters because it changes not just what a notebook is, but how it can be shared. A notebook becomes something you can publish like any web page: static, lightweight, and infinitely scalable. Whether you’re hosting it on GitHub Pages, a university server, or a personal blog, you can serve thousands of users with minimal resources. There is no backend to manage, no accounts to create, no queues to wait in. The gap between “here is a notebook” and “here, try it” disappears, and with it, many of the barriers that have traditionally limited access to computational tools.

This makes JupyterLite ideal for education, where students can experiment with code without installing anything; for live documentation, where readers can tweak and run examples from a project’s docs; and for citizen science, where communities can collaborate on data analysis without specialized infrastructure. Fortran, a language built for heavy computation, is now as accessible as a webpage.

Building the kernel on xeus carried it most of the way to the browser. The same xeus kernel code that ships on conda-forge for the desktop is built for WebAssembly by emscripten-forge and loaded into JupyterLite by jupyterlite-xeus, and many of the JupyterLite kernels available today are xeus kernels: Python, R, C++, Lua. The desktop and browser kernels are the same code.

The last obstacle was the interesting one. In interactive mode, LFortran leans on LLVM’s just-in-time compilation: each cell is compiled as it arrives and executed on the spot. A browser will not do that. WebAssembly keeps code and data in separate address spaces, so a program cannot write machine code into its own memory and then jump to it, and the classic JIT move is unavailable. What the browser does allow is loading a module: new code enters a running WebAssembly program by being handed to the host and instantiated, never by being written into memory. The problem becomes one of routing the compiler’s output through that door.

This is the wall xeus-cpp hit when it brought the clang-repl C++ interpreter to WebAssembly, and the solution here is the one devised there. Instead of executing generated code in place, the kernel passes it to the WebAssembly toolchain, itself compiled to WebAssembly and running in the same tab, which links it into a side module (the browser’s equivalent of a shared library) and loads it into the running kernel. Parsing, compilation, and linking all happen in the browser; the notebook’s program grows one module at a time.

Try it

You can try it right now on notebook.link, with nothing but a browser. Just click HERE! The demo ships with a handful of notebooks to start from, including a Mandelbrot example and one showing rich display output — plots and images rendered straight from Fortran cells.

The kernel itself is available as a package on emscripten-forge, a software distribution for WebAssembly. That means Fortran is now just another entry in an environment.yml: any JupyterLite deployment built with jupyterlite-xeus can include it. The deployment is a static site you can host anywhere (GitHub Pages, a blog) with no server costs and no limits on the number of users. For a Fortran project, this could mean embedding live, executable examples directly in your documentation, letting users experiment with your code without ever leaving the browser.

For a regular Jupyter installation, the kernel is still available the classic way:

conda install -c conda-forge lfortran jupyterlab

Inside the LFortran compiler

Another reason why LFortran is a particularly good fit for the notebook workflow lies in its compiler pipeline: its Abstract Semantic Representation, or ASR. While the AST records the syntax the user wrote, ASR records what the program means, with types, symbols, procedure interfaces, and other semantic information made explicit. LFortran can therefore perform high-level, Fortran-aware transformations on a valid program before lowering it to LLVM IR, leaving LLVM to handle lower-level optimization and code generation.

Fortran source → AST → ASR → LLVM IR → native code / WebAssembly

For interactive use, each notebook cell has its own ASR translation unit. Its symbol table points to the translation unit from the preceding cell, forming a chain through the session. A cell can therefore use variables, modules, and procedures defined earlier, while redefining a name shadows the old definition instead of modifying code that has already been compiled. Cell-level symbols are given distinct generated names, allowing old and new definitions to remain alive at the same time. Once the current cell has been lowered to an LLVM module, it is handed to the native On Request Compilation (ORC) JIT or to the browser executor described above.

Epilogue

When Fortran arrived in 1957, using it meant punching cards, standing in line at the keypunch and the card reader, and waiting for the batch to come back around. Today it means opening a URL, typing a few lines, and pressing Shift+Enter. The formulas look much the same as they did sixty-nine years ago.

What changed is everything around them.

About the authors

Ondřej Čertík is the creator of LFortran, which he started in 2017 and has led ever since; he is also the original author of SymPy and SymEngine, a co-founder of the fortran-lang community, and a Principal Software Engineer at Microsoft.

Anutosh Bhat did the work of bringing this kernel to the browser: he ported both xeus-cpp and the Fortran kernel to WebAssembly, and iterated extensively on the kernel itself. Anutosh is a scientific software engineer at QuantStack, an LLVM maintainer, and a co-author of xeus-cpp; his contributions span clang-repl, LLVM’s WebAssembly backend, and their packaging for emscripten-forge.

Acknowledgements

The work by Anutosh on the porting the LFortran kernel to WebAssembly was funded by QuantStack, the team behind JupyterLite, xeus, and the emscripten-forge distribution.

LFortran itself is the product of many years of sustained work by more than 130 contributors, made possible by the support of the Sovereign Tech Agency; Los Alamos National Laboratory, which supported its early development; GSI Technology; FLOSS/fund; QuantStack; John D. Cook; NumFOCUS; and Google Summer of Code, through which many contributors first joined the project. We also thank the individuals and organizations who have sponsored LFortran through GitHub and OpenCollective.

GNU Octave Meets JupyterLite: Compute Anywhere, Anytime!

2025-10-16T15:04:00+00:00

We are thrilled to announce the newest member of our JupyterLite kernel ecosystem: Xeus-Octave. Xeus-Octave allows you to run GNU Octave code directly on your browser. GNU Octave is a free and open-source Scientific Programming Language that can be used to run Matlab scripts. In this article, we present the challenges encountered when targeting WebAssembly, the current state of the Xeus-Octave kernel, and the future plans for expanding the GNU Octave ecosystem.

Earlier this year, we introduced the JupyterLite kernel for R, Xeus-R-Lite. Much like R, cross-compiling GNU Octave to WebAssembly required the same custom toolchain to enable the compilation of Fortran code, combining LLVM Flang and Emscripten.

Similar to many other mathematically oriented language packages, GNU Octave requires a BLAS/LAPACK implementation. Fortunately, OpenBLAS and the Netlib implementations of BLAS/LAPACK had already been added to the emscripten-forge WebAssembly distribution. Initially, OpenBLAS was the preferred implementation, but for the successful compilation of Octave, Netlib LAPACK was selected as it presented fewer hurdles during the build process.

Cross-Compilation of GNU Octave

One of the complications of cross-compiling Octave to WebAssembly, which had not been encountered with the R source code, was the extensive use of Fortran common symbols blocks in the internal libraries of Octave such as odepack.

C Source: liboctave/external/odepack/slsode.f
C-----------------------------------------------------------------------
C The following internal Common block contains
C (a) variables which are local to any subroutine but whose values must
C     be preserved between calls to the routine ("own" variables), and
C (b) variables which are communicated between subroutines.
C The block SLS001 is declared in subroutines SLSODE, SINTDY, SSTODE,
C SPREPJ, and SSOLSY.
C Groups of variables are replaced by dummy arrays in the Common
C declarations in routines where those variables are not used.
C-----------------------------------------------------------------------
      COMMON /SLS001/ CONIT, CRATE, EL(13), ELCO(13,12),
     1   HOLD, RMAX, TESCO(3,12),
     1   CCMAX, EL0, H, HMIN, HMXI, HU, RC, TN, UROUND,
     2   INIT, MXSTEP, MXHNIL, NHNIL, NSLAST, NYH,
     3   IALTH, IPUP, LMAX, MEO, NQNYH, NSLP,
     3   ICF, IERPJ, IERSL, JCUR, JSTART, KFLAG, L,
     4   LYH, LEWT, LACOR, LSAVF, LWM, LIWM, METH, MITER,
     5   MAXORD, MAXCOR, MSBP, MXNCF, N, NQ, NST, NFE, NJE, NQU

Initially, it was not possible to cross-compile these common blocks to WebAssembly because the latest version of LLVM (v20 at the time of testing) did not support common symbol linkage.

// Source: llvm/lib/MC/MCWasmStreamer.cpp
void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
  llvm_unreachable("Common symbols are not yet implemented for Wasm");
}

As a temporary solution, LLVM was patched with the help of Serge Guelton to simulate common symbols as weak symbols.

void MCWasmStreamer::emitCommonSymbol(MCSymbol *S, uint64_t Size,
                                      Align ByteAlignment) {
-  llvm_unreachable("Common symbols are not yet implemented for Wasm");
+  auto *Symbol = cast<MCSymbolWasm>(S);
+  getAssembler().registerSymbol(*Symbol);
+  Symbol->setWeak(true);
+  Symbol->setExternal(true);
}

A proper solution to enable support of common symbols is currently in progress and will likely be included in the next release of LLVM v22 (see llvm-project/pull/151478). For curious readers, the patched version of LLVM can be found here (linux only).

In addition to the patches for LLVM, GNU Octave required a few minor modifications to target WebAssembly; mainly this entailed disabling the GUI functionalities and consolidating the Fortran function signatures and calling conventions. A full list of patches can be found in the recipe directory on emscripten-forge.

Xeus-Octave

Once GNU Octave had been successfully packaged for WebAssembly, bringing Xeus-Octave to JupyterLite was a simple matter of adding a recipe to emscripten-forge!

Try Xeus-Octave in JupyterLite!

Future Work

For our next steps, the team is planning on expanding the Octave ecosystem by adding Octave packages to both conda-forge and emscripten-forge. The packaging work will require defining a process where Octave packages can be installed in predetermined conda environments, perhaps with some minor modifications to the existing pkg utility.

About the Author

Isabel Paredes, who led the charge on bringing GNU Octave to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the R programming language and the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This project synthesizes work from many open-source contributors.

Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes, Anutosh Bhat, Martin Renou, Ian Thomas, Wolf Vollprecht, and Johan Mabille.
JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Xeus-Octave, the Xeus-based Jupyter kernel for GNU Octave, was created by Giulio Girardi and Antoine Prouvost.

C++ in Jupyter — Interpreting C++ in the Web

2025-06-19T15:29:00+00:00

Scientists and engineers utilize programming languages not only to build software systems but also to drive interactive exploratory workflows. They leverage developer tools to explore and reason through problems effectively. This process involves executing code, examining visualizations, loading data, and running additional code. Rapid iteration is particularly crucial during the exploratory phase of any technical project.

For this reason, the most popular tools for exploratory work tend to be interpreted languages, such as Python or R. The “Read-Eval-Print-Loop” (REPL) places the user at the center of these iterative workflows. While compiled languages such as C++ are essential for performance-sensitive applications, engineers accustomed to interactive tools often struggle in such environments.

At CERN, the C++ programming language has been pivotal to their data analysis platform for particle physics, notably in the ROOT framework. ROOT is underpinned by the Cling C++ interpreter, which is built on the Clang and LLVM stack. The Cling interpreter, co-created by Vassil Vassilev and Axel Naumann in the 2010s, effectively turned C++ into an agile scientific scripting language!

These interactive workflows are where the Jupyter stack shines compared to traditional IDEs. Indeed, the Jupyter project was designed with exploratory computing in mind. Crucially, it was built from the ground up to be language-agnostic, accommodating multiple programming languages seamlessly. In 2017, the Xeus-Cling Jupyter kernel was announced. Developed by Sylvain Corlay, Johan Mabille and Loic Gouarin, it integrated Cling in the Jupyter ecosystem, taking advantage of the rich features of Jupyter, such as rich mime type display and interactive widgets.

An example C++ Jupyter notebook powered by the original Xeus-Cling kernel

Packaging the Cling project proved challenging as it required patching the code base of Clang, and was not released as frequently as the LLVM stack, causing difficulties with rolling software distributions like conda-forge.

In an effort led by Cling co-creator Vassil Vassilev, the Clang-Repl project was born, which consists in building the foundations for interactive C++ in the core of Clang. Clang-Repl has been undergoing active development for a few years now and a Jupyter kernel for Clang-Repl, Xeus-Cpp, has also been developed, providing a similar set of features to Xeus-Cling.

Building the new C++ kernel upon Xeus was crucial as it enabled the porting of the C++ kernel to the Web browser with JupyterLite.

Scaling Jupyter with JupyterLite

The standard approach for Jupyter-based computing relies on providing each user with computing resources in the backend. Serving a few thousand end users necessitates substantial resources and complex deployment strategies.

The JupyterLite project shifts this paradigm by running code directly in the user’s Web browser, by relying on a WebAssembly build of the kernel and required libraries. This eliminates the linear relationship between the number of users and the required resources.

For instance, the NumPy.org website features a JupyterLite code console on its main page, offering a computational environment to thousands of monthly visitors without incurring any cloud costs. Similarly, the Capytale deployment of Jupyter, used in French high schools, operates on the same model. It accounts for over half a million registered users and hosts more than 200,000 user sessions weekly.

In order to provide a broad range of packages and libraries for this new platform, the JupyterLite team at QuantStack developed the Emscripten-forge distribution, utilizing the conda package manager for creating WebAssembly environments. Emscripten-forge comprises the main packages of the Python scientific computing ecosystem, as well as the R programming language, and several command-line applications powering the JupyterLite terminal.

Today, we are proud to announce that Xeus-Cpp and LLVM are available in emscripten-forge, effectively enabling interpreted C++ in the browser.

The new C++ Jupyter kernel in action in JupyterLite.

Feel free to try it out in your browser by clicking the link below!

We now dive into the details on the internals of the WebAssembly build of Xeus-Cpp, and the key features of the project.

Walking up the stack

Emulated Just-in-Time Compilation and WebAssembly

A fundamental challenge inherent to the WebAssembly is that it operates in a sandboxed Harvard architecture: code and data reside in completely distinct memory spaces. This makes conventional JIT compilation unfeasible in the browser, as it prohibits dynamic modification of executable memory.

The WebAssembly backend for Clang-Repl was introduced during the LLVM 17 development cycle, with the introduction of a WASM-specific IncrementalExecutor, sidestepping the standard LLVM JIT approach used on native platforms. This new WasmIncrementalExecutor class handled the WebAssembly-specific execution model as follows:

The LLVM IR from the REPL’s Partial Translation Unit (PTU) is compiled into an object file.
This object file is passed to wasm-ld with flags to generate a standalone WebAssembly module.
The resulting .wasm module acts as a dynamically linked side module, designed to be loaded on top of a persistent main module at runtime.

These side modules, while similar to shared libraries, are treated as auxiliary WASM modules that are dynamically attached to the main application using Emscripten’s dlopen mechanism. They share memory with the main module and can resolve symbols defined in previous executions. As a result, each REPL input produces a corresponding .wasm module that is compiled, linked, and loaded at runtime, incrementally extending the current execution environment.

Following the initial version in LLVM 17, the support for WebAssembly was incrementally consolidated by a series of pull requests addressing various issues and corner cases with respect to symbol duplication, memory corruption, and dynamic linking failures. We believe the version available today in LLVM 20 is a solid foundation to build upon.

For a deeper understanding of the constraints of the WebAssembly platform for JIT compilation, refer to the GSOC report: Anubhab Ghosh — WASM Clang-Repl.

Porting Xeus-Cpp to Emscripten-forge

In addition to challenges with emulating JIT compilation in WebAssembly, packaging Xeus-Cpp for emscripten-forge presented another challenge: managing shared objects within the initial WebAssembly executable, which are typically built statically for other xeus kernels. This issue was addressed in the following PRS:

As a result of this work, required side modules are now specified as metadata in the Jupyter kernelspec kernel.json file, enabling JupyterLite to automatically detect and utilize them.

Features

Inline documentation

To fully integrate C++ as a first-class citizen within the Jupyter ecosystem, supporting inline documentation is crucial. Similar to the Python kernel, utilizing the ? magic command should allow users to inspect the following type, perform a lookup in the available documentation, and display the relevant information.

The C++ kernel comes equipped with a lookup file for the standard library, making it possible to execute queries like ?std::vector seamlessly.

Inline help feature in the C++ JupyterLite kernel

This can be enabled for third-party libraries. Check out the documentation to enable inline help for your package.

This feature, inherited from the Xeus-Cling project, was initially created by Loic Gouarin.

Rich display

One of the core benefits of integrating C++ into the Jupyter ecosystem is the ability to leverage Jupyter’s rich MIME display system. Rather than limiting output to plain text, Xeus-Cpp allows you to render rich content like images, HTML tables, LaTeX, or even custom visualizations, all directly from C++.

This is made possible by the function xcpp::display, which sends a MIME bundle to the frontend. Each bundle can include multiple representations of the same object, such as text/plain, image/png, or even custom mime types.

This is illustrated in the following example:

Rich mime type rendering feature of the Xeus-Cpp kernel

A custom renderer for any given type can be defined by defining the mime_bundle_repr function in the corresponding namespace, which is then picked up by Xeus-Cpp’s display system through Argument-Dependent Lookup (ADL). A large collection of mime types are supported by the JupyterLab frontend, which can also be extended with plugins to support custom mime types.

Advanced Graphics

Another compelling example leveraging the rich rendering capabilities, is to combine it with frameworks such as SDL for rich graphic content. Emscripten has built-in support for SDL (enabled with the -s USE_SDL=2 compilation flag), which allows us to leverage its power in C++ notebooks.

To illustrate this, we ported Kevin Beason’s legendary smallpt global illumination renderer (a path tracer in 99 lines of C++) into a C++ notebook. The scene is rendered using SDL onto an in-memory canvas, and the resulting image is captured and displayed with xcpp::display.

This setup showcases the seamless integration of SDL and the Jupyter display system:

Use SDL for real-time or offscreen rendering
Then pipe the result through xcpp::display(…) as a PNG or pixel buffer.

PR #299 added an example notebook supporting smallpt with SDL-based rendering. Feel free to access the notebook through our lite link. Since the number of samples per pixel (spp) directly correlates with how long the process runs (approximately 2 minutes in our case), we’ve included debug logs to keep users informed during rendering, so they aren’t left wondering about the progress.

Smallpt tiny raytracer in action in JupyterLite

This approach paves the way for interactive C++ graphics demos, ray tracing notebooks, and even the prototyping of browser-based video games from within WebAssembly C++ notebooks.

Loading Third-Party Libraries

For a user’s perspective, a key difference between a Jupyter kernel for a compiled language like C++ and e.g. Python concerns how third-party libraries can be utilized. In Python, one merely needs to import the relevant installed packages, while in the case of C++, the relevant compiled libraries must be loaded in addition to including the corresponding headers.

Thankfully, Clang-Repl provides the LoadDynamicLibrary to dynamically load shared objects, which was adapted to the WebAssembly based in LLVM PR #133037. Instead of relying directly on raw dlopen calls, this function ensures that memory layout, symbol resolution, and constructor invocation are handled correctly — even in a WebAssembly environment.

Building on top of this, the CppInterOp project made these features available via a high-level C++ API.

Showcasing the dynamic loading of shared objects with CppInterOp

Library authors interested in facilitating the use of their package with the Jupyter kernel can include the calls to CppInterop’s LoadLibrary can simply be included in user-facing headers.

This has been done for example, in the Symengine library.

Symbolic Computing with Symengine

A compelling example of the Jupyter rich display system’s advanced capabilities with Xeus-Cpp is the SymEngine package, a powerful C++ library for symbolic computing.

Originally designed as a backend for symbolic Python libraries like SymPy, SymEngine offers native support for expression trees, calculus and Latex/MathJax rendering of mathematical expressions. We have integrated SymEngine into the emscripten-forge distribution, making it readily available for use. Once installed, it works seamlessly with Xeus-Cpp, requiring no additional setup.

SymEngine expressions can be rendered via xcpp::display(), which produces LaTeX-formatted expressions and delegates their rendering to MathJax in the frontend.

Rich mime type rendering used with Symengine

Array-based Computing

Jupyter notebooks are particularly popular among data scientists, students, and practitioners of scientific computing. Array-based computing is at the center of the practice, be it in Python with NumPy, in R, GNU Octave, or Julia.

A framework for array-based computing is C++ is Xtensor. Xtensor’s syntax closely mirrors that of NumPy, supporting features like array broadcasting, lazy evaluation, and element-wise operations with an idiomatic C++ API. Using Xtensor in combination with the C++ Jupyter kernel provides a similar user experience to that of NumPy with the Python Jupyter kernel.

JupyterLite used in combination with Xtensor, utilizing rich display for tables.

Built on top of Xtensor, Xtensor-BLAS brings the high-level syntax of Xtensor with the performance of OpenBLAS. It provides convenient, NumPy-style access to BLAS and LAPACK routines making operations like matrix inversion or system solving easy to express and execute.

JupyterLite used in combination with Xtensor-BLAS

To use these libraries inside a JupyterLite notebook, all you need is to include Xtensor-BLAS in your environment (available via emscripten-forge). Under the hood, Xtensor-BLAS dynamically loads libopenblas, which is bundled automatically as a dependency.

SIMD Acceleration

We can take a step further with SIMD (Single Instruction, Multiple Data) acceleration.

Using raw WebAssembly intrinsics : WebAssembly includes 128-bit SIMD instructions, which are provided in Clang’s <wasm_simd128.h> header. They can be used to perform vectorized operations directly.

Xeus-Cpp kernel showcasing the use of SIMD instructions

Using Xsimd:

Rewriting specialized vectorized routines for WebAssembly can be cumbersome. This is where frameworks like Xsimd come in. It provides a high-level SIMD API in C++ that abstracts platform-specific SIMD instructions. With Xsimd, you can efficiently operate on batches of numbers using familiar arithmetic operators or mathematical functions, while benefiting from optimized performance under the hood.

Xsimd powers SIMD acceleration in Xtensor and is also used in many other projects, including Apache Arrow, Firefox, Velox, Pythran, and Krita. Here’s how the same example would look when when using Xsimd instead of raw intrinsics:

Xeus-Cpp kernel showcasing the Xsimd library

Magic Commands

In addition to interpreting standard C++ code, Xeus-Cpp also supports magic commands, special notebook commands prefixed with % or %% that allow for extended functionality not defined by the C++ language itself.

These magics let you interact with the runtime environment, manage files, or perform meta-operations, all from within a code cell. The following magics are available:

%file : The %file magic enables file creation and editing directly from a code cell. You can append to an existing file using the -a flag

Using the %file magics in Xeus-Cpp-lite
%timeit : A %timeit magic has also been prototyped via PR #289 by @kr-2003, enabling performance measurement for expressions in the notebook.

However, due to the lack of last value printing support in Clang-Repl, this feature currently relies on a workaround and hasn’t yet been merged. Once upstream support is in place, %timeit will be integrated cleanly and available out of the box.

Here’s a preview of the %timeit magic in action:

Using the %timeit magics in Xeus-Cpp-lite

Magic commands open the door to a more expressive and interactive notebook experience for C++. While support is currently limited to a few core magics, this is just the beginning — with more capabilities and enhancements expected in the near future.

Interactive Widgets

Beyond rich mime type rendering and inline documentation, the Jupyter widgets system allows users to create interactive components in the Jupyter notebook, leveraging by-directional communication with the kernel for more interactivity.

The xwidgets package is a C++ implementation of the Jupyter widget protocol, comprising an implementation of all the core Jupyter widgets, and which underlies other interactive visualization libraries like xleaflet or xcanvas.

Here are a couple examples demonstrating interactive widgets through xwidgets & xcanvas with Xeus-Cpp-lite.

Displaying a slider using xwidgets

Displaying a canvas through xcanvas

If you inspect the codebase of xwidgets or xcanvas, you will notice that it contains very little code. All that is required on the C++ end is a declaration of the attributes that should be synchronized with the frontend, and serialization methods for complex data types.

Most of the programmatic logic for Jupyter interactive widgets lies in the frontend implementation. The consequence for the C++ Jupyter kernel is that with seemingly little implementation effort, we could enable an entire ecosystem of widget libraries, covering a large range of use cases. Interactive widgets for 2-D plotting (bqplot, plotly) and 3-D plotting (ipyvolume), GIS (ipyleaflet), efficient rendering of meshes (ipygany), graph visualization (ipycytoscape), and many more could be almost automatically connected to the C++ world.

How to deploy JupyterLite with Xeus-Cpp

The GitHub repository https://github.com/jupyterlite/xeus-lite-demo is a template for creating a JupyterLite deployment on GitHub pages that includes the packages specified in a conda environment.

The process is as follows:

Create a new repository from the GitHub template.
Enable the deployment on GitHub pages from a GitHub action, as shown in the README.
Edit the environment file to include the desired packages.

For example, to deploy an C++ kernel with Symengine & Xtensor-blas installed, the environment.yml file would contain the following:

name: xeus-cpp
channels:
  - https://repo.prefix.dev/emscripten-forge-dev
  - conda-forge
dependencies:
  - xeus-cpp
  - symengine
  - xtensor-blas

The deployment linked at the beginning of this article was created with this template.

Future work

The journey of interactive C++ in Jupyter, both natively and in the browser, is still evolving. While Xeus-Cpp already offer powerful foundations, several exciting developments are underway that will further expand their capabilities across platforms

Debugger Support

Currently, only a few Jupyter kernels like Xeus-Python offer full integration with the Jupyter Debug Protocol to enable the Jupyterlab debugger. We’re working to bring that experience to C++ as well.

This summer, thanks to Google Summer of Code 2025, Abhinav Kumar will be implementing native debugging support in Xeus-Cpp.The approach is based on using LLDB and its Debug Adapter Protocol (lldb-dap). Modeled after Xeus-Python, it leverages LLDB’s Clang and JIT debugging capabilities to enable breakpoints, variable inspection, and step-through execution directly within the JupyterLab debugger interface.

We already have a proof-of-concept demonstrating that debugging C++ in Jupyter is possible. You can track this effort here: xeus-cpp issue #282.

In the longer term, we aim to explore extending debugging to the browser through Xeus-Cpp.

Cuda Support in Xeus-Cpp

CUDA support in Clang-Repl was broken by the time LLVM 20 was released, but that changed with this upstream patch:

LLVM PR: #136404 — Enable CUDA support in clang-repl

We’re now preparing to bring CUDA support directly to Xeus-Cpp, enabling GPU programming from a Jupyter notebook. Early planning is underway:

GitHub issue: #300 — CUDA REPL support in xeus-cpp

Last Value Printing

In cling, a widely appreciated feature is last value printing, simply writing a variable at the end of a cell automatically displays its value, without needing any manual printing. This behavior naturally carried over to xeus-cling, making interactive exploration even smoother.

This feature stands almost done in Clang-Repl:

LLVM PR: #84769 — Last Value Printing in clang-repl

Once merged, this will be integrated into Xeus-Cpp, enabling seamless REPL experience for C++ users.

How to contribute?

Whether you’re a developer, package maintainer, educator, or simply curious about C++ in the browser, contributions are always welcome!

Packaging for the Browser

Many of the capabilities in xeus-cpp-lite rely on packages built for the WebAssembly platform via emscripten-forge. If you’re interested in porting a library or creating a new recipe, please visit emscripten-forge.org for detailed guidelines and examples.

PRs and discussions are welcome at github.com/emscripten-forge, especially if you’re looking to bring more numerical, graphical, or scientific libraries into the browser.

Improving Xeus-Cpp, CppInterOp and Clang-Repl

We welcome contributions to the broader xeus project, which is part of the Project Jupyter ecosystem. Xeus-Cpp in particular is actively seeking contributors, whether to improve kernel behavior, expand feature support, or explore new magic commands and UI integrations.

On the LLVM side, the Clang-Repl component is still under active development and can greatly benefit from community contributions, particularly around WebAssembly backend improvements and incremental execution features.

Areas of Need & Opportunities for Collaboration

In addition to code contributions, we are seeking funding opportunities to advance the project. We’ve identified several areas where focused effort could significantly enhance the ecosystem, and we warmly welcome collaborators or potential funding partners interested in helping us move these efforts forward, including:

Integration testing: LLVM currently lacks tests for Clang-Repl + WebAssembly. Establishing a reliable test harness in this area is both challenging and critical.
Interactive plotting: A robust plotting library or plugin for Xeus-Cpp would make the kernel far more expressive for teaching, data exploration, and scientific computing.
Plugin system for magics: Further decoupling kernel extensions would simplify development and allow community-driven growth. A concrete description of this project is framed on CERN’s website.

If you or your organization shares the vision of making modern C++ more accessible, especially in-browser, we’d love to hear from you.

About the Authors

Anutosh Bhat is a scientific software engineer at QuantStack. His contributions range from the core of LLVM and Xeus, to their packaging for the Emscripten-forge software distribution. In the course of this endeavor, Anutosh was granted commit rights to the LLVM project.

Vassil Vassilev is a Research Software Consultant with Princeton at CERN. As the co-creator of Cling and the lead developer of the Clang-Repl and CppInterOp projects, he has spearheaded the development of interpreted C++ for over a decade.

Acknowledgements

The work by Vassil Vassilev on this project is done in the context of the Compiler Research group, funded by the National Science Foundation grant OAC-2311471.

The work by Anutosh Bhat on this project is funded by QuantStack, as part of a broader initiative to enable the data science ecosystem within the Web browser.

Work on prototyping the WebAssembly infrastructure for Clang-Repl was done by Anubhab Ghosh, supported through Google Summer of Code 2023 via the LLVM mentoring organization.

A significant portion of the Xeus-Cpp codebase was inherited from the Xeus-Cling project and adapted for the Clang-Repl interpreter. This includes work by Sylvain Corlay, Johan Mabille, Loic Gouarin, and Martin Renou.

We are grateful to Thorsten Beier, the creator and lead developer of the emscripten-forge software distribution, for his support in packaging LLVM and Xeus-Cpp.

We extend our gratitude to Martin Renou and Anastasiia Sliusar for devising the processing of conda packages in the frontend, which underlies the support of third-party libraries in WebAssembly environments used by Xeus-Cpp and xeus-r. We are grateful to Johan Mabille for creating xeus and for his code reviews within the xeus stack. We also thank Jeremy Tuloup for creating JupyterLite.

We are grateful to Matthew Barton, Tharun Anandh & Abhinav Kumar for their numerous and significant contributions to the Xeus-Cpp project.

Finally, we are immensely grateful to the authors and maintainers of Emscripten and LLVM for building such remarkable technologies and for their engagement and support during our numerous interactions on GitHub.

R in the Browser: Announcing Our WebAssembly Distribution

2025-02-28T08:31:00+00:00

R is now available in emscripten-forge, enabling the Xeus-R kernel in JupyterLite.

In the past few months, QuantStack has received funding from the Bill & Melinda Gates Foundation through a grant to CourseKata to develop support for the R programming language in JupyterLite and the Emscripten-forge distribution for WebAssembly.

Today, we are thrilled to announce the availability of the R programming language in the Emscripten-Forge distribution for WebAssembly, which enables R workflows in JupyterLite through the Xeus-R Jupyter kernel.

JupyterLite with an R Jupyter notebook in use

To experience R in JupyterLite, simply click on the link below:

In this article, we present the details of our efforts to enable R workflows in JupyterLite. We cover the challenges encountered in packaging R and its dependencies for this platform, the solutions we implemented, and the process of handling R mamba packages on the frontend.

Why WebAssembly? Scaling scientific computing workflows in Jupyter to millions of users

The main challenge in scaling Jupyter deployments lies in providing a live computing environment on the server side for each user session.

While Kubernetes-based JupyterHub deployments have been successfully implemented at a university scale (such as the UC Berkeley Data 8 class, or the JupyterHub deployment of Université Paris Saclay), they demand substantial computing resources and a skilled DevOps team to achieve and maintain this level of scalability.

In contrast, JupyterLite represents a fundamentally different paradigm. As a browser-based distribution of Jupyter, it runs language kernels directly in the browser, enabling significantly greater scalability and ease of deployment. Notable JupyterLite-based applications include:

A JupyterLite console integrated into NumPy’s front page, providing immediate access to NumPy functionality without requiring installation.
A similar console application on SymPy’s project page, fully hosted statically.
The official Jupyter website^[1].
The examples in Scikit-Learn’s documentation.

These examples illustrate how WebAssembly can be transformative by providing ready-to-use computing environments for scientific workflows, in documentation, blog posts, and scientific communications. The WebAssembly environments are made available to the hundreds of thousands of monthly visitors of these websites without requiring any computing resources on the backend.

JupyterLite deployment displayed on the NumPy website

JupyterLite deployment displayed on the SymPy website

JupyterLite deployment displayed on the Jupyter website

Beyond Python, enabling R in the browser, and more

While the initial focus of the JupyterLite team has been on the Python programming language, the JupyterLite infrastructure was designed to be language-agnostic from the outset. The same principle applies to the package management stack we have been contributing to over the past few years, namely, mamba and conda-forge.

When we began targeting the WebAssembly platform, we chose to build our efforts on the mamba and conda-forge ecosystem. This decision allowed us to provide customizable environments for WebAssembly and expand beyond Python. This enabled applications such as the JupyterLite terminal, created by Ian Thomas, which comprises a bash-like shell written in TypeScript called Cockle, and WebAssembly builds of native shell commands such as cat, grep, ls, and vim. You can read the earlier announcement introducing the JupyterLite terminal.

Today, we are excited to reveal the availability of R in emscripten-forge, enabling Jupyter kernels such as Xeus-R to be used in JupyterLite.

With the addition of R, emscripten-forge now provides a unified packaging solution for Python, R, and terminal applications for WebAssembly. This covers the scopes of Pyodide (a Python distribution for WebAssembly), WebR (an R distribution for WebAssembly), and Sandbox.bio (a shell emulator with WebAssembly commands).

Building R for Emscripten-forge

Emscripten-forge is a GitHub organization containing recipes to build conda packages for the emscripten-wasm32 platform; it is also the first distribution of conda packages targeting this platform. It is built upon a modernized stack, leveraging rattler-build (the successor to the conda-build package builder) and mamba. The emscripten-forge project, which was created and is still led by Thorsten Beier, is now a team effort covering a broad range of packages.

In order to build R with emscripten-forge, we first required a toolchain capable of converting Fortran and C/C++ code into WebAssembly. Since R relies on BLAS and LAPACK (which are written in Fortran) and many essential R packages wrap native libraries that use Fortran, this conversion capability was crucial.

Fortran

By default, the emscripten-forge toolchain can cross-compile any C/C++ packages into WebAssembly; however, the first challenge in building the R stack was expanding the toolchain to also compile Fortran code. The two compilers that we considered for this task are LFortran and Flang.

LFortran is a compiler built on top of LLVM and designed from the ground up as a cross-compiler. Despite its potential, LFortran is still in its early stages compared to Flang and does not yet fully support all Fortran features.
LLVM Flang is the Fortran compiler of the LLVM project. Unfortunately, it does not support cross-compilation.

Working with Serge Guelton and Kerim Birgi, we initially experimented with the LFortran compiler and contributed to addressing issues encountered when building Netlib LAPACK. The BLAS implementation from this package served as a valuable testing ground since BLAS is one of the core dependencies for R.

After careful consideration, we found that Flang, with its ability to compile a wider variety of Fortran projects, brought us closer to building the Fortran components of R to WebAssembly. George Stagg, the main developer of the WebR project, has written a detailed article on using Flang to produce WebAssembly, which was adapted to the (then) latest version of LLVM (v18). Inspired by George Stagg’s endeavors, Serge posted upstream contributions to the Flang project (#99465, #99822, #101242, #105589), resolving issues with 32-bit platforms that were impacting our effort to support WebAssembly. All of these changes will be included in LLVM/Flang 20.0.

The remaining patches which cannot be merged upstream at this point involve:

(1) enabling code generation that targets WebAssembly (198c0e1),
(2) encoding target-specific size definitions for various data types given that the host and target platforms have different architectures (1da66cd), and
(3) disabling support for 128-bit floating-point types (f20c7c0).

A version of (2) that could be contributed upstream is in the works, which would be one step closer to making WebAssembly an official target for Flang. Additionally, (3) will no longer be needed starting with Flang v20.

These patches were integrated by Axel Obermeier into the Flang build recipe for the conda-forge distribution, making the modified build accessible to everyone. It can be installed on Linux with mamba or micromamba by executing:

micromamba install conda-forge/label/emscripten::flang libllvm19 --no-channel-priority

The final ingredient for our toolchain involved creating a compatible Fortran runtime library. Since Flang implements its runtime library in C/C++, compiling this library with Emscripten proved to be straightforward. We added a recipe for libflang to emscripten-forge, which packages the runtime library and includes all the aforementioned patches.

In the following example, we invoke Flang on a simple Fortran script, and execute the resulting JavaScript and WebAssembly with NodeJS.

> ls
hello.f90  libFortranRuntime.a
> cat hello.f90
File: hello.f90
program hello
 print *, "Hello, Fortran!"
end program hello
> source ~/emsdk/emsdk_env.sh
> flang-new --target=wasm32-unknown-emscripten -c hello.f90 -o hello.o
> emcc hello.o libFortranRuntime.a -o hello.js
> ls
hello.f90  hello.js  hello.o  hello.wasm  libFortranRuntime.a
> node hello.js
Hello, Fortran!

With a working Fortran-to-WebAssembly toolchain, we managed to compile BLAS and LAPACK (See the LAPACK recipe). But before we could get started with building R, we needed to tackle the remaining R dependencies.

Walking up the stack

In R (v4.4), there are a handful of dependencies that are required to provide basic functionality; these are: libiconv, zlib, bzip2, xz, pcre2, and libcurl. Thanks to the packaging efforts led by Thorsten Beier and Wolf Vollprecht, these dependencies are available on emscripten-forge with the exception of libcurl. Because of its socket-based networking architecture and reliance on system-level resource access, libcurl faces fundamental incompatibilities with WebAssembly’s sandboxed environment constraints. Therefore, libcurl had to be disabled in the R source code. As a consequence, downstream R packages which depend on libcurl cannot be ported to WebAssembly until a suitable libcurl replacement is available.

Apart from the core dependencies, R also requires a BLAS implementation. Although external BLAS libraries can be used with R, the R code base contains linear algebra routines from Netlib BLAS and a subset of LAPACK where both libraries are written in C and Fortran. To simplify configuration, we opted for these internal implementations instead of using external BLAS and LAPACK libraries.

Furthermore, several key optional dependencies were cross-compiled to WebAssembly to enhance R’s graphical capabilities; these include libpng, libtiff, cairo, and pango, along with their respective subdependencies. One particularly challenging aspect of packaging these libraries was the cross-compilation of glib, which both cairo and pango depend on. Compiling glib was only possible thanks to the efforts of Kleis Auke Wolthuizen whose patches made glib compatible with Emscripten, and Johan Mabille who disabled multi-threading.

The R dependencies are summarized in the following table:

Table of R dependencies

Cross-compilation of R

Although R typically configures and builds seamlessly on most unix platforms, cross-compiling is not straightforward because R uses a bootstrap approach; first it creates a minimal version of R which is then used to run several R scripts to complete the build process. Bootstrapping is a challenge because the built binaries for the target platform (WebAssembly) are not executable (at least not directly) from the host platform. Therefore, the r-base package necessitates two build phases. First, we compile R for the host platform (Linux) with GCC and Flang; this generates the R and Rscript executables needed for the second phase. And then, we cross-compile to WebAssembly with Emscripten and Flang.

One of the major challenges of this setup was generating the R Data Base (.rdb) and R Data Index (.rdx) files for the internal packages of r-base (base, compiler, grDevices, graphics, grid, methods, parallel, splines, stats, stats4, tools, and utils). To accomplish this, it was necessary to dynamically load some of the internal shared libraries (tools, grDevices, graphics, utils, stats, and methods) which provide essential functionality to generate the data and index files. However, the R executables we created during the first phase of the build are incompatible with the WebAssembly shared libraries. In order to work around this, we temporarily replaced the WebAssembly libraries of the internal packages with the linux shared libraries we built during the first phase (See cross_libraries.sh). This makeshift solution is also used to cross-compile all other R packages; the setup is carried out by the activation script of the cross-r-base package (See activate-cross-r-base.sh).

Additionally, the R source code required a few modifications to enable cross-compilation to WebAssembly.

The Emscripten platform was added to select the correct configuration for the platform.
The dependency on libcurl was removed (as mentioned in the previous section) and as a consequence, the internet package which relies on libcurl was disabled.
System calls which are not compatible with our WebAssembly environment were disabled.
Cairo replaced Xlib as the default bitmap type to enable graphics.

The complete list of patches applied to R is available here: r-base patches.

The R package ecosystem

Building R itself was just the beginning. The next challenge was to provide a wide array of R packages to create a robust development environment. Fortunately, many packages are written in pure R and are available as “noarch” packages on conda-forge, making them directly compatible with emscripten-forge environments. However, for the R packages that require compilation, a package recipe needs to be created and added to emscripten-forge. Thanks to the help of Anutosh Bhat, the following packages have been compiled and are available to use.

r-askpass     r-digest    r-haven      r-magrittr  r-purrr    r-tidyr
r-base64enc   r-dplyr     r-hexbin     r-mass      r-rcpp     r-tzdb
r-bit         r-ellipsis  r-htmltools  r-matrix    r-readr    r-utf8
r-bit64       r-fansi     r-isoband    r-mgcv      r-rlang    r-vctrs
r-cachem      r-farver    r-jsonlite   r-mime      r-sp       r-vroom
r-cli         r-fastmap   r-later      r-nlme      r-stringi  r-xfun
r-colorspace  r-ggrepel   r-lattice    r-plyr      r-sys      r-yaml
r-data.table  r-glue      r-lazyeval   r-promises  r-tibble

Most of the time, cross-compiling an R package is very simple because the bulk of the work is handled by r-base. Building a package typically boils down to a single line:

$R CMD INSTALL $R_ARGS .

Occasionally, minor patches are applied to the package source code to enable the WebAssembly target platform or to simplify cross-compilation. Some R packages, such as r-nlme, contain Fortran code; for such packages, the custom flang compiler needs to be installed before the package can be cross-compiled. Should any package be needed for your use case, feel free to contribute them to emscripten-forge!

Xeus-R

Once a sufficient set of packages is available for the platform, the next step is to provide a development environment for end users. Our chosen tool for this purpose is Jupyter.

The most popular Jupyter kernel for R is IRkernel, which has been developed over several years. It includes complete ZeroMQ bindings for the R language and comprehensive coverage of the Jupyter protocol. However, since IRkernel is tied to ZeroMQ, we opted to work with the Xeus-R project instead, which was announced last year. Xeus-R was developed by Romain François in collaboration with our team and with JupyterLite in mind. It is built upon the Xeus library, which provides a native implementation of the Jupyter protocol and is adapted to work with the JupyterLite frontend.

Fortunately, the components of IRkernel responsible for the rich representation of R objects and rich display in Jupyter frontends were split into two separate R packages: IRdisplay and repr. These packages can be reused in Xeus-R. As a result, Xeus-R and IRkernel offer the same rich display functionality, ensuring that Jupyter notebooks created with one kernel will work with the other. However, Xeus-R provides a different implementation of the communication layer through jupyterlite-xeus, which enables integration with the JupyterLite stack. This integration was previously implemented for Xeus-Python and Xeus-Lua.

One key aspect of this integration is the populating of the in-memory file system used by JupyterLite kernels with the required runtime dependencies, particularly shared libraries. This process is done upon bootstrapping the kernel. For this purpose, Anastiasiia Sliusar and Martin Renou developed mambajs, which, in combination with jupyterlite-xeus, processes conda packages in the frontend and installs them in the kernel filesystem at startup. In WebAssembly, shared libraries need to be handled with special care in order to function as expected. Drawing inspiration from the Python community, we adapted the pioneering work of Pyodide to properly load shared libraries into the filesystem.

How to make a deployment with custom packages

The GitHub repository https://github.com/jupyterlite/xeus-lite-demo is a template for creating a JupyterLite deployment on GitHub pages that includes the packages specified in a conda environment.

The process is as follows:

Create a new repository from the GitHub template.
Enable the deployment on GitHub pages from a GitHub action, as shown in the README.
Edit the environment file to include the desired packages.

For example, to deploy an R kernel with coursekata installed, the environment.yml file would contain the following:

name: xeus-r
channels:
  - https://repo.mamba.pm/emscripten-forge
  - conda-forge
dependencies:
  - xeus-r
  - r-coursekata

The deployment linked at the beginning of this article was created with this template.

How to contribute

Contributors are always welcome! New R packages can be requested by opening an issue on emscripten-forge, or added directly by opening a pull request with the desired package recipe. For instructions on how to contribute, please visit https://emscripten-forge.org/.

What is in the works

Currently, we are developing the infrastructure to dynamically install emscripten-forge and conda-forge noarch packages in JupyterLite kernel environments. This will apply to all Xeus kernels that make use of these packages, such as Xeus-Python, Xeus-R, and Xeus-Lua. In the case of R, this may be directly bound to install.packages.

Additionally, we are working on more advanced features for the R kernel, including support for Jupyter interactive widgets and the split of the pure R parts of xeus-r into a separate package, which would facilitate a possible convergence with IRkernel.

Finally, other language kernels based on the same package management stack are also in the works. Stay tuned for future announcements!

About the Author

Isabel Paredes, who led the charge on bringing R to emscripten-forge, is a senior scientific software developer at QuantStack. Prior to working on this project, she focused on porting the Robot Operating System (ROS) framework to WebAssembly.

Acknowledgments

This effort brought together several endeavors from many open-source developers.

JupyterLite, the Jupyter distribution that runs entirely in the web browser, was created by Jeremy Tuloup.
Xeus, the C++ library implementing the Jupyter kernel protocol, enabling a custom communication layer, and is foundational to kernels like xeus-r, xeus-python, running in JupyterLite, was created by Johan Mabille and is maintained by a broader team including Martin Renou, Sylvain Corlay, and Thorsten Beier, who worked on the first integration with JupyterLite.
Emscripten-forge, the distribution of conda packages for WebAssembly, was created by Thorsten Beier, who continues to lead the project. Many recipes were contributed by Isabel Paredes (for the R ecosystem), but also Kerim Birgi, Anutosh Bhat, Martin Renou, Wolf Vollprecht, and Johan Mabille.
Xeus-R, the Xeus-based Jupyter kernel for R, was created by Romain François.
IRDisplay and repr are pure R packages providing rich display and rich mime type rendering of many core R types, which are used in xeus-r. These two packages stem from the IRkernel project, a Jupyter kernel for R written in R, which was started by Thomas Kluyver, and has been maintained over the years by Philipp Angerer and Jan Katins.
MambaJS, the library enabling the processing of conda packages in the frontend, was developed by Anastasiia Sliusar and Martin Renou.
Conda-forge, the main distribution of conda packages, includes a large number of “noarch” recipes, including for R packages, and is developed by a broad community of contributors worldwide.
WebR, an R distribution for WebAssembly, was created by George Stagg. Although not directly used in this project, it documented numerous patches and changes to the Flang compiler that were adapted for this effort.

Funding

The work by Isabel Paredes and other QuantStack team members on bringing R and Xeus-R to Emscripten-Forge was supported by the Bill & Melinda Gates Foundation through a grant to CourseKata. CourseKata is a nonprofit project dedicated to improving statistics and data science education through interactive, research-backed curricula that integrate professional tools like R and Jupyter notebooks. This grant was part of CourseKata’s broader mission to scale its innovative curriculum, equipping students with the skills and confidence to engage with statistics, data science, and coding in meaningful ways.

Even though it powers the deployment of Jupyter on the main Jupyter website, JupyterLite is not an officially sanctioned Jupyter project. ↩︎

Meet Xeus-R: a future-proof Jupyter kernel for R

2024-01-22T17:20:00+00:00

JupyterLab session with a notebook powered by the Xeus-R kernel for R

The Jupyter ecosystem provides a versatile and user-friendly platform for data analysis and scientific computing, it has democratised access to powerful tools for researchers, scientists, educators, and data professionals.

The name Jupyter, a blend of Julia, Python, and R has always hinted at its relevance in multiple scientific programming languages and has since conquered new horizons thanks to its flexible and extensible design.

Underneath the interactive surface of Jupyter lie the Jupyter kernels, a set of language specific engines orchestrating the dynamic execution and exploration of code. Kernels implement the Jupyter protocol, a shared specification of the communication process between notebooks and kernels, defining how messages, commands and outputs are exchanged and processed.

The Xeus framework is a central piece of the puzzle, offering a reference to facilitate Jupyter kernel implementations. The speed, extensibility, and alignment with the official protocol make Xeus a perfect technology to implement new kernels.

Xeus-R

Today, we, a collaborative team led by Romain François and supported by QuantStack, are thrilled to announce the initial release of Xeus-R, a future-proof Jupyter kernel for R. Xeus-R builds upon the components of Xeus to facilitate the use of R in Jupyter notebooks and already supports typical capabilities: R code execution, inspection, completion, and contextual help.

The recent advancements in support for WebAssembly in the R world led by the WebR project, the flexible design of Xeus, and the rise of JupyterLite all contribute to our strong belief that Xeus-R has the potential to deliver Jupyter in-browser experience for data scientists using R.

Xeus-R is currently at the stage of a proof of concept, but we wanted to communicate about it early to broadcast our excitement about its potential, share some of our thoughts, and initiate a conversation.

Relationship with IRkernel

While the IRkernel project paved the way and is the current reference implementation of a Jupyter kernel for R, it does not take advantage of the Xeus infrastructure and includes a complete implementation of the Jupyter protocol, down to the composition of all of the JSON messages and most importantly how they are transported through ZeroMQ sockets.

In contrast, Xeus-R fully embraces and leverages the capabilities of Xeus, while borrowing some higher level concepts from IRkernel and reusing features from its dependent packages IRdisplay and repr.

We believe this path will benefit future development as it gives us the freedom to work on innovations that are specific to the needs of R users, and let Xeus absorb evolutions of the Jupyter protocol.

Key features

Xeus-R implements the basic building blocks of what you would expect from a Jupyter kernel. The results of a code cell, i.e. the value of its last expression are displayed in the output, taking advantage of ANSI escape codes for colours as were made popular by the crayon and later cli R packages.

Traditional and ggplot2-powered graphics are captured and displayed as images. The aspect ratio of plots can be controlled by setting the repr.plot.height and repr.plot.width options either globally, or locally via the cell_options() function.

Thanks to the scaffolding in place in the IRdisplay and repr packages, Xeus-R is also able to emit richer displays through mime bundles.

In contrast with IRkernel, the rich displays need to be explicitly requested with the display() function. There are cases however, such as HTML widgets, where automatic rich display makes the most sense because a text/plain representation is not valuable.

Xeus-R also supports contextual help and includes a basic completion system, based on the internal utils::.getTokenFromLine() system.

Installation and Getting Started

At the time of writing this article, Xeus-R is only available and tested to work on Unix-based systems. We consciously decided to focus initial efforts on features rather than platform comprehensiveness, but we should be able to fix this soon enough.

The easiest way to get up and running with Xeus-R is to install it from conda-forge:

mamba install xeus-r

Instructions to build from source are available in the github repository for Xeus-R: https://github.com/jupyter-xeus/xeus-r

Looking ahead: JupyterLite Integration

Another benefit of a Xeus-based approach is that it has been designed in a way that supports in-browser experience to power JupyterLite. Together with the recent developments around WebR, we can envision R becoming a first-class citizen in the JupyterLite world.

This is an exciting vision and will serve as our North Star ⭐️ for future developments of Xeus-R.

Conclusion

Join us in welcoming Xeus-R, a Jupyter kernel for R that leverages the robust foundation of Xeus architecture. R is a language of paramount importance for Data Science and Scientific Computing and it rightfully deserves its place as a first-class citizen within the Jupyter ecosystem.

We firmly believe that Xeus-R has the potential to become a reference implementation for an R kernel, well positioned to tackle the challenges of the ever-evolving landscape of in-browser Scientific Computing, a promising future led by the JupyterLite and WebR projects.

Acknowledgements

The work by Romain François on Xeus-R was funded by QuantStack.

Author Bio

Romain François is a prominent figure in the R community, recognised for his longstanding contributions to R programming and Data Analysis. With a profound passion for open-source software and commitment to advancing the capabilities of R, he has earned a well-deserved reputation as a respected expert.

Romain’s expertise encompasses a wide spectrum of R-related fields, including package development, data manipulation, and interoperability with other programming languages. He is widely recognized for his work on packages like dplyr, which has become a cornerstone for efficient data manipulation in R, and for his contributions to Rcpp, a library that bridges the power of C++ with R for optimizing R packages and enhancing performance.

Plug your application into the Jupyter world

2023-10-23T06:09:00+00:00

Kernels are a simple but powerful abstraction in the Jupyter architecture. They encapsulate language interpreters and make them accessible through a standardized interface. This is the key to Jupyter’s remarkable versatility, with over 100 supported languages. Embedding a kernel in your custom application can seamlessly expose it to the Jupyter ecosystem. Not only can you attach JupyterLab to your program for e.g. state inspection, as you would do with a debugger, but you can truly extend your application with all the power of the Jupyter ecosystem.

Applications: GUI and API

Several applications such as FreeCAD, Blender, or QGIS provide a Python API to access their functionalities programmatically, and include an embedded Python console which allows using this API from within the application. Users can both use the graphical user interface and operate on the application in a scripted fashion depending on their needs.

Python consoles in FreeCAD, Blender, and QGIS (from left to right).

If you think about it, these are the kinds of workflows that Jupyter has always been about, but Jupyter has evolved in the opposite direction. Instead of adding scripting to GUIs, Jupyter added GUIs to scripting. At first, Jupyter was IPython, a console for interactive computing. Then it went to the Web and became IPython Notebook, renamed to Jupyter Notebook. With the browser it became possible to display richer content and create small GUIs, using Jupyter Widgets. Then JupyterLab was created and allowed to create fully-fledged desktop-like applications in the browser, like 3-D CAD modelers.

JupyterLab showing notebooks using JupyterCAD and ipyleaflet.

Jupyter put a lot of attention on building a good experience interacting with the Python REPL, which is not the main focus of domain-specific desktop applications. The Python console embedded in desktop applications is often quite raw. Allowing users to connect from Jupyter can enable a much better experience.

What if there was a bridge that allowed you to connect both worlds, your application on one side and Jupyter on the other side? What is the common denominator allowing code to interact with an application, while also being shared with Jupyter? In the Jupyter infrastructure, we call it a kernel.

Jupyter kernels

Kernels provide a standardized way to expose language interpreters to applications, in a language-agnostic fashion. You can think of it as a server that accepts requests such as “execute this piece of code” and that replies with a result. The result is actually a data representation, it can be as simple as “3” for an input of “1+2”, or a complex interactive plot. A kernel usually lives in a separate process on your machine, but it can also run in the same process, or even remotely on another machine. Someone wanting to connect to this kernel will need the connection information such as the IP address and the ports of the different communication “channels”. It is stored in what is called a connection file.

The Jupyter kernel protocol allows clients to interact with different languages.

Anyone who has access to this connection file can then execute code in the kernel. This is “shared code”, meaning that every client to this kernel eventually runs code in the same interpreter. If one client sets a variable “a=1”, other clients will see this variable. They are all running the “same program”. If the kernel runs in the same process as the application, then the kernel can interact directly with it. This way of sharing the execution engine serves as a bridge between the application and Jupyter.

Now it is just a matter of exposing the connection file so that e.g. JupyterLab can see it. The application launching the kernel can save the connection file to a specific directory, and give the path to JupyterLab. And that’s it, JupyterLab can run a notebook using this kernel, allowing it to interact with the application.

Getting started

The xeus-qt-python repository is a great place to start and see how to embed a Python kernel inside a Qt application. Under the hood, it uses xeus-qt, an integration of the xeus kernel architecture in the Qt event loop. The Qt application actually runs JupyterLab in a Qt WebEngine window. Users can run code in a notebook and interact with the application, which in turn can interact with the notebook. Here, we create a Qt button from a notebook. When clicking on the button, a callback prints a message in the notebook. This demonstrates a two-way binding between the Qt application and JupyterLab.

Example of a Qt application embedding JupyterLab.

SlicerJupyter from Kitware was one of the first applications to take advantage of this approach. You can read more in this blog post. We hope that this technology will unlock a new brand of use-cases and allow applications to reach a new audience through the Jupyter ecosystem. Please reach out to us if you need help!

About the author

David Brochart is a technical director at QuantStack. He mostly works on the Jupyter backend stack. He contributed to the development of real-time collaboration in JupyterLab, and created the Jupyverse project, a next-generation implementation of the Jupyter server.

Acknowledgments

This work was made possible thanks to several contributors. Mariana Meireles was the initial author of xeus-qt. Thorsten Beier is a scientific software developer at QuantStack and contributes to the xeus kernel ecosystem. He is the initial creator of the xeus-qt-python demonstration. Johan Mabille is a technical director at QuantStack and he is the main author of xeus. Afshin Darian is a technical director at QuantStack and worked on the UI in JupyterLab 4.0 allowing to use an external kernel.

A Jupyter kernel for GNU Octave

2023-01-11T17:33:00+00:00

GNU Octave is a long-standing member of the scientific computing ecosystem, featuring a mathematics-oriented syntax with built-in 2D/3D plotting and visualization tools, and compatible with the Matlab syntax. It is a natural candidate for high-quality integration in the Jupyter ecosystem.

Jupyter has been devised with language agnosticism in mind since it was spun off from the IPython project. The Kernel (the part of the infrastructure responsible for executing the code input by the user) is a key extension point to Jupyter, and dozens of language kernels exist in the ecosystem.

Prior art on Jupyter / GNU Octave integration includes the Calysto/Octave_kernel project by Steven Silvester (who is also a core Jupyter maintainer and the co-creator of JupyterLab). This existing kernel is part of the family of kernels built on top of the ipykernel reference implementation of the Jupyter protocol, in Python.

Today, we are happy to announce the xeus-octave project, a Jupyter kernel for GNU Octave. Xeus-octave was created by Giulio Girardi, recently joined by Antoine Prouvost — and has been incorporated into the Project Jupyter governance.

Xeus-octave is built upon the Xeus library, a C++ implementation of the Jupyter kernel protocol that enables fast development of new kernels using programming languages’ native APIs, unlocking new possibilities without parsing standard outputs or requiring a Python interpreter at runtime. After all, a Jupyter kernel is merely an executable providing a well-defined communication protocol. It is not bound to Python APIs.

A fully-featured kernel

We strove to make the xeus-octave kernel as complete as possible for this first iteration.

Multiple graphical toolkits

Notebook

A notebook Octave graphical toolkit can be used to render plots natively, and present the figure as an image in the notebook.

The classic Octave sombrero plot rendered as an image inside the notebook

Plotly

An experimental plotly Octave graphical toolkit can also use Plotly to render code in the browser, with dynamic views, zooms, and hover tooltips.

Multiple plots rendered in a single plotly figure

Rich display

The kernel binds with the C++ interface of GNU Octave to provide a complete Octave experience. Having access to the exact internal type of the objects, xeus-octave can present data with rich output.

Matrices can be presented as HTML or LaTeX tables.

Symbolic expressions can be presented as LaTeX formulas.

Structured data can be visualized as interactive tree views

Interactive help

Formatted help, extracted from the GNU Octave reference manual, can be queried using the ? syntax.

Trying it online

You can try the new Jupyter kernel for GNU Octave by clicking on the link below:

The Binder linked above includes examples of notebooks in the fields of Electronics, demonstrating the usability of xeus-octave in real-life complex scenarios.

This interactive demo is also featured on the Jupyter website at https://jupyter.org/try#kernels.

Installing xeus-octave

Xeus-octave has been packaged on conda-forge and can be installed with mamba or conda.

$ mamba install xeus-octave -c conda-forge

Future Work

In future iterations, we will work towards consolidating the xeus-octave experience, by adding cell magics and improving the Plotly integration.

A major ongoing development is the implementation of an Octave backend for Jupyter interactive widgets, based on the xwidgets project. Combined with Voilà, it will be possible to use GNU Octave to create standalone web dashboards.

Another ongoing endeavor is to port a growing portion of the scientific computing stack to WebAssembly (using the emscripten-forge project). Producing a Wasm build of GNU Octave will enable its use with Jupyterlite, enabling large-scale deployments on websites, blogs, and arbitrary web pages without any need for scalable cloud infrastructure.

About the authors

Antoine Prouvost is a Scientific Software Engineer at QuantStack and a xeus-octave maintainer. Previously, Antoine was researching Machine Learning and Combinatorial Optimization at the DS4DM research chair and Mila.

Giulio Girardi is a Master’s degree student in Electronic Engineering at the UNIPD, currently working as an Electronic Engineer at Protech Engineering. Giulio is the creator and a maintainer of the xeus-octave project.

Mamba meets JupyterLite

2022-07-14T11:19:00+00:00

JupyterLite is a Jupyter distribution that runs entirely in the web browser without any server components. To achieve this, all language kernels must also run in the browser.

JupyterLite running in the browser as a static website

A significant benefit of this approach is the ease of deployment. With JupyterLite, the only requirement to provide a live computing environment is a collection of static assets. It makes it possible to embed a console or a notebook interface on any static page or blog without having to deal with a server architecture deployment. The scalability of this approach allowed several major projects of our ecosystem (NumPy, SymPy, Pandas, PyMC, and many more) to embed interactive examples on their websites, which are visited by millions of users monthly.

JupyterLite is the easiest and most scalable way to embed an interactive console or notebook on a web page without any server component.

The most prominent JupyterLite kernel is the Pyolite Python kernel, which is based on the Pyodide distribution for WebAssembly. Beyond the CPython interpreter, Pyodide includes many popular scientific computing packages such as NumPy, Pandas, and Matplotlib. Pyodide also provides a foreign function interface (FFI) that allows calling Python from JavaScript and vice versa.

The JupyterLite inline console embedded on the SymPy project website

Pyodide — and beyond

While Pyodide provides many scientific computing packages, its monolithic distribution model does not allow to specify package versions, although versions of pure python packages installed on top can be set. Our goal is to enable the composability of computing environments allowed by package managers and to adopt the conda-forge model for large-scale software distribution crowdsourcing.

Being able to pin down package versions in an environment is a strong requirement for software reproducibility. In fact, a locked WebAssembly environment could be seen as a reproducibility time capsule. As WebAssembly is a recognized web standard, it ought to be runnable for much longer than native binary packages: these are bound to a combination of architecture and platform and will eventually require an emulator.

This is why we developed a mamba-based distribution of WebAssembly packages built with Emscripten.

Emscripten-forge

The choice of the Mamba/Conda package manager was natural. Its main strength is the conda-forge community-maintained distribution, which has become the de facto standard source of packages for scientific computing.

Beyond its solid technological foundations and the multi-platform nature of the conda-forge distribution, its main strength is its social model. It allowed for a crowdsourcing approach of the packaging problem, with a balance of separation of concerns between maintainer teams and across-the-board automation, plus an amazing maintainers community.

We plan on contributing this work to the conda-forge project, so that all recipes live in the same space.

The Mamba/Conda package manager has support for many platforms and architectures such as Linux, OS X (for both x86 and arm64), and Windows. However, the WebAssembly family of platforms is not supported yet.

Adding support for WebAssembly to mamba & conda

To create conda packages for the WebAssembly platform, we relied on the Emscripten toolchain.

We defined a new target platform for conda-build and boa, namely wasm32-unknown-emscripten for which we use the emscripten-32 shorthand name. We then associated the Emscripten compiler, wrapped in a conda package as the C/C++ compiler for this new target.
This already allowed us to build many packages, including simple libraries like bzip2 and zlib, but also more complex packages like Python.
For Python extension modules we used crossenv which can create virtual environments for cross-compiling, and cross-python which integrates crossenv into conda. All the code and recipes for cross-compilation are hosted on the emscripten-forge GitHub repository.
We then used GitHub actions to build packages with Emscripten and upload them to a package server.
Packages are hosted on a deployment of the Quetz open-source server.

With this, you can easily create an environment for the emscripten-32 target:

micromamba create -n my-env --platform=emscripten-32 \
    -c https://repo.mamba.pm/emscripten-forge \
    -c https://repo.mamba.pm/conda-forge \
    python ipython numpy jedi

Note that we not only added emscripten-forge as a channel, but also conda-forge. This means all noarch packages can be used.

Adding new packages to the emscripten-forge channel

Adding new packages is a simple procedure:

fork the repository https://github.com/emscripten-forge/recipes
create a folder for your package in recipes/recipes_emscripten/<my_package>
add a recipe.yaml for your package in recipes/recipes_emscripten/<your_package>
create a pull request containing the recipe. Once the pull request is merged, the package is automatically uploaded to the emscripten-forge channel.

Integration with JupyterLite

Even though this is a general-purpose conda-based distribution for Emscripten packages, we had one particular application in mind for this first iteration: JupyterLite. The existing Pyolite kernel is too tightly coupled with the Pyodide distribution, so we decided to go with xeus-python instead.

The main reason for picking xeus-python (over ipykernel) is that with xeus-based kernels, it is possible to override the communication layer of the kernel (switching e.g. from ZMQ to HTTP/2). In the case of JupyterLite, the implementation simply relies on direct JavaScript function calls.

You can check out our earlier blog post for more details on the JupyterLite xeus-based kernels.

Providing a complete Python development experience

Some remaining intrinsic limitations to the WebAssembly platform need to be worked around to provide a complete experience to end-users. For example, sockets cannot be created in WebAssembly, preventing the use of the default asyncio event loop implementation. Luckily, the Pyodide authors developed a custom asyncio event-loop called WebLoop: it wraps the browser event loop using the Python — JavaScript foreign function interface (FFI) provided with Pyodide.

Pyjs

Since it is non-trivial to extract Pyodide’s FFI and use it for other projects, we created a modern Python - JavaScript FFI from scratch. This was done with the following tricks:

Pybind11 is used to call Python from C++ and vice versa,
Embind is used to call JavaScript from C++ and vice versa.

When we use Pybind11 and Embind together we can call Python from JavaScript and vice versa, with C++ as a man in the middle. This not only allows us to write a simple FFI from scratch with relatively little code but also avoids calling any low-level CPython APIs and enables using high-level constructs — like pybind11::object and emscripten::val— instead.
The code is available in the pyjs repository. The API is very similar to Pyodide’s so that it can be used as a drop-in replacement in code, like Pyodide’s WebLoop implementation.

Deployment

The xeus-python-kernel allows conda packages to be pre-installed in the Python runtime. This can be done by passing the XeusPythonEnv.packages CLI option to jupyter lite build. The following command will install NumPy, Matplotlib, and ipyleaflet:

jupyter lite build --XeusPythonEnv.packages=\
    numpy,\
    matplotlib,\
    ipyleaflet

Running the xeus-python kernel with the ipyleaflet widget in JupyterLite

More details can be found in the xeus-python-kernel GitHub repository.

What about the future?

This combination of JupyterLite and Mamba has the potential to open Jupyter to millions of additional users.
Given its scalability, ease of deployment, reproducibility, and accessibility, JupyterLite will be everywhere: countries, organizations, and schools that don’t have access to sovereign cloud infrastructure will be able to deploy Jupyter-based education platforms on servers that they truly own, without endangering the data of their students or becoming too reliant on resources that they do not control.

In the short term, we are working on the following “next steps”:

Mambalite: To support the installation of packages at runtime. Similar to Pyodide’s pip-lite, it will allow downloading packages at runtime.
Fortran: Compiling Fortran code with Emscripten is currently not supported, but it is necessary for key packages like SciPy. Pyodide relies on f2c, a Fortran-to-C converter, in conjunction with a set of patches to compile Fortran code with Emscripten. We are working on a more direct approach: compiling SciPy natively with LFortran.
Binderlite: Binder converts a repository of notebooks into an executable JupyterLab environment, making code immediately reproducible by anyone, anywhere. Emscripten-forge is the missing piece to build BinderLite, a version of Binder relying on JupyterLite instances instead of vanilla JupyterLab instances.
Rust/PyO3 support: We are working on integrating the work of the Pyodide team on Rust/PyO3 support in emscripten-forge. This will be important to build Rust extension modules like cryptography.

Credits

This was built upon the work of a much bigger crowd!

The Pyodide team

The Pyodide project was started at the Mozilla foundation by Michael Droettboom and is now maintained by Hood Chatham, Roman Yurchak, and Gyeongjae Choi.

The foundational work of the Pyodide project pioneered the use of Python in the browser and made all of the rest possible, from JupyterLite to this work.

The Emscripten team

Both Pyodide and emscripten-forge are built upon the Emscripten toolchain, which provides the foundational components to be able to meaningfully run WebAssembly programs in the browser.

The JupyterLite team

The JupyterLite project was started by Jeremy Tuloup, with significant contributions from Nick Bollweg and Martin Renou.

The Mamba Org team

The mamba ecosystem has been instrumental in making these developments possible. We use the Quetz open-source server for hosting the packages and the Boa tool to build them. In the mamba development team, we should highlight the work of Wolf Vollprecht, Johan Mabille, Joel Lamotte, and Andreas Trawöger.

The Xeus team

The xeus project was started by Johan Mabille and Sylvain Corlay. It is at the foundation of the JupyterLite integration and helped to get all the pieces together (Xeus, Mamba, Jupyter). We should especially credit the work of Martin Renou and Thorsten Beier on this integration with JupyterLite.

Acknowledgment

The work of Thorsten Beier, Johan Mabille, Martin Renou, Sylvain Corlay, Wolf Vollprecht, Joel Lamotte, and Andreas Trawoger at QuantStack was funded by Bloomberg.

About the Authors

Thorsten Beier

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open-source developer, Thorsten worked on a variety of projects, from xeus and xtensor in C++ to inferno, kipoi, ilastik, and napari-splineit in Python.

Martin Renou

Martin Renou is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he studied at the French Aerospace Engineering School SUPAERO. He also worked at Logilab in Paris and Enthought in Cambridge. As an open-source developer at QuantStack, Martin worked on a variety of projects, from xsimd, xtensor, and xframe in C++ to ipyleaflet and ipywebrtc in Python and JavaScript.

Xeus-Lite

2021-10-25T09:40:00+00:00

JupyterLite is a JupyterLab distribution that runs entirely in the web browser, backed by in-browser language kernels. Xeus is C++ library for writing Jupyter kernels. In this blogpost we show how these two can work together.

JupyterLite

With vanilla Jupyter, kernels are running in separate processes. The communication between the kernels and the client is achieved with ZeroMQ. In contrast to this, JupyterLite runs entirely in the browser, including the language kernels. The following kernels are already implemented in the JupyterLite framework:

JavaScript Kernel
P5 Kernel: Since P5 is a JavaScript library, it is easy to provide a P5 kernel running in the browser.
A Python kernel based on pyodide called pyolite: While the two JavaScript based kernels are trivial to implement as in-browser kernels, pyodide is a Python interpreter compiled to WebAssembly via Emscripten. Not only Python itself has been compiled to WebAssembly, but also the whole Scientific Stack including packages like: Numpy, Scipy, Pandas and many more. An overview of packages can be found here. Furthermore any pure Python package can be installed from within pyolite.

Since JupyterLite runs entirely in the browser, it is very easy to try it out!

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing.
With the help of Xeus, it is very easy to write Jupyter kernels . In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-lua is a Xeus-based Lua kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework.
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

Since it is very easy to write Xeus based kernels for Jupyter, we also want to leverage Xeus for writing JupyterLite kernels. This has been the main motivation for Xeus 2.0.

Xeus Architecture

Xeus is has three extension points as depicted below

The Server is the middleware component responsible for receiving and sending messages to the Jupyter client. The default server it is built upon ZeroMQ and handles the concurrency model of the application.
As we will see later, a custom server is needed to connect Xeus with JupyterLite.
The Interpreter provides the interface that kernel authors must implement.
The Debugger provides the interface that can to be implemented when the kernel shall have debug capabilities, but we do not yet support this for Xeus-JupyterLite-Kernels.

JupyterLite Xeus-Kernels

Traditionally, the Jupyter kernel and the Client to the Jupyter kernel run in separate processes as depicted below:

The communication between the kernels and the client (ie the Jupyter-server) are done with inter-process communication via ZeroMQ. But with JupyterLite the kernel runs directly in the browser. In that case, ZeroMQ is not required anymore to communicate with the frontend. This was the motivation for Xeus 2.0, where it is possible to build Xeus without ZeroMQ. To implement JupyterLite Kernels with Xeus, we need to implement a custom Server which handles the Emscripten/Embind-based communication layer

The overall architecture is best described with the picture below: The C++ Xeus-Kernel is compiled to WebAssembly with Emscripten / Embind. The WebAssembly compiled code runs in a dedicated WebWorker thread. The WebWorker communicates via Message Passing with a thin layer of TypeScript which implements the IKernel API of JupyterLite. The communication between JupyterLite and the JupyterLite k ernel is done entirely in JavaScript.

The C++ code needed to compile an existing xeus-kernel to WebAssembly is very minimal:

Furthermore one needs a thin layer of TypeScript / JavaScript to implement the JupyterLite IKernel API. A good example of this code is given in the JupyterLite Xeus-Lua repository. While most of the repositories code is boilerplate, one needs to provide a simple Dockerfile which compiles the Xeus-Kernel to WebAssembly.

Xeus-Lua

We recently released a Xeus based Lua kernel. Even though its worthwhile to have Xeus-Lua kernel for vanilla Jupyter, the main purpose of Xeus-Lua was to experiment with Xeus-based JupyterLite kernels.
We choose Lua as the first language to try out Xeus-based JupyterLite kernels since Lua is easy to compile to WebAssembly.
A JupyterLite instance with Xeus-Lua included can be found here

Xeus-Lua has built in support for interactive widgets

Xeus-Wren

Wren is relatively new “Classy Little scripting Language” created by the author of the great book “Crafting Interpreters”. Wren can be described with the following:

Think Smalltalk in a Lua-sized package with a dash of Erlang and wrapped up in a familiar, modern syntax.

We choose Wren as a second language to experiment with Xeus + JupyterLite since it is small, simple to compile to WebAssembly and easy to embed in C++. A JupyterLite instance with Xeus-Wren included can be found here.

Xeus-Wren has built in support for displaying data of various MIME-types

Outlook

There are many languages which can be compiled to WebAssembly like:

Julia: https://github.com/Keno/julia-wasm
Octave: https://news.ycombinator.com/item?id=22926120
C++: https://github.com/binji/wasm-clang
Python: https://github.com/pyodide/pyodide

Also databases as sqlite can be compiled to WebAssembly:

Sqlite: https://github.com/sql-js/sql.js/

Our goal is to provide Xeus-based JupyterLite kernels for these projects to make the JupyterLite ecosystem even more powerful.

Acknowledgements

We like to thank

Jeremy Tuloup, Madhur Tandon, Martin Renou, and Nicholas Bollweg for their work on JupyterLite.
Johan Mabille for Xeus and his help implementing the Xeus-Emscripten server.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty and vigra in C++ to inferno, kipoi and ilastik in Python.

Xeus 2.0

2021-09-28T09:41:00+00:00

We have just released Xeus 2. This is a major release of the library. While it includes backward-incompatible changes, they are very limited and upgrading your kernels should be relatively easy.

How to upgrade Xeus-based kernels

Let’s illustrate that with the code of a simple kernel before and after upgrading to Version 2. Most typically, only the kernel’s main file will have to be updated:

diff --git a/example/src/main.cpp b/example/src/main.cpp
index 29de437..eb8fdba 100644
--- a/example/src/main.cpp
+++ b/example/src/main.cpp
@@ -21,12 +21,14 @@ int main(int argc, char* argv[])
     std::string file_name = (argc == 1) ? "connection.json" : argv[2];
     xeus::xconfiguration config = xeus::load_configuration(file_name);

+    auto context = xeus::make_context<zmq::context_t>();
+
     // Create interpreter instance
     using interpreter_ptr = std::unique_ptr<custom::custom_interpreter>;
     interpreter_ptr interpreter = interpreter_ptr(new custom::custom_interpreter());

     // Create kernel instance and start it
-    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(interpreter), xeus::make_xserver_zmq);
+    xeus::xkernel kernel(config, xeus::get_user_name(), std::move(context), std::move(interpreter), xeus::make_xserver_zmq);
     kernel.start();

     return 0;

The first change is that you now have to instantiate and pass an additional argument to the kernel constructor: the context. Fortunately, Xeus provides a function to build such a context object, making the update straightforward. We will detail the reason for this change in the next section.

The second change is that the kernel constructor now requires a function to build the server while a default value was provided for it in Xeus 1.x. Again, the motivations for this change will be detailed in the next section. Finally the functions for building the servers have moved to different headers.

To summarize, here is the list of the breaking changes:

The kernel’s constructor now requires an additional argument, the context. Xeus provides a function to easily instantiate it.
The order of kernel’s constructor arguments has changed, and the function for building the server is now required.
The functions for building the servers have moved to the headers where the corresponding servers are defined.

Motivation for these changes

When we created Xeus, kernels were expected to run in separate processes. The communication between the kernels and the client was achieved with the ZeroMQ middleware framework, and it was natural to heavily rely on it in Xeus.

Recent developments have pushed in a different direction, where the kernel can run directly in the browser. In that case, a middleware library is not required anymore to communicate with the frontend. The architecture of Xeus was not suitable for this new kind of kernels. An update was required so that it is possible to build Xeus without the dependency on ZeroMQ.

About the developers

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of Xeus, and developed the debugger extension to xeus-python.

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored Xeus and xeus-python.

From Jupyter to the Moon

2021-09-24T12:24:00+00:00

Jupyter architecture in a nutshell

A key principle in Jupyter’s design is language agnosticism, and one of the main extension points of the Jupyter ecosystem is the kernel, the part of the architecture responsible for executing the user’s code.

In the end, a Jupyter kernel is merely an executable implementing a well-defined inter-process communication protocol.

A reference implementation of the kernel is ipykernel, which provides the default Python kernel, and utilities to build other language kernels. Another reference implementation of the protocol is Xeus.

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing. There is a long list of scripting languages which can be embedded in C/C++. With the help of Xeus, it is very easy to write kernels for such languages. In fact, Xeus has already been used for making several language kernels:

xeus-cling is a Xeus-based C++ kernel.
xeus-python is a Xeus-based Python kernel.
xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
xeus-robot is a Jupyter kernel for the Robot Framework.
LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

xwidgets

The xwidget library is a C++ implementation of the Jupyter Interactive Widget protocol. Xwidgets were initially developed for xeus-cling, the Xeus C++ kernel, but any Xeus-based kernel can reuse these components.

A xeus-based Lua kernel

Today, I am pleased to announce the release of xeus-lua, a new Jupyter kernel for Lua based on Xeus.

Lua

Lua, which means “Moon” in Portuguese, is a lightweight embeddable scripting language written in C. While Python is one of the leading scripting languages for scientific application, Lua is the leading scripting language used in games / game engines. The list of video games using Lua is long and includes triple-a games like Crisys and Farcry. Not only games rely on Lua, but also applications like Adobe Photoshop Lightroom.

Sol2

With the Sol2 library, Lua can be embedded into C++ in a very elegant way.

While Lua itself can be embedded in a C++ application quite easy, it is even easier when using Sol2. Sol2 is a C++ library that simplifies the usage of Lua from C++. Furthermore Sol2 can be used to expose C++ classes to Lua, in a similar fashion as pybind11 can be used to expose C++ classes to Python.

Xeus-lua

The compulsory “Hello, World!” from xeus-lua. This blog-post would be incomplete without it.

I am pleased to announce xeus-lua, a new Jupyter kernel for the Lua language based on Xeus. Using Sol2 and Xeus, it was very easy to implement a feature rich kernel using modern C++. With the help of Sol2, it was trivial to expose the xwidgets classes to Lua such that we can use the widgets within xeus-lua notebooks.

But why a Lua kernel?

While Lua might not be the hippest language in 2021, there are multiple reasons while I choose to implement such a Lua kernel, and why such a kernel can be useful:

Writing a Xeus-based kernel is the best way to learn and familiarize oneself with the xeus ecosystem. And since Lua is so easy to build and embed, Lua and Xeus are perfect match. Implementing this kernel gave me great insights into Xeus and Jupyter itself.
Learning Lua with xeus-lua is fun! Before writing xeus-lua, I had zero Lua experience. While implementing xeus-lua I learned Lua from within Jupyter by playing around with xwidgets from within Lua — this was a very joyful experience.
The xeus-lua implementation might serve as reference for future kernel authors. In particular the integration of xwidgets in xeus-lua might be useful for future implementations.
Since Lua is so easy to build, xeus-lua can be built for WebAssembly and will be integrated in JupyterLite — stay tuned!

Features

Obviously xeus-lua supports the execution of code.

Simple code execution

Streams are redirected to the notebooks output

Redirection of streams

Errors messages are shown in the notebooks output

Error Handling

Input streams are redirected to the notebooks input system

User Input

Statements can be auto-completed with a press of tab

Code Completion

Data of different sources as Latex, JSON and html can be displayed.

Display Data

Interactive widgets can be added to the notebook!

Widgets

We can feed the widgets with data!

Binary buffers for widgets

Try It Out

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Other Lua kernels

There are several Lua kernels available:

lupyter: A Python based implementation of a Lua Jupyter kernel. The heavy lifting is done in a C-extension.
ILua: A feature rich implementation. Ilua is implemented in Python and does the communication with Lua via IPC.
IPyLua: IPyLua is a pure Lua implementation.

While these implementations are feature rich and work well, none of the above supports interactive widgets. Since xeus-lua is based on Xeus, we can reuse components from the Xeus ecosystem like xwidgets with minimal effort.

Acknowledgments

We are grateful to Juliette Taka for the illustration.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty, and vigra in C++ to inferno, kipoi and ilastik in Python.

Abracadabra! Bringing the magics to xeus-python

2021-02-18T13:30:00+00:00

Last year, we set ourselves to implement a visual debugger for JupyterLab. This endeavor required major developments in the JupyterLab front-end, in core-Jupyter protocols, and on the kernel side (the part of the Jupyter infrastructure responsible for executing the code).

For kernel-side development, we decided to start with the xeus-python kernel, a lightweight implementation of a Jupyter kernel for the Python programming language. Based on xeus, xeus-python has a flexible concurrency model that was required to e.g. add breakpoints while executing code.

Unfortunately, xeus-python did not provide all ipykernel features (magics, Matplotlib support, etc.). Furthermore, many notebooks depend on IPython, as they import it explicitely or make use of the IPython configuration system.

Today, we are proud to announce that xeus-python supports 100% of the IPython magics! This was achieved by leveraging the core IPython package. This is getting us closer to feature parity with ipykernel.

Magics

xeus-python now supports all magics that IPython provides and even user-defined magics!

Support for builtin IPython magics and user-defined magics

Matplotlib

Most Matplotlib back-ends are now supported by xeus-python, allowing you to show static plots with the inline back-end or interactive plots with ipympl in your Notebook:

Matplotlib inline back-end and ipympl interactive back-end

Extending

IPython has a rich configuration system that you can tweak in order to get an enhanced Notebook experience. Xeus-python now supports any configuration you setup for IPython:

Automatically import NumPy in the IPython settings

Visual Debugging

The latest JupyterLab version introduced a visual debugger in its interface. xeus-python was the first Jupyter kernel to add support for it!

Stepping into Python code in JupyterLab with the visual debugger

xeus-python is not the only Jupyter kernel that supports debugging anymore: xeus-robot is a xeus-based Jupyter kernel for the RobotFramework language that supports the JupyterLab visual debugger.

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Installation

You can install the latest xeus-python version using mamba or conda:

mamba install xeus-python -c conda-forge

conda install xeus-python -c conda-forge

You can also compile it yourself, instructions can be found in the repository:
https://github.com/jupyter-xeus/xeus-python

Acknowledgments

My work on xeus and xeus-python at QuantStack was funded by Bloomberg.

About the Author

My name is Martin Renou, I am a Scientific Software Engineer at QuantStack. Before joining QuantStack, I studied at the aerospace engineering school SUPAERO in Toulouse, France. I also worked at Logilab in Paris, France and Enthought in Cambridge, UK. As an open-source developer at QuantStack, I work on a variety of projects, from xtensor and xeus-python in C++ to ipyleaflet and bqplot in Python and JavaScript/TypeScript.

An SQL solution for Jupyter

2021-02-03T12:03:00+00:00

A few months ago we released xeus-sqlite, a Jupyter kernel that allows users to make SQLite queries directly from the notebook. With the needs of the Jupyter data science community in mind we decided to expand on this project.

RDBMSes supported by xeus-sql

Today, we’re happy to announce the next step of this endeavour, bringing you xeus-sql a general-purpose database access tool for Jupyter, which allows you to make SQL queries on many different databases:

MySQL
PostgreSQL
SQLite3
DB2
Oracle
Firebird
and any database with an ODBC driver.

To offer all these integrations we rely on the SOCI library as the backbone of the project. SOCI abstracts all the different database connections and query details behind a unified C++ API. xeus-sql uses SOCI and xeus to expose SQL functionalities to Jupyter.

Xeus-sql, as well as the MySQL, PostgreSQL and SQLite3 backends of SOCI are available on conda-forge, meaning you can install them easily with conda or mamba:

# Installing xeus-sql and the mysql backend
mamba install xeus-sql soci-mysql -c conda-forge

# Installing xeus-sql and the postgresql backend
mamba install xeus-sql soci-postgresql -c conda-forge

# Installing xeus-sql and the sqlite backend
mamba install xeus-sql soci-sqlite -c conda-forge

There are idiosyncrasies between these different SQL backends which we detail in the xeus-sql documentation and examples.

Visualizing your queries

HTML and text table visualizations for different Jupyter interfaces

You’re probably familiar with visualizing your SQL tables and query results in the form of… tables, of course. Jupyter’s rich display system offers the option of showing them in rich-text display or plain-text display depending on which application you’re using.

Beyond displaying queries with tables, you can now easily create Vega-Lite graphs from your query results directly in the notebook:

Running a series of examples on a notebook

To support this feature xeus-sql relies on xvega, a C++ backend to vega, and custom Jupyter magics for plotting through a mini-language implemented in the xvega-bindings utility library. Besides using the mini-language you can directly provide the JSON specification for your visualization, thanks to contributions by Wang Fenjin.

Vega-Lite is a powerful library and there are many different visualizations you can create from your relational data using xeus-sql. You can see some other examples below:

Examples of different visualizations generated with xeus-sql and xvega-lite by Wang Fenjin

We’re looking forward to exchange with users and learn more about their use cases for xeus-sql and xvega, so if you have any questions please drop them on the public QuantStack chat, or open an issue at the project’s GitHub page. As always, contributions are welcome!

Try it online!

You can try the stable version on Binder without the need of installing anything on your computer just by clicking on the image below:

Acknowledgements

We are grateful to Wang Fenjin for his contributions with respect to the use of vega specs in xeus-sql. The plotting mini-language was built upon the xvega library by Madhur Tandon. We are also grateful to the SOCI contributors for their awesome package that made this work possible.

About the author

My name is Mariana Meireles and I’m a software developer working for QuantStack. I care deeply about the impacts that technology has in the world and try my best to be the change I want to see by contributing to open source projects that stand upon libre and diverse standards.

A Curiously Recurring Widget Library

2021-01-27T14:22:00+00:00

Interactive widgets allow Jupyter users to create user interfaces inline in their notebooks, and to turn them into standalone applications with tools such as Voilà.

Language backends for Jupyter interactive widgets exist in Python (with ipywidgets), and C++ (with xwidgets, and the xeus-cling C++ Jupyter kernel), reusing the same frontend implementation of the widgets in JavaScript.

Simple interactive widgets at play in JupyterLab with the xeus-cling C++ kernel

In this article, we dive into some of the C++ techniques used in the implementation of the xwidgets library, including

discussions on value semantics and RAII (Resource Acquisition Is Initialization),
an original application of CRTP (Curiously Recurring Template Pattern),
an original implementation of the observer pattern, xproperty,
a plea for allowing the overloading of the dot operator in C++, to enable better proxy types.

For readers interested in knowing more about interpreted C++, we recommend the following posts:

Interactive workflows for C++ with Jupyter (Jupyter blog), by Sylvain Corlay, Loic Gouarin, Johan Mabille, and Wolf Vollprecht

Interactive C++ for Data Science (LLVM blog), by Vassil Vassilev, David Lange, Simeon Ehrig, and Sylvain Corlay

Interpreted C++ for GIS with Jupyter (Jupyter blog), by Martin Renou

Jupyter widgets are special objects that trigger the creation of a counterpart JavaScript model object in the Jupyter frontend upon creation. The state of the object in the backend is synchronized with the state of the JavaScript frontend object. Views of that widget model are instantiated upon display.

The MVC (Model View Controller) architecture of Jupyter widgets, and synchronization with the backend

This MVC (Model-View-Controller) architecture for Jupyter interactive widgets allowed us to reuse all of the frontend implementation, by simply providing an alternative backend in C++, implementing the same messaging protocol.

In order to tie the lifetime of the kernel and frontend objects, we decided to use the RAII (Resource Acquisition Is Initialization) pattern. RAII is a common programming idiom that consists of tying the lifetime of an object with the holding of a resource. Most typically, the resource is

acquired in the constructor of the object,
released in the destructor of the object.

The RAII pattern is not common in garbage-collected languages because unlike in C++, the time when objects are destroyed is not deterministic. The Python programming language mitigates that issue by introducing context managers, often used to e.g. open and close files.

A consequence of relying on object lifetime for resource management in C++ is to adopt the “value semantics” for widget instances, instead of “reference semantics” which is more typical for widget frameworks like Qt.

Note: value semantics vs reference semantics

If you are not familiar with the concepts of value and reference semantics in C++, I recommend reading the excellent FAQ of isocpp.org.

To summarize:

with value semantics, objects hold actual values, and copying an object copies their attributes. With value semantics, one should provide implementations of copy, and move constructors, as well as copy and move assignment operators. Besides, values should not have virtual methods.

with reference semantics, objects are manipulated through references (or pointers) and never by value. Reference semantics is typically used for polymorphic programming with virtual methods. With reference semantics, it is recommended to delete copy and move constructors, as well as copy and move assignment operators to avoid accidental copies when passing objects to functions taking arguments by values, causing object slicing. (They can also be made private). Explicit cloning of a reference semantics object is generally allowed via a call to a clone virtual method.

In a C++ codebase, the existence of public copy or move constructors alongside virtual methods in the same class is generally a sign of a bad design.

The use of value semantics for xwidgets provides clear lifetime management for associated resources. Careful use of the move semantics provides fine-grained control.

Illustration of the move semantics for xwidgets

Another advantage of using value semantics in xwidgets is that C++ beginners who are typical users of the Jupyter notebook (often used by instructors) can easily manipulate “widgets as values” without having to deal with manual memory allocation etc.

With value semantics, addressing the lifetime of objects becomes the responsibility of the framework author, in a carefull implementation of (copy and move) constructors, destructors, and (copy and move) assignment operators.

Static polymorphism and the CRTP pattern

While the use of value semantics provides fine-grained control over the lifetime of widgets, it prevents the use of virtual methods in their implementation. Code reuse is achieved with static polymorphism techniques and specifically the CRTP pattern.

The Curiously Recurring Template Pattern (CRTP) is the practice of making a class X derive from a class template instantiation using X itself as a template argument.

class X : public base<X>

CRTP is commonly used by matrix or tensor algebra libraries making use of expression templates (such as xtensor, eigen, or blitz) to prevent the overhead of virtual function dispatch for operations likely to be performed in a loop, such as element access.

Placing common code in a CRTP base allows code reuse without the overhead of virtual dispatch. However, the template base class is specialized for each final type increasing the resulting binary size. This can be mitigated by factoring as much of the code in a base class that would not be templated by the derived type.

CRTP in xwidgets — closing the recursion

In order to allow for code reuse without virtual inheritance, xwidgets’ class hierarchy is entirely based on CRTP. Our naming scheme is that CRTP bases, which should not be instantiated directly are prefixed with the letter x and final concrete widget types are not.

Upon construction of e.g. the button widget, constructors of base types are called in the order of inheritance, which is why we need to establish the connection with the frontend in the constructor of the most derived type, after all attributes have been initialized, and send a message to the frontend with all the values.

The pattern for creating the most derived type being always the same, we defined the xmaterialize template class closing the CRTP hierarchy with

using button = xmaterialize<xbutton>;

The xmaterialize template class is defined as final, to prevent further inheritance. The constructors and assignment operators forward to those of the CRTP bases and implement the RAII pattern.

template <template <class> class B, class... P>
class xmaterialize final : public B<xmaterialize<B, P...>>
{
public:

    using self_type = xmaterialize<B, P...>;
    using base_type = B<self_type>;

    template <class... A>
    xmaterialize(A&&...)
        : base_type(std::forward<A>(args)...)
    {
        this->open();      //  RAII: create the frontend model.
    }

    template <template <class> class B, class... P>
    inline xmaterialize<B, P...>::~xmaterialize()
    {
        if (!this->moved_from())
        {
            this->close();  // RAII: delete the frontend model.
        }
    }

    ...

};

The full implementation of xmaterialize (as of xwidgets 0.25) is available here. Beyond the logic described in this section, it also includes the handling of the move semantics and the method chaining API which is the subject of a later section.

Precompilation and binary size optimization

Even though xwidgets is fully based on template types, we decided to precompile all final widget types for faster interactive use with the xeus-cling kernel. This is achieved with

an extern declaration in the header (here in xbutton.hpp)
```
extern template class xmaterialize<xbutton>;
```
and in the source file (here in xbutton.cpp), an instruction for the precompilation
```
template class XWIDGETS_API xmaterialize<xbutton>;
```

Doing this for all widget types of the library initially resulted in a large compiled binary size. Using the button widget as an example, we see that base types are templated by the final type in the class hierarchy button -> xbutton<button> -> xwidgets<button> -> xobject<button>, and therefore, their binary representation is duplicated for each final type.

A strategy for reducing the binary size has been to factor out as much of the logic of xobject<D> in a non-template base xcommon improving compilation speed and preventing binary code duplication.

Xproperty: an implementation of the observer pattern

In order to update the frontend upon changes of widget properties, xwidgets relies on an implementation of the observer pattern called xproperty. xproperty is to xwidgets what traitlets are to ipywidgets.

In order to trigger observers and validators in the owner object upon assignment of new values, xproperty relies on the overload of the assignment operator =.

template <class T, class O>
template <class V>
inline auto xproperty<T, O>::operator=(V&& value) -> reference
{
    // Before assigning the new value, invoke validators
    // which may also mutate the value.
    m_value = owner()->template invoke_validators<T>(m_name, std::forward<V>(value));

    // Call class-level observer.
    owner()->notify(m_name, m_value);

    // Call registered observers for that attribute
    owner()->invoke_observers(m_name);

    // Return the new value 
    return m_value;
}

Jupyter interactive widgets have many attributes that may be specified at construction time.

In the Python implementation, this is handled with keyword arguments, but the C++ programming language does not support keyword arguments. There exist various approaches to enable this feature with advanced metaprogramming techniques. In the case of xwidgets, this need is limited to the initialization of xproperty attributes, which allowed us to adopt a more scoped approach: a method chaining API for property initialization:

auto slider = slider<double>::initialize()
  .min(-1.0)
  .max(1.0)
  .description("Another slider")
.finalize();

this was enabled by overriding the function call operator () on xproperties to pass an intial value (only when the said property is an rvalue). A static initialize method is used instead of the default constructor to prevent the initialization of the JavaScript counterpart while all attributes may not have been set yet. The frontend counterpart is only acquired with the finalize() call.

Building upon xwidgets

Jupyter interactive widgets are not limited to the controls available in the core package. In fact, there is a rich ecosystem of widget libraries built upon the core framework: ipyvolume (3-D plotting), ipyleaflet (maps visualization), bqplot (2-D plotting), ipygany (3-D mesh visualization), ipycanvas (generic drawing), ipywebrtc (streaming video and audio), and many many more.

For the C++ programming language, we have already provided:

xleaflet (the C++ equivalent to ipyleaflet, reusing the same frontend),
xwebrtc (the C++ equivalent to ipywebrtc, reusing the same frontend).

Screencast of xleaflet in JupyterLab, loading and visualizing a GeoJSON dataset in a C++ notebook.

Potentially, C++ backend to all Jupyter interactive widget packages could be provided, creating a huge opportunity for interactive data visualization in C++.

C++ should allow overloading the dot operator

xwidgets and xproperty makes heavy use of value semantics, and proxy objects.

At the moment, to access an attribute or method of a value held in an xproperty object, we must first call the function call operator () to access the undelying object first, which is cumbersome.
This is also an issue in other places in the xwidgets stack, when making use of e.g. reference proxies.

Being able to automatically map all methods of the underlying type to be accessible in the xproperty would be incredibly powerful, and remove the need for explicitly accessing the underlying.

Interestingly, the C++ standard does have the equivalent operator overload when it comes to pointer semantics, with the arrow operator ->. If overloading the dot operator . was allowed, this could enable this kind of usecase:

struct V
{
    void f();
};

struct X
{
    V& operator.() { return m_value; }
    V m_value;
};

in which case, X::f would call V::f .

Allowing the overloading of the dot operator . would also enable usecases such as smart references (similar to smart pointers, but with value semantics) and fully-fledged reference proxies (such as the return type of operator[] for std::vector<bool>).

Try it online!

Thanks to MyBinder, you can try xwidgets without the need of installing anything on your computer. Just follow this link:

About the author

Sylvain Corlay is the founder and CEO of QuantStack, an open-source software development studio comprising maintainers of key projects of the scientific computing ecosystem.

As an open-source developer, Sylvain is very active in the Jupyter project, contributing to the Xeus stack, Jupyter interactive widgets, Voilà dashboards. He is a member of the Jupyter steering committee and was the vice chair of JupyterCon 2020. Sylvain also contributes to the conda-forge project and he is the co-creator of the Xtensor C++ tensor algebra library.

Robotic Process Automation with JupyterLab

2021-01-18T10:36:00+00:00

Robotic Process Automation (RPA) differs from classical automation tools in that the actions to automate can be developed by observing a user perform a task in a graphical user interface, across multiple applications.

It is a means to lower the entry barrier of process automation and enable the use of tools that don’t provide a programmatic interface.

Most typically, RPA developers will use a mixed approach between textual programming and performing actions manually. The resulting programs are typically called software robots. Therefore, interactive computing tools like Jupyter are a natural environment for RPA, as the interactive nature of Jupyter allows for quick iterations and trial-and-errors when developing such robots.

While many RPA tools are commercial software, Robot Framework and the tooling developed by Robocorp provide an open-source RPA programming language, with a high-level syntax, extensible with Python plugins. It has a rich ecosystem of libraries and tools that are developed as separate projects.

Robot Framework and Project Jupyter

Today, we are happy to announce the first release of xeus-robot, a Jupyter kernel for Robot Framework based on xeus, supporting the new JupyterLab 3.0 visual debugger, auto-completion, and much more!

But before we dive into xeus-robot, we should recognize the earlier work by Nick Bollweg and Asko Soukka, who developed two Jupyter kernels for Robot Framework, irobotframework, and robotkernel, and gave an amazing talk together at RoboCon in 2019!

The motivation for us to reboot this effort with a kernel based on xeus was to enable the JupyterLab Visual Debugger for this kernel. This requires a different concurrency model than that of ipykernel, which underlies both irobotframework and robotkernel. In the end, we were able to provide the same features and more, including e.g. code completion in Python cells, debugging etc.

Diving into Xeus-robot

xeus-robot is a reboot of the already existing robotkernel, based on xeus.

Like most language kernels, xeus-robot supports code completion, inspection, error handling, etc. It also allows using Python cells to define custom robot keywords in Python, those Python cells support code completion as well.

Code Completion in Robot Framework

By using the libraries robotframework-seleniumlibrary and robotframework-seleniumscreenshots, you can even complete the selection of elements on the page you are currently testing!

Code completion with an action of the user in the UI.

The %%python cell magic makes it possible to extend Robotframework with Python modules in the notebook.

Extending RobotFramework with Python, and code completion in Python cells

Widget-based UIs are provided to test Robotframework “keywords”.

Testing Robot Framework “keywords” with Jupyter widgets

Last but not least, xeus-robot comes with full support for the JupyterLab Visual Debugger! After the xeus-python kernel, it is the second Jupyter kernel to support the Jupyter Debugger Protocol! We hope that many more will come.

You can set breakpoints, step in defined keywords (the equivalent of functions), inspect variables, and see the callstack, as shown below.

The JupyterLab Visual Debugger in action with Robot Framework

Installing xeus-robot

Xeus-robot is available for all platforms on conda-forge, and can be installed with conda or mamba.

mamba install xeus-robot -c conda-forge

The jupyterlab-robotmode package, which provides JupyterLab syntax highlighting for Robot Framework will also be installed as a dependency.

For your conda installation, we recommend starting from mambaforge or miniforge which are available for download here, and default to the conda-forge channel (making the -c conda-forge argument unnecessary).

Try it online

Thanks to MyBinder, you can try it out without the need of installing anything on your computer. Just follow this link:

Acknowledgments

The work on xeus-robot by QuantStack was funded by Robocorp, and is part of their initiative to create an open-source RPA ecosystem. Beyond the xeus-robot kernel, Robocorp also funds the integration of Robot Framework with Visual Studio Code, RPA Framework, and other development to push the Robot Framework project forward.

The implementation of the debugger in xeus-robot relies on robotframework-lsp by Fabio Zadrozny and we want to thank him for his help in integrating it in xeus-robot.

About the authors

Martin Renou is a scientific software developer at QuantStack. He is the creator of the xeus-python and xeus-robot kernels.

Johan Mabille is a scientific software developer at QuantStack. He is a co-author of the xeus project, and developed the debugger extensions to xeus-python and xeus-robot. He also co-authored the front-end for the JupyterLab visual debugger.

SlicerJupyter: a 3D Slicer kernel for interactive publications

2020-07-08T16:29:00+00:00

Use Jupyter and 3D Slicer kernel to implement biomedical data processing workflows in a notebook.

The Jupyter ecosystem is a powerful platform for exploratory computational science, and now it can connect with some of the deep and rich domain-specific desktop applications that have decades of feature development already invested in them. With the integration of 3D Slicer with Jupyter through the xeus-python’s interpreter, we demonstrate how a Qt-based graphical desktop application with 3D visualization provided by Visualization Toolkit (VTK), image processing provided by the Insight Toolkit (ITK), can be used through a Jupyter notebook. This approach is available on the SlicerJupyter GitHub repository and could be extended to other applications that embed Python, such as Blender, FreeCAD, or ParaView.

This xeus-python integration is beneficial both for the Jupyter ecosystem and desktop applications. Features that have been developed for decades for desktop applications become readily available for Jupyter users without learning a new working environment or redeveloping features. For a desktop application, the Jupyter notebook can serve as a way to create reproducible data processing workflows, scientific publications, and maintainable tutorials without requiring local software installation. From Jupyter, you can now quickly create simple medical imaging applications with extremely rich interactivity from 3D Slicer. Read below for a discussion of the features available and the history of the project.

Powerful Medical Imaging Capabilities Available Through Jupyter

3D Slicer (or Slicer for short) is a C++ desktop application that uses Qt, ITK, and VTK libraries for visualization and medical image analysis. Slicer’s embedded Python interpreter makes all its features accessible with the Python programming language. Slicer has a simple built-in console to run Python commands interactively and can run Python scripts from files, but these are not as convenient as cell-base interactive notebooks, which have become popular among data scientists and researchers in recent years.

By integrating the xeus-python kernel, we can use a Slicer process as a Jupyter kernel. xeus-python leverages the xeus C++ implementation of the Jupyter kernel protocol. xeus-python is an alternative to ipykernel, which can be used with a vanilla CPython interpreter, interfacing to standard CPython Jupyter widgets like itkwidgets, but it can also be coupled with custom interpreters and GUI event loops, like the Slicer interpreter and its Qt event loop. This allows you to represent a complete scene in Medical Reality Markup Language, MRML, Slicer’s internal data structure. The kernel exposes the full medical imaging API and representation of your data in a meaningful way for Python developers, allowing access through standard Python ecosystem formats such as pandas dataframes and NumPy arrays in the Notebook.

Interactivity Levels

You can also use Jupyter interactive widgets (sliders, buttons, etc.) to control Slicer, modify data, or adjust processing and visualization parameters. Interactivity can be implemented at different levels.

Level 1: Standard Jupyter widgets display application specific objects by automatic conversion of application-specific data objects to standard Python objects. For example, Slicer markup fiducial lists are displayed as a nicely formatted table and model nodes are rendered as 3D objects.
Level 2: Static image widgets display content that the desktop application renders. These widgets can be made interactive by modifying data and rendering parameters using additional standard widgets. This makes rich visualization capabilities — sophisticated rendering various data types, rendering of very large data sets, etc. directly available in Jupyter.
Level 3: Dynamic viewer widgets display 2D and 3D views rendered by the desktop application. Mouse and keyboard events are forwarded to the desktop application that allows zooming/rotating views, and to utilize all 3D interactions implemented in the desktop such as placing annotations, making measurements, or segmenting images the same way as if it was done on the desktop appication’s screen. This is implemented in the Slicer Jupyter kernel using ipycanvas and ipyevents packages.
Level 4: Full desktop graphical user interface integration. Users can see parts of the application window rendered in notebook cells, including standard desktop widgets (sliders, menus, etc.). It is implemented using noVNC and TigerVNC in Slicer Jupyter. This is particularly useful when the application runs on a remote server.

Video demonstrating how to run 3D Slicer using Binder

These interactive tools allow developers to implement complete data processing workflows in a notebook, even if certain steps require manual user inputs such as segmenting 3D regions or setting seed points.

Try it online!

Since Jupyter notebooks can be used from any web browser, it can essentially turn any desktop application to a web application. By setting up a remote Jupyter server, users do not have to install anything on their computers. We have set up a demonstration of this using Binder (www.mybinder.org) that anybody can try at https://mybinder.org/v2/gh/Slicer/SlicerNotebooks/master.

Click on the binder image to launch the demo

The docker image that installs and configures Slicer and all dependencies (ipycanvas, ipyevents, VNC, etc.) is available at https://github.com/Slicer/SlicerDocker/tree/master/slicer-notebook. For deployment to non-technical users, applications can be deployed using Voilà (https://voila.readthedocs.io/), which only shows relevant content and interactive widgets, so the notebook looks like a simple dynamic web page.

The current implementation is already stable and offers a wide range of features, but there is still room for design and performance improvements. For example, we could not implement fully automatic conversion of application-specific data objects to displayable Python objects (due to complex implementation of display hooks); xeus-python debugger’s threading model needs to be improved to allow using it without locking the application’s main thread; and dynamic viewer widget’s performance (level 3 interaction) could be optimized to achieve higher refresh rates.

History of Slicer and this integration

Built over two decades with support from the NIH and a worldwide open source developer community, 3D Slicer is a unique, multi-platform desktop application for analysis, integration, and visualization of medical images that is heavily used by researchers globally for basic and applied research in a wide range of topics. The 3D Slicer Community includes physician-scientists with disease-specific knowledge of clinical challenges, computer scientists and physicists who develop novel algorithms, imaging informatics researchers, software engineers with the ability to understand clinical problems and create reliable tools, and application engineers with the multidisciplinary skills to deploy these tools in a range of cancer research settings. Slicer is maintained by Kitware, Inc., and the NAMIC consortium, and has an active open source software community. Slicer is used in hospitals and by researchers, with more than 10k academic citations and more than 150k downloads in the last year alone.

Click on the image to download 3D Slicer

It can be used for image processing workflows on 2D, 3D, and 4D images. What makes Slicer so useful is its huge set of community contributed modules that extend its functionality, as well as its ability to read DICOM and a wide array of exotic file formats.

Slicer has been scriptable in Python for well over a decade, but a robust SlicerJupyter extension is born from significant development effort by the community. The idea for this integration dates back to SciPy in 2014, when Jean-Christophe Fillion-Robin of Kitware organized a sprint intended to integrate Slicer and an IPython notebook, motivated by the dream of creating interactive and fun tutorials for Slicer. Mike Sarahan and Jean-Christophe created this proof of concept https://github.com/commontk/QEmbedIPython#qt-embed-ipython, but it was far from usable. In 2015 Matt McCormick of Kitware, created the SlicerDocker repository to support headless builds and rendering in a Docker image. Then in June 2018, while attending the Slicer Project Week, Andras Lasso (Queen’s University) and Jean-Christophe learned about Xeus, a C++ implementation of the Jupyter kernel protocol developed by QuantStack that would help streamline the integration of Slicer with Jupyter. To support this effort, Andras and Jean-Christophe created the Slicer/SlicerJupyter GitHub repository. They also contributed changes to Xeus to support this new integration paradigm where the event loop of the kernel is driven by a Qt-based Desktop application (see here). Building on this foundation, Isaiah Norton, then working at Brigham and Women’s Hospital, contributed additional improvements like a better auto-completion using jedi (see https://github.com/Slicer/SlicerJupyter/pull/12) as well as integration with Binder. More recently, Jean-Christophe and Sylvain Corlay (QuantStack) met after the Slicer project week while attending SciPy in Austin. Following the creation of a new project called xeus-python (started by Martin Renou at QuantStack), we took the integration of Jupyter and Slicer to the next level by adding support for improved interactive use and MRML data node visualization directly in the notebook.

About the authors

Alphabetically ordered

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Sylvain Corlay

Jean-Christophe Fillion-Robin is an open-source enthusiast, original author of the SlicerJupyter extension and a principal engineer at Kitware Inc where he leads the development of “3D Slicer” based commercial applications. J-Christophe also maintains scikit-build, an improved build system generator for CPython C/C++/Fortran/Cython extensions.

Jean-Christophe Fillion-Robin

Mike Grauer is a Technical Leader on the data and analytics team at Kitware. He is specialized in building scalable server-side processing frameworks that enable scientific workflows over web platforms.

Mike Grauer

Andras Lasso is an original author of the SlicerJupyter extension, Senior Research Engineer and Associate Director of the Laboratory for Percutaneous Surgery at Queen’s University.

Andras Lasso

Matt McCormick is an open source, medical imaging researcher working at Kitware Inc. Matt is an active, contributing member of scientific open source software efforts such as the Insight Toolkit (ITK) and scientific Python (SciPy) communities, and maintains the Jupyter 3D widget, itkwidgets.

Matt McCormick

Isaiah Norton is a Senior Software Developer at TileDB, Inc. with experience in digital pathology and image-guided surgical navigation.

Isaiah Norton

Steve Pieper is a Chief Architect and active developer of the 3D Slicer application for well over a decade. He is CEO of Isomics, Inc, where he uses a range of software technologies to perform medical imaging research with leading universities and companies.

Steve Pieper

Martin Renou is a scientific software developer at QuantStack. He is the original author of xeus-python, the xeus-based Python kernel, and contributed to the new concurrency model used for the debugger.

Martin Renou

Mike Sarahan is a software engineer at RStudio, PBC working on bringing language ecosystems together. He is passionate about making software work for data scientists in ways that are easy to maintain and improve.

Mike Sarahan

A Jupyter kernel for SQLite

2020-06-11T08:42:00+00:00

While it is well known in the Python scientific computing community, Jupyter is in fact a language-agnostic development environment. High-quality language kernels exist for the main languages of data sciences, such as Python, C++, R or Julia.
But another important tool for data science is the SQL family of programming languages. Today, we announce the release of a Jupyter kernel for SQLite.

This new kernel allows the user to use the complete SQLite syntax as well as some extra operations such as opening or closing a database file, or visualizing the data in different ways using Jupyter magics.

HTML and Text table visualizations for different Jupyter interfaces

To make the task of implementing a new Kernel for Jupyter easier, we make use of Xeus, a library providing a solid implementation of the Jupyter kernel protocol, so that we can focus on implementing the language-specific parts of the kernel.

As a connection between SQLite’s native C implementation and C++ we’re using SRombauts’ library SQLiteCpp.

Current state

xeus-SQLite is still under active development but it offers a fully functional SQLite interface and magics to perform higher-level operations that are outside of the scope of the SQL syntax, such as creating, opening, or closing SQLite databases. Some of the things you can do with xeus-SQLite are creating a new database, loading it, backing it up or deleting it. You can also check if a table exists, set and reset keys of a database and get information about it. You can find more about the use of these methods in SQLite’s documentation.

xeus-SQLite provides rich HTML display of tables in the Jupyter Notebook and Jupyter Lab. For the Jupyter Console we make use of the tabulate library for textual display.

Different commands being executed on xeus-SQLite

The future

For the future of xeus-SQLite we want to create an intuitive form of visualizing data: creating plots, graphs, charts, maps and much more from your SQLite query results, all in the same notebook. Currently we’re exploring the use of Vega in xeus-SQLite, a declarative language for creating interactive visualization designs and can do bar plots using jupyter magics:

SQLite query result with bar and table outputs for same query

This feature is still in very early stages and being developed in this branch.

We also plan on producing a static build of xeus-SQLite bundling xeus and the SQLite library into a single executable that can be easily distributed.

Try it online!

You can try the stable version without the need of installing anything on your computer just by clicking on the image below:

This won’t grant you access to the experimental branch where we’re testing, if you’re interested in trying it please check the installation guide and make sure you’re in the right branch.

Installation

Note that you first need to have Jupyter installed on your computer. You can install xeus-sqlite using mamba:

mamba install xeus-sqlite -c conda-forge

Or conda:

conda install xeus-sqlite -c conda-forge

About the author

Prior to QuantStack I worked as a developer on the PySide team at the Qt Company and as a web performance developer at Mozilla.

A visual debugger for Jupyter

2020-03-25T15:53:00+00:00

Most of the progress made in software projects comes from incrementalism. The ability to quickly see the outcome of an execution and iterate has been one of the main reasons for the success of Jupyter, especially in scientific exploratory workflows.

Jupyter users like to experiment in the notebook, and to use the notebook as an interactive communication tool. However, for more classical software development tasks such as the refactoring of a large codebase, they often switch to general-purpose IDEs.

The JupyterLab environment.

The Jupyter project has made strides in the past few years towards filling that gap, notably with the JupyterLab project, which enables a richer UI including a file browser, text editors, consoles, notebooks, and a rich layout system.

However, a missing piece (which has remained one of the main reasons for users to switch to a different tool) is a visual debugger. This feature has long been requested by users, especially those accustomed to general-purpose development environments.

A debugger for Jupyter

Today, after several months of development, we are glad to announce the first public release of the Jupyter visual debugger!

This is just the first release, but we can already set breakpoints in notebook cells and source files, inspect variables, navigate the call stack and more.

Screencast of the JupyterLab visual debugger in action

Try the debugger on binder

You can also try the debugger online with binder. Just click on the binder link:

Click on the binder link to launch the demo

Installation

The debugger front-end can be installed as a JupyterLab extension.

jupyter labextension install @jupyterlab/debugger

The debugger front-end will be included in JupyterLab by default in a future release.

In the back-end, a kernel implementing the Jupyter Debug Protocol (which will be detailed in the next section) is required. The only kernel implementing this protocol, for now, is xeus-python a new Jupyter kernel for the Python programming language. (Support for the debugger protocol in ipykernel is also on the roadmap).

conda install xeus-python -c conda-forge

Once xeus-python and the debugger extension are installed, you should be all set to use the Jupyter visual debugger!

Note: Depending on the platform, PyPI wheels are available for xeus-python, but they are still experimental.

The Jupyter Debug Protocol

New message types for the Control and IOPub channels

Jupyter kernels (the part of the infrastructure that executes the user’s code) communicate with the rest of the infrastructure with a well-specified inter-process communication protocol.

Several communication channels exist, such as

the Shell channel, which is a request/reply channel for e.g. execution requests
the IOPub channel, which is a one-directional communication channel from the kernel to the client, and is used e.g. to forward the content of the standard output streams (stdout and stderr).

The Control channel is similar to Shell but operates on a separate socket so that messages are not queued behind execution requests, and have a higher priority. Control was already used for Interrupt and Shutdown requests, and we decided to use the same channel for the commands sent to the debugger.

Two message types were added to the protocol:

the debug_[request/reply] to request specific actions to be performed by the debugger such as adding a breakpoint or stepping into a code, which is sent to the Control channel.
the debug_event uni-directional message used by debugging kernels to send debugging events to the front-end. Debug events are sent over the IOPub channel.

Extending the Debug Adapter Protocol

A key principle to the Jupyter design is the agnosticism to the programming language. It is important for the Jupyter debug protocol to be adaptable to other kernel implementations.

A popular standard for debugging is Microsoft’s “Debug Adapter Protocol” (DAP) which is a JSON-based protocol underlying the debugger of Visual Studio Code and for which there already exist multiple language back-ends.

It was therefore natural for us to use the DAP messages over the debug_[request/reply] and debug_event messages that we just added.

However, it was not quite sufficient in the case of Jupyter. Indeed

In order to support page reloading, or a client connecting at a later stage, Jupyter kernels must store the state of the debugger (breakpoints, whether the debugger is currently stopped). The front-end can request that state over with a debug_request message.
In order to support the debugging of notebook cells and of Jupyter consoles, which are not based on source files, we also needed messages to submit code to the debugger to which breakpoints can be added.

Besides these two differences, the content of the debug requests and replies corresponds to the debug adapter protocol.

All these extensions to the Jupyter kernel protocol have been proposed for inclusion in the official specification. The JEP (Jupyter Enhancement Proposal) can be found here.

Xeus-python, the first Jupyter Kernel to support debugging

Xeus is a C++ implementation of the Jupyter kernel protocol. It is not a kernel by itself but a library that helps kernel authoring. Xeus is useful when developing a kernel for a language that has a C or a C++API (like Python, Lua, or SQL). It takes the cumbersome task of implementing the Jupyter messaging protocol for the kernel author to focus on the core interpreter tasks: executing code, inspecting, etc.

Several kernels have been developed with xeus, including the popular xeus-cling kernel for the C++ programming language, based on the cling C++ interpreter from CERN. The xeus-python kernel is an alternative Python kernel to ipykernel, based on xeus. The first release of the xeus-python kernel was announced on this blog earlier this year: A new Python kernel for Jupyter

Xeus-python was an appropriate choice for this first implementation of the debugging protocol because

it has a pluggable concurrency model, which allowed running the processing of the Control channel in a different thread.
it has a lighter-weight codebase which made it a convenient sandbox to iterate upon. Implementing the first version of the protocol in ipykernel would have required more significant refactoring and consensus building at an early stage.

The roadmap of Xeus-python

The short-term roadmap for xeus-python includes

adding support for IPython magics in xeus-python, which is the main missing feature with respect to ipykernel.
improving the PyPI wheels of xeus-python.

What about other kernels?

The work in the front-end is valid for any kernel implementing the extended kernel protocol.

We will be working in 2020 to enable debugging with as many kernels as possible.

This will soon be the case for other xeus-based kernels which share a large part of the implementation with xeus-python, such as xeus-cling.

Diving into the debugger front-end architecture

The debugger extension for JupyterLab provides what users would typically expect from an IDE:

a sidebar with a variable explorer, a list of breakpoints, a source preview and the possibility to navigate the call stack
the ability to set breakpoints directly next to the code, namely in code cells and code consoles
visual markers to indicate where the current execution has stopped

When working with Jupyter notebooks, the state of the execution is kept in the kernel. But a cell can be executed and then deleted from the notebook. What should happen when a user wants to step in deleted code?

The extension supports that particular use case and enables retrieving a read-only view of the previously executed cell.

Stepping into a deleted cell

Consoles and files also have support for debugging.

Debugging code consoles in JupyterLab

Debugging files in JupyterLab

Debugging can be enabled on a notebook level, which lets users debug a notebook and work on a different one at the same time.

Debugging multiple notebooks simultaneously

Variables can be inspected using a tree viewer and a table viewer:

The variable explorer

The debugger extension for JupyterLab has been designed to work with any kernel that supports debugging.

By relying on the Debug Adapter Protocol, the debugger extension abstracts away language-specific features and provides a consistent debugging interface to the user.

The following diagram shows how the debug messages flow between the user, the JupyterLab extension and the kernel during a debugging session.

Using the Debug Adapter Protocol in the debugger extension (source)

Future developments

In 2020, we plan on making major improvements to the debugger experience:

Support for rich mime type rendering in the variable explorer.
Support for conditional breakpoints in the UI.
General improvements of the debugger user experience.
Enable the debugging of Voilà dashboards, from the JupyterLab Voilà preview extension.

Acknowledgements

The JupyterLab debugger is the result of the collaboration and coordination of developers from several institutions, including QuantStack, Two Sigma, and Bloomberg.

The work on both the front-end and the back-end at QuantStack by Jeremy Tuloup, Johan Mabille, Martin Renou, and Sylvain Corlay, was funded by Bloomberg.
The work on the Jupyter debugger by Borys Palka and Afshin Darian was made possible by Two Sigma.

About the developers

Jeremy Tuloup

Jeremy Tuloup is a Scientific Software developer at QuantStack. He authored a large part of the front-end of the JupyterLab debugger.

Borys Palka

Borys Palka is a software developer at Codete. He authored a large part of the front-end of the JupyterLab debugger.

Johan Mabille

Johan Mabille is a scientific software developer at QuantStack. Johan is a co-author of xeus, and developed the debugger extension to xeus-python. He also authored a large part of the debugger front-end.

Martin Renou

Afshin Darian

Afshin Darian is a software developer at Two Sigma, and one of the authors of JupyterLab.

Sylvain Corlay

Sylvain Corlay is the founder and CEO of QuantStack, and a core Jupyter developer. He co-authored xeus and xeus-python.

Xeus is now a Jupyter subproject

2020-02-04T12:08:00+00:00

It is a great pleasure to announce that the Xeus project has been incorporated as a Jupyter subproject. Xeus will now be subject to the Jupyter governance and code of conduct.

For reference, the Jupyter Enhancement Proposal (JEP) for the Xeus incorporation is available here.

What is Xeus?

The Xeus project is a C++ implementation of the Jupyter kernel protocol. Xeus is not a kernel, but a library meant to facilitate the authoring of kernels.
Several Jupyter kernels have been created with Xeus:

xeus-cling, a kernel for the C++ programming language, based on the Cling C++ interpreter. The cling project comes from CERN and is at the foundation of the ROOT project.

The xeus-cling Jupyter kernel for the C++ programming language.
xeus-python, a kernel for the Python programming language, embedding the Python interpreter.

The xeus-python Jupyter kernel for the Python programming language
xeus-calc, a calculator kernel, meant as an educational example on how to make Jupyter kernels with Xeus.

Beyond these three kernels built on top of Xeus by the Xeus maintainers, third-parties have developed other Jupyter kernels with Xeus:

JuniperKernel, a kernel for the R programming language by Spencer Aiello.
xeus-fift, a kernel for the fift programming language by Michael Zaikin. The fift programming language was developed by Telegram to create TON blockchain contracts.
SlicerJupyter, a kernel for the Python programming language by Kitware which integrates into the Qt event loop of the Kitware “Slicer” project.

Finally, the xeus-python kernel includes a first implementation of the Jupyter debugger protocol used by the Jupyter debugger project. xeus-python enables the Debug Adapter Protocol over the Control channel through new debug request/reply and debug event messages.

Why moving Xeus under the Jupyter governance?

While Xeus started as a side project for QuantStack engineers, the project now has several stakeholders who depend on it. We think that moving the project to an open governance organization may be a better way to reflect this situation.

Acknowledgements

Xeus was started by the team of open-source developers at QuantStack as a separate project, but with the full intent to incorporate it into Jupyter. The initial project development at QuantStack was funded by Bloomberg.
Now, Xeus contributors work in many institutions, including Université Paris-Saclay and École Polytechnique.

About the Author

Johan Mabille is a Scientific Software Developer at QuantStack, specializing in high-performance computing in C++. He holds master’s degree in computer science from Centrale-Supelec.

As an open source developer, Johan coauthored xtensor, xeus, and xsimd. He also made major contributions to the JupyterLab debugger project and bqplot.

Building a Calculator Jupyter Kernel

2019-07-16T08:36:00+00:00

An early device for interactive computing

In order to provide a language-agnostic scientific development environment, the Jupyter project is built upon a well-specified protocol to communicate with the Kernel, the part of the infrastructure responsible for executing the code.

For a programming language to leverage the potential of the Jupyter ecosystem, such as JupyterHub, JupyterLab, and interactive widgets, all that is needed is a Kernel to be created for that language that is, an executable implementing the specified inter-process communication. Dozens of kernels have already been implemented bringing Jupyter to many programming languages.

We are completing our engineering degree and interning at QuantStack. We recently attended the Jupyter Community Workshop on the kernel protocol that took place in Paris in late May. In this occasion, we set ourselves to write a new Jupyter kernel.

Today, we are proud to announce the first release of xeus-calc, a calculator kernel for Jupyter! xeus-calc is meant to serve as a minimal, self-containedexample of Jupyter kernel. It is built upon the xeus project, a modern C++ implementation of the protocol. This article is a step-by-step description on how the kernel was implemented.

You may find this post especially useful if you are creating a new programming language and you want it to work in Jupyter from the start.

Xeus

Implementing the Jupyter kernel protocol from scratch may be a tedious and difficult task. One needs to deal with ZMQ sockets and complex concurrency issues, rely on third-party libraries for cryptographically signing messages or parsing JSON efficiently. This is where the xeus project comes into play: it takes all of that burden so that developers can focus on the parts that are specific to their use case.

In the end, the kernel author only needs to implement a small number of virtual functions inherited from the xinterpreter class.

#include "xeus/xinterpreter.hpp"
#include "nlohmann/json.hpp"

using xeus::xinterpreter;
namespace nl = nlohmann;

namespace custom
{
    class custom_interpreter : public xinterpreter
    {
    public:

        custom_interpreter() = default;
        virtual ~custom_interpreter() = default;

     private:

        void configure() override;
        nl::json execute_request_impl(int execution_counter,
                                      const std::string& code, 
                                      bool silent, 
                                      bool store_history,
                      const nl::json::node_type* user_expressions,
                                      bool allow_stdin) override;
    
        nl::json complete_request_impl(const std::string& code,
                                       int cursor_pos) override;

        nl::json inspect_request_impl(const std::string& code,
                                      int cursor_pos,
                                      int detail_level) override;

        nl::json is_complete_request_impl(const std::string& code)
        override;

        nl::json kernel_info_request_impl() override;
    };
}

Typically, a kernel author will make use of the C or C++ API of the target programming language and embed the interpreter into the application.

This differs from the wrapper kernel approach documented in the ipykernel package where kernel authors make use of the kernel protocol implementation of ipykernel, typically spawning a separate process for the interpreter and capturing its standard output.

Jupyter kernels based on xeus include:

xeus-cling: a C++ kernel built upon the cling C++ interpreter from CERN
xeus-python: a new Python kernel for Jupyter.
JuniperKernel: a new R kernel for Jupyter based on xeus.

In this post, instead of calling into the API of an external interpreter, we implement the internal logic of the calculator in the kernel itself.

Exposing the xeus calculator to Jupyter

A calculator project

First, to implement your own Jupyter kernel, you should install Xeus. You can either download it with conda, or install it from sources as detailed in the readme.

Now that the installation is out of the way, let’s focus on the implementation itself.

Recall that the main class for the calculator kernel must inherit from the xinterpreter class so that Xeus can correctly route the messages received from the front-end.

This class defines the behavior of the kernel for each message type that is received from the front-end.

kernel_info_request_impl: returns the information about the kernel, such as the name, the version or even a “banner”, that is a message that is prompted to console clients upon launch. This is a good place to be creative with ASCII art.
complete_request_impl: checks if the code can be completed, by that we mean semantic completion, and makes a suggestion accordingly. This way the user can receive a proposition for an adequate completion to the code he is currently writing. We did not use it during our implementation as you will see later, it is safe to return a JSON with a status value only, if you do not want to handle completion.
is_complete_request_impl: whether the submitted code is complete and ready for evaluation. For example, if brackets are not all closed, there is probably more to be typed. This message is not used by the notebook front-end but is required for the console, which shows a continuation prompt for further input if it is deemed incomplete. It also checks whether the code is valid or not. Since the calculator expects single-line inputs, it is safe to return an empty JSON object. This may be refined in the future.
inspect_request_impl: concerns documentation. It inspects the code to show useful information to the user. We did not use it in our case and went with the default implementation (that is to return an empty JSON object).
execute_request_impl: the main function. An execute_request message is sent by the front-end to ask the kernel to execute the code on behalf of the user. In the case of the calculator, this means parsing the mathematical expression, evaluating it and returning the result, as described in the next section.

Implementation of the calculator

First things first, we need to find a way to parse mathematical expressions. To do so, we turn the user input into Reverse Polish Notation (or RPN), a name full of meaning for the wisest among our readers (or at least the oldest) who used RPN calculators in high school.

The RPN, also called Postfix notation, presents the mathematical expression in a specific way : the operands go first followed by the operator. The main advantage of this notation is how it implicitly displays the precedence of operators.

Reverse Polish Notation illustration

The main logic of the calculator is provided by two main functions dealing respectively with parsing and evaluating the user expression and a third one for handling spaces in the expression.

First we have the parsing function (parse_rpn) transforming the expression into this representation. For this purpose we implement the Shunting-yard algorithm.

It is based on the use of a stack data structure to change the order of the elements in the expression, depending on their type : operator, operand or parenthesis.

Transforming a user expression into RPN

Now that we have the expression turned into RPN (with spaces delimiting operands and operators) we need to do the computation. For this purpose we have the function compute_rpn. Its implementation is based on a loop through a stringstream (hence the need for space delimiters) which performs operations in the right order.

Note that the result is not returned as an execute_reply message but is sent on a broadcasting channel instead, so that other clients to the kernel can also see it. The function execute_reply_impl actually returns the status of the execution only, as you may see in the code below.

nl::json interpreter::execute_request_impl(int execution_counter,
                                           const std::string& code,
                                           bool /*silent*/,
                                           bool /*store_history*/,
                                           nl::json /*user_exprs*/,
                                           bool /*allow_stdin*/)
{
    nl::json pub_data;
    std::string result = "Result = ";
    auto publish = [this](const std::string& name, 
                          const std::string& text) {
        this->publish_stream(name,text);
    };
    try
    {
        std::string spaced_code = formating_expr(code);
        result += std::to_string(compute_rpn(parse_rpn(spaced_code,
                                                       publish),
                                             publish));
        pub_data["text/plain"] = result;
        publish_execution_result(execution_counter,
                                 std::move(pub_data),
                                 nl::json::object());
        nl::json jresult;
        jresult["status"] = "ok";
        jresult["payload"] = nl::json::array();
        jresult["user_expressions"] = nl::json::object();
        return jresult;
    }
    catch (const std::runtime_error& err)
    {
        nl::json jresult;
        publish_stream("stderr", err.what());
        jresult["status"] = "error";
        return jresult;
    }
}

And that’s it for our calculator! It is as simple as that.

Yet remember that Xeus is a library, not a kernel by itself. We still have to create an executable that gathers the interpreter and the library. This is done in a main function whose implementation looks like:

int main(int argc, char* argv[])                       
{                           
    // Load configuration file                           
    std::string file_name = 
        (argc == 1) ? "connection.json" : argv[2];                           
    xeus::xconfiguration config = 
        xeus::load_configuration(file_name);                                                   
    
    // Create interpreter instance                           
    using interpreter_ptr = std::unique_ptr<xeus_calc::interpreter>;                           
    interpreter_ptr interpreter = 
        std::make_unique<xeus_calc::interpreter>();                              
    
    // Create kernel instance and start it                           
    xeus::xkernel kernel(config, 
                         xeus::get_user_name(), 
                         std::move(interpreter));                           
    kernel.start();                                                   
    return 0;
}

First, we need to load the configuration file. To do so, we check if one was passed as an argument, otherwise, we look for the connection.json file.

Then, we instantiate the interpreter that we previously set up. Finally, we can create the kernel with all that we defined beforehand. The kernel constructor accepts more parameters that allow customizing some predefined behaviors. You can find more details in the Xeus documentation. Start the kernel and we are good to go!

Now that everything is set, we can test out our homemade calculator kernel.

As you can see in the demonstration below, the code displays step-by-step how the computation is done with RPN. This is done with publish_stream statements, which is equivalent to std::cout for the Jupyter notebook, very useful for debugging purposes.

The final result, a functional calculator!

You should now have all the information you need to implement your own Jupyter kernel. As you noticed, the Xeus library makes this task quite simple. All that you have to do is to inherit from the xinterpreter virtual class and implement the functions related to the messaging protocol. Nothing more is required.

This project can be found on GitHub. Feel free to contribute to the project if you wish to improve it, keeping in mind that xeus-calc should remain lean and simple!

Note that the current implementation only supports arithmetical operators. However it can be easily extended and we may add functional support in the near future.

Acknowledgments

We would like to thank the whole QuantStack team for their help throughout the process of making this blog post.

We are also grateful to the organizers of the Jupyter community workshop on kernels as we actually started to endeavor during the event.

About the authors

Vasavan Thiru is completing a master’s degree at Sorbonne Université Pierre & Marie Curie in applied mathematics for mechanics. He is currently interning as a scientific software developer at QuantStack.

Thibault Lacharme is finishing a master’s degree in Quantitative Finance at Université Paris Dauphine. Thibault is currently on his internship as a scientific software developer at QuantStack.