From Jupyter to the Moon

A new Jupyter kernel for Lua based on Xeus

Jupyter architecture in a nutshell

A key principle in Jupyter’s design is language agnosticism, and one of the main extension points of the Jupyter ecosystem is the kernel, the part of the architecture responsible for executing the user’s code.

In the end, a Jupyter kernel is merely an executable implementing a well-defined inter-process communication protocol.

A reference implementation of the kernel is ipykernel, which provides the default Python kernel, and utilities to build other language kernels. Another reference implementation of the protocol is Xeus.

Xeus

Xeus is C++ library for writing Jupyter kernels. It is not a kernel but a library that simplifies the process of writing kernels. Xeus implements the Jupyter messaging protocol, so that kernel authors only need to implement a handful of methods related to the actual interpreter, like executing pieces of code and auto-completing. There is a long list of scripting languages which can be embedded in C/C++. With the help of Xeus, it is very easy to write kernels for such languages. In fact, Xeus has already been used for making several language kernels:

• xeus-cling is a Xeus-based C++ kernel.
• xeus-python is a Xeus-based Python kernel.
• xeus-sql / xeus-sqlite are Xeus-based kernel for SQL / SQLite databases.
• xeus-robot is a Jupyter kernel for the Robot Framework .
• LFortran is a modern interactive Fortran compiler with a Xeus-based Jupyter kernel.

xwidgets

The xwidget library is a C++ implementation of the Jupyter Interactive Widget protocol. Xwidgets were initially developed for xeus-cling, the Xeus C++ kernel, but any Xeus-based kernel can reuse these components.

A xeus-based Lua kernel

Today, I am pleased to announce the release of xeus-lua, a new Jupyter kernel for Lua based on Xeus.

Lua

Lua, which means “Moon” in Portuguese, is a lightweight embeddable scripting language written in C. While Python is one of the leading scripting languages for scientific application, Lua is the leading scripting language used in games / game engines. The list of video games using Lua is long and includes triple-a games like Crisys and Farcry. Not only games rely on Lua, but also applications like Adobe Photoshop Lightroom.

Sol2

With the Sol2 library, Lua can be embedded into C++ in a very elegant way.

While Lua itself can be embedded in a C++ application quite easy, it is even easier when using Sol2. Sol2 is a C++ library that simplifies the usage of Lua from C++. Furthermore Sol2 can be used to expose C++ classes to Lua, in a similar fashion as pybind11 can be used to expose C++ classes to Python.

Xeus-lua

The compulsory “Hello, World!” from xeus-lua. This blog-post` would be incomplete without it.

I am pleased to announce xeus-lua, a new Jupyter kernel for the Lua language based on Xeus. Using Sol2 and Xeus, it was very easy to implement a feature rich kernel using modern C++. With the help of Sol2, it was trivial to expose the xwidgets classes to Lua such that we can use the widgets within xeus-lua notebooks.

But why a Lua kernel?

While Lua might not be the hippest language in 2021, there are multiple reasons while I choose to implement such a Lua kernel, and why such a kernel can be useful:

• Writing a Xeus-based kernel is the best way to learn and familiarize oneself with the xeus ecosystem. And since Lua is so easy to build and embed, Lua and Xeus are perfect match. Implementing this kernel gave me great insights into Xeus and Jupyter itself.
• Learning Lua with xeus-lua is fun! Before writing xeus-lua, I had zero Lua experience. While implementing xeus-lua I learned Lua from within Jupyter by playing around with xwidgets from within Lua — this was a very joyful experience.
• The xeus-lua implementation might serve as reference for future kernel authors. In particular the integration of xwidgets in xeus-lua might be useful for future implementations.
• Since Lua is so easy to build, xeus-lua can be built for WebAssembly and will be integrated in JupyterLite — stay tuned!

Features

Obviously xeus-lua supports the execution of code.

Simple code execution

Streams are redirected to the notebooks output

Redirection of streams

Errors messages are shown in the notebooks output

Error Handling

Input streams are redirected to the notebooks input system

User Input

Statements can be auto-completed with a press of tab

Code Completion

Data of different sources as Latex, JSON and html can be displayed.

Display Data

Interactive widgets can be added to the notebook!

Widgets

We can feed the widgets with data!

Binary buffers for widgets

Try It Out

You can try it right now with binder, without the need of installing anything on your computer, just by clicking on this link.

Other Lua kernels

There are several Lua kernels available:

• lupyter: A Python based implementation of a Lua Jupyter kernel. The heavy lifting is done in a C-extension.
• ILua: A feature rich implementation. Ilua is implemented in Python and does the communication with Lua via IPC.
• IPyLua: IPyLua is a pure Lua implementation.

While these implementations are feature rich and work well, none of the above supports interactive widgets. Since xeus-lua is based on Xeus, we can reuse components from the Xeus ecosystem like xwidgets with minimal effort.

Acknowledgments

We are grateful to Juliette Taka for the illustration.

About the Author

Thorsten Beier is a Scientific Software Engineer at QuantStack. Before joining QuantStack, he graduated in computer science at the University of Heidelberg and worked at the EMBL. As an open source developer, Thorsten worked on a variety of projects, from nifty, and vigra in C++ to inferno, kipoi and ilastik in Python.