Building a Calculator Jupyter Kernel
A step-by-step guide for authoring language kernels with Xeus
In order to provide a language-agnostic scientific development environment, the Jupyter project is built upon a well-specified protocol to communicate with the Kernel, the part of the infrastructure responsible for executing the code.
For a programming language to leverage the potential of the Jupyter ecosystem, such as JupyterHub, JupyterLab, and interactive widgets, all that is needed is a Kernel to be created for that language that is, an executable implementing the specified inter-process communication. Dozens of kernels have already been implemented bringing Jupyter to many programming languages.
We are completing our engineering degree and interning at QuantStack. We recently attended the Jupyter Community Workshop on the kernel protocol that took place in Paris in late May. In this occasion, we set ourselves to write a new Jupyter kernel.
Today, we are proud to announce the first release of xeus-calc, a calculator kernel for Jupyter! xeus-calc is meant to serve as a minimal, self-containedexample of Jupyter kernel. It is built upon the xeus project, a modern C++ implementation of the protocol. This article is a step-by-step description on how the kernel was implemented.
You may find this post especially useful if you are creating a new programming language and you want it to work in Jupyter from the start.
Xeus
Implementing the Jupyter kernel protocol from scratch may be a tedious and difficult task. One needs to deal with ZMQ sockets and complex concurrency issues, rely on third-party libraries for cryptographically signing messages or parsing JSON efficiently. This is where the xeus project comes into play: it takes all of that burden so that developers can focus on the parts that are specific to their use case.
In the end, the kernel author only needs to implement a small number of virtual functions inherited from the xinterpreter class.
#include "xeus/xinterpreter.hpp"
#include "nlohmann/json.hpp"using xeus::xinterpreter;
namespace nl = nlohmann;namespace custom
{
class custom_interpreter : public xinterpreter
{
public: custom_interpreter() = default;
virtual ~custom_interpreter() = default; private: void configure() override;
nl::json execute_request_impl(int execution_counter,
const std::string& code,
bool silent,
bool store_history,
const nl::json::node_type* user_expressions,
bool allow_stdin) override;
nl::json complete_request_impl(const std::string& code,
int cursor_pos) override; nl::json inspect_request_impl(const std::string& code,
int cursor_pos,
int detail_level) override; nl::json is_complete_request_impl(const std::string& code)
override; nl::json kernel_info_request_impl() override;
};
}
Typically, a kernel author will make use of the C or C++ API of the target programming language and embed the interpreter into the application.
This differs from the wrapper kernel approach documented in the ipykernel package where kernel authors make use of the kernel protocol implementation of ipykernel, typically spawning a separate process for the interpreter and capturing its standard output.
Jupyter kernels based on xeus include:
- xeus-cling: a C++ kernel built upon the cling C++ interpreter from CERN
- xeus-python: a new Python kernel for Jupyter.
- JuniperKernel: a new R kernel for Jupyter based on xeus.
In this post, instead of calling into the API of an external interpreter, we implement the internal logic of the calculator in the kernel itself.
A calculator project
First, to implement your own Jupyter kernel, you should install Xeus. You can either download it with conda, or install it from sources as detailed in the readme.
Now that the installation is out of the way, let’s focus on the implementation itself.
Recall that the main class for the calculator kernel must inherit from the xinterpreterclass so that Xeus can correctly route the messages received from the front-end.
This class defines the behavior of the kernel for each message type that is received from the front-end.
kernel_info_request_impl: returns the information about the kernel, such as the name, the version or even a “banner”, that is a message that is prompted to console clients upon launch. This is a good place to be creative with ASCII art.complete_request_impl: checks if the code can be completed, by that we mean semantic completion, and makes a suggestion accordingly. This way the user can receive a proposition for an adequate completion to the code he is currently writing. We did not use it during our implementation as you will see later, it is safe to return a JSON with a status value only, if you do not want to handle completion.is_complete_request_impl: whether the submitted code is complete and ready for evaluation. For example, if brackets are not all closed, there is probably more to be typed. This message is not used by the notebook front-end but is required for the console, which shows a continuation prompt for further input if it is deemed incomplete. It also checks whether the code is valid or not. Since the calculator expects single-line inputs, it is safe to return an empty JSON object. This may be refined in the future.inspect_request_impl: concerns documentation. It inspects the code to show useful information to the user. We did not use it in our case and went with the default implementation (that is to return an empty JSON object).execute_request_impl: the main function. Anexecute_requestmessage is sent by the front-end to ask the kernel to execute the code on behalf of the user. In the case of the calculator, this means parsing the mathematical expression, evaluating it and returning the result, as described in the next section.
Implementation of the calculator
First things first, we need to find a way to parse mathematical expressions. To do so, we turn the user input into Reverse Polish Notation (or RPN), a name full of meaning for the wisest among our readers (or at least the oldest) who used RPN calculators in high school.
The RPN, also called Postfix notation, presents the mathematical expression in a specific way : the operands go first followed by the operator. The main advantage of this notation is how it implicitly displays the precedence of operators.
The main logic of the calculator is provided by two main functions dealing respectively with parsing and evaluating the user expression and a third one for handling spaces in the expression.
First we have the parsing function (parse_rpn) transforming the expression into this representation. For this purpose we implement the Shunting-yard algorithm.
It is based on the use of a stack data structure to change the order of the elements in the expression, depending on their type : operator, operand or parenthesis.
Now that we have the expression turned into RPN (with spaces delimiting operands and operators) we need to do the computation. For this purpose we have the function compute_rpn. Its implementation is based on a loop through a stringstream (hence the need for space delimiters) which performs operations in the right order.
Note that the result is not returned as an execute_reply message but is sent on a broadcasting channel instead, so that other clients to the kernel can also see it. The function execute_reply_impl actually returns the status of the execution only, as you may see in the code below.
nl::json interpreter::execute_request_impl(int execution_counter,
const std::string& code,
bool /*silent*/,
bool /*store_history*/,
nl::json /*user_exprs*/,
bool /*allow_stdin*/)
{
nl::json pub_data;
std::string result = "Result = ";
auto publish = [this](const std::string& name,
const std::string& text) {
this->publish_stream(name,text);
};
try
{
std::string spaced_code = formating_expr(code);
result += std::to_string(compute_rpn(parse_rpn(spaced_code,
publish),
publish));
pub_data["text/plain"] = result;
publish_execution_result(execution_counter,
std::move(pub_data),
nl::json::object());
nl::json jresult;
jresult["status"] = "ok";
jresult["payload"] = nl::json::array();
jresult["user_expressions"] = nl::json::object();
return jresult;
}
catch (const std::runtime_error& err)
{
nl::json jresult;
publish_stream("stderr", err.what());
jresult["status"] = "error";
return jresult;
}
}And that’s it for our calculator! It is as simple as that.
Yet remember that Xeus is a library, not a kernel by itself. We still have to create an executable that gathers the interpreter and the library. This is done in a main function whose implementation looks like:
int main(int argc, char* argv[])
{
// Load configuration file
std::string file_name =
(argc == 1) ? "connection.json" : argv[2];
xeus::xconfiguration config =
xeus::load_configuration(file_name);
// Create interpreter instance
using interpreter_ptr = std::unique_ptr<xeus_calc::interpreter>;
interpreter_ptr interpreter =
std::make_unique<xeus_calc::interpreter>();
// Create kernel instance and start it
xeus::xkernel kernel(config,
xeus::get_user_name(),
std::move(interpreter));
kernel.start();
return 0;
}First, we need to load the configuration file. To do so, we check if one was passed as an argument, otherwise, we look for the connection.json file.
Then, we instantiate the interpreter that we previously set up. Finally, we can create the kernel with all that we defined beforehand. The kernel constructor accepts more parameters that allow customizing some predefined behaviors. You can find more details in the Xeus documentation. Start the kernel and we are good to go!
Now that everything is set, we can test out our homemade calculator kernel.
As you can see in the demonstration below, the code displays step-by-step how the computation is done with RPN. This is done with publish_streamstatements, which is equivalent to std::cout for the Jupyter notebook, very useful for debugging purposes.
You should now have all the information you need to implement your own Jupyter kernel. As you noticed, the Xeus library makes this task quite simple. All that you have to do is to inherit from the xinterpreter virtual class and implement the functions related to the messaging protocol. Nothing more is required.
This project can be found on GitHub. Feel free to contribute to the project if you wish to improve it, keeping in mind that xeus-calc should remain lean and simple!
Note that the current implementation only supports arithmetical operators. However it can be easily extended and we may add functional support in the near future.
Acknowledgments
We would like to thank the whole QuantStack team for their help throughout the process of making this blog post.
We are also grateful to the organizers of the Jupyter community workshop on kernels as we actually started to endeavor during the event.
About the authors
Vasavan Thiru is completing a master’s degree at Sorbonne Université Pierre & Marie Curie in applied mathematics for mechanics. He is currently interning as a scientific software developer at QuantStack.
Thibault Lacharme is finishing a master’s degree in Quantitative Finance at Université Paris Dauphine. Thibault is currently on his internship as a scientific software developer at QuantStack.
