Introducing Jupyter Enterprise Gateway
by Luciano Resende, Kevin Bates, Alan Chin
Yesterday, the Jupyter Steering Council voted to make Jupyter Enterprise Gateway a top-level Jupyter Project. I want to thank everyone for their contributions so far — code from my teammates at IBM and the community in general; advice from the Jupyter development team and mentors; and questions, issues, and requirements from end users.
As we become an official Jupyter project, I would like to take the opportunity to give an update on the project’s progress during our incubation period.
What is Jupyter Enterprise Gateway?
Jupyter Enterprise Gateway enables Jupyter Notebook to launch remote kernels in a distributed cluster, including Apache Spark managed by YARN, IBM Spectrum Conductor or Kubernetes.
Although Enterprise Gateway is mostly kernel agnostic, it provides out of the box configuration examples for the following kernels:
· Python using IPython kernel
· R using IRkernel
· Scala using Apache Toree kernel
Jupyter Enterprise Gateway does not manage multiple Jupyter Notebook deployments, for that you should look for JupyterHub. Having said that, Enterprise Gateway can enable JupyterHub to launch remote kernels as individual Kubernetes pods, providing better resource allocation and enabling better environment management as each pod can be based on different images (e.g. TensorFlow, Anaconda, etc)
Supported Platforms
Jupyter Enterprise Gateway currently enables remote kernels in the following platforms:
Distributed Kernels in Apache Spark
Jupyter Enterprise Gateway leverages different resource managers to enable distributed kernels in Apache Spark clusters. One example shown below describes kernels being launched in YARN cluster mode across all nodes of a cluster.
Note that, Jupyter Enterprise Gateway also provides some other value-added capabilities such as enhanced security and multiuser support with user impersonation.
Distributed Kernels in Kubernetes
Jupyter Enterprise Gateway support for Kubernetes enables decoupling the Jupyter Notebook Server and its kernels into multiple pods. This enables running Notebook server pods with minimally necessary resources based on the workload being processed.
Jupyter Enterprise Gateway and JupyterHub
JupyterHub is a multi-user server that manages and proxies multiple instances of the single-user Jupyter notebook server. Particularly in a Kubernetes environment, Jupyter Enterprise Gateway can enable JupyterHub to launch remote kernels as individual Kubernetes pods, providing better resource allocation and enabling better environment management as each pod can be based on different images (e.g. TensorFlow, Anaconda, etc). This has proven to be very desired, particularly when working on Deep Learning related Notebooks.
Some project metrics
The following stats have been collected from the Jupyter Enterprise Gateway GitHub repository during the incubation period:
- 10 releases
- 12 individual contributors
- 90 Stars
- 34 Forks
Source code, documentation, and other community resources
The Jupyter Enterprise Gateway community provides multiple resources that both users and contributors can use:
Source Code available at GitHub
https://github.com/jupyter/enterprise_gateway
Documentation available at ReadTheDocs
http://jupyter-enterprise-gateway.readthedocs.io/en/latest/
Automated builds available at Travis.CI
https://travis-ci.org/jupyter/enterprise_gateway
Releases available at PyPi.org and Conda Forge
https://pypi.org/project/jupyter_enterprise_gateway/
https://github.com/conda-forge/jupyter_enterprise_gateway-feedstock
Related Docker Images available at Elyra organization at DockerHub
https://hub.docker.com/u/elyra/dashboard/
What’s next?
We are eager to build an even greater community around the project, and tailor the project roadmap based on community advise.
Currently, we are busy working on advancing our Kubernetes support and integration with JupyterHub.
As always, we welcome questions, comments, and suggestions from users and the community in general.
