Jupyter Blog - JupyterHub

JupyterHub 6.0

2026-09-11T08:54:00+00:00

JupyterHub is a collection of tools that provides standardized, user-friendly computing environments for education and research. Today, we’re pleased to announce the release of JupyterHub 6.0. This is a substantial release with several new features. The breaking changes ought to be minimal, and should affect very few deployments. There is a small database schema upgrade, so make sure to backup your database prior to upgrading, as usual.

Some changes to highlight:

Stricter named server rules: Named servers now have stricter rules on their names, and a new less-restricted “display name” field. This might be the most significant change affecting you! See the documentation for more details.
Python 3.10 and database upgrade: JupyterHub 6.0 now requires Python 3.10 and includes a database schema upgrade. Make sure to back up your database before upgrading.
Updates to metrics: The old StatsD metrics have been removed, in favor of the Prometheus metrics that most people are already using. For Spawner (and spawn hook) authors, JupyterHub 6.0 also introduces SpawnException, which provides more granular information for classifying spawn failures in metrics, to help deployments separate errors from rejections.
Internal communication changes: JupyterHub now uses aiohttp for internal HTTP requests, which may require tuning for large deployments since internal HTTP requests. This update has facilitated using unix sockets for internal communication on single-node deployments.
Improved granularity in API access: JupyterHub provides new API endpoints for working with single servers, along with more flexible ways to define permissions, such as start:servers scope and extra_user_scopes.
See the documentation for more information.

Beyond these main changes, there have been lots of other improvements and bugfixes. You can find more details in the upgrading documentation and changelog.

Thank you to our contributors

The following people contributed discussions, new ideas, code and documentation contributions, and review: agoose77, anny0811, ateucher, betatim, bl-aire, clhedrick, consideRatio, dometto, dragz, erawn, ericvd-ucb, jezwilkinson, jnywong, jrdnbradford, kellyrowland, kreuzert, LUC4R4T0R, manics, mayli, milliams, minrk, mynkyu, nsychev, Paul2708, Rameshkumar31595, rgaiacs, shaneknapp, sixonenines, tbizouerne, Tech-Abhang, wilkinvr, willingc, yuvipanda

Positron Server available for academic use via JupyterHub

2026-07-29T18:19:00+00:00

Editor’s note: This is a guest blog post from Posit, a Jupyter Foundation member

Academic institutions can now offer Positron directly within their existing JupyterHub environments, giving students access to a modern, full-featured data science IDE without requiring local installation or additional infrastructure. Licensed under the permissive, source-available Elastic License 2.0, Positron as a desktop app is free for everyone to use, build on, and extend across personal, academic, and commercial projects. Through a free educational license, universities and other academic organizations can additionally make Positron Server available to currently enrolled students for coursework, helping create a consistent learning experience across classrooms, labs, and remote environments.

By integrating Positron into JupyterHub, educators can provide students with a powerful development environment alongside the tools they already use. Students simply launch Positron from the JupyterHub interface, just as they would open JupyterLab or a different environment, and can begin working immediately.

Bringing a Modern Data Science IDE to the Classroom

Positron combines the familiarity of notebook-based workflows with the capabilities of a professional integrated development environment (IDE). Once launched through JupyterHub, students have access to the complete Positron experience, including:

Comprehensive support for both Python and R
Access to the OpenVSX extension marketplace
Built-in data viewer and variables explorer
Integrated help pane, debugging tools, and version control
Advanced development features that help students grow their data science skills over time

This approach allows instructors to standardize course environments while reducing the technical barriers often associated with software installation and configuration.

Designed to Work with Existing JupyterHub Deployments

Positron Server is built to integrate directly into existing JupyterHub environments, making adoption straightforward for administrators already managing Jupyter-based infrastructure.

The solution is compatible with:

JupyterHub deployments running JupyterLab 4
Python 3.9 and later environments

Installation is provided through the jupyter-positron-server Python package, which is built on Jupyter Server Proxy. For administrators who have previously configured services within JupyterHub, the setup process will feel familiar.

Rather than functioning as a standalone desktop application, Positron Server extends existing JupyterHub deployments by making the Positron IDE available as another launch option within the environment.

Free Educational Licensing for Teaching

Academic institutions using Positron for instructional purposes are eligible for a free teaching license. Under this license, access can be provided to:

Currently enrolled students
Course participants
Faculty and staff involved in delivering educational programs

This licensing model enables institutions to incorporate Positron into coursework without additional software costs, helping educators provide professional-grade tools to learners.

Getting Started

Institutions interested in offering Positron through JupyterHub can get started in three simple steps:

Review the eligibility requirements outlined in the Positron Education License Rider.
Contact Posit to request a free teaching license.
Follow the jupyter-positron-server installation and configuration documentation to deploy Positron within your JupyterHub environment.

Supporting the Next Generation of Data Scientists

As data science and computational research continue to grow across disciplines, educational institutions are looking for ways to provide students with professional tools that are easy to access and manage. By bringing Positron directly into JupyterHub, universities can deliver a modern, scalable development environment that supports both teaching and learning while minimizing setup complexity.

Whether students are writing Python code, analyzing data in R, or exploring advanced development workflows, Positron Server helps create a seamless experience from the first day of class through more advanced coursework and research projects.

Becoming the new JupyterHub and Jupyter Book community manager

2026-06-15T07:01:00+00:00

There are moments in life when things, simply, converge. You work and study, study and work, and then, simply, it happens. It starts one day when somebody tells you “there is this new thing called Jupyter Notebook, you should give it a try!”. So you install it, play around with it. It works like a charm. It’s easy, fast, nearly playful. It’s beautiful–and if you grow up in Italy, beauty always matters. Then, you have the urge to tell others about it. You organize a workshop at a conference in musculoskeletal image analysis and introduce marvelous novel tools for computational research: “You know? Nowadays we can do open science easily because somebody has created Jupyter Notebook, Binder, new Python libraries!” And then, together with some friends, you write a proposal for a Jupyter Community Workshop, and–unbelievable!–you get it. Initially, you think you are just organizing a workshop. But somehow, the number of people interested becomes larger and larger, and you find yourself co-founding and coordinating what is now the Open and Reproducible Musculoskeletal Imaging Research (ORMIR) Community. In ORMIR’s Python packages, the examples are in Jupyter Notebook, the documentation is in Jupyter Book. Sooner or later, we will have our JupyterHubs too; it’s just a matter of time. In parallel, life gives you some professional challenges. So you start teaching Python online, one-to-one. You look for material that fits your style, but you cannot find any. So you start teaching it your way, using Jupyter Notebook in JupyterLab and it works. Students find it easy, independent of their background. The teaching material seems to work, so you transcribe it into a book entitled Learn Python with Jupyter, and you make it open. At the same time, you are obsessed with keeping up to date. You go to JupyterCon, meet the people, watch their YouTube videos, participate in online meetings. And then, there is a job posting for a community manager. And you are like: “Nah, they will never hire me”, so you set it aside. But then somebody out of the blue tells you to apply because “100% of lottery winners played!”. So you give it a chance, and here I am: my name is Serena Bonaretti and I am the new JupyterHub and Jupyter Book community manager!

In this new role, I am very much looking forward to working with the JupyterHub and Jupyter Book communities! I will collaborate with maintainers to understand and document projects’ priorities and define the best processes to achieve their vision. I will support contributors in finding their way into the projects and make the real-world impact that they hope to achieve. I will dialogue with users to make sure that the tools we create are useful for their work. And I will share our activities monthly and what we learn with anybody interested, so that we can all benefit from this journey!

And you? How did you start using the tools of the Jupyter ecosystem? Do you work in JupyterHub? Have you ever written a Jupyter Book? Would you like to be a contributor? Are you an enthusiastic user? Write me on Discord (Jupyter Book/MyST) or Zulip (JupyterHub). The JupyterHub and Jupyter Book communities and I are looking forward to getting to know you!

Better sharing UX with nbgitpuller and contextual error handling

2026-04-09T15:02:00+00:00

TL;DR

nbgitpuller now has improved UX context-aware error handling. Update to version 1.3.0 and let us know what you think by opening an issue 🚀

What is nbgitpuller?

nbgitpuller lets you distribute content to a Jupyter user via the click of a button! This allows your users to focus on the content without needing to understand git or other version control machinery.

nbgitpuller provides automatic, opinionated conflict resolution by using git under the hood. It is commonly used to distribute content to multiple users of a JupyterHub, though it works just fine on an individual person’s computer, if they have Jupyter installed.

Here’s an example of nbgitpuller in action:

The author uses the nbgitpuller link generator to create a link.
The consumer clicks the link, and the content is pulled into a live Jupyter session.

Example use cases for sharing with nbgitpuller include:

Interactive demos
Workshop and training scenarios
University exams

and many more!

How does nbgitpuller work?

The nbgitpuller package is installed in the compute environment. The compute environment now has a /git-pull endpoint URL that can understand where to pull content from using URL parameters. Syncing content critically depends on git operations to fetch, checkout, clone, merge, commit, etc.

When it goes wrong

Based on the data sent through by the kind folks running the Berkeley DataHub, there were 2163 logs available, of which:

983 (45%) were ‘merge’ conflicts
493 (23%) were ‘fetch’ errors
467 (22%) were ‘ls-remote’ errors

Bar chart showing the most common errors users faced.

The remaining errors (10%) were mostly ‘checkout’ errors. This represents the most frequent errors as seen by students.

Bar chart showing the most unique errors link authors made.

In the same set of logs, there were 172 unique errors: most ‘ls-remote’ errors come from mistakes in the content repo URL. This represents the most uniquely common errors made by instructors.

Merge conflicts

If the link author changes content after the consumer clicks a link, then nbgitpuller needs to sync updates for the consumer on subsequent link clicks. The nbgitpuller merging strategy makes opinionated choices so that the link consumer never has to interact with git, and will always preserve the consumer’s working changes.

Things can go wrong when

Consumers can diverge the git history if they perform a git commit
Authors can diverge the git history if they perform force push commits

Error UX (old)

Old UX for merge errors.

Problems with the old UX include:

a scary terminal
difficulty for the user to figure out what went wrong
no suggestion for the user to fix the problem or a signpost to continue to the compute platform

Error UX (new)

New UX for merge errors.

New improvements to the UX include:

The terminal is closed by default, but you can optionally toggle it open
There is a copy to clipboard button to easily share the error log from the terminal
A more user-friendly and context-aware helper message is displayed
A link to the general documentation for reference
A ‘Proceed without syncing’ button takes the user to the compute platform without making any changes
In the case of merge errors, an extra Backup and resync button option is presented

Learn more

nbgitpuller documentation

Acknowledgements

UC Berkeley and the CloudBank Classroom project
CAL ICOR for co-funding this development work
Eric Van Dusen and Sean Morris for championing this work
Balaji Alwar for providing the error data and sharing feedback
Nicolas M. Thiéry for feedback on the UX design
cmarmo for contributing the ‘Backup and resync button’

Call for applications: Community Manager for JupyterHub and Jupyter Book

2026-03-05T16:44:00+00:00

The JupyterHub and Jupyter Book communities are hiring a community manager.

Join our community and support us all to grow, nurture, and build things together. Credit: The Turing Way and Scriberia, used under a CC-BY license https://doi.org/10.5281/zenodo.3332808

In JupyterHub, we are democratizing access to interactive computing environments to allow everyone, no matter where they are, to explore data, develop code, and advance knowledge without barriers. In Jupyter Book, we are building the next generation of open, community-driven technical publication tooling, leveraging the MyST document engine to create modular, reusable, and reproducible computational narratives.

We are excited to fund dedicated time for someone — maybe you! — to join these teams.

The community manager will be responsible for identifying and developing activities and communication pathways that align maintainer and contributor actions with the goals of each project, and that build bridges across the Jupyter ecosystem.

You can read more about our motivation for this role in the proposal, part of the Jupyter Foundation’s Community Funding initiative. We particularly encourage applicants to review the section “Impact Metrics: How do we define success?”

In this blog post we have outlined the community manager’s responsibilities, their preferred expertise, and how to apply.

This is a contract role, funded at $90,000 to be spent by the end of April 2027. The community manager will be engaged as a contractor of the Linux Foundation and will follow the standard Linux Foundation contractor onboarding process.

The deadline to apply is 29 March 2026.

Responsibilities

Strategic alignment and contributor guidance

Collaborate with the maintainer teams from both projects to understand, document, and broadcast each project’s priorities to their communities.
Actively connect contributors to open issues and workstreams that align with the respective project’s goals, and redirect well-intentioned contributions that are not aligned with the project’s currently prioritized efforts.
Efficiently connect contributors with maintainers to provide them the guidance and review needed to make efficient progress, managing workload on both sides of the mentorship pair.

Contributor growth and leadership development

Design, visualize, and manage a clear contributor journey, including identifying requisite skills and project context knowledge, for each community.
Identify engaged community members and create pathways for them to take on more responsibility, such as triaging issues, reviewing pull requests, or mentoring others within their project of interest.

Fostering knowledge sharing and collaboration

Identify and create opportunities for the community of each project to share information, best practices, and technical knowledge, while respecting their distinct roadmaps.
Facilitate cross-community events, such as running “collaboration cafes” for both JupyterHub and JupyterBook, and strategically shape them into meetings that provide value and foster connection for both projects.
Develop re-usable templates for explaining contributor pathways, such as onboarding documentation, skills maps/matrices for progression into contribution and maintenance roles, and collaboration cafe guides.

Learn in the open

Ensure all materials are publicly available and discoverable, including progress against our community health metrics described below.
Promote these learnings across the Jupyter community including within the Zulip channel, at regular community calls, and by “visiting” other sub-projects (at synchronous meetings or in asynchronous discussion spaces) to discuss community management efforts as requested.
Join the Jupyter Community Building Working Group and be an active member throughout their paid time on the project.

Preferred expertise

We have not marked any of the expertise categories listed here as required. We recognize that we are looking for a unicorn, and they are curiously hard to find! If you do not have expertise across all of these categories we encourage you to apply. We are looking forward to getting to know candidates better, including where they have identified opportunities for career and skill development as part of the JupyterHub and Jupyter Book teams.

Our assessment pathway is included as part of this list to guide applicants in curating their strongest application. If you are uncertain how to answer any of the questions and/or prompts, please ask in our Jupyter Hub Zulip channel or email Kirstie Whitaker directly at kirstie.whitaker@berkeley.edu.

Track record of contributions to open source projects.

Assessment: Cover letter & links

Understanding of multi-stakeholder, community governed open source community dynamics.

Assessment: Interview question. Describe two situations — one featuring a success and one a failure — where you have sought to align diverse stakeholders around a common goal. For both, share what you learned from the experiences.

Understanding of the challenges that Jupyter Hub and Jupyter Book are designed to solve (or reduce) in the world.

Assessment: Interview question. Describe two deployments — one each of Jupyter Hub and Jupyter Book — that each meet a need in either education, research or technical development. You do not need to have run these deployments yourself.

Technical skills in Python and/or TypeScript.

Assessment: Cover letter & links

Track record of identifying and completing tasks in the presence of significant uncertainty.

Assessment: Cover letter

Ability to work effectively with two distinct and diverse teams whose community members are geographically distributed with significant time zone differences.

Assessment: Cover letter. All instructions from this recruitment call followed, the letter is clearly and concisely written, includes concrete examples, and publicly accessible links where appropriate.
Assessment: Interview question. Given your knowledge of Jupyter Hub and Jupyter Book, and your previous experience, how will you know if you are working effectively within these two communities?

Ability to ask for help and self manage time allocations and tasks.

Assessment: Interview question. Which aspects of this role do you think will be hard for you? Which aspects will be easy (or easier) for you? Where do you expect to ask for help and/or support in the role?

How to apply

Submit a cover letter and resume through this Google Form: https://forms.gle/Qw6ddyJ8rzZY5vQm6

Please include:

Your motivation to apply for this role.

Links to up to 3 publicly available GitHub repositories that highlight engagement in issues, pull requests, community discussions (or equivalent conversations on alternative git forge platforms).

If you don’t have publicly available links, provide a letter from a team member from a private initiative who can endorse these collaboration skills.
If you can not meet this requirement, please note that in your application.

Links to up to 3 examples of code you have written in Python and/or TypeScript.

If you don’t have publicly available links, provide a letter from a team member on a private initiative endorsing these development skills.
If you can not meet this requirement, please note that in your application.

A description of a previous role (paid or volunteer) where you have identified and delivered outcomes in the presence of uncertainty.

Include links to task documentation (issue / PR / report) if publicly available.
If you can not meet this requirement, please note that in your application.

What to expect

We’ll accept applications from now through 29 March, 2026.

Once applications are in, here’s our process:

Resume and cover letter review: The recruitment panel will review all applications and invite up to 5 applicants for an interview on zoom at a time that is convenient for the applicant and the panel members.
Interview: The recruitment panel will meet with applicants for 30 minutes over Zoom. They will ask the same questions of every applicant.
Panel discussion & alignment: The recruitment panel will discuss interview outcomes and align on a decision.
Offer: We expect to communicate a final decision to all applicants by 30 April, 2026.

Timeline

29 March 2026, AoE: Deadline for resume and cover letter applications.
Before 5 April 2026: Invite up to 5 applicants to interview.
8 April — 17 April 2026: Interviews conducted over Zoom.
Before 11 May 2026: Communicate final decision to all applicants.
May 2026: Post holder coordinates a contract with the Linux Foundation, following their standard onboarding process.

Acknowledgements

This job description was co-created by April Johnson, Yuvi, and Kirstie Whitaker, with guidance from Chris Holdgraf, Min Ragan-Kelley, Raniere Silva, and Stéfan van der Walt, along with community input at the JupyterHub and Jupyter Book collaboration cafes.

Scaling “Maintainer Intuition” with Pull Request Triage Boards

2025-11-16T21:22:00+00:00

When I helped start 2i2c.org, one of my goals for the non-profit was to experiment with new and different ways of supporting the Jupyter ecosystem’s long term health. As part of that, we have identified how foundational contributions are very important (and distinct from directed contributions), and have been experimenting with new ways to make foundational contributions in sustainable and structured ways.

One such way was via systematically doing code reviews for Pull Requests from non-maintainers in the JupyterHub ecosystem. Reviewing PRs is a critical way that maintainers keep an open source project moving forward, but identifying PRs that can productively be merged is hard. This is a post describing our system for scaling this in our team. Each 2 week sprint, we:

Looked at all open PRs in the JupyterHub org who were not maintainers
Picked a PR that I deemed was reviewable and ideally mergeable within this time window
Have an engineer on the team pick up that PR, and try to get it to close
Report back on what we have learnt, so we can iterate on our process

We managed to do this for a majority of sprints over the last roughly 12 months! One key bottleneck we identified in the process was Step 2. In particular, I was relying on my maintainer intuition to pick a single PR that I believe can be merged, so others in the team can do review work. I started exploring what this intuition is, and if it can be scaled.

What is this maintainer intuition?

How did I pick a PR from a long list of open PRs? Observing my own behavior a few times, I noticed I was looking for:

PRs that aren’t too big, and are a reasonable size that can be merged within a 2 week window
CI tests passing, so at least our automated checks haven’t caught any issues with it
Features or bug fixes that I believe add value to the project and move us in the right direction towards being able to support our users as they need (this is the hardest!)
If the author of the PR is a newish contributor, as I want to encourage them to stick around by being responsive to their gift. All PRs are gifts that we may or may not choose to accept, but should do so with grace.
How long ago the PR was opened. There is such a big difference between a response to your PR 2 days after you make it vs 2 months vs 2 years. I prioritized newer PRs.
What kind of contribution is it primarily? Different engineers on our team have different skillsets (JS, Python, etc) and I wanted to match the PR to what the engineer preferred code reviewing.

While (3) is hard to scale, everything else seemed like something we could build systems for that let others follow a process, thus removing myself as a bottleneck. I experimented with some GitHub issue filters and project automations, and after finding them lacking, built out a brand new open source project to do this: pr-triage-board-bot!

PR Triage GitHub Boards

pr-triage-board-bot automatically maintains PR Triage Boards for a Github organization. You can check out the current boards to get a sense of how it looks: JupyterHub, JupyterLab, and GeoJupyter.

For each PR Triage Board, the bot will:

Add all open, ready for review PRs
Remove all closed or draft PRs
Annotate each PR with additional deterministic project fields that allow for sorting and filtering in the project board. To start with, it populates the following fields:
1. Author kind (Maintainer, Seasoned Contributor, First Time Contributor, Bot)
2. Size (Number of lines touched)
3. Date it was opened
4. If a maintainer has already interacted with the PR (One, Many, None)
5. Are there merge conflicts?
6. What kind of files were mostly changed? (Python, JS, Docs)
Keep this up to date by automatically running every hour via GitHub actions

Once these project fields are populated automatically, maintainers can create different Project Views for themselves to help with different workflows via filters and sorting. For example, the JupyterHub board contains the following views:

PRs split by author kind, sorted by newness and size so we can try to respond to PRs from new users as early as possible
PRs that have not had a single maintainer interaction on them, so we can acknowledge people’s contributions to us even if we can’t fully review it at the moment
Bot PRs with all tests passing and no merge conflicts, so we can more easily stay on top of automated updates
PRs that have been approved by a maintainer but not merged yet, as sometimes a maintainer wants to give others time to object but needs to go back and hit merge.
PRs that are primarily updating documentation, as the review process for this can be different

For our original purpose of getting more people to do code review, this board has served well — we can roughly say ‘Pick a PR that looks good to you from the top of the “First Time Contributor” or “Seasoned Contributor” list’, and that relieves me from being the bottleneck quite a bit. Other maintainers are also finding a lot of value in this board. For example, if you only have 15 min, you can probably get a clean bot PR reviewed and merged. Or look for PRs that haven’t been acknowledged by a maintainer and engage with them. Our next step is to help run structured social experiments, where we try to establish specific ceremonies in the open source ecosystem to get specific lists of open PRs (such as “PRs with no maintainer engagement, or PRs older than 2y”) down to zero. Stay tuned to hear more :)

We are essentially using GitHub Project Fields as a database, adding additional fields that are very valuable to maintainers but not available in GitHub by default. While ideally these would be contributed by us directly into GitHub, only Microsoft has control over what gets added to the product. So we find interesting workarounds like this to accomplish our goals :)

Successful Adoption

I was just playing with this, and Raniere from the JupyterHub team spotted it and asked about it on Zulip. Jason Grout from the JupyterLab team was also super interested, and with contributions from him, the bot quickly got adopted by the JupyterLab GitHub org as well. We cleaned this up a bit more, and found adoption within the GeoJupyter project too with help from Matt Fisher. As with everything we do at 2i2c, I had tried to design this to be widely useful to many orgs and maintainers rather than just us, and looks like I have wildly succeeded :)

I am happy to announce today that 2i2c is officially donating pr-triage-board-bot to Project Jupyter! I will still continue to contribute to maintaining the project, and welcome contributions from everyone else too!

If this looks useful to your open source project, consider adopting it by following these instructions. The project is still fairly new, so if you run into issues please let us know on the Project Jupyter Zulip chat or by opening an issue.

Simplifying and speeding up Binder builds with BuildKit

2025-03-03T14:05:00+00:00

The Binder Project allows users to build reproducible, sharable environments for interactive computing. To accomplish this, Binder uses a tool called repo2docker to generate an executable Docker image using the Reproducible Execution Environment Specification.

The first time a repository is launched on Binder, repo2docker must build the reproducible environment for it. This process can take a long time because of all the dependencies that need to be installed and turned into the image. As a result, Binder launches can feel slow and clunky, which is a poor UX for workflows that are designed around quick interactive sessions.

repo2docker was built several years ago, and followed patterns that were commonplace at the time. However, in the past few years the Docker community has made significant advances in optimizing the image building process. One-such improvement is the creation of BuildKit, a replacement for Docker’s historical build system that is much more sophisticated. However, repo2docker hasn’t leveraged these improvements because it was still using the original Docker Build system.

So, we’ve decided to spend a few cycles modernizing repo2docker’s image building logic by using the more modern BuildKit API (via docker buildx build). This allows for optimizations like build parallelization, better build caching, and supporting some Dockerfile features that Binder didn’t support earlier (particularly, COPY --chown). It also lays a foundation for significantly simplifying the repo2docker build infrastructure and leveraging more of BuildKit’s parallelization functionality. For example, we’d like to leverage BuildKit’s Kubernetes driver which distributes builds much more efficiently and in parallel.

Authors of Binder repositories won’t need to take any action^[1], and they’ll simply notice that mybinder.org (and any other community-run BinderHub instance) will be a bit snappier at building images.

If you’d like to learn more about the changes that enabled this, check out this mybinder.org pull request which has links to the repo2docker pull requests that added this functionality. We’re excited keep improving Binder, and are hopeful that this makes the experience of using mybinder.org and community Binders a little bit better.

Authors and acknowledgements

Yuvi Panda is a co-founder and the Technical Lead at 2i2c. He is passionate about building participatory open infrastructure for scientific & educational use cases. He is a Project Jupyter team member primarily focused on infrastructure related projects (JupyterHub, Binder, etc). He also wrote most of the code for this docker buildx transition, and shepherded it through to deployment on mybinder.org as well. He is ex-Wikimedia and ex-GNOME. Let’s eliminate accidental complexities wherever we find them.

Chris Holdgraf is a co-founder and the Executive Director of 2i2c. He is on the Executive Council of Project Jupyter, and co-leads the JupyterHub and Binder team as well as the Jupyter Book team). He was previously a post-doctoral researcher in the Department of Statistics at UC Berkeley, and a Community Architect with the Division of Data Science at Berkeley. He’s interested in using open infrastructure to support interactive computing workflows in research and education.

Many thanks to @minrk, @manics, and @consideRatio for their help reviewing and shaping this work.

Unless they were relying on undocumented implementation details of the old builder — in particular, the presence of a /.dockerenv file to detect if you are running in repo2docker scikit-learn/scikit-learn#30835 has an example. ↩︎

Join JupyterHub on Zulip chat!

2025-02-06T16:05:00+00:00

The JupyterHub team have just moved all their chat to a Zulip channel!

The Jupyter Zulip instance is a Jupyter-wide chat space allowing cross-team communication in channels, some of which can be viewed publicly without an account.

For the JupyterHub team, we have found better engagement from team members and across the wider eco-system by using the Zulip instance. To that end, we recently updated all of our Gitter rooms to “invite-only” so no new members could join, and left a pinned message with a link to the JupyterHub channel on Zulip and an invite to join the conversation there. We have also updated any of our repositories that had Gitter links to point to Zulip as well.

If you would like to keep in touch with the JupyterHub, we would love to hear from you over on Zulip!

Desktop GIS software in the cloud with JupyterHub

2023-08-02T16:23:00+00:00

👋 We are Trey Stafford and Matt Fisher, co-authors of the QGreenland data package’s source code. This year, we had the pleasure of running a hands-on geospatial data and open science QGreenland Researcher Workshop. It was important for attendees to participate in the workshop in a hands-on way while minimizing the negative impacts of installing software, requiring expensive personal computers, and troubleshooting unique computer configurations. We felt a JupyterHub was a good fit for our workshop for this reason, if it could accommodate our need for running QGIS — a desktop application.

In this blog post, we will introduce QGreenland, describe our experience using JupyterHub in the cloud for our workshop’s computing environment, and discuss challenges we overcame to enable our attendees to use QGIS in a cloud graphical desktop environment. Finally, we will highlight some workshop outcomes and discuss opportunities for enhancement based on new developments in the Jupyter ecosystem.

In our workshop, 25–30 international learners (including from Germany, India, France, Canada, Poland, and the United States) used QGIS in a JupyterHub’s browser-based Linux desktop environment to collaboratively test, explore, visualize, and process Earth science data simultaneously with the same user experience they expect from using QGIS on their personal computers! Better yet, getting started was as simple as logging in. Our workshop was a success story not just in education, but also in open source and collaborative development, and we want to share what we learned.

Note the browser tabs at the top of this screenshot; this is a full desktop-based GIS environment running QGreenland in the cloud!

The JupyterHub used by the QGreenland 2023 Researcher Workshop was generously provided by the NASA CryoCloud team, whose mission is to help researchers transition to cloud-based collaboration.

About QGreenland

QGreenland is an open-source Greenland-focused geospatial data package for QGIS, a community-owned graphical Geographic Information System (GIS) platform. Researchers and members of the public leverage QGreenland’s ready-to-use interdisciplinary datasets to do field planning, teach about glaciers, and much more.

QGreenland’s MIT-licensed source code uses community-maintained open software like GDAL and PyQGIS to automate data normalization and populate the QGIS project with important information like data provenance and the order of layers in the QGIS Layers Panel. Check out our documentation to learn more! QGreenland also has a YouTube channel with tutorials produced by CIRES Education and Outreach.

Once QGIS is installed, opening QGreenland is as easy as double-clicking the included “.qgs” file. Here, a representative view of QGreenland v3 alpha in QGIS is displayed with newly updated layers: Arctic sea routes (National Geospatial Intelligence Agency), bathymetric depth (General Bathymetric Chart of the Oceans (GEBCO)), and ice thickness (IceBridge BedMachine Greenland v5) layers.

Based on user research, QGreenland has enabled:

the public to more easily access data gathered by researchers visiting Greenland: “In Greenland, people are often asking, ‘how can we find the data the foreign scientists bring back from Greenland?’ Now we can directly utilize much of it.”
researchers to plan field work: “Being able to use QGreenland at our field station was critical to our research process!”
educators to develop interactive lessons about Greenland and climate change: “…using QGreenland for presentations because it is presentation quality already.”

QGreenland’s 2023 researcher workshop

One of the QGreenland team’s most important forms of direct user interaction and support is facilitating workshops. Most recently, we hosted a 3-day (total of 9 hours) virtual workshop for researchers focused on working with geospatial data in an open science framework. All of the materials covered in the workshop were built using open-source tools and are MIT-licensed and published on GitHub.

A “personal computer” in the cloud

We decided early on that we wanted to use JupyterHub to solve the diverse problems that come with “bring your own device” workshops. We experimented with administering our own JupyterHub on Kubernetes, but the setup overhead was too high for our short workshop. CryoCloud’s JupyterHub enabled us to avoid this overhead and focus on serving our participants. Because the software that comprises CryoCloud is open-source and developed in collaboration with the communities CryoCloud serves, we could directly contribute to curating a computing environment ideal for our participants.

JupyterHub is known for providing access to Jupyter Notebooks via JupyterLab, but it turns out it can also be used to host pretty much any interactive web based application! The jupyter-server-proxy project enables this, and there are additional packages that make running specific applications easier. jupyter-rsession-proxy makes it easy to run RStudio inside JupyterHub, jupyter-vscode-proxy allows running code-server (fully open source self-hosted version of Visual Studio Code) inside a JupyterHub, etc. Pertinent to our use case is jupyter-remote-desktop-proxy, which lets you run a complete Linux desktop environment inside your JupyterHub! This was critical for our workshop, as it allowed us to use QGIS — purely desktop software, not adapted for the web — from inside a web browser. Workshop participants did not need to install anything. This enabled participants to focus on the content of our workshop rather than the logistics of setting up and debugging tools on their varied machines.

The CryoCloud JupyterHub enabled each of our workshop participants to provision their own compute environment (JupyterLab + Linux Desktop) with all of our workshop’s dependencies pre-installed. It also set everyone on equitable footing — someone accessing the workshop on a 10 year old laptop would get the same computing resources as someone on a brand new MacBook Pro.

Challenges scaling QGreenland

The CryoCloud JupyterHub already had jupyter-remote-desktop-proxy and QGIS installed, so we could validate this approach to our workshop quickly. However, to use QGreenland at this scale, we needed to solve a couple of usability problems. The first issue was a user experience problem: the operating system did not have appropriate file type associations for QGIS, so files like the QGreenland project file would not open in QGIS when double-clicked in the desktop file browser. We quickly discovered a solution and integrated it with a simple pull request to the Docker image we were using.

The second problem was a performance problem: QGIS would take several minutes to open QGreenland from the hub’s shared storage drive. After some investigation, it turned out this was due to us loading multiple GB of data from an NFS share! While a long term solution might involve getting QGIS to load data directly from cloud object storage (like S3), we instead decided to go a different route — provision each user a small, fast and temporary Elastic Block Store disk. At the start of the workshop, we provided all users a small script that would copy the dataset from NFS to this faster disk once, and this drastically reduced load times from about 5 minutes to under 3 seconds! You can follow our debugging process on this issue, and find the JupyterHub config used to provision these disks here.

By overcoming these challenges, we created a smooth, intuitive, and performant computing experience for all of our participants, most of whom had never been exposed to this sort of collaborative computing environment.

Outcomes

The workshop participants engaged in small group work to complete various exercises, group discussions, and data scenarios. Each group produced Jupyter Notebooks and GitHub Discussions posts as deliverables. We created an outcomes webpage to summarize our participants’ accomplishments. One highlight was participants’ insightful commentary on FAIR & CARE principles.

Based on these outcomes, we consider our workshop a success. While we put in a significant amount of time creating our materials, CryoCloud’s cloud costs and our time investment in preparing computing resources were relatively small. For approximately 25 people, our cloud costs break down to roughly $1/person/day!

Conclusion

The CryoCloud JupyterHub met our workshop needs and provided a delightful experience for administrators and participants alike, and we are excited for what’s next. JupyterLab 4 and jupyter_collaboration v1.0.0, a real-time collaboration extension, were just announced, and the CryoCloud team is currently working to integrate these new releases into their hub. Real-time collaboration will enable exciting cloud use cases, like small groups working together on the same notebook without a screen share, or organizers providing technical support in a live notebook. We anticipate running this workshop again. We are excited to use JupyterHub again and look forward to experimenting with these new features!

Acknowledgements

Reviewers

In alphabetical order, thanks to Twila Moon, Yuvi Panda, Tasha Snow, and Alyse Thurber for their time contributing to this post!

CryoCloud

Snow, Tasha, Millstein, Joanna, Scheick, Jessica, Sauthoff, Wilson, Leong, Wei Ji, Colliander, James, Pérez, Fernando, James Munroe, Felikson, Denis, Sutterley, Tyler, & Siegfried, Matthew. (2023). CryoCloud JupyterBook (2023.01.26). Zenodo.https://doi.org/10.5281/zenodo.7576602

2i2c

2i2c is a non-profit organization that runs open-source infrastructure for collaborative computing, and maintains the CryoCloud JupyterHub used in this workshop. You can see the complete configuration of this JupyterHub in this public repository.

From intern to mentor

2023-06-19T08:02:00+00:00

Four years ago, Project Jupyter participated for the first time in the Outreachy program. And at the end of last year, during the December round, three more awesome interns joined the project. Because I was lucky enough to be part of this program both times, first time as an intern and second time as a mentor, in this post I will share the parts that were the most impactful and contributed the most to my growth as both mentor and engineer. I hope this will inspire other interns, to start their journey, but also other community members into becoming mentors.

When I first interacted with the Jupyter community, I remember I was impressed by how welcoming everyone was, how well organized they seemed, how promptly questions got answered and how fast code got reviewed. This was especially impressive during the contribution phase when prospect interns were everywhere at the same time, we were generating crazy amount of work for mentors and the community when asking for guidance and tackling tasks.

When the internship ended I realized how much I appreciated this process compared to other internships. The contribution phase made me feel very motivated because of the opportunity to work on real tasks, with impact for the entire community.

All the work traffic, immense quantity of GitHub notifications, emails, questions, etc., were totally opaque to me as an intern, they seemed to just be magically handled by mentors. I now know this seamless interaction is made of some very special ingredients like team work, time commitment and hard work, nothing magic about it.

Scoping the project, the most challenging part

I believe scoping the project was the part that I found the most challenging as a first time mentor.

The plan I had in mind for how to create the project roadmap, was to define a few tasks that were not very tightly scoped, choose one to start with, allocate time for the intern to research about it, have synchronous meetings to collaboratively refine it, then implement it, review, and then repeat for next task. And although this was a good first plan, I failed to recognize when it stopped working and didn’t realize it’s important to pause and adapt the plan.

It took me some time and guidance from others in the community, to realize that sometimes, constraints are beneficial and allowing complete implementation freedom over a task, can be overwhelming. For finding the appropriate [freedom — constraint] balance I needed to experimented with different task sizes, until tasks had a less overwhelming learning curve. This fosters and grows confidence and improves time management skills.

I found this to be particularly tricky because this balance can be hard to achieve when creating a project roadmap. The hidden complexities of a project, existing skills, or any prior experience managing a project from scratch, are hard to know and plan around in the very first iteration.

Another challenge of being a mentor that is usually hidden is the context switch penalty between mentor time and daily job responsibilities. A mentor needs to fine tune the amount of intern interaction and involvement in the project implementation so that context switches from daily responsibilities are as soft as possible. It’s also critical that a mentor has full support and transparent communication with the daily job management so that work responsibilities don’t spill over into mentoring time, otherwise this can lead to stress and a sub-optimal experience for the intern.

Asking questions publicly, the biggest achievement

The open source comes with a challenge attached to it by default, which is being comfortable communicating in the open and asking questions publicly. The time and work that needs to go into fostering this skill needs to be taken into account from the very beginning and it’s extremely important to consider it from the project proposal and roadmap definition phase. It will help set the correct project expectations, and expose an otherwise hidden complexity of open source project maintainability.

This was a challenge that was more difficult to overcome than expected, because in our case it involved, a first time mentor, a first time intern, and a project that required diving into the codebase of lots of other different projects in the ecosystem, so guidance from the entire community and asking questions was crucial.

My intern, Sheila was very determined and courageous and she received great care and support from the entire community in overcoming this obstacle. She is now a key maintainer of the internship project she worked on, asking and answering questions without issues. Checkout out her many achievements captured in this list (not complete) and her presentation about the Reusable JupyterHub Pytest Plugin at JupyterCon. A recording of her talk should soon be available on JupyterCon’s YouTube channel.

Mentor training, mentor shadowing, co-mentoring, the best allies

I am extremely grateful to have had the support of the other mentors and the community during theses three mentoring months.

In particular, I am very grateful for Sarah Gibson, the Jupyter strategic lead and our Community Coordinator with Outreachy, for organizing for us to get mentor training beforehand through the OLS community. I found that to be very helping and ground setting and it made me feel more empowered.

I also appreciated and found really useful the mentor shadowing sessions that I did with Sarah and the demo session that Min RK hold with all the interns and mentors. They were both great opportunities for me to observe and learn from the other mentors and get feedback from them.

Co-mentoring was something that was considered during this round, but because we wanted to select all these great interns we ended up not mentoring in teams. But this appears as a great approach of introducing first-time mentors.

It was all worth it, the takeway

The JupyterHub project got to improve its documentation, accessibility and testing thanks to this program and to the three amazing interns that worked with us during this round ❤️. In addition, this internship round helped us create a place where people can learn about JupyterHub’s experience with Outreachy and a template repository to be used for quickly generating other JupyterHub Python sub-repositories that follow the standards of the organization.

On the personal side now that the internship is over, I can happily say that it was very enlightening and humbling to be on the mentoring side as well. Even if it’s challenging, you get an amazing feeling when you give back and channel all the effort and kindness that shaped you and that you received as a mentee from other people.

Accurately counting Daily, Weekly & Monthly active users on JupyterHub

2023-03-27T09:16:00+00:00

Being able to say ‘we served X unique users over the last month’ (the Monthly Active User metric) is very helpful when advocating for resources for a JupyterHub your organization is running. However, until now, figuring out that number accurately has been difficult, requiring keeping and analysing JupyterHub logs.

That changes with JupyterHub 3.1! This Pull Request adds daily, weekly, and monthly active user metrics to JupyterHub, accessible via the prometheus interface by hitting the /metrics URL on your JupyterHub. These metrics are calculated by JupyterHub itself, and are pretty accurate as it already keeps track of when a user was last active. This relies metric on each JupyterHub user matching an actual user, so if you are using your JupyterHub deployment purely as an API with ephemeral users (as binderhub does, for example) or delete inactive users, these will not be useful numbers.

Ideally, you should have a prometheus instance to scrape and store metrics from your JupyterHub over time, so you can track this metric over time. However, in a pinch, you can also go directly to https://<your-hub-url>/hub/api/metrics and see the current value of these metrics. They are in the prometheus exposition format, but you can look for something like:

# HELP jupyterhub_active_users number of users who were active in the given time period
# TYPE jupyterhub_active_users gauge
jupyterhub_active_users{period="24h"} 610.0
jupyterhub_active_users{period="7d"} 2800.0
jupyterhub_active_users{period="30d"} 4526.0

This denotes that this particular hub had 610 daily active users (active over the last 24 hours), 2800 weekly users (over last 7d) and 4526 monthly ones (over the last 30 days). Very helpful if you want to quickly add numbers to a report :)

Note that depending on your hub’s configuration, access to the /metrics endpoint might require authentication.

If you do have a prometheus installation for your JupyterHub, you may benefit from deploying the JupyterHub Grafana Dashboards. These are targeted at installations of zero-to-jupyterhub on kubernetes, and provide a lot of useful usage & diagnostic information.

Grafana Dashboard showing “Hub Usage Stats” for a JupyterHub deployment. Four panels, clockwise: Current Active Users, Daily Active Users, Weekly Active Users, Monthly Active Users.

Happy report writing!

Online Collaboration Café launch: JupyterHub team meetings to become more collaborative spaces!

2023-02-27T08:49:00+00:00

This illustration is created by Scriberia with The Turing Way community. Used under a CC-BY 4.0 licence. DOI: 10.5281/zenodo.3332807

The JupyterHub team are refactoring our monthly meeting into a collaborative, co-working space that is more accessible and inclusive to those who are just getting started in the community — an Online Collaboration Café! This blog post aims to explain what that means, and what to expect when you attend. We plan to hold our first Online Collaboration Café on 21st March 2023. Please come along and let us know your feedback!

TL;DR

The team meeting will be held in a new format for 2 hours in the original time slot. Instead of a typical meeting style with an agenda, the group will be divided into breakout rooms to work more collaboratively on ideas and perhaps begin actioning them. There will always be breakout rooms available for onboarding newcomers and those who wish some dedicated time to undertake maintenance tasks, as well as a quiet working space in the main room. Participants should feel free to swap rooms and join/drop out as they need.

Why the change?

The JupyterHub community often cite the team meetings as a touchpoint for newcomers to familiarise themselves with the JupyterHub project. However, this is not always the case. The team meetings have an emergent agenda built by the community members — which is great! — but it also means that it is a potluck as to whether the meeting you happen to attend will actually be useful depending on the agenda, and a newcomer may have to attend multiple meetings over a long period before feeling comfortable to ask questions and know where they can begin to help.

By reformatting the meeting into a collaborative co-working space using breakout rooms, we can cater for both the need of newcomers to be oriented to the project, and for the community to discuss in-depth topics in an emergent nature.

What is an Online Collaboration Café?

The Collaboration Café is a concept that was developed by The Turing Way community, and you can read more about it and how they are run in their Community Handbook.

In short, an Online Collaboration Café utilises breakout rooms and pomodoro sprints to allow groups of community members to work together on a topic that best suits them. The space between the pomodoros are used as shareouts to the rest of the group, or as biobreaks.

What are the logistics?

This is an online café. We will meet on a video call.
We will be using the same alternating time slot that the team meetings used to occur in, but the slot will now be two hours. You can view the Team Calendar for details.
A chair will always be present in the main room to manage breakout rooms and greet folk who may arrive mid-sprint.
There will always be some default breakout rooms available:
- Onboarding: For new arrivals to the community. A member of the team will be available on the call to support anyone wanting to learn more about collaborating on GitHub, getting a virtual tour of our GitHub organisation, and help you in any way we can.
- Maintenance: Folks working in this room will be triaging issues, reviewing pull requests, and other maintenance-related activities across the JupyterHub organisation.
- Quiet working in the main room: If you would just like some quiet time dedicated to any JupyterHub-related work you have, you are invited to hang out in the main room (and keep the chair company 😉)

What will happen when I attend?

When you arrive to the Online Collaboration Café, there will first be some housekeeping, such as introductions, Code of Conduct review. There will then be some goal setting around what folks are hoping to achieve during the time. These goals will determine the topics for the breakout rooms. Any attendee is welcome to: suggest their own topic, join a suggested topic, join one of the default rooms, or work quietly in the main room. The breakouts will then run in sprints with breaks to share their progress. The Café is closed with some reflections, for example: how did your work progress, what should the project think about working towards next?

Here is an example schedule of how an Online Collaboration Café could happen inspired by The Turing Way:

Time	Activity
Start	Welcome, CoC review
10 mins	Introductions and goal setting
20 mins	Pomodoro 1
5 mins	Break
20 mins	Pomodoro 2
5 mins	Break
20 mins	Open discussion: celebrations, reflections, future plans
5 mins	Close

2 hours seems like a long time… Why not multiple meetings?

Various individual meetings covering separate topics, such as onboarding and maintenance, were considered. However given that the majority of JupyterHub’s community are volunteers, it didn’t seem practical to fill up the calendar with lots of meetings. Another reason to use parallel breakout rooms is that participants can swap rooms if the current conversation doesn’t appeal to them, as if you were at tables in a real café. This practice is highly encouraged.

The extension to 2 hours is important because this is a pivot towards collaboration and co-working, and we want to provide a dedicated time for folk to achieve that and begin actioning ideas. However it is a large chunk of time to devote when we are all busy, and so it is not a requirement to arrive on time and participate for the full 2 hours. Joining when you can and dropping when you need to is encouraged and doesn’t require apologies.

Will there be cake at this café?

It is an online café so there will be as many virtual cakes as you like! 🍰🍰🍰

This is an informal space, so you are encouraged to bring along any beverages and/or snacks. We hope you will join us!

Introducing JupyterHub’s Outreachy interns! — December 2022 Cohort

2022-12-07T07:45:00+00:00

As part of the community strategic support project funded by CZI’s EOSS grant series, the JupyterHub sub-project has funding to support Outreachy interns through four cohorts over the next two years. We would like to introduce you to the first cohort and the projects they will be working on!

Ogoh Blessing Onyowoicho — Improve Accessibility in JupyterHub

Accessibility is the ability of tools (in our case web tools) to be used by a variety of communities with different disabilities. There are a variety of standards and tools for evaluating and ensuring that a web page can be used effectively by as many people as possible. Work by the Accessibility team is ongoing to define a set of standard tools to improve accessibility across the Jupyter ecosystem.

The JupyterHub project is working to improve the accessibility of its pages to ensure we are providing tools that are as useful as they can be to as many people as we can. During the internship, we will evaluate JupyterHub’s accessibility, find ways to improve it, and integrate accessibility testing into the development process, in collaboration with the Accessibility team, to ensure we do a better job going forward.

Ogoh Blessing says:

My name is Ogoh Blessing Onyowoicho. I am a self-taught Frontend developer based in Lagos, Nigeria and I am an Outreachy intern working on Improving the accessibility of JupyterHub.
I am excited to work on JupyterHub because it is giving me the opportunity to use skills that I have accrued over the years to proffer solutions to problems that affect people’s lives directly. On hearing accessibility, the first thought one has is that it involves building web products that people with special needs can use seamlessly. Though this is part of it, accessibility goes way beyond this. It involves building products that different users (e.g users at different locations, users with different devices etc) can use easily. The thought of contributing to improving the experience of so many people alone excites me.
In the coming months, I hope to learn and continue to hone my skills as I am guided by my mentors and members of the community I get to interact with.

Allan Wasega — Restructure and improve JupyterHub documentation by implementing the Diátaxis framework

JupyterHub has a range of documentation that covers both developer and user audiences in order to help them deploy, maintain, and use their own instance of a JupyterHub. The success of an open source software project to (i) be adopted by users, and (ii) receive meaningful contributions relies heavily on the quality, navigability and accessibility of documentation so that users and developers have all the information they need to achieve what they want to do.

A framework for organising technical documentation has arisen called diátaxis. It takes a systematic approach to understanding user requirements of documentation throughout the lifecycle of interaction with a product and posits that different user needs require different approaches in creation of the documentation, as well as a layout to navigate these different “modes” of documentation.

This project will focus on a refactoring of the documentation for the JupyterHub package. We will begin by performing a review of the present documentation, categorise these into the diataxis framework, and then restructure the documentation files in the repository. Once we have transformed the documentation into this framework, it will be much easier to identify missing and unclear documentation (those that were difficult to categorise). We can then begin to curate resources that can fill the gaps and improve documentation that is not specific enough.

Allan says:

I am Allan Wasega, from Nairobi, Kenya. Broadly, I like to describe myself as a researcher and a writer. Researcher because looking into things to find patterns or hidden information has always been of interest to me. Writer because I figured early on that words allow me to express myself and to communicate to a larger audience than that inside my head :). As a Computer Science student, I looked for ways to bring these two skills together within the realm of computing and that is how I found myself in the technical writing space.
As an undergraduate student, I used Jupyter Notebooks extensively for most of my programming assignments and projects. As a result, when making my Outreachy application, the JupyterHub project immediately piqued my interest because I did not know that there was a way to serve Jupyter Notebooks to multiple users, such as within a classroom setting. Presently, I serve as a mentor at KamiLimu, which is a mentorship program for students pursuing technology-aligned courses in tertiary institutions in Kenya. One of the objectives of KamiLimu is to introduce students to tech specializations such as Data Science and Machine Learning, which, from experience, uses Jupyter Notebooks! Therefore, I am excited to work on JupyterHub because I will be learning about and helping to improve a product I hold dear and which I can use to advance the skills of the next generation of computer technologists in Kenya.
First, I hope to gain a deep understanding of how JupyterHub works so as to spread the word about it and its functionalities. Second, I hope to apply and advance my technical writing skills. I encountered the Diataxis Framework earlier this year while working as a technical writer at Tingle Software, a Nairobi-based software company. I have also been following Daniele Procida (the author of the framework) on Twitter for a while now and his work has been quite inspiring. Through this project, I will apply the Diataxis Framework to restructure JupyterHub’s documentation and, in doing so, further my understanding of the framework. Finally, I am passionate about mentorship, especially in the tech field. Therefore, besides expanding my professional and personal networks, this project will enable me to learn how to conduct mentorship within a global setting, and I can, in turn, apply this knowledge within my local community.

Sheila Kahwai — Create a reusable JupyterHub pytest plugin

JupyterHub is a modular and extensible project, with parts, like the proxy, authenticator and spawner, that can be easily changed and extended. Testing the functionality of these components against JupyterHub is important and it requires various hub setups that can sometimes become complicated.

Currently, each of these hub components and the hub itself define their own testing infrastructure, building everything from the ground up using the pytest framework. But some of this complex work is either repetitive across JupyterHub sub-projects, or under-specified for some of them.

This project will abstract out these common parts into a separate testing framework. This framework should be a pip-installable pytest plugin that would provide various hub functionalities through pytest fixtures. These fixtures can then be re-used by JupyterHub and its components to bootstrap their own testing suite.

Integrating this plugin will drive some important refactoring work of the current testing architecture of JupyterHub and have a great impact in improving the overall test coverage, maintainability and continuity of the JupyterHub project.

Sheila says:

My name is Sheila Kahwai. I am a self-taught python developer from Nairobi, Kenya, working to specialize in back-end engineering.
I am excited to work on JupyterHub because it has made many products I have used throughout my learning journey. It is a privilege to work with a diverse community that has created products that are very useful to equally diverse users like myself.
While creating a reusable JupyterHub pytest plugin, I hope to gain more insight into creating plugins for massive codebases to improve maintainability and scalability with clean and reusable code. Through this project, I look forward to improving the overall testing infrastructure of the various JupyterHub components.

Welcome to the interns! We’re so excited to start working with you!

Securely pushing to GitHub from a JupyterHub with gh-scoped-creds

2022-04-21T16:54:00+00:00

Many JupyterHub users want to push and pull their content from GitHub in order to collaborate and share their work. However, working on a JupyterHub means working on shared infrastructure, not your own laptop, and this poses some extra security risks that have made two-way sync with GitHub more difficult. This post describes gh-scoped-creds, a new tool to make it quick and easy to authorize a JupyterHub session with push access to GitHub in a secure and simple manner.

GitHub user credentials are high value targets for cybercriminals in today’s security environment, and any system that stores these credentials long term paints an unwanted target on itself. Current solutions — putting an ssh key on the JupyterHub, using a personal access token or deploy keys — involve storing long term valid GitHub credentials in the filesystem. As users can do this by themselves without admin intervention, admins often are not aware these (often unencrypted) credentials are on their filesystems. If an attacker compromises an ssh key or a personal access token, they have unlimited access to all GitHub repos the compromised user had access to, including repos in high-impact GitHub organizations. In the recent credential theft incident, Travis-CI and Heroku were ‘lucky’ in that the attackers accessed npm infrastructure — and since npm is owned by GitHub, GitHub was able to detect that Travis CI and Heroku had compromised credentials. You and the users of repositories you have rights to might not be so lucky. It’s 2022, and supply chain attacks are everywhere — you aren’t special, you’re just one link in a long chain attackers use to get to someone else.

There is a clear need for a simple solution that lets users push to GitHub from JupyterHub in a secure manner without admins having to worry about securing high-value GitHub credentials long term. It is not acceptable to “Just Say No” to users wanting this functionality either — if you try to ‘sacrifice’ usability for security, you end up getting neither.

gh-scoped-creds attempts to solve this problem by allowing users to grant time-limited push access to specific repositories to specific JupyterHub installations in a user friendly way.

Here’s a quick GIF running through the user workflow.

Push access is scoped both by time (credentials expire after 8 hours) as well as repository (access is granted per-repository, per-hub). While you need to refresh credentials every 8 hours, the list of repositories is remembered until you explicitly revoke access. You can always grant access to your own personal repositories, but repositories belonging to organisations might require admins to approve push access to them.

You can also run the command from the terminal as gh-scoped-creds instead of using the IPython magic %ghscopedcreds as shown in the demo. This way, you can also use this from a HPC system, not just a JupyterHub!

Setting this up for your JupyterHub requires a tiny bit of work from the admin — see the project README for more details. Shouldn’t take long, and it’s a one-time task. Once that’s set up, your users can securely push to GitHub from the comfort of their JupyterHubs!

Thanks to Fernando Perez for using his stat159 class at UC Berkeley to test this project out.

JupyterHub 2.0

2021-12-01T14:15:00+00:00

We’ve just published JupyterHub 2.0, our biggest release in a while.

Roles and scopes (RBAC)

Thanks to the work of Omar Richardson and Ivana Huskova and more contributors along the way, JupyterHub now has fine-grained control of permissions, via scopes (a role is a named collection of scopes).

This makes JupyterHub more secure because you no longer need to give every user or service full admin permissions if they need to take one elevated action (such as a cull-idle service). Now, you can give users and services exactly the permissions they actually need. admin still exists, but nothing should need to be an admin anymore. Fine-grained scopes also allow new configurations that weren’t previously available, such as granting instructors permission to access their students’ servers, which is useful with JupyterLab’s new real-time collaboration features, without needing to grant them full admin permissions, or even access to servers of users not in their specific class.

The significant change in how permissions work means that upgrading to 2.0 requires some downtime. You will need to shut down all services and single-user servers across the upgrade, and make sure you upgrade JupyterHub to 2.0 in both the user environment and the Hub environment at the same time.

All tokens issued prior to the 2.0 upgrade will be revoked.

As always, back up your database before upgrading!

Read more about roles and scopes in the docs.

JupyterLab by default

2.0 also marks JupyterHub’s transition to using JupyterLab as the default UI. You can stick with the classic notebook server with:

c.Spawner.environment = {
    "JUPYTERHUB_SINGLEUSER_APP": "notebook.notebookapp.NotebookApp",
}

or try RetroLab, a classic-style UI built on the newer Jupyter Server:

c.Spawner.default_url = "/retro/"

(assuming you’ve installed it in the user environment, with e.g. pip install retrolab).

And more

There are various other changes and improvements in 2.0, including a new admin UI, API pagination, and no longer relying on any command-line arguments for single-user servers. Check out the changelog for more details.

We encourage you to take 2.0 for a spin, try out the roles and scopes, and let us know how it goes.

Thanks to all the contributors who helped us with 2.0, especially prerelease testers!

Setting up a “Production Ready” TLJH deployment

2021-06-04T17:05:00+00:00

The Littlest JupyterHub is an extremely capable hub distribution that I’d recommend for situations where you expect, on average, under 100 active users.

The Littlest JupyterHub is a distribution of JupyterHub for single VM instances, best-used with 1–100 users.

Why not Kubernetes?

The primary reason to use Zero to JupyterHub on k8s over TLJH in cases with a smaller number of users is to reduce costs — Kubernetes can spin down nodes when not in use. However, you’ll always have at least one node running (for the hub / proxy pods) and the extra complexity that comes with it — particularly around needing to build your own docker images — may not be worth it. TLJH works perfectly well for these cases!

What is ‘production’?

A JupyterHub that you can run securely without lots of intervention from the person who created it is what I’ll call a production-ready JupyterHub. It’s a pretty arbitrary standard. In this blog post, I’ll lay out what I want in the TLJH hubs I run before I let users on them.

Authentication

Use a real Authenticator, not the default FirstUseAuthenticator. The default authenticator is pretty insecure, and should really not be used in production. If you don’t know what to use, I’ll suggest the Google or GitHub authenticators.

Enable HTTPS

Enable HTTPS. An absolute security requirement now, and TLJH makes it quite easy. You do need to get a domain for this to work, which can be a source of friction. Totally worth it, though.

Resource Limits

In many systems, a single user can often write code that accidentally crashes the whole system. By default, TLJH doesn’t have any memory limits enforced per-user, but it is very easy to configure it to enforce memory limits. Tuning these to match your needs will help prevent a single student from accidentally taking down your whole hub. I’d highly recommend checking how much memory your typical notebook uses, and making sure you have user limits set to above that.

Sizing your VM correctly

If you choose a VM that’s too big, you’ll end up spending a lot of cash for unused resources. If it’s too small, your users will not have the resources they need to do their work. TLJH provides some helpful docs estimating your VM size, and you can always resize your VM afterwards if you get it wrong.

Disk backups

TLJH contains everything on the VM’s disk — your user environment, users’ home directories, current hub configuration, etc. It is very important you back this up, to recover in case of disasters. Automated disk snapshots from your cloud provider are an easy way to do this. Most major cloud providers offer a way to do this — Google Cloud, Digital Ocean, AWS, etc. Some let you automate it as well — Google & AWS certainly do, I’m not sure about other cloud providers. This isn’t the best way to do backup — there’s approximately 1 billion ways to do so. However, this is an absolute minimum, and it might just be enough.

If you want to be more fancy, I’d suggest using a separate disk / volume for your user home directories, possibly on ZFS, and snapshot much more aggressively. Talk to your nearest google search bar for your options.

Pin your public IP

Some cloud providers change your VM’s public IP address if you start / stop them. This can be pretty bad — you’ll have to change your domain’s DNS entry, and re-acquire HTTPS. A hassle! You can tell your cloud provider to hang on to your IP even if your VM goes down / changes. And you should! DigitalOcean doesn’t require this, but Google Cloud does. I think AWS does too, but I’m not sure how you can reserve the public IP for it — since it’s usually a domain name itself.

Base environment setup + snapshot

TLJH has a shared conda environment that is used by all users. Everyone can read from it, but only users who are admin can write to it (via sudo). This is one of TLJH’s core design trade-offs - admins can install packages the way they are used to, without requiring a separate image-build step. But it also means the admin can mess it up - conda environments can be sometimes fickle! So it’s not a bad idea to spend some time in the beginning setting everything up - python packages, JupyterLab extensions, etc. Then make a disk snapshot, so you can revert to it if things go bad. This is where having a separate disk for your user home directories comes in handy, so you can reset your hub environment without losing your user home directories.

SSH admin access

The TLJH documentation strives hard to make sure SSH isn’t required for setup and most common usage. However, if your TLJH breaks in certain ways, you can no longer access the machine — since all access is via TLJH! For this, I recommend making sure someone who is admin has SSH access to the VM. Most cloud providers offer a way to set the root ssh key on creation. If not, you can follow the many guides on the internet to making it happen.

You can also just put your ssh keys in $HOME/.ssh/authorized_keys, and ssh in as jupyter-<username>@<hub-ip>. This works for any / all users!

Others?

I’m sure this isn’t the end — probably need something about firewalls, monitoring and automated system package upgrades. But hey, great start!

Lessons learned from Jupyter’s Contributor in Residence pilot

2021-03-11T20:22:00+00:00

Project Jupyter, like most open source communities, often lacks the time and people power to maintain existing projects. Existing maintainers are often overloaded with work, while new volunteer contributions are often driven by the excitement of new feature development. Although on-boarding new maintainers is a sustained long-term team effort, it is something necessary and beneficial not only for the project, but also for existing maintainers and new contributors alike.

For this reason, we decided to pilot a Contributor in Residence program that was generously funded for a year through CZI’s Essential Open Source Science program. Last year we ran the first iteration of the JupyterHub Contributor in Residence role. This post is a short report about what we learned, in the hopes that it guides other projects in pursuing similar programs.

tl;dr

The Contributor in Residence program is a great way to:

Introduce new members into your community
Create pathways for others to get more experience, and grow in their role and leadership
Get a lot of work done that is hard to motivate volunteers to do
Spend time thinking at a “meta” level about how to improve dynamics in the community
Have a paid opportunity to support individuals who can’t contribute via volunteerism alone

However, doing this well requires planning ahead of time and attention throughout the program. It is crucial that you develop a plan and pathways for integrating the CIR into your community, and ensuring that they have the right perspective to support the community. If you do this, the CIR program can be a fantastic win-win situation for everyone involved.

What is the Contributor in Residence?

The Contributor in Residence (CIR) carries out a variety of technical and community-focused actions, looking for opportunities to keep momentum moving forward in conversations, improving infrastructure to minimize human toil and streamline information, and making improvements to the codebase that are focused more around sustainability, stability, and understandability.

For more information about our CIR program, check out these blog posts:

The JupyterHub welcome bot image, hand-drawn by our first Contributor in Residence, Georgiana

How management and planning worked

We didn’t define explicit “metrics-based” outcomes for this Contributor in Residence pilot, as we were not sure the behaviors that would be most beneficial in the long-run. Instead, Georgiana kept track of the activities that she worked on throughout the year, and reflected with us often in identifying challenges to overcome, and opportunities to improve the work she was doing.

In practice, Georgiana did not have a sole formal ongoing mentor providing her regular guidance throughout the project. She received mentorship from a variety of JupyterHub Team members across the repositories, and via participation in issues in GitHub (more on that later).

Georgiana kept track of her actions at this GitHub project. This contains a collection of issues, to-do items, and pull-requests throughout the JupyterHub community, and is a public record of much of the work that she focused on. While we scoped the CIR to cover all of the JupyterHub repositories, these efforts had a general focus on The Littlest JupyterHub, where she is now the primary maintainer and community liaison for the project. In addition to these development-focused efforts, Georgiana also spent a lot of time improving infrastructure and team processes at the JupyterHub-wide level, generally via the JupyterHub Team Compass.

Tips for running a CIR program

The following are a few best-practices that we picked up during our pilot.

Define an explicit work plan and system for managing tasks

The biggest challenge Georgiana faced was in identifying specific projects to work on, and working through the ambiguity of a position that explicitly had no single clear workstream. To get around this, Georgiana sought out guidance and advice from others in the community, and over time gained an understanding of ways that she could contribute. This will be a challenge for any Contributor in Residence moving forward, and we will need to think of new ways to support this role with concrete suggestions. Georgiana wrote a bit about this experience in her CIR update blog post.

Dedicate management and mentorship

A related challenge was in ensuring that the CIR felt like they had a clear idea of their expectations, to-do items, and progress. In future iterations, we would like to dedicate more resources towards providing mentorship and management of the person. This role shouldn’t be a “task giver,” but rather someone to help guide, triage, and prioritize for the benefit of the community. They should have regular touch-points with the CIR to make sure they are supported in their role.

Have an onboarding session to help them navigate your project

Regardless of ongoing mentorship, you should definitely have an onboarding session. Many open source communities have multiple projects, varying stages of contributor documentation, different spaces for conversation and interaction. Moreover, the technical infrastructure may have underlying architectures and organization that is hard to learn on your own. Have a core maintainer (or the mentor if you’re using one) go over your technology’s structure and layout, and double-check the things the CIR is unsure of, and introduce them to various discussions and action points to learn from. This may require multiple meetings in the first few weeks/months.

Assess your current strategy every 3 months and adjust based on current needs

Choose a time interval when you revisit your plan. Project needs, community dynamics, and time availability will change over time, so take the time to think about what parts of the plan need to be modified to fit the situation at hand. An option could be bringing this up during team meetings to brainstorm ideas and get feedback, then write it down to put everything in order and make it real. Consider writing 3-month blog posts of your progress and status — these become public records of the experience gained from the program, and are good opportunities to reflect on how things are going.

Actively invite them into the community

Remember that most open source communities are just that — communities. Take time to make sure others in the community know who the CIR is, invite them to participate in community conversations, welcome them in gitter/slack/issues/etc. Anything you can do to make them feel welcome is great. Make sure that you keep doing this over time, not just once at the beginning of their fellowship. When someone feels that they are welcome and a part of a community, they will be much more comfortable at navigating how to support it themselves.

Tailor the role to fit the CIR as a person too

The CIR responsibilities are usually already sketched up before actually having a person to fill the role. Having an overall plan is an awesome idea and a very useful program compass. However, once the CIR is identified, it is important to re-address the overall focus and goals of the role to ensure that it aligns with the interests and skillsets of the incoming CIR. This includes things like familiarity with the project, future career directions for the CIR, and working styles for people involved.

Start slower

Jumping directly into a new project with a lot of complexity and many sub-projects is not sustainable. In our initial plan, we decided to track about 10 main repositories in the first three months of the CIR pilot. This was not sustainable, and led to an overwhelming feeling because of the constant influx of new issues, questions, etc. Especially for someone new to the community and their new role, this led to a feeling of “I’m not learning fast enough,” and difficulty concentrating on the plan. So please, start slow, take the time to acclimate to the project and go repository-by-repository.

Remember that knowing everything was never part of the role

Being a CIR for a complex project can be very overwhelming. There may be different sub-projects, technology stacks, and team processes across a community. It is crucial to consider the CIR program as a two-way street — it is primarily a learning experience and a chance to grow for the CIR, as well as an opportunity for the project to get some much-needed assistance. The CIR should not be expected to know or do everything either at the beginning or the end of their tenure — instead, celebrate the process and the learning that has come from it.

Ideas for things the CIR can work on

Here are a few things that we found particularly useful for the CIR’s workstream, that will likely be portable across other communities as well.

Identify automation and standardization action points

Observe whether a repetitive task could be performed by an automated system like a bot or a CI system, as this has the potential to ease the workload for maintainers.

For example, the JupyterHub GitHub org has about 60 repositories. Some of them had issue and pull requests templates, but most of them didn’t. Some had instructions of how to make a new release of the project, others didn’t. Some had their instructions written in a README and others had documentation, etc.

When the CIR project took off, the JupyterHub and Binder team already started the quest of automating and standardizing its sub-projects. The CIR’s job was to help the other team members propagate the changes (bots setup, templates, badges, docs) to as many repositories as possible.

Research and discover best-practices from other communities

CIRs can also act as helpful eyes and ears to watch other communities and understand the practices that they adopt. Many practices are useful across projects, and a person serving in this “meta” role is in a good position to think at a high level about opportunities.

For example — a simple “thank you” to a contributor can boost their experience a lot, especially when it is their first time contributing to a project. GitHub has a mechanism of letting you know when someone opens an issue or a PR to a project for the first time, but that little notification can be missed when under a huge load of work. For this, there is “the welcome bot” that can be set up to express the community’s gratitude towards their first time contributors.

Our CIR noticed that “The Turing Way” — an open source community-driven guide to reproducible, ethical, inclusive and collaborative data science — made excellent use of GitHub’s welcome bot, including illustrations to make the message even more pleasant and celebratory. Georgiana was inspired by this, which led to adding welcome bots across the JupyterHub repository (including our own hand-drawn images!). Every time a contributor reacts with a ❤ to the bot’s message reinforces the feeling that the bot has its desired, positive effect.

Welcome images that Georgiana drew for the “welcome bots” that she deployed across the JupyterHub repositories.

Spend time on fixing bugs and improving testing infrastructure

Having a working testing infrastructure helps maintainers and contributors alike. Various updates to the underlying infrastructure of the CI system, or even something as simple as a dependency version bump, can break it. So keeping an eye on the health of the testing infrastructure is something that should concern a CIR. This is particularly useful as the world of “maintenance automation” changes very rapidly, with new frameworks, plugins, and standards. Having a pair of eyes constantly thinking about how to improve this process is very helpful.

Having just a handful of people knowing how to to release a new version of a project is not scalable and puts an enormous pressure on that group. Instead, each project should have public instructions on how to do this, so that other contributors that have the permissions can cut a new project release.

The process of creating RELEASE.md files with these instructions for JupyterHub sub-projects, just like the standardization process, had already been started inside the JupyterHub org by the team members before the CIR started. The CIR’s job was to extend this idea to as many sub-projects as possible. This standardization and improvement quest continues to move forward thanks to the CIR and to Erik’s and Simon’s coordination efforts!

More generally — the CIR is often someone relatively new to a project, which makes them a valuable perspective to understand where team documentation is unclear to newcomers. Use the fresh eyes of the CIR to understand how to make the project more welcoming and easy to navigate, and it will clear a pathway for more to join the project in the future.

What we still haven’t figured out

While we learned a lot during this pilot phase, there are a few things we still haven’t figured out.

How to define a pipeline for new CIRs

The tenure of a CIR seems like it will be relatively short — maybe one to two years on average. This means that we need a pipeline of new individuals that are ready to learn and take on this role once a previous CIR moves on. Being effective as a CIR was only possible for someone with some context for the JupyterHub repositories and team processes, and we likely would not have had success hiring someone on a one-year contract who had never worked in the JupyterHub community before. In future iterations we will need a reliable pipeline of newcomers to the project to potentially serve as CIRs. Programs like Outreachy and Google Summer of Code are excellent sources of future CIRs, but we need to think about the “pipeline” for gracefully using them to power the CIR project.

Find mentorship that isn’t “nights and weekends”

While we may be able to find funding for mentorship and management, many people in the JupyterHub community already had full-time jobs. Having funding to pay for a part of someone’s time is helpful, but this is only possible if you can actually buy out part of somebody’s time (often not possible in certain careers and roles). It may be possible to spend extra time on mentorship in the evenings and weekends, but a sustainable model for this program will require somebody dedicating part of their “FTE time” on mentorship. We need to identify certain kinds of roles / individuals that have this flexibility in their work-load, and bake it into the strategy of the program.

Define a funding mechanism for CIRs

Finally, this CIR pilot was funded by a one-year grant from CZI. In future iterations we will need to have a strategy for funding this position in a reliable manner (or at least, in a repeatable manner). To make this sustainable, we must define a pattern to follow and require resources to do a good job. This will both reduce the amount of effort needed to plan a new round for the CIR, and will make it easier for other organizations to understand how their financial contributions could help fund another round of the program. Perhaps this can be an ongoing fundraising effort, similar to how other programs have “target funding levels” in their annual giving strategy, or perhaps it can be a coordinated effort across projects to join resources together for these positions. Either way, it will be important to have reliable funding to ensure that the CIR program isn’t in a constant state of sputtering and jumpstarting.

We hope that this post was an informative and useful set of lessons-learned from the JupyterHub team. If you’ve got any suggestions, comments, or questions for us, we’d love to chat. Feel free to open up an issue in the Jupyter Community Forum! We look forward to continuing this experiment, and hope to see other communities experimenting with the same model in the future.

Plasma: A learning platform powered by Jupyter

2020-05-11T07:55:00+00:00

Jupyter has been a great choice for education for many years. The Jupyter Notebook has become one of the most popular tools to conduct workshops, tutorials, and teach online classes.

Recently we have seen the emergence and adoption of JupyterHub distributions to facilitate the deployment of Jupyter-based platforms, both on private servers and in the cloud.

We would like to share with you an open-source learning platform called Plasma, built with Jupyter at its core.

JupyterHub Distributions

JupyterHub is a highly customizable and modular framework. To simplify its adoption, JupyterHub distributions target specific deployment scenarios with opinionated defaults. They make it easier to deploy JupyterHub on a single server and in the cloud.

There are currently two popular JupyterHub distributions:

The Littlest JupyterHub (TLJH): for running JupyterHub on a single machine, targeting between 1 and 100 users
Zero To JupyterHub on Kubernetes (ZTJH): for running JupyterHub on top of Kubernetes, to scale to a large number of users and machines

Although not an official distribution, jupyterhub-deploy-docker is also a good resource for a full Docker-based setup. It runs JupyterHub itself in a Docker container and orchestrates the stack with Docker Compose.

The Plasma Stack

Plasma stands for PLateforme d’e-Learning pour l’Analyse de données Scientifiques MAssives, which can be translated to “An e-learning platform for massive scientific data analysis”.

The platform is typically meant to be deployed on high-end machines with multiple cores and GB of RAM.

Some of the requirements for the Plasma project fall somewhere between the TLJH and ZTJH distributions:

The deployment should be on a single server, and reproducible on other machines too (running on Ubuntu 18.04+)
It should support multiple user environments with different sets of dependencies
Users should authenticate as system users and their data should be persisted in their home directories on the host machine

Although TLJH doesn’t officially support container technology, its plugin system opens the door to many other use cases.

Because of this and to foster the TLJH plugin ecosystem, we decided to develop the Plasma stack as a plugin for TLJH and consolidate the deployment story with Ansible playbooks.

Plasma is an opinionated JupyterHub stack powered by The Littlest JupyterHub, with the following defaults:

PAMAuthenticator to authenticate JupyterHub as users existing on the host machine
SystemUserSpawner to start single-user servers in Docker containers, using the system user home directories for data persistence

To enable extra functionalities, the Plasma stack relies on:

The tljh-repo2docker plugin to create multiple user environments
The tljh-plasma plugin to define the JupyterHub config
Ansible playbooks to automate the deployment and configuration of the server and add additional tools

The Plasma stack also contains tools to monitor the system, create and configure users, and add hub admins. It can be visualized with the diagram below:

The Plasma Overview Diagram

The stack is defined in the following repository: https://github.com/plasmabio/plasma

There is also extensive documentation with detailed explanations on how to deploy the stack on a new server: https://docs.plasmabio.org

A repo2docker plugin for The Littlest JupyterHub

The tljh-repo2docker plugin lets JupyterHub admins create new user environments using repo2docker. This plugin starts a JupyterHub service to manage user environments from the JupyterHub UI.

For those already using Binder, the idea will sound very familiar. Under the hood, the tljh-repo2docker plugin also uses repo2docker to build the Docker images. It follows the same patterns and naming conventions as Binder, which makes it easier and more natural to test the environments on Binder before adding them to JupyterHub.

New environments can be added by clicking on the Add New button and providing a URL to the repository. Optional names, memory, and CPU limits can also be set for the environment:

Adding a new environment

The Environments page shows the list of built environments, as well as the ones currently being built:

Building a new environment

The status of the environment changes once the underlying Docker image has been built:

The list of available user environments

Once ready, the environments can be selected from the JupyterHub spawn page:

Selecting an environment

Because it is separate from the Plasma stack, this plugin can also be used for other temporary TLJH deployments on a virtual machine. For example, the organizer of a workshop can prepare a list of environments before the event, just like they would with Binder.

Automating deployments with Ansible

To minimize the number of manual steps involved in the setup of the stack, Plasma also provides a list of Ansible playbooks.

Ansible is an open-source tool to automate the provisioning of servers, configuration management, and application deployment.

Playbooks define a list of tasks that should be executed and declare the desired state of the server.

The list of playbooks and instructions on how to use them are provided in the Installation section of the documentation.

Overall, the playbooks make it easier to perform upgrades, to automate the deployment process, and to replicate the setup at other institutions and universities.

Here is an example of what an upgrade looks like:

Upgrading the stack with an Ansible playbook

This playbook defines the tasks to:

download the TLJH installer
execute the TLJH installer to perform the upgrade
update the TLJH plugins
set the idle culler timeout
set the default memory and CPU limits
reload JupyterHub
pulls the latest jupyter/repo2docker Docker image

Acknowledgements

The development of the Plasma stack at QuantStack was funded as part of the Plasma project, led by Claire Vandiedonck, Pierre Poulain, and Sandrine Caburet, associate professors at Université de Paris.

Sponsors to the Plasma initiative include:

Région Île-de-France, via the “Trophées franciliens de l’innovation numérique dans le supérieur” (EdTech 2018) grant program,
Université de Paris, via the Initiative of Excellence (IdEx) Label and its “inovating teaching” grant program,
EUR G.E.N.E., the graduate school on Genetics and Epigenetics,
the university training “Création, analyse et valorisation de données biologiques omiques” (DU Omiques).

The Jupyter Contributor In Residence, update 1

2020-05-07T15:22:00+00:00

This year Jupyter was awarded a one-year “Essential Open Source Software for Science” grant from the Chan Zuckerberg Initiative, which we are using to bring on our first “Contributor in Residence” (many thanks, CZI ❤). This is a short report back from Jupyter’s first “Contributor in Residence”, Georgiana Dolocan. We are now about one third of the way through her tenure as CIR. Editor’s note: Georgiana has also made all of her own GIFs, because she’s just awesome like that ;-)

At the end of last year, I found out that the CZI grant (so nicely put together by Chris Holdgraf), was accepted. I was very happy to have gotten the chance to continue working alongside an amazing community that I grew very fond of from day one that I started contributing.

Plan 1.0

Being the first project of its kind, we had millions of ideas about what the Contributor in Residence (CIR) project should be about. We discussed and converged towards a common plan, then we decided to continuously iterate once we got some real life feedback.

See this GitHub issue with the end result of the brainstorming, and with various milestones set in place. In short, the purpose of the CIR is to work on the little day-to-day tasks that occur across a project (respond to questions, review PRs, fix bugs, write documentation, maintain infrastructure), nothing fancy, no big projects.

The biggest little achievements

Among the little day-to-day tasks, there are a few achievements that are more noticeable and worthwhile to mention, that will potentially have a bigger impact than others.

🚩 The first blog post about The-Littlest-JupyterHub is out

The superheroes and the magic wand is the first blog post about TLJH on the Jupyter Blog. It first started as a technical blog post but while writing and constantly relating to TLJH as having superpowers to explain how it works, it turned rapidly into a superhero story.

TLJH is an important project in the Jupyter ecosystem and one that I care about a lot, so stay tuned as there are more blog posts to come about it, and although technical this time, I won’t restrain myself from talking about its superpowers.

🚩 The TLJH CI tests were refactored and an upgrade test was added

Are you familiar with that feeling when a pull request looks OK, it passes all tests and all the bots are happy, so you hit that merge button? Yet, soon after, the unexpected happens and bug reports start showing up. 🙈

Even though you manage to find the issue and solve it, one question remains: “Why did the checks pass?” The answer isn’t easy to hear: “The tests need to be improved”.

TLJH operates on a “rolling release” model, where people just install from master, so it’s very important to have strong tests. However, our issue wasn’t exactly with the tests themselves, but rather our CI didn’t match the production installation workflow, and that small difference between the workflows left TLJH in an uninstallable state. Merging that PR in the first place along with finding the divergence point was the key to solve the problem.

Bonus point: whenever someone wants to upgrade TLJH, they do it by running the installer again. Thus, the changes we’re adding by merging a PR, don’t just need to pass the tests, but also they must not break the upgrade process. So, we also added an upgrade test to our CI to solve any potential future upgrade problems.

🚩 Three repositories in the JupyterHub organization are now being watched by a support bot

The Jupyter community currently uses GitHub, Gitter and Discourse (the newest of them) to connect to one another. Having three communication channels might be confusing if there isn’t any clear distinction between them in terms of their scope. So we needed to find a way to organize our discussion to make the contributing experience better for everyone (users, core developers, and everyone in between).

The Discourse forum was already the place where most of our general discussions happened. So why not make this place even more popular and encourage everybody to share their ideas and questions there? This way, the forum becomes the place where we can help and inspire each-other, and GitHub issues are where we talk about cool new features, find bugs, and figure out how to solve the issues.

The support bot will help spread the word about the forum and will make the transition easier. Currently, it is plugged into the jupyterhub, binderhub and tljh repositories and will act each time the “support” label is added to a GitHub issue that should be on Discourse.

We kindly encourage everybody to use Discourse to start or participate in awesome discussions or ask questions of any kind. A great place to start is the Introduce yourself thread where you can just stop by and say “Hi”.

Struggles and solutions

Working on little tasks sounds simpler and easier than starting a big, complex project from scratch. Well, this is not entirely true… Those small tasks are happening around big projects that are used by a lot of people and are constantly evolving. So, even a small, not very complicated task, can be hard when you’re still figuring out the “hows” and “wheres” of a project.

Struggle: distributing focus across multiple repositories and issues

When working on a bigger project, I used to switch between tasks as a means to unwind when running out of ideas and things seemed impossible. I guess one could call this multitasking. However, switching between tasks of different projects that you know little about is challenging enough to cause stress, resuscitate that impostor syndrome, and ultimately turn into burnout.

Solution: Although I don’t have a solution to this and I’m still having troubles mastering this skill, I find inspiration and hope among the people in the Jupyter community. Even if maintaining lots of projects at once, they manage to successfully juggle between tasks, sometimes even when this isn’t their main job, so they deserve a huge shout out ❤️

Struggle: keeping track of little things

The difference between working on a big project and solving small tasks here and there (besides the obvious size distinction) is that solving smaller tasks makes you prone to loosing a sense of progress. Though not a project in the traditional way, the CIR is a concept that would be worth extending if successful, so evaluating its progress is a must and can help determine and improve those weak points — and why not — even point out when it’s time to stop. But keeping track of little things is tricky.

Solution: GitHub issues, comments or PRs can automatically be retrieved through GitHub’s API (Chris Holdgraf has a great tool for this called github-activity). However, to get a sense of accomplishment and progress at the end of the day, I tried to keep my own list of todos and tasks done. I experimented with two approaches in this sense: keeping a HackMD “journal” and creating a GitHub project. After these three months I think I like the GitHub project better because it allows me to keep a todo list with small enhancements I’d like to work when that notification bell doesn’t ring that often and it’s less copy and pasting links.

Struggle: getting out of the comfort zone

Implementing that cool new feature or solving that annoying bug when feeling adventurous are part of the fun of software development. However, being a CIR is more than writing code. Things like writing blog posts and reviewing PRs are core responsibilities as well. But these are all new territory, so much that, I think up until last year I could count on the fingers of one hand the number of blog posts I had written or the number of PRs I had reviewed. So, the only thing left to say is “Hello new challenges! Farewell comfort zone 👋”.

Solution: Accept the challenge and practice every day until things don’t feel that scary anymore. Patience is key.

Plan 2.0

Version 1 of the plan was solid enough not to have to modify it completely.
So, things like tracking down the bugs and failing tests won’t be halted.
Instead, during these next two months, the plan will only suffer some little focus-related adjustments.

Specifically, I want to focus more on:

Getting more familiar with some of the repositories that didn’t get enough attention during the first phase of the project (zero-to-jupyterhub-k8s, jupyterhub-deploy-docker, binderhub, dockerspawner, kubespawner, repo2docker), potentially improving the documentation along the way. No more dipping the toes in the water, just dive in 🌊.
Writing more blog posts about the superpowers of TLJH.
Have more bots 🤖 helping the community.

In the process, I hope I can check the boxes for some of the items on my “learn and improve” wish-list:

technical writing
Kubernetes
UI/UX
courage

Stay tuned, because the next blog post about the JupyterHub and Binder Contributor in Residence adventures should be out in two months time.

The superheroes and the magic wand

2020-03-19T18:23:00+00:00

This is the first of a series of posts describing tools in the JupyterHub ecosystem, written by our wonderful Contributor in Residence, Georgiana. For our first post, we’ll share some lore of JupyterHub, and tell you a story of how it all began…

In a place far, far away, on a planet called Jupyter, magic happens every day. This land is special because it’s full of magical tools with all kind of powers, devoted to one common purpose: to help people. Everyone sympathizing with this goal either advocates, uses, or cares for these tools. So, in the blink of an eye, these people with sometimes nothing else in common than the same drive to help others, gathered together and formed a community.

This special group is known in the galaxy as “The Jovyans” and new recruits join every day. 🚀

This image was created by Scriberia for The Turing Way community and is used under a CC-BY licence. Zenodo record.

But somebody needed to take care of these magical tools. So, the first Jovyans decided that from then on, they will become tool-keepers. Because their greatest responsibility was to teach new Jovyans how to use the magic, shortly after, they created a set of guiding laws. Some call this “the Documentation”.

The magic wand

One greatly cherished tool on planet Jupyter is the magical wand. This wand’s very special power is to help people work together as a team and find solutions to important problems.

The wand is called “JupyterHub”.

But the wand is of vast complexity and because of this, so are the guiding laws that control it.

As a consequence, people trying to benefit from the magic of JupyterHub, spent a lot of time reading and understanding the instructions.

The superheros

The tool-keepers noticed that most of the people coming to planet Jupyter to become Jovyans were from a big city in the cloud 🌤 called Kubernetes. So they gathered together and debated what is the best way to help the people of Kubernetes.

After 3 days and 3 nights of intense discussions (also lots of pizza breaks of course) they decided that one of them needed to get special training, rent a house in Kubernetes and teach the people there the wonders of the JupyterHub magic wand.

The Chosen One gained the people’s trust, and it got better and better at anticipating and understanding their needs. So the locals started seeing it as a superhero and they even gave it a name, “Z2JH”. 👓

The news of these events started to spread far and wide, and people living in little towns outside of Kubernetes felt that they deserved the support of a superhero too. They also had great ideas and little time and needed to use the wand’s magic to do good.

The tool-keepers were inspired by the magnificent accomplishments of these little towns, taking place even without a superhero around. So the littlest of them all, volunteered to go into superhero training to help the people of the little towns do good, faster. Though it was little, in no time, it grew to become the helper these small groups needed.

The residents called it “The Littlest JupyterHub” or “TLJH” because of its stature. But everybody knew that its stature didn’t reflect its enormous tenacity, speed and ability to do great things. This ended up inspiring people all over to believe that not all superheros wear capes, nor do they need to know how to fly in the cloud to be cool.

Nowadays, people from all over join their forces and help the superheros learn new skills. Thanks to them, TLJH and Z2JH get stronger and better and have more special powers than they had when first created by the tool-keepers. Join them, become a Jovyan! ツ