Jupyter Blog - funding

2026 Jupyter Community Call For Funding Proposals

2026-07-06T19:32:00+00:00

The Jupyter Executive Council and Jupyter Foundation are pleased to announce a new Call for Proposals (CFP) for funding the Jupyter community to improve Jupyter. Visit the Jupyter Foundation Community Proposals webpage to learn more about the process and how to submit proposals. Submit a proposal by Wednesday, September 9, 2026.

The Jupyter Foundation began its operations in early 2025 with a mission to use its resources to support the Jupyter community. As part of this mission, we funded a round of community proposals in 2025. This funded work continues to bring many benefits to the Jupyter ecosystem (see the progress reports here). In this next call for proposals, we are iterating on this success.

Visit the Community Proposals webpage to learn more about past funded proposals and the Jupyter Foundation team compass for more detailed background about this funding call. In this second round, our goal is to continue learning how to best manage a community funding program. Based on feedback, we are updating the proposal process in several ways, including:

A new section of the proposal template has been added for Jupyter council or committee endorsements. We strongly encourage authors to seek and include endorsements from Jupyter councils or committees affected by the proposal. Each endorsement should be a brief paragraph about the impact of the proposed work from the council or committee’s perspective, and express support for the work and a willingness to receive it. We encourage authors to communicate early with relevant Jupyter councils and committees to make this endorsement process easier.
The de-risk section of the proposal is now optional for smaller funding requests.
The CFP is open for several weeks longer than the previous round. We do not anticipate extending the CFP deadline this time.

Submit a proposal by Wednesday, September 9, 2026.

As with the previous round, to help inspire proposals we are particularly excited to fund, we’ve defined funding priorities for the Jupyter Foundation. These are key outcomes that we must improve in order to grow the overall health and impact of the ecosystem. Briefly summarizing, these priorities are:

Grow the contributor capacity of Jupyter. We would like to grow the number of contributors, improve the efficiency of existing contributors, and facilitate learning across Jupyter subprojects. We’d love to see proposals that help us mobilize and support the total pool of energy available to contribute to Jupyter. Examples of recent activities in this area include hiring a community manager and hosting community workshops.
Improve the reliability, security, and consistency of Jupyter’s software and team practices. We believe that our technology will be more impactful and easier to contribute to and deploy if we improve our development infrastructure and team practices in ways that contribute to reliability, security, and consistency. Examples of recent activities in this area include investing in better testing infrastructure and linting rules, refactoring build systems, improving accessibility, funding security issue triage, and hosting a subproject roadmap alignment workshop.

While we’re particularly excited about these two outcomes and aim to focus funding on proposals that clearly feed into one or both of these priorities, we also encourage people to get creative and submit proposals for outcomes that have high impact and strategic value for the Jupyter community. We’re excited to see what others come up with, and are eager to work with you in unlocking critical funding in support of Jupyter’s community.

Announcing our first Jupyter Community Funded Proposals

2025-11-04T19:19:00+00:00

We are thrilled to announce the results of our first round of Jupyter Community Funding proposals. This is the first funding call from the Jupyter Foundation to directly solicit proposals from the Jupyter community.

We received many outstanding ideas from community members. We are so grateful for the time and effort everyone put into their submissions. This is a new process for us, and we’ve already learned a lot. We’re excited to take this first step in directly supporting community-driven work.

After a careful review, we are happy to fund the following proposals:

More reliable visual testing for Jupyter frontends: This project will improve visual regression and integration testing across Jupyter frontends. We hope this will lower maintenance costs and reduce maintainer burnout by making testing more reliable.
Separating JupyterLab’s build system for easier maintenance: This work will finish separating JupyterLab’s build system into a standalone tool. This will make maintenance easier for both core developers and extension authors.
Improving keyboard accessibility in JupyterLab: This proposal will focus on making JupyterLab fully navigable via keyboard and testing with disabled users. The goal is to find and fix key accessibility blockers, making the tools more usable for everyone.
Growing the contributor community for JupyterHub and JupyterBook: This project will tackle efforts like team coordination and mentorship pipelines with clear entry points for contributions. We hope this will grow our contributor base by making it easier to attract, retain, and leverage community contributions.
Improving code quality with custom linting rules: This proposal will develop custom linting rules to enforce project-specific standards in the JupyterLab and Notebook codebases. This will help catch common issues early and improve the long-term health of the code.
Reviving the extension playground for easier prototyping: This project will revive and improve the extension playground, a tool for prototyping new extensions. We’re excited for this to lower the entry barrier for extension development and help the ecosystem grow.

A huge thank you to everyone who submitted a proposal and participated in the review. We also want to thank the authors of proposals that were not funded in this round. We particularly appreciate their effort and hope they will continue to engage with this process.

We’re excited to see these projects develop and support the dedicated people behind them. This is just the beginning, and we look forward to more community funding opportunities in the future.

We want to send a special thank-you to Savannah Ostrowski, former Jupyter Foundation board treasurer, for her work in setting up this initial round of funding.

Your Ideas, Our Support: Jupyter Community Call For Funding Proposals

2025-08-28T15:18:00+00:00

The Jupyter Executive Council and Jupyter Foundation are pleased to announce a call for proposals for the Jupyter community. Visit the Jupyter Foundation Community Proposals webpage to learn more about the process and how to submit proposals. The call for proposals is open through Sunday, October 5th, 2025.

edit: We’ve decided to extend this by one week to give a few in-progress proposals time to write and submit! The new deadline is October 5th.

The Jupyter Foundation began its operations in early 2025 with a mission to use its resources to support the Jupyter community. We’ve set up the operations, structure, goals, and strategy of the Jupyter Foundation and published them on the Foundation team compass. We’ve had several rounds of feedback from the Jupyter community and defined the Foundation’s overall goals for its first year of operation.

Our first funding allocation supported events that grow the connections within the Jupyter community via the community workshops series as well as our connection with the broader data science community via JupyterCon 2025.

For the next wave of funding for the community, we’re running an experiment to directly solicit funding proposals from the Jupyter community. We think this will result in a strong clear connection between the Foundation’s funds and the needs of Jupyter’s maintainers, contributors, power users, and advocates.

The call for proposals is open through Sunday, October 5th, 2025.

Visit the Community Proposals webpage to learn how to submit a proposal and the Jupyter Foundation team compass for more detailed background about this funding call.

In this first round, one of our goals is to learn. We’d like to learn how to administer a community call like this as well as what kind of value it delivers to the Jupyter community. We’ve set minimum targets for funding disbursement and hope to grow our funding allotment as we iterate from here.

To help inspire proposals we are particularly excited to fund, we’ve defined funding priorities for the Jupyter Foundation. These are key outcomes that we must improve in order to grow the overall health and impact of the ecosystem. Briefly summarizing, these priorities are:

Grow the contributor capacity of Jupyter. We would like to grow the number of contributors, improve the efficiency of existing contributors, and facilitate learning across Jupyter subprojects. We’d love to see proposals that help us mobilize and support the total pool of energy available to contribute to Jupyter.
Improve the reliability, security, and consistency of Jupyter’s software and team practices. We believe that our technology will be more impactful and easier to contribute to and deploy if we improve our development infrastructure and team practices in ways that contribute to reliability, security, and consistency.

While we’re particularly excited about these outcomes and aim to focus funding on proposals that clearly feed into one or both of these priorities, we also encourage people to get creative and submit proposals for outcomes that have high impact and strategic value for the Jupyter community. We’re excited to see what others come up with, and are eager to work with you in unlocking critical funding in support of Jupyter’s community.

Acknowledgements

We are grateful for the support of the Jupyter Foundation and its members that makes this opportunity possible. If your organization would like to support this program, please contact the Jupyter Foundation.

JupyterGIS

2024-06-12T17:09:00+00:00

We are thrilled to announce that the European Space Agency (ESA) is funding our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS.”

The consortium spearheading this project comprises QuantStack and Simula Research Lab (Simula), two organizations with a long history of contributions to the Jupyter project and the broader open-source scientific computing ecosystem.

The goal of the project is to build solid foundations for a versatile web-based user interface for Geographic Information Systems (GIS) workflows. It will comprise several components, including a JupyterLab extension for collaboratively editing QGIS project files and the integration of these APIs in the Jupyter Notebook. We will then explore integration with platforms like the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC).

Collaborative workflows in geosciences

Collaborative editing of documents has become an integral part of our digital lives and has made us collectively more productive. Gone are the days of cumbersome email exchanges with documents shuttling back and forth.

Looking ahead, the potential of co-editing extends far beyond text documents and will apply to all authoring UIs, from CAD to image processing. We think that the shift to collaborative editing will be even more transformative for larger and more complex projects, which are inherently social and require the concerted effort of large teams. Whether you are designing a stadium, a plane, or an ocean liner, you need to coordinate a diverse expertise to build a unified model. In geosciences, it may range from climate modeling to agriculture, ecology, urban planning, and many more areas of expertise. For such endeavors, we must embrace tools favoring collaboration, and this applies to the future web-based user interfaces for geoscience research.

We have been working on collaborative editing in the core of JupyterLab for the past three years. Our approach is based on the Yjs framework, an implementation of CRDT data structures (Conflict-free Replicated Data Type). We have learned that since it is so tied to the data model, retrofitting these features into an existing application is considerably more arduous than building the initial data model on the appropriate paradigm from inception. This is why JupyterGIS will be built from the ground up with collaborative editing in mind.

JupyterGIS will be the first open-source GIS tool to provide collaborative editing features.

Importantly, JupyterGIS is part of a broader mission to enable collaborative workflows in open-source technical computing. The JupyterCAD project, a collaborative CAD modeler, is another example of this endeavor. By sharing knowledge and resources, both JupyterGIS and JupyterCAD will benefit from and contribute to each other’s progress, ultimately driving innovation and productivity in their respective fields.

From the desktop to the web

Advanced authoring tools, including IDEs, CAD modelers, image processing software, and GIS applications, are essential for professionals who rely on them for extended periods. These users have high expectations and demand key features to optimize their productivity and workflow. Among these features are extensibility with plugins, configurable keyboard shortcuts, themability, internationalization, scriptability, a unified settings system, and the ability to operate across multiple browser windows and devices.

Developing a new application from the ground up that meets all these requirements is a formidable challenge. This is where the JupyterLab application framework proves invaluable. By leveraging this framework, developers can build custom, feature-rich authoring tools that incorporate these essential features from the outset, making it an ideal solution for creating modern, user-centric applications.

Beyond the pure authoring user interface, for which JupyterLab is a great tool, it is also the perfect tool for integration with notebook-based workflows. As part of this project, we will develop an advanced Python API to manage JupyterGIS sessions, leveraging Jupyter’s robust display system to incorporate sophisticated GIS features inline in Jupyter notebooks and consoles. This integration will further enhance the capabilities and versatility of our application. This feature will follow the same architecture as that of JupyterCAD for the integration with the notebook user interface.

Real-world applications

JupyterGIS is meant to become a general-purpose tool. However, we will work with teams of practitioners on specific real-world applications to ensure that it addresses their needs.

One such project concerns the use of digital and GIS solutions for emergency management. While practitioners involved in emergency management already have experience with such tools, we are convinced that web-based applications built with collaboration in mind from the start can foster improved coordination and collaboration, and therefore the effectiveness of the response.

Response teams comprise different organizations, including the affected municipalities, police, emergency response organizations like the Red Cross, and advisers such as the U.S. Army Corps of Engineers (USACE) in the United States or the Norwegian Water Resources and Energy Directorate (NVE) in Norway. Even when using the same incident management system, miscommunications and misunderstandings can occur between these stakeholders, resulting in delays or improper use of resources. An improved integration of collaborative GIS software can improve upon the existing tooling.

Building collaborations in Europe and beyond

We will partner with key practitioners and make sure JupyterGIS addresses their use cases. Integration in the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC) will be key to the adoption of JupyterGIS and demonstrate its deployment in such environments.

Beyond Europe, we are currently working on including this project into a broader scope. We are excited to be partnering with the “LIVE-Env” project, an open-source initiative spearheaded by Alyssa Goodman at Harvard University. Furthermore, we are establishing a key collaboration with the Berkeley Institute for Data Sciences (BIDS) and the Schmidt Center for Data Science and Environment (DSE) at UC Berkeley and 2i2c, with a focus on Jupyter-based geosciences.

Affiliations

Sylvain Corlay: QuantStack
Anne Fouilloux: Simula
Monika Weissschnur: Simula

European Commission Funds Jupyter Bug Bounty Program

2023-07-28T19:46:00+00:00

Three Jupyter Subprojects – Jupyter Server, JupyterLab, and JupyterHub –are participating in a bug bounty program sponsored by the European Commission and hosted on the Intigriti platform.

Intigriti is a cybersecurity company that specializes in crowdsourced security services like bug bounty programs, hybrid penetration testing, and hosting live hacking events. The financial support covers bounties from €250 to €5,000 and is part of the European Commission’s Open Source Software Strategy 2020–2023, which:

Promotes the sharing and reuse of software solutions, knowledge and expertise, to deliver better European services that benefit society and lower costs to that society. The Commission commits to increasing its use of open source not only in practical areas such as IT, but also in areas where it can be strategic.

This is a great opportunity for Project Jupyter to reduce the number of potential vulnerabilities in critical Jupyter components and to evaluate our vulnerability handling processes.

Project Jupyter was selected for sponsorship at the end of 2022. After discussions involving several Jupyter Subprojects and other stakeholders, the three Jupyter software components were chosen for inclusion. Representatives from Jupyter Server, JupyterLab, and JupyterHub have committed to validating and responding to submissions for their area. Focusing on a limited number of key Jupyter components was a way to balance the benefits of identifying vulnerabilities against developer obligations to review vulnerability submissions. The Jupyter Security Subproject is contributing to this effort by drafting the bug bounty program details and being a liaison between Intigriti and the other Jupyter Subprojects.

Submissions are triaged by Intigriti, including reproducing potential vulnerabilities and assigning a severity. The triage process ensures that submissions fall within the scope (software components, versions, etc.) defined by the program. Vulnerability severity is based on Intigriti’s contextualized Common Vulnerability Scoring System (CVSS) and assigned a rating of low, medium, high, critical, or exceptional. The severity determines both the bounty and the response time for Project Jupyter to validate a submission. After a submission passes triage, it is sent to Project Jupyter for validation.

Project Jupyter appreciates the financial support from the European Commission and the help by Intigriti representatives to establish the program. Security researchers and others interested in the bug bounty program can view the details on the Intigriti website.

CZI awards three EOSS grants to Jupyter community members

2021-10-14T20:30:00+00:00

The Jupyter community is pleased to announce that members of its community have received three awards from the CZI Essential Open Source Software grant series. This grant series has previously funded work in the Jupyter ecosystem such as real-time collaboration in JupyterLab and JupyterHub’s Contributor in Residence pilot. Read below for a brief overview of the recently-funded proposals!

CZI’s Essential Open Source Software for Science program supports software maintenance, growth, development, and community engagement for critical open source tools.

Accessibility in Jupyter

One of the biggest challenges in building complex web-based applications is ensuring that they are accessible to a broad and diverse audience of users. Improving accessibility requires dedicated time and significant expertise in understanding the major challenges to overcome. Tania (@trallard), Isabela (@isabela-pf), and Tony (@tonyfast) were recently awarded an EOSS award to dedicate time and development towards making Jupyter tools more accessible. You can follow some of their work in the Accessibility repository and see the original text of this proposal here. This grant will be led by Tania, Isabela, and Tony (each at QuanSight Labs), and administered by the team at QuanSight Labs.

Interactive documentation

Because Jupyter is a dynamic and web-based interface, you can do a lot more with interactive documentation than is possible on traditional static websites. @carreau was awarded an EOSS grant to improve the state of interactive documentation within Jupyter. You can check out some of his early ideas in this blog post. This grant will be led by Matthias Bussonnier at QuanSight Labs, and administered by the team at QuanSight Labs.

Community strategic support for JupyterHub

The JupyterHub community built off of its Contributor in Residence pilot to propose a dedicated role for designing and developing community strategy for the JupyterHub sub-project. This role will spearhead efforts to ensure that there are inclusive and effective pathways into and throughout the JupyterHub community, with the goal of sharing best-practices with others in Jupyter and PyData ecosystems. You can follow along with this work in the JupyterHub Team Compass and find the original text of this proposal here. This effort will be led by Sarah Gibson(@sgibson91) and Chris Holdgraf (@choldgraf) (both at 2i2c), and Sarah will serve as our first community strategic lead. The grant will be administered by the team at NumFocus.

Congrats to the recipients of these awards, we’re excited to see what comes out of this work! And many thanks to all of the community members that helped write and refine these proposals.

The JupyterHub and Binder Contributor in Residence!

2019-11-26T19:52:00+00:00

The Chan Zuckerberg Initiative and Project Jupyter are teaming up to pilot a new role in the Jupyter community: the JupyterHub Contributor in Residence

Thanks to funding from the CZI Essential Open Source Software for Science initiative we are welcoming Georgiana Elena as our first Contributor in Residence!

Managing a codebase that spans many repositories and sub-projects as well as operating mybinder.org is a significant effort. Over time we have identified a need for dedicated support to ensure that activity and communication within these repositories is efficient and productive. Driven by this need and inspired by the Django Fellowship model, we shaped the idea of a JupyterHub/Binder Contributor in Residence (CIR).

Thanks to the Chan Zuckerberg Initiative, a Contributor in Residence within the community is now possible. Starting this year, CZI launched the Essential Open Source Software for Science grant program. It is aimed at helping open source communities that make up the core foundation of the scientific stack. We applaud CZI in providing support for core infrastructure across the sciences, and we’re honored to be part of CZI’s vision of funding maintenance, growth, development, and community engagement for open-source projects.

In this post we’ll introduce the community to our first Contributor in Residence (Georgiana) and her mentor (Tim)! The rest of this post is in the form of a conversation between Georgiana and Tim. We hope you enjoy it and that it let’s you get to know the team a bit better.

Tim: Hello Georgiana. Congrats on becoming the inaugural CIR. We’ve been lucky to have you already as a contributor on JupyterHub especially your work adding support for using Traefik as a proxy.
Geo: Hey Tim! Thank you, it’s been a really amazing experience to work on JupyterHub. I’m really excited for this next year’s journey.
Tim: You are the very first Contributor in Residence. Can you explain a little what this means?
Geo: The CIR role is about helping make the little things around the projects be amazing. Little things like answering newly opened issues, guide people towards the right place for discussion, improve bits of documentation or code, helping with releases, give people the right tools to make great contributions. And I have a great team of existing contributors to work with.
Tim: That’s right! My role in all this is to be a mentor and guide. JupyterHub and Binder have a small team that does a good job looking after the main repositories, but the universe of JupyterHub and Binder repositories is huge. So sometimes things get overlooked or missed. For example Pull Requests or Issues in one of the less-popular repositories can go for a long time without getting attention. This may hold up progress somewhere else, and people have a bad experience contributing to the project.
Geo: Hopefully we can improve on this a bit so that more people have a great experience when contributing.

Tim: Let’s find out a bit more about you. What did you do before getting involved with JupyterHub?
Geo: I studied computer science in Bucharest spending the last semester developing my bachelor’s project as an Erasmus exchange student in Madrid. During summers I worked in the industry, doing a few internships abroad. I tried to gain as much practical experience as possible, to get in touch with different people and cultures, and interning seemed like the right move.
Tim: How did you get involved with JupyterHub?
Geo: I got involved with JupyterHub thanks to the amazing Outreachy community. The Outreachy program introduced me to the world of opensource and guided me through my first steps. I now feel very lucky to have applied for the Outreachy internship at the same time that JupyterHub came up with two amazing projects. I chose JupyterHub because I “clicked” with the TraefikProxy project and I was happy to find a very welcoming community that made contributing a very pleasant and not at all a scary experience.
Tim: Have you turned into someone who spends all their time contributing or do you still have other hobbies?
Georgiana: Because I currently live in the busiest/loudest city of Romania, I love to escape it at the end of the week and spend some time in nature. My favorite place to be is the village I grew up in, where I recharge my batteries walking my dogs and cooking with my mother and sister. I like painting and photography and sometimes I do both using the photos I take as an inspiration for my paintings.

Tim: Moving on to the things you will be doing as CIR, have you already picked out something to get started with?
Geo: At first, I will be focusing on trying to get an overview of what issues are feature requests, support questions and which have been solved already. I will also spend some time getting familiar with more of the JH/Binder projects. And what better way to learn about a project than to try and use it, and to be a first time contributor.
Tim: If you had to name the top three goals you hope to achieve while being contributor in residence, what would those be?
Geo: From the perspective of someone with still a lot to learn and figure out, I would be extremely happy if during this year I will grow to be the CIR this community needs. I would love to help improve the contributing experience, increase the overall productivity of the projects and become a better contributor myself.

Tim: How can people interact with you?
Geo: People can use the usual community communication channels, GitHub, Gitter and Discourse to provide feedback or ask questions and I’ll do my best to help out.
Tim: Now that we have a CIR, do people still need to contribute to the projects?
Geo: Definitely! People are always welcome to contribute. The CIR role wasn’t created to replace anything, but to be an extra pair of hands helping the daily activities around the community become more efficient.
Tim: How can people stay engaged with the CIR program or the broader JupyterHub community moving forward?
Geo: As the CIR, I am a team member just like anyone else, and the best way to stay connected is to connect is through our team channels — we have team discussions in the Jupyter Community Forum, and team information in the JupyterHub Team Compass repository (including monthly team meetings that anyone is welcome to join).

Jupyter meets the Earth

2019-09-09T17:48:00+00:00

We are thrilled to announce that the NSF is funding our EarthCube proposal “Jupyter meets the Earth: Enabling discovery in geoscience through interactive computing at scale” (pdf). The team working on this project consists of Fernando Pérez [1, 2, 3], Joe Hamman [4], Laurel Larsen [5], Kevin Paul [6], Lindsey Heagy [1], Chris Holdgraf [1, 2] and Yuvi Panda [7]. Our project team includes members from the Jupyter and Pangeo communities, with representation across the geosciences including climate modeling, water resource applications, and geophysics. Three active research projects, one in each domain, will motivate developments in the Jupyter and Pangeo ecosystems. Each of these research applications demonstrates aspects of a research workflow which requires scalable, interactive computational tools.

In this project we intend to follow the patterns that have made Jupyter an effective and successful platform: we will drive the development of computational machinery by concrete use cases from our own experience and research needs, and then find the appropriate points for extension, abstraction, and generalization. We are motivated to advancing research of contemporary importance in geoscience, and are equally committed to producing work that leads to broad impact, general use infrastructure that benefits scientists, educators, industry, and the general community.

The adoption of open languages such as Python and the coalescence of communities of practice around open-source tools, is visible in nearly every domain of science. This is a fundamental shift in how science is conducted and shared. In recent years, there have been several high-profile examples in which open tools from the Python and Jupyter ecosystems played an integral role in the research, from data analysis to the dissemination of results. These include the first image of a black hole from the Event Horizon Telescope team and the detection of gravitational waves by the LIGO collaboration. The utility of open-source software in projects like these and the success of open communities such as Pangeo, provide evidence of the force-multiplying impact of investing in an ecosystem of open, community-driven tools. Through this project, we will advance this open paradigm in geoscience research, while strengthening and improving the infrastructure that supports it. We made this argument when discussing the intended impacts of our proposal, and we are pleased that the NSF is investing in this vision.

Geoscience use cases

Given our project’s aims and approach, participating actively in domain research is crucial to our success. The following descriptions are meant to offer a flavour of the research questions we are tackling, each led by a geoscientist in the team.

CMIP6 climate data analysis (Hamman). The World Climate Research Program’s Coupled Model Intercomparison Project is now in its sixth phase and is expected to provide the most comprehensive and robust projections of future climate predictions. When complete, the archive is expected to exceed 18 PB in size. In the coming years, this collection of climate model experiments will form the basis for fundamental research, climate adaptation studies, and policy initiatives. While the CMIP6 dataset is likely to hold new answers to many pressing climate questions, the sheer volume of data is likely to present significant challenges to researchers. Indeed, new tools for scalable data analysis, machine learning, and inference are required to make the most out of these data.

Large-Scale Hydrologic Modeling (Larsen). Streamflow forecasts are a valuable tool for flood mitigation and water management. Creating these forecasts requires that a variety of data types be brought together including model-generated streamflow estimates, sensor-based observations of water discharge, and hydrometeorological forcing factors, such as precipitation, temperature, relative humidity, and snow-water equivalent. The integration of simulated and observed data over disparate spatial and temporal scales creates new avenues for exploring data science techniques for effective water management.

Geophysical inversions (Heagy). Geophysical inversions construct models of the subsurface by combining simulations of the governing physics with optimization techniques. These models are critical tools for locating and managing natural resources, such as groundwater, or for assessing the risk from natural hazards, such as volcanoes. Today, we need models applicable to increasingly complex scenarios, such as the socially delicate task of developing groundwater management policies in water-limited regions. This will require the development of new techniques for combining multiple geophysical data sets in a joint inversion, as well as the use of statistical and data science methods for including geologic and hydrologic data in the construction of 3D models.

These scientific problems exhibit, each with its own flavour, similar technical challenges with respect to handling large volumes of data, performing expensive computations, and doing both of these as a part of the interactive, exploratory workflow that is necessary for scientific discovery.

Jupyter & Pangeo: empowering scientists

Jupyter and Pangeo are both open communities that share the goal of developing tools and practices for interactive scientific workflows; these tools aim to deliver practical value to scientists who, in the course of everyday research, face a combination of big data and large-scale computing needs.

Project Jupyter creates open-source tools and standards for interactive computing. These span the spectrum from low-level protocols for running code interactively up to the web-based JupyterLab interface that a researcher uses. Jupyter is agnostic of programming language: over 130 different Jupyter kernels exist, and they provide support for most programming languages in widespread use today. Jupyter can be run on a laptop, in an HPC center, or in the cloud. Shared-infrastructure deployments (e.g. HPC / cloud) are enabled by JupyterHub, a component in the Jupyter toolbox that supports the deployment and management of Jupyter sessions for multiple users. The development process of tools in the Jupyter ecosystem is community-oriented and includes a diverse set of stakeholders across research, education, and industry. The project has a strong tradition of building tools that are first designed to solve specific problems, and then generalized to other users and applications.

A Pangeo Platform is a modular composition of open, community-driven projects, tailored to the scientific needs of a specific scientific domain. In its simplest form, it is based on the following generic components: a browser-based user interface (Jupyter), a data model and analytics toolkit (Xarray), a parallel job distribution system (Dask), a resource management system (either Kubernetes or a job queuing system such as PBS), and a storage system (either cloud object store or traditional HPC file system). These are complemented by problem- and domain-specific libraries.

This modular design allows for individual components to be readily exchanged and the system to be applied in new use cases. Pangeo was created by, and for, geoscientists faced with large-scale data and computation challenges, but such problems are now common in science. Researchers in a variety of disciplines including neuroscience and astrophysics are working to adapt the Pangeo design pattern for their communities. Beyond the initial Pangeo deployments supported by the NSF EarthCube grant for Pangeo, the platform has been adopted internationally, including by the UK Met office. It has also supported new research and education efforts from other federal agencies, such as NASA’s HackWeek focused on the analysis of ICESat-2 data.

User-Centered Development

Pushing the boundaries of any toolset unveils areas for improvement and opportunities for developments that streamline and upgrade the user experience. We aim to take a holistic view of the scientific discovery process, from initial data acquisition through computational analysis to the dissemination of findings. Our development efforts will make technological improvements within the Jupyter and Pangeo ecosystems in order to reduce pain-points along the discovery lifecycle and to advance the infrastructure that serves scientists. Following established patterns in Jupyter’s development, we take a user-first, needs-driven approach and then generalize these ideas to work across related fields. Broadly, there are 4 areas along the research lifecycle where we will invest development efforts, which we discuss next.

Data discovery. An early step in the research process is locating and acquiring data of interest. Data catalogs provide a way to expose datasets to the community in a way that is structured. Within the geosciences, there are a number of emerging community standards for data catalogs (e.g. THREDDS, STAC). To streamline access to such data sets, we plan to develop JupyterLab extensions which provide a user-interface that exposes these catalogs to researchers. This work will build upon the JupyterLab Data Registry, which will provide a consistent set of standards for how data can be consumed and displayed by extensions in the Jupyter ecosystem, as well as Intake, a lightweight library for finding, loading, and sharing data which is already serving the Pangeo community. Our communities are already discussing potential avenues for integration between these tools.

Scientific discovery through interactive computing. The Jupyter Notebook has been adopted by many scientists because it supports an iterative, exploratory workflow combining code and narrative. Beyond code, text, and images, Jupyter supports the creation of Graphical User Interfaces (GUIs) with minimal programming effort on the part of the scientist. The Jupyter widgets framework lets scientists create a “Research GUI” that combines scientific code with interactive elements such as sliders, buttons and menus in just a single line of code, while still allowing for extensive customization and more complex interfaces when required. In this project, we will develop custom widgets tailored at the specific scientific needs of each of our driving use cases.

Beyond their utility in the exploratory phase of research, interactive interfaces, or “dashboards” provide a mechanism for delivering custom scientific displays to collaborators, stakeholders, and students for whom the details of the code may not be pertinent. Voilà is a project, led by the QuantStack team, that enables dashboards to be generated from Jupyter notebooks. We plan to develop interactive dashboards using Voilà for our geoscience use-cases, contribute generic improvements to the Voilà codebase, and provide a demonstration of how researchers can deploy dashboards to share their research.

Research GUIs to explore Maxwell’s equations in research and education. Photo credit: SEOGI KANG

Established tools and data visualization. Many widely-used tools, particularly for visualization (e.g. Ncview, Paraview), are desktop-based applications and therefore cannot easily be used in cloud or HPC workflows. In some cases, modern, open-source alternatives are available. But often for specialized tasks, modern tools may not yet have functionality equivalent to the desktop version. JupyterHub can readily serve non-Jupyter web-native software applications such as RStudio, Shiny applications, and Stencila to users; under this project we aim to extend JupyterHub to also be able to serve desktop-native applications.

Using and managing shared computational infrastructure. JupyterHub makes it possible to manage computing resources, user accounts, and provide access to computational environments online. Currently, JupyterHubs in the Pangeo project are deployed and maintained using the Zero to JupyterHub guide alongside the HubPloy library. Together, these libraries have simplified the initial setup and automated upgrades to the Hubs. There are still many improvements that can make JupyterHub more suitable for larger, more complex deployments, both in terms of managing users and efficiently allocating resources. Under this project, we plan to build tools which collect metrics such as CPU and memory usage and expose these to both users and administrators so they can make more efficient use of the Hub. For shared deployments, we will improve user management so that user-groups can be used to manage permissions and allocations, and so that usage can be tracked and appropriately billed to the relevant grants. Within the HubPloy library, we also plan to make improvements to streamline continuous deployments so that installation and upgrade processes are repeatable and reliable.

An opportunity for meaningful impact — join us!

The impacts of climate change and the need for data-driven management of resources are some of the most critical and complex challenges facing society today. In recent years, we have experienced severe droughts in California and are in the midst of water management crises in the Central Valley, while devastating wildfires have destroyed entire communities. Through this partnership between researchers and Jupyter developers, we hope to contribute to the advancement of science-based solutions to these challenges, both by contributing directly to the research and by improving the open ecosystem of tools available to researchers and the stakeholders impacted by these issues.

If developing open-source tech to advance research in geoscience and beyond excites you, then please get in touch! At UC Berkeley, we will be hiring in 2 positions: a dev-ops position focussed on JupyterHub and shared infrastructure deployments, and a JupyterLab-oriented role focussed on extensions, dashboards, and interactivity. There will also be a position opening up at NCAR for a software engineer targeting improvements to improving the user experience of Xarray and Dask workflows.

Even if you aren’t looking for a new job, there are other ways to get involved with both the Jupyter and Pangeo communities. We welcome new participants to the weekly Pangeo meetings (on Wednesdays alternating between 4p GMT and 8p GMT) and there are monthly Jupyter community calls, which are open and meant to be accessible to a wide audience. Outside of calls, general Jupyter conversations happen on the Jupyter discourse and Pangeo conversations are typically on the Pangeo GitHub.

In closing: a step toward sustainable open science

This project provides our team with $2 Million in funding over 3 years as a part of the NSF EarthCube program. It also represents the first time federal funding is being allocated for the development of core Jupyter infrastructure.

The open source ecosystem that Jupyter and Pangeo belong to has become part of the backbone that supports much of today’s computation in science, from astronomy and cosmology to microbiology and subatomic physics. Such broad usage represents a victory for this open and collaborative model of building scientific tools. Much of this success has come through the efforts of scientists and engineers who are committed to an open model of science, but who have had to work with little direct funding, minimal institutional support, and few viable career paths within science.

There is real strategic risk to continuing with the implicit assumption that scientific open-source tools can be developed and maintained “for free.” If open, community-driven tools are to sustainably grow into the computational backbone of science, we need to recognize this role and support those who create them as regular members of the scientific community (we recently talked about this in more detail in a talk at NSF headquarters). Projects like ours, where funding and resources are explicitly allocated toward this goal, are a step in the right direction. We hope that our experiences will contribute to ongoing conversations in the scientific community around these complex issues.

In the past, we have tried to maintain a close relationship between domain problems and software development in Jupyter. However, this has typically been done in an ad-hoc manner, either by “hiding” the software development under the cover of science or by having funding to Jupyter alone. This is the first project where we explicitly partner with a team of domain scientists to simultaneously drive forward domain research and the development of Jupyter infrastructure.

We are excited about this opportunity and hope to be able to demonstrate that investing in open tools can be a force-multiplier of resources. As always, our work will be done openly, transparently, and with constant community engagement. We look forward to your critiques, ideas, and contributions to make this effort as successful as possible.

Acknowledgments

Thanks to Joe Hamman, Chris Holdgraf, and Doug Oldenburg for constructive feedback and edits on this blog post.

Many thanks to Ryan Abernathey (Columbia), Paul Bedrosian (USGS), Sylvain Corlay (QuantStack), Rich Signell (USGS), and Rollin Thomas (NERSC), who provided us with letters of support for this project; we look forward to working with you all! We are also grateful to Shree Mishra, our NSF Program Director on this project, and to Dave Stuart for their support as we move forward with the project. This project is a part of the to the EarthCube program and we look forward to engaging with its working group.

Finally, the sustained growth of Jupyter to the large-scale project that it has become would not have happened without the generous support of the Alfred P. Sloan, the Gordon and Betty Moore, and the Helmsley Foundations, as well as the leadership of Josh Greenberg and Chris Mentzel respectively at Sloan and Moore.

This work is supported by the NSF EarthCube program under awards 1928406, 1928374

Affiliations

[1] UC Berkeley, Statistics Department

[2] UC Berkeley, Berkeley Institute for Data Science

[3] Lawrence Berkeley National Lab, Computational Research Division

[4] National Center for Atmospheric Research, Climate and Global Dynamics Laboratory

[5] UC Berkeley, Department of Geography

[6] National Center for Atmospheric Research, Computational Information Systems Laboratory

[7] UC Berkeley, Division of Data Sciences

Continuum Analytics donates $100k to Project Jupyter

2015-12-15T16:59:00+00:00

If you are a user of the Jupyter Notebook, you are probably familiar with Continuum Analytics. Continuum is well known in the data science space for their Anaconda Python Distribution, which eases the pain of installing the Jupyter Notebook and many other open-source Python and R packages.

Continuum is also an Institutional Partner and long time collaborator with Project Jupyter. Damian Avila (Jupyter Steering Council member) and a number of other Jupyter Contributors work with and for Continuum. Furthermore, the open-source project, phosphor.js, is being developed at Continuum under the leadership of Chris Colbert, in partnership with Bloomberg. Phosphor.js is being used as the foundation for the next-generation of the official Jupyter Notebook frontend, called JupyterLab.

Today we are pleased to announce a new aspect of Continuum’s support for Project Jupyter. Continuum has made a research gift of $100,000 to the project through Fernando Perez (UC Berkeley and the Lawrence Berkeley National Laboratory) and Brian Granger (Cal Poly). This funding will enable the project to strengthen its core activities and expand into new areas that existing funding doesn’t cover.

Fernando Perez, creator of IPython and co-founder of Project Jupyter, describes the significance of the funding in the following terms:

As the adoption and usage of the Jupyter Notebook has grown dramatically over the last few years, the project faces new challenges expanding in multiple directions at once. Continuum’s financial support helps us build the project’s foundations to ensure we can continue creating a valuable research, education and industrial data science platform for years to come.

We would like to thank Continuum for their contributions, and are looking forward to working with them in the future.

New funding for Jupyter

2015-07-08T03:34:00+00:00

We are pleased to announce that the Jupyter/IPython project has received $6M in funding from three organisations:

The grant, which is being made both to the University of California, Berkeley and California Polytechnic State University, San Luis Obispo, will support the project for three years and includes new collaborations with the University of Southampton, and the Simula Research Lab in Norway.

With these resources, our team will expand the reach of the Jupyter Notebook in research, education and industry, emphasizing collaborative data science, the creation of interactive dashboards, and tools for using the notebook in rich documentation workflows.

The complete text of our grant proposal can be found here: ( html | pdf ). In brief, these are some of the main areas on which we will focus during the next three years:

We aim to greatly improve the experience of collaboration and sharing with notebooks. Our team has already shown how to deploy Jupyter Hub in complex educational scenarios. Building on this, we will define a sharing and access model that makes sense when code execution is an integral part of the documents.
We also want to tackle the complex challenge of real-time, multi-user collaborative editing and execution of notebooks. This continues our ongoing partnersnip with researchers at Google.
We want to make components of the notebook readily reusable in completely different applications, so that various interfaces can quickly leverage our JavaScript infrastructure for interactive computation. Projects like Thebe from O’Reilly, and the Hydrogen editor plug in are early examples of such applications; we plan to make the architecture more flexible and easier to use.
We will develop new tools to improve the management of notebooks as documents. This includes better documentation of our own project as well as integration with technical documenation systems like Sphinx, electronic publishing formats like ePub, and stronger partnerships with the publishing community, from traditional journals to novel models. In particular, we want to leverage tmpnb and Thebe to allow any project to have live, executable examples in their documentation.
The community is a key component to our success, and we want to recognize that more. It is always thrilling to see other projects using our architecture; with more than 40 kernels now available, and a growing ecosystem of frontends and related tools, it is hard to keep track of all things Jupyter. This is why we will start organizing an annual Jupyter Conference, targeted at developers who want to build their own tools with our platform, and as a way for our team to better understand the needs of our community. We also have reserved resources for Jupyter Days, small, community-driven events that we can support where we hope to seed local knowledge and activity beyond our core team. We will communicate more on all these in the coming weeks and months.

Watch this space as we ramp up with our new resources. We will be making a number of announcements for job openings, including some great opportunities at Cal Poly that are already out, so apply if you are interested!

We would like to thank the generous support of these three foundations, and look forward to working with them over the next three years on advancing this effort.

Project Jupyter: Computational Narratives as the Engine of Collaborative Data Science

2015-07-08T01:45:00+00:00

Note: this is the full text of the grant proposal that was funded by the Helmsley Trust, the Gordon and Betty Moore Foundation and the Alfred P. Sloan Foundation on April 2015, as described on these two announcements from UC Berkeley and Cal Poly, and press releases from the Helmsley Trust and the Moore Foundation. A PDF version of this document can be found here.

Fernando Perez, Lawrence Berkeley National Lab & UC Berkeley

Brian E. Granger, Cal Poly San Luis Obispo

Statement of the problem and audience

Computers are good at consuming, producing and processing data. Humans, on the other hand, process the world through narratives. Thus, in order for data, and the computations that process and visualize that data, to be useful for humans, they must be embedded into a narrative — a computational narrative — that tells a story for a particular audience and context. There are three fundamental aspects of these computational narratives that frame the problem we seek to address. First, a single computational narrative needs to span a wide range of contexts and audiences. For example, a biomedical researcher might originally perform statistical analyses and visualizations for a highly technical paper to be published in an academic journal. Over time, however, that same individual will give talks to other researchers, or even non-technical audiences. Eventually, it may even be important to enable non-coding lab scientists to perform that same statistical analyses and visualizations on data from new samples using a simplified graphical user interface. Across all of these different audiences and contexts, core aspects of the computational narrative remain invariant. Second, these computational narratives need to be reproducible. That is, other people — including the same scientist six months later — need to be able to understand exactly what was done (code, data and narrative) and be able to reliably reproduce the work in order to build new ideas off it. Reproducibility has long been one of the foundations of the scientific method, but the rise of data science brings new challenges to scientific reproducibility, while simultaneously extending these questions to other domains like policy making, government or journalism. Third, computational narratives are created in collaboration. Multiple individuals need the ability to work together at the same time, on the code, data and narrative. Collaboration is present in nearly all contexts where computational narratives are created: between two postdocs and a professor in the same research group; between the writers, editors and visual designers of an online news site; between the data scientists and business strategists at a large internet company; or between a teacher and students in a university classroom.

Given this background, the core problem we are trying to solve is the collaborative creation of reproducible computational narratives that can be used across a wide range of audiences and contexts. We propose to accomplish this through Project Jupyter (formerly IPython), a set of open-source software tools for interactive and exploratory computing. These software projects support scientific computing and data science across a wide range of programming languages (Python, Julia, R, etc.) and already provide basic reproducibility and collaboration features. This grant aims at making major progress atop this foundation. The main application offered by Project Jupyter is the Jupyter Notebook, a web-based interactive computing platform that allows users to author computational narratives that combine live code, equations, narrative text, interactive user interfaces and other rich media. These documents provide a complete record of a computation that can be converted to a number of formats (HTML, PDF, etc.) and shared with others through email, Dropbox, GitHub, etc. They can also be published online thanks to our Jupyter Notebook Viewer, a free service we operate that allows anyone on the web to view a notebook as a regular web page.

In this section, we describe related work in two areas: interactive computing and online collaboration software. After that, we detail the various organizations with which we have significant collaborations. There are a number of interactive computing environments that have similarities to our work with Project Jupyter. The largest group of products, by number of users, are the traditional commercial interactive computing environments: Matlab^[1], Mathematica^[2], SAS^[3], SPSS^[4] and Microsoft Excel. While these products are extremely popular, their proprietary nature and expensive licensing fees make them unattractive for open and reproducible scientific research and data science. On the open source side, there are the popular Sage^[5] and RStudio^[6] projects and the newer Spyder IDE^[7], Beaker Notebook^[8], Zeppelin Project^[9], SageMathCloud^[10] and Wakari^[11](Wakari is a proprietary project based on open-source tools). A number of these projects (Sage, Spyder, Beaker, SageMathCloud and Wakari) rely on and provide integration with the Jupyter/IPython architecture. In the area of online collaboration software, there are two gold standards. First, Google Drive^[12] has, quite literally, invented modern online collaboration by offering a productive environment that allows multiple, distributed users to simultaneously edit documents, spreadsheets, and slide presentations. For many organizations, these real-time collaboration capabilities of Google Drive have transformed how distributed teams get work done together. Second, for code and data, git^[13] and GitHub^[14] have played a similar transformative role in distributed collaboration. The git project is an open source distributed version control system that programmers use to track and share changes in complex software. GitHub is a commercial (but free for public usage) collaboration platform built around git that has become invaluable for companies, open source projects and scientists alike. SageMathCloud and Wakari expose the Jupyter Notebook online and provide some collaboration features. While there are other online code and document collaboration platforms (Bitbucket, Office 365, Hackpad, Etherpad, etc.) all of these are largely inspired by Google Drive and git/GitHub.

Over the past few years, we have spent significant amounts of time and effort investing in relationships with other individuals and organizations that have overlapping missions, impact areas, user groups and technologies as Project Jupyter. In the area of academic research and education, we have ongoing collaborations with individuals and departments at Stanford, UW, NYU, MIT, Harvard, Bryn Mawr, U. Southampton, U. Sheffield and Simula Research Lab (Norway). In the area of open science, we coordinate efforts with the Center for Open Science (Brian Nosek and Jeff Spies) and Software Carpentry (Greg Wilson). In traditional journalism, we have relationships with staff at 538, BuzzFeed and the New York Times focused around data-driven journalism. In open source software, we collaborate closely with the core developers of all the major scientific computing and data science projects in Python (NumPy, SciPy, Pandas, Matplotlib, Scikit-Learn, etc.), Julia (core developers) and R (rOpenSci).

We also work closely with a number of companies that are building products based on the Jupyter Notebook, contribute code and financial resources to the project and serve as advisors on a wide range of technical and strategic topics. Because these collaborations are so important for the ongoing sustainability of Project Jupyter, we wish to highlight a few of these.

GitHub is an online collaboration and hosting site for software projects and code. For us, GitHub is significant because many users of the Jupyter Notebook host and share their computational narratives, as Jupyter Notebook documents, on GitHub. Our own Notebook Viewer service renders notebooks stored on GitHub as static HTML pages, which can be shared with anyone in the world without their installing anything. We are currently working with Arfon Smith and Tim Clem of GitHub to explore other integration points between GitHub and Project Jupyter.

Rackspace is a commercial cloud hosting company that supports a wide range of open source projects, including Project Jupyter. For the last year, Rackspace has provided significant hosting resources for our popular Notebook Viewer service. This includes one of Rackspace’s engineers, Kyle Kelley, building and maintaining this deployment and making significant contributions to the Jupyter codebase. In the summer of 2014, Kyle, with Rackspace’s explicit support, began a much more ambitious effort to offer cloud-hosted Jupyter Notebooks where users could instantly try a live Jupyter Notebook to run Python, R and Julia code. Thanks to this work, we were able to embed a live demo of the Jupyter Notebook in an article about the project that was published in November of 2014 in Nature^[15].

Microsoft has been collaborating with the PIs of this grant since 2008, when Dr. Brian Granger added support for Microsoft’s job scheduler to IPython’s parallel computing framework. Since then, Microsoft has added IPython integration to the popular Python Tools for Visual Studio, demonstrated the Jupyter Notebook running in the Microsoft cloud (Azure) and donated $100,000 to the project through NumFOCUS. We are currently working closely with Microsoft to identify future areas of collaboration.

Bloomberg is one of the largest financial data and news companies in the world. Two individuals at Bloomberg (Jason Grout and Sylvain Corlay) have been approved to contribute to Project Jupyter in an official capacity. Both are regular contributors and have been critical in the design of the project’s interactive widget architecture. This collaboration has led to our working with Chris Colbert and others at Bloomberg to begin designing the next generation of web-based user interfaces. We will use the open-source phosphor.js JavaScript library developed at Bloomberg, needed for this grant’s deliverables. Bloomberg also has an official open source program and policies, including hosting of “Open Source Days” at their headquarters in NYC and London.

Google approached us in 2014 regarding a prototype of a project, called coLaboratory, that integrates the Jupyter Notebook into Google Drive. Most importantly, this prototype demonstrated that building real-time collaboration into the Jupyter Notebook would be possible. However, this prototype also revealed the incredible technical challenges of doing so. In late 2014, Google donated $100,000 to Fernando Perez at UC Berkeley to hire a postdoc (Matthias Bussonnier) that would begin to extract the Google Drive integration from coLaboratory into Project Jupyter itself. We want to emphasize two aspects of this collaboration. First, the technical challenges in building real-time collaboration into the Jupyter Notebook are so significant that we could not do this without close collaboration with Google. Second, the effort required to implement this in Project Jupyter requires resources that extend far beyond those provided to us by Google.

O’Reilly Media is a major publisher of technology focused books and online content as well as the organizer of the most significant conferences in the data science and open source spaces (Strata, Hadoop World, OSCON, etc.). In 2014, we began working with the CTO of O’Reilly, Andrew Odewahn, to explore ways of integrating the Jupyter architecture into their publishing platform, to enable both authors and readers of O’Reilly content to experience books as live, computational entities. O’Reilly already has multiple books that include code examples as Jupyter Notebooks.

Organizational background

Mission and Background

Project Jupyter’s mission is to create open source tools for interactive scientific computing and data science in research, education and industry, with an emphasis on usability, collaboration and reproducibility.

Today’s Jupyter evolved from the IPython project, created in 2001 as an interactive Python shell by Dr. Fernando Perez. Dr. Brian Granger joined the IPython project in 2004, and the two of us have led the project since then. The core development team has grown to roughly a dozen active contributors and a “long tail” of community contributors currently numbering over 400, who participate with various degrees of regularity.

For the first decade, IPython focused strictly on scientific and interactive computing in the Python language, providing a rich interactive shell well suited to the workflow of everyday research, as well as tools for parallel computing. It was part of an organic ecosystem of open-source projects for scientific computing in Python, informally known as the “SciPy Stack”.

Around 2010, IPython evolved from providing only a terminal-based interactive shell into a generic architecture for interactive computing and computational narratives in any programming language. This design allowed us to build the web-based Notebook described in this proposal. This expansion beyond Python led to a renaming of all the non-Python specific parts to Project Jupyter. Today, this architecture supports over 20 different programming languages, with most implementations having been created by third-parties.

Core problems and constituency

While scientists have always used computers as a research tool, they use them differently than industrial software engineers: in science, the computer is a kind of “abstract microscope” that enables the scientist to peek into data and models that represent or summarize the real world. Software engineers tend to write programs to solve reasonably well-defined and independently specified problems, and their deliverable is a software artifact: a standalone application, library or system. While standalone software libraries exist in science (say the building of a library to solve differential equations), we target a more common scenario: the iterative exploration of a problem via computation and the interactive study of intermediate results. In this kind of computational work, scientists evolve their codes iteratively, executing small test programs or fragments and using the results of each iteration as insight that informs the next step. The computations inform their understanding of their scientific questions, and those questions shape the process of computing. The nature of this process means that, for scientists, an interactive computing system is of paramount importance: the ability to execute very small fragments of code (possibly a single line) and immediately see the results is at the heart of their workflow.

Furthermore, the purpose of computation in science is precisely to advance science itself. In the famous words of R. Hamming, “the purpose of computing is insight, not numbers.” For this reason, computation in science is ultimately in service of a result that needs to be woven into the bigger narrative of the questions under study: that result will be part of a paper, will support or contest a theory, will advance our understanding of a domain. And those insights are communicated in papers, books and lectures: narratives of various formats.

The problem the Jupyter project tackles is precisely this intersection: creating tools to support in the best possible ways the computational workflow of scientific inquiry, and providing the environment to create the proper narrative around that central act of computation. We refer to this as Literate Computing, in contrast to Knuth’s concept of Literate Programming, where the emphasis is on narrating algorithms and programs. In a Literate Computing environment, the author weaves human language with live code and the results of the code, and it is the combination of all that produces a computational narrative.

We consider this problem while acknowledging that science is, by definition, an open, collaborative enterprise founded on the principle of independent validation of all knowledge. This means that supporting collaboration and reproducibility are central guiding principles of the project.

Finally, while all the above has been cast in the context of scientific research, the rise of ubiquitous data science means that these same questions are now not only the purview of physicists or biologists. Today policy makers, journalists, business analysts, financial model builders, all work with the same tools and challenges: their data may come from a population census or the stock market, and instead of an academic paper they may be writing a blog post or a sales report for a client, but ultimately the process is similar. They need to extract insight from data using computational tools, and they need to communicate that insight to an audience in the form of a narrative that resonates with that audience.

So today, Project Jupyter serves not only the academic and scientific communities, but also a much broader constituency of data scientists in research, education, industry and journalism. Given the importance of computing across modern society, we see uses of our tools that range from high school education in programming to the nation’s supercomputing facilities and the leaders of the tech industry mentioned above.

Basically, anyone who needs to execute code an interactive programming environment can be legitimately considered as served by our project. As computation and data analysis become pervasively woven into the fabric of society, our constituency continues to broaden. The challenge for our organization is to maintain a focused research agenda where we provide a coherent vision of the future in interactive computation, a clean set of abstractions and tools, and a sustainable community model. These things, combined, should serve as the foundation on which others can then build the solutions they need in their specific contexts. The purpose of this proposal is to advance the state of the art in those core questions.

Project organization

Project Jupyter is organized around an open-source model that allows for individual Contributors to join the effort based on their personal interest, resources and availability. Along side this open community of Contributors, the project has a thin layer of formal organizational structure and governance. A summary of that structure and governance follows^[16]:

A large, public, open and inclusive community of Contributors participate in the creation of Project Jupyter’s software. Some contribute code, others documentation, ideas or bug fixes. Nearly all technical decisions are made through the informal consensus of this open community.
Through a record of sustained activity, Contributors can be nominated to have more rights and responsibilities in the development of specific parts of the project. This is done by providing them with write privileges (known as “commit rights”) in the code repositories of the organization, hosted on GitHub. We currently have roughly 20 people in this capacity.
Contributors who demonstrate significant leadership and contributions for longer than 1 year can be nominated to the Jupyter Steering Committee. The Steering Committee is the formal governing body for the project and is ultimately responsible its the technical, strategic and communal health. We emphasize, however, that the Steering Committee delegates essentially all technical decisions to the open community.
From a legal perspective, Project Jupyter is part of the NumFOCUS Foundation, a 501(c)3 organization dedicated to support research, development and education in open source scientific computing and data science. NumFOCUS provides legal structure and fiscal sponsorship for small amounts of community-focused project funds.
The project also has Institutional Partners: companies, universities and other legal entities who have at least one employee on the Steering Committee. By raising money (donations, grants, for-profit business models) and employing project Contributors and Steering Committee members these Institutional Partners are the main source of financial support for the project.
We emphasize, however, that Contributor or Steering Committee status is always based on the technical participation of individuals, rather than the financial weight of the Institutional Partners; it is impossible to “buy your way” onto the Steering Committee. The current Institutional Partners are UC Berkeley, Cal Poly, Rackspace and Continuum Analytics.
The PIs on this grant are project Contributors, Steering Council members and employees of the UC Berkeley (Fernando Perez) and Cal Poly (Brian Granger) Institutional Partners. Through their seniority and long time (14 and 10 years respectively) commitment, leadership and contributions, they effectively lead the Steering Council and project.
Project Jupyter deliberately has no full time employees through NumFOCUS; all full time staff positions are handled through Institutional Partners. UC Berkeley currently has 1 full time software engineer and two postdocs (in addition to F. Perez). Cal Poly currently has one full time software engineer, who is currently paid as an independent contractor through funds from Microsoft/NumFOCUS (in addition to B. Granger). Other Steering Council members are employed by Rackspace (Kyle Kelley) and Continuum Analytics (Damian Avila).

Project activities and highlights

The main project activities, supported by a combination of open source volunteers, funded researchers and industry partners, are:

The creation of open source software, hosted on the Github site under the ipython and jupyter organizations. We currently host 46 public repositories.
Hosting online services powered by our software, currently supported by Rackspace:
- The Notebook Viewer (http://nbviewer.ipython.org): renders the URL of any notebook as a static web page, enabling effortless sharing of notebooks. This service gets currently ~ 800,000 page views per month, from ~ 200,000 visitors.
- TryJupyter/tmpnb (http://try.jupyter.org): an ephemeral, anonymous live Jupyter Notebook. This lets anyone log into a Notebook server and experiment with the provided example notebooks or type their own code.
- A live demo for the Nature Journal: since November 2014, as a companion for an article about IPython published by Nature, we have hosted an instance of the ephemeral notebook service that lives in the Nature.com domain. This has served over 20,000 live sessions and broke readership records for simultaneous users on the Nature site.
A public chat room (https://gitter.im/ipython/ipython/help) where our project developers help members of the public with general questions.
A public mailing list where more long-form discussions take place.
Weekly development meetings publicly broadcast via Google+ Hangouts and archived on YouTube. This improves our community engagement and the transparency of our process. Multiple other open source projects have adopted this model since we introduced it in 2013.

Estimated user base. It is very hard to get accurate user counts for an open source project that can be downloaded freely from multiple sources. But we estimate at least 2 million users for IPython. This is a rough number, but if anything, a conservative undercount. We justify this number as follows:

Estimates of Linux users (not datacenter servers) range from 20M for Ubuntu to ~70M across Linux Distributions^[17].
The Debian Linux distribution tracks package installations with the ‘popcon’ tool. This shows IPython to be regularly installed in ~ 5% of Debian systems^[18].
If we use Debian as a baseline, and estimate total Linux user counts at ~50M (rough average of the above two numbers), we get about 2.5M installs of IPython on Linux.
This doesn’t count many other sources IPython can be installed from, such as Github, Python’s package repository, the Continuum Anaconda distribution, Enthought Canopy, etc. Nor does it count the increasing number of server-side hosted deployments we see more and more of.

Other highlights. A few other relevant achievements of the project over the last few years:

Very large user base.
Three books have been published devoted to IPython ^[19] ^[20] ^[21], and several more cover IPython in detail (one or more chapters). Two books have also been written either entirely as Notebooks or containing extensive Notebook collections^[22] ^[23], and we know several more are being written.
Courses at top universities in the US and abroad use Jupyter Notebooks as core educational technology. We know of over a dozen at UC Berkeley, Cal Poly, U. Santa Clara, Harvard, Columbia, U. Claude Bernard Lyon (France), and more.
There have been 18 academic (peer-reviewed or preprints) articles ^[24] that provide IPython Notebooks to support reproducibility.
There are independent implementations of the Jupyter protocol that provide kernels in over 25 different programming languages.
Google Research created and released the CoLaboratory system for integration of Notebooks with Google Drive as an app in the Chrome web store. This effort led to a funded collaboration with our team.
Professors Lorena Barba (George Washington U), Ian Hawke (U. Southampton) and Carlos Jerez (U. Pontificia Católica de Chile) taught in 2014 a MOOC on numerical computing whose teaching materials consist entirely of IPython Notebooks^[25].

There are many more teaching materials, conference talks, blog posts and projects using our architecture and tools, that we can not fit in this space. We refer the reader to the Notebook Gallery and list of Projects using IPython in our wiki for a comprehensive list (https://github.com/ipython/ipython/wiki).

Approach and strategy

This project is structured in a “3+1” format, with three main focus areas of research and development and one extra topic of ongoing work. The three focus areas are Interactive Computing, Computational Narratives and Collaboration. The problem of Sustainability will require ongoing attention but is conceptually distinct from the first three, as it doesn’t focus on specific research questions or deliverables.

Interactive Computing

At the heart of the entire Jupyter architecture lies the idea of interactive computing: humans executing small pieces of code in various programming languages, and immediately seeing the results of their computation. Interactive computing is central to data science because scientific problems benefit from an exploratory process where the results of each computation inform the next step and guide the formation of insights about the problem at hand. In this Interactive Computing focus area, we will create new tools and abstractions that improve the reproducibility of interactive computations and widen their usage in different contexts and audiences.

Notebooks as interactive applications

The Jupyter Notebook has an architecture for interactive widgets that bind data and code in the backend (Python, Julia, R, etc.) to interactive user interfaces running in the browser. This enables users to quickly explore code and data by manipulating user interface controls (buttons, sliders, etc.). This architecture is being used to create custom user interfaces in the notebook using powerful JavaScript libraries such as d3.js. These widgets open the door for non-technical “consumers” of notebooks to explore data in a notebook, without coding. We will extend the widget architecture into a full-blown application framework.

First, we will enhance our widgets to handle more complex hierarchies of objects with events, parent/child relationships, etc. Second, we will enable these widgets to be deployed in contexts outside the traditional notebook, where a live computational kernel is not running, such as nbviewer or a static HTML page. Third, we will create a system that allows users to bundle, share and deploy sets of widgets as independent “apps.” This will allow users and developers to leverage the notebook for highly customized, but still data and code driven, user interfaces that can be used with non-technical audiences.

Modular, reusable UI/UX

The Notebook user interface (UI) is the main way that users interface with our project on a daily basis. In order to create a more humane and productive environment for users, we will invest significant effort on improving the UI and user experience (UX).

First, we will create a more modular set of UI components to enable users and third party developers to build purpose specific UIs with custom components, such as file browsers, debuggers, variable inspectors, documentation panes, etc. This work will be done in collaboration with Bloomberg (see above for the details of this collaboration). Second, by hiring a full time UI/UX person, we will begin to perform studies of the usability of our various UIs. This will include qualitative user testing and quantitative A/B testing. Third, we will add a set of richer actions that can be performed on single or multiple notebook documents. These include multicell operations (cut/copy/paste), structural operations that allow different sections and subsections to be collapsed/expanded and moved atomically, and an improved dashboard for working with directories of files and notebooks.

Software engineering with notebooks

The Jupyter Notebook emerged out of a need to capture otherwise transient interactive computations in a form that could be reproduced and shared with others. However, as single notebooks grow in size and complexity, they become difficult to work with from the software engineering perspective (testing, documentation, reuse, modularity, etc.). In this grant we will improve the software engineering aspects of notebook based workflows.

First, we will improve the ability of users to transition from a single large notebook, to a smaller notebook that calls code contained in external modules that can be tested and documented separately. Second, we will work to enable notebooks themselves to be treated as importable Python packages. This will enable code in one notebook to be used in other notebooks or projects more easily. Third, we will build tools that allow notebooks to be tested in similar ways as traditional software projects are. Fourth, we will build tools that can verify that a notebook is reproducible; that it gives the same results when run again. This verification will be performed by rerunning the original notebook, comparing the output of the rerun notebook with that of the original, and then creating a human readable “reproducibility report” that summarizes the differences, if any. This will also enable us to develop quantitative metrics that measure the degree of reproducibility.

Capstone

As a capstone to the Interactive Computing focus area, we will create a prototype of a spreadsheet app/widget that is integrated with the notebook architecture. This prototype will validate our improvements to widgets, our UI/UX modularization work and our work towards “application” oriented usages of our architecture.

Computational Narratives

Jupyter Notebooks enable users to create and share code and data driven narratives. These narratives end up being used in a wide variety of contexts: academic publications, blog posts, books, traditional journalism articles, technical documentation, government reports, grant applications, industry research and commercial products. Furthermore, a single notebook could be communicated in a number of different formats (PDF, live demo, web page, slideshow) to different audiences. Today, users encounter significant pain along this path. In this Computational Narratives focus area, we will build tools to enable the notebook to be used more easily across different contexts, formats and audiences.

nbconvert

nbconvert is the command line tool that Jupyter offers for converting notebooks to different formats (currently LaTeX/PDF, HTML, Markdown and reveal.js). We will target the following areas of work on nbconvert in this grant.

First, we will create an EPUB exporter for nbconvert. Because EPUB is based on HTML/CSS/JavaScript, it is a much better format than LaTeX/PDF for representing the rich content found in notebooks. Most importantly, EPUB is an ideal format for interfacing with publishers in the academic and technical spaces. EPUB support will help our ongoing collaborations with O’Reilly Media and Nature. Second, the nbconvert software needs significant refactoring and improvements to its command line and programming APIs to enable users and developers to customize it more easily. This includes improved documentation and examples. Third, we will explore the feasibility of exporting notebooks to Microsoft Word. This will be done through our ongoing collaboration with the data, development and machine learning teams at Microsoft, which is described above.

Element filtering

Notebooks contain code (source code and output) and markdown (narrative) cells. When a notebook is used across different contexts and audiences, it is useful to filter what types of content is visible. For example, when a notebook is read by non-technical users, it is helpful to hide all of the source code, but show the narrative text, visualizations and widgets. To address these needs, we will create a system that allows the notebook content to be selectively filtered, based on the intended context and audience.

First, we will enable cells to be tagged with user selected labels (“homework”, “testing”, etc.). Users will add labels to cells using an appropriate user interface and the labels will be stored in the notebook metadata. Second, we will create an interactive query syntax that allows content to be selectively hidden and shown based on the cell type, cell labels, widgets, input/output, etc. This query syntax will be integrated into the live notebook, nbconvert and nbviewer.

Documentation

Documentation is one of the primary ways that users interact with open source software. Existing documentation for Jupyter and other open source projects is written as static web pages generated by markup languages such as Markdown. Developers are forced to manually copy and paste code samples into this format. This leads to documentation that easily falls out of dates and cannot be tested. For users, these static web pages are not integrated with the notebook, can’t be searched and most importantly, can’t be run as live code. Furthermore, all of the documentation for each open source project is hosted on different places on the web.

To address these issues, we will create a notebook based documentation system for our own and other open source software projects. This documentation system will allow developers to write documentation as notebooks and package them with their own project as live code. Upon installation of these packages, users will be able to browse and search all of the documentation from within the notebook. Most importantly, users will be able to run the documentation as live code. To test this approach, we will write our own documentation using this system.

Capstone

As a capstone to this Computational Narratives focus, we will test our deliverables in the context of collaborations with publishers, both traditional and web-oriented. These publishers include O’Reilly, Nature, GitHub, BuzzFeed and 538.

Collaboration

Since the Jupyter Notebook was released in 2011, better support for collaborative workflows has been our users’ most common request. This is for good reason. In our modern, web-enabled companies, universities, research labs and non-profits, data science and scientific computing are carried out by distributed teams whose work and contributions are tightly coupled. For static content, this is enabled by technologies such as email, video chat, online comment/review systems, GitHub and Google Docs. Today, the Jupyter notebook has almost no support or these types for synchronous and asynchronous collaborations, which limits the impact and usefulness of the notebook in collaboration rich contexts such as education and scientific research.

Real time collaboration

In this grant, we will add real-time collaboration capabilities to the notebook that are modelled on the abstractions and architecture of Google Drive/Docs. This will allow multiple users to share notebooks with each other online, and edit those notebooks together in real time. To this collaborative editing system we will add user presence, commenting and cloud based document storage.

As described above, this work is extremely technical and will require major rewrites of significant portions of our architecture. Furthermore, there are significant security issues to work through. Because of the difficulty and scope of this work, we are working directly with Google Research to help us design the underlying architectures and implement them in our software (see above for the details of this collaboration). The initial implementation of these features will rely on open Google APIs (Drive API, Real Time API), however, we plan on building abstractions and APIs that will allow us to plug into a number of different collaborative backends (Firebase, etc.).

JupyterHub

The basic Jupyter Notebook is a single user web application that most users install and run on their own laptop or desktop. JupyterHub is a multiuser version of the notebook server than can be run on a central server(s) or in the cloud. JupyterHub eases the installation and deployment of the notebook to large numbers of users and opens the door for novel collaboration possibilities, However, the version of JupyterHub that exists today has very limited sharing capabilities. In this grant we will improve the collaboration capabilities of JupyterHub in the following ways.

First, we will define richer and finer grained sharing semantics that allow users to share notebooks with other individuals or groups. This will include user interfaces that make this easy to do across different storage and deployment backends. Second, we will work to create tools that ease the deployment of JupyterHub in different contexts. Third, we will allow users of JupyterHub to “publish” notebook to other users of JupyterHub or the public.

Capstone

The individual tasks in this Collaboration focus area involve creating new, but semi-separate, collaboration capabilities for the different sub-projects (Jupyter Notebook, JupyterHub). As a capstone, we will begin to integrate these different collaboration approaches to create an integrated, ubiquitous system for notebook based collaboration.

Sustainability

As the scale of Jupyter’s usage and development expands, it is important to create for us to create a sustainable technical project, community and organization. This focus area is conceptually different from the above described 3 main technical areas and will involve an ongoing set of activities throughout the project period.

People

People are the backbone of our sustainability plan. While the project has had over 400 contributors, most of the major work has been done by a few key individuals. Thus, our first goal is to expand the set of these key contributors.

First, we will set up a robust training program that leverages senior project staff to manage and train new undergraduates, graduate students and postdocs to work on the project at Cal Poly and UC Berkeley. New data science programs at both of these universities will be used as the needed source of initial human capital for these efforts. We have an excellent track record of training students; two of our most senior Contributors and Steering Council members were previously undergraduate students of Brian Granger.

Second, we will send these newly trained individuals out into academia and industry where they can expand our network of collaborators, contributors and Institutional Partners even further. We know of multiple companies that are currently interested in hiring project newly trained Contributors and Steering Council members. To initiate this “sending out” this grant will fund two of our current Steering Council members (Min Ragan-Kelley and Thomas Kluyver) to move from UC Berkeley to institutions in Norway and the UK. We plan on working with them to build their own, independent, European based funding sources in the future. Their move will also create two new Institutional Partners: Simula Research Lab and the University of Southampton or Sheffield (which of these two will join is being determined, but one of them will).

Finally, a key concern of the project, inscribed in a larger societal discussion of the problem, is improving the diversity of our community. We address it in detail in the required Appendix.

Events

Jupyter related events will enable us to build a more sustainable community of users, developers and collaborators.

First, we will continue to have week long developer meetings twice per year. These meetings bring together 5–15 core developers and designers to review the project’s progress, discuss major technical and architectural issues and plan the future roadmap of the project. Because our core developers are geographically distributed, these in person meetings are critically important for us to build a cohesive developer community and project.

Second, for the first time, we plan on organizing JupyterCon, an annual conference to bring together all of Jupyter’s developers, users, collaborators and Institutional Partners. This conference will be a 2–3 day event in the Bay Area or New York that has time for talks, coding sprints, brainstorming, etc. As the project grows in size, JupyterCon will be an important way for us to bring our community together in a focused event. JupyterCon will also enable Institutional Partners to provide input to the project; eventually we forsee Insitutional Partners as becoming an advisory board for the project that works alongside the Steering Council. This centralized conference will also be used to seed other, smaller community organized outreach event in cities throughout the world (JupyterDays).

Third, we will continue to disseminate the results of our work to an ever wider range of communities in academia and industry. This will include talks at academic and industry focused conferences and workshops and the publication of articles about our work in academic journals.

Year by year output

Our approach to building software

We want to clarify our approach to building software and describe how that relates to the deliverables of this grant. The approach described here has emerged from our own experience in building open source software over the last 14 years as well as a careful study and application of the methods described by Eric Ries in his book, the The Lean Startup, as well as the books and courses of Steve Blank.

First, for each deliverable, we always begin by creating an Minimum Viable Product (MVP). The MVP is an initial implementation of that deliverable that provides the absolute minimal set of features we hypothesize will be useful to our users. The MVP always has a very limited scope and lacks features present in the final version.

Second, we immediately release the MVP to our users and begin watching how they respond. Our goal in this phase is to collect as much information as possible to validate our hypotheses about the deliverable. During this stage, we also identify the individuals and organizations who are stakeholders and collaborators in building that particular deliverable.

Third, we then incorporate the information gathered through validation to build the final version of the deliverable that has the exact set of features required by users. The final version typically has much larger scope than the MVP and is built with the collaborators identified in the validation stage.

Given this background, we expect the funding provided in this grant will be sufficient for us to complete the MVP and validation stages of each deliverable. For some of the smaller deliverables, we also expect to start building the final versions. However, we expect the final versions of all deliverables to be out of scope of this grant for two reasons. First, the validation stage is completely unpredictable. Features are used in unexpected ways, new groups of users emerge, other developers extend and reuse our work in innovative ways, and new collaborators and stakeholders emerge. Second, the validation phase typically expands the scope of the deliverable far beyond our original plans and budget.

However, we want to emphasize that the validated MVPs produced through this grant’s activities will be highly functional and have a deep impact on our users. To set the scale appropriately, we consider most of our current software, including the Jupyter Notebook, to be at the validated MVP stage.

Year-by-year plan of deliverables

The following table details our year-by-year plan of deliverables in the core focus areas of Interactive Computing, Computational Narratives and Collaboration. The numbers in the table represent the number of full time staff technical staff working on that deliverable at UC Berkeley, Cal Poly, Simula Research Lab and the University of Southampton. This table only includes our eight software engineers and postdocs, as well as ½ FTE on Y1 for a technical writing consultant. We expect the UI/UX designer, Project Manager and 2 PIs to work across all deliverables each year.

Deliverable	Year 1	Year 2	Year 3
NB as Apps ⟂	0.5	1	1
UI/UX ⟂	1.5	2	2
NB Software Eng. ∥	1
nbconvert ∥		1	1
Element Filtering ⟂	1	0.5	0.5
Documentation ∥	1.5	1
Real Time Collab ⟂	2	2.5	2.5
JupyterHub ⟂	1		1

Measuring effectiveness

We will measure the effectiveness of our work through the following metrics. All target numbers are three year totals unless otherwise specified.

Number of talks given at conferences and workshops we have never attended before (target=10).
Ethnic and gender diversity of active Jupyter developers in our broader community (target=3 from currently underrepresented groups. This would be a very significant increase, given our core development team has roughly a dozen regularly active members).
Traffic on project web site and web services (nbviewer and tmpnb) (target=4x current traffic).
Number of deployments of JupyterHub by 3rd parties in research, education, industry (target=direct knowledge of 25).
The number of universities, companies and research labs that adopt the Jupyter Notebook and related technologies at an institutional level (target=6).
Institutions and projects that use our building blocks as infrastructure to create other software and products (target=12).
The number of books, academic publications, education course materials, journalism articles that use the notebook as a primary or secondary mechanism to deliver content (target=100).
The number of new collaborations with large companies in the data science space (target=6).
The number of companies providing funding for the project (target=12 companies).
The number of new Institutional Partners of the project (target=4)
The number of undergraduate, graduate and postdoc students that are trained through the grant activities and placed in related jobs (target=12).
Number of deliverables for which the MVP and validation stages are completed (target=all).
The MVP and validation stages increase the impact, scope and reached users of our deliverables so significantly that we have to do additional fundraising to complete final versions.

Budget justification

The main expenses for this grant are for the salaries, travel and supplies for full time project staff who will work at UC Berkeley, Cal Poly, Simula Research Lab and the University of Southampton or Sheffield. Because of this, our budget justification is organized around these staff positions. The amounts allocated for travel are based on our actual travel numbers over the past two years. For project PIs, this amounts to approximately 10 week long trips per year and for other project staff this amounts to 1–3 week long trips per year. First year supplies are higher to enable us to purchase computers and monitors for staff.

Project PIs (Dr. Fernando Perez and Dr. Brian Granger). Core activities of the project PIs include managing and training staff, creating the strategic direction of the project, building relationships with Institutional Partners and key collaborators, giving talks at conferences and workshops, fundraising, community building, hiring project staff, etc. To enable both PIs to focus on Project Jupyter, the budget funds significant fractions of the PIs’ time. Dr. Perez will lead the UC Berkeley team and bridge the project with related activities in data science at the Berkeley Institute for Data Science and Lawrence Berkeley National Laboratory, where he holds a Staff Scientist appointment. Dr. Brian Granger will lead the team at Cal Poly, where he is an Associate Professor of Physics and Data Science. This position gives him access to highly motivated and talented students that can be hired and trained to work on the project.

Existing full time staff (Dr. Min Ragan-Kelley, Dr. Thomas Kluyver, Dr. Matthias Bussonnier and Jon Frederic). These individuals are core Contributors and Steering Council members who form the technical backbone of the project; without their full time work on the project, technical activity would completely grind to a halt. Dr. Ragan-Kelley will be working as a postdoc at Simula Research Lab. Dr. Kluyver will be working as a postdoc at the University of Southampton. Dr. Bussonnier will be a postdoc at UC Berkeley. Mr. Frederic will be a senior software engineer at Cal Poly.

Project Manager. This grant will scale our full time project staff from 6 to 14. We feel it is critical for us to hire a new full time Project Manager (at UC Berkeley) to help the PIs manage the increased project scope and technical staff. This role will enable us to segment the deliverables into smaller pieces that can be tackled in parallel by more independent teams, while still keeping the project wide vision and approach consistent.

UX/UI Designer. Significant amounts of our current and proposed work involves visual and interactive design, as well as frontend web development. We currently have no designers on the team; this has created a significant bottleneck for us, even with our current scope. To enable us to tackle the ambitious user interface work of this grant, we propose to hire a new full time user interface/experience (UI/UX) designer who will work with the different teams to design, build and test these user interfaces.

Software engineering. The deliverables of this grant will expand the scope, complexity and effort of the project significantly. The popularity of the project has also put incredible pressure on our existing developers to give talks at conferences and support users online. To enable us to tackle these challenges, we propose to hire four new full time software engineers: two postdocs at UC Berkeley and two software engineers at Cal Poly.

Administrative help. Currently the project PIs have to do all project administration: filing reimbursements, managing the budget, travel and event planning, etc. To handle the increased budget and activities of this grant we propose to hire two new project administrators (full time at Cal Poly and ½ time at UC Berkeley). These administrators will free the PIs, Project Manager and technical staff to focus on the deliverable of the grant. The full time administrator at Cal Poly will also be the lead on planning various project events: core developer meetings, JupyterCon and JupyterDays.

Consultants. During the first year we will hire two consultants. First, we will hire a web/design firm to overhaul our entire web presence, which has seen nearly zero development in years, is a poor representation of the project and lacks key usability aspects. Second, we will hire a technical writer to go over our entire body of documentation and help us triage, rewrite, clarify and organize it. Beyond the first year, our full time technical staff will take over the maintenance of these resources, however, their current state is so bad that we need extra help catching up.

Students. This grant will fund 4 summer undergraduate students each year at Cal Poly. These students are a critical part of our long term sustainability plan. Our goal is to train these students to work on Project Jupyter in different capacities (software engineering, technical writing, design, etc.) and then send them out to jobs at current and future Institutional Partners. Dr. Granger’s position at Cal Poly gives him access to talented students in the new Data Science and Computer Science degree programs.

JupyterCon/Days. We plan on organizing a new Jupyter focused conference each year (JupyterCon). This will be a 2–3 day event that will bring together contributors, steering council members, institutional partners, third party developers and users. Over the long term, we plan on this conference becoming self supporting through industry sponsorship and registration fees. The grant budget includes seed money we know will be required to get this conference off the ground. Some of this seed money will also be used to seed single day Jupyter events that are organized by the larger community (JupyterDays).

Core Developer Workshops. Over the past two years, we have bought our core project Contributors together twice a year for a week long in person meeting. During these meetings, we review the project’s progress and set a roadmap for the following six months. Given the distributed nature of our project, these in person meetings are absolutely critical in building our team and setting the strategic vision of the project. The grant budget includes funding for our core Contributors to attend these events.

Other sources of support

Through NumFOCUS, Project Jupyter currently has funding from Microsoft, which donated $100,000 in 2013 to NumFOCUS for general project work. This money has been used to hire Steering Council member (Jonathan Frederic) as an independent contractor working at Cal Poly with PI Brian Granger.

The Institutional Partner, Rackspace, is donating significant cloud hosting resources and the time of Kyle Kelley, a Rackspace employee and Steering Council member.

The PIs of this grant, through the Institutional Partners UC Berkeley and Cal Poly have the following Jupyter related funding:

A $1.15M grant from the Alfred P. Sloan Foundation during 2013 and 2014 for the creation of nbconvert, interactive widgets and a prototype of JupyterHub. We expect this grant’s funds to run out in March 2015.
A $100,000 grant from Google in the fall of 2014 to UC Berkeley to hire Steering Council member (Matthias Bussonnier) as a postdoc to begin integrating the Jupyter Notebook and Google Drive.
A $100,000 grant from the Simons Foundation to F. Perez at UC Berkeley, that supports the integration of the Jupyter Notebook into a system for data sharing in neuroimaging. This grant is in collaboration with the Stanford Center for Cognitive and Neurobiological Imaging, led by prof. Brian Wandell.

We also plan to raise additional funding through UC Berkeley and Cal Poly in the coming years. This additional funding will be used primarily to fund new work not funded by this grant. Examples include:

A replacement or extension to Markdown syntax that handles the full complexity of academic and book publishing.
Features found in traditional Interactive Development Environments (IDEs) (IntelliJ, Microsoft Visual Studio and Eclipse) such as an interactive debugger, variable inspector, refactoring tools, etc.
Technologies that enable the Jupyter Notebook and JupyterHub to scale to internet sized user groups.
Developing tmpnb and nbviewer into full blown platforms for sharing, indexing, reviewing/commenting and searching notebook based content.
Robust security for the Jupyter Notebook that addresses deployments on the open internet and in highly secure settings.
A federated architecture for live Jupyter Notebooks in the cloud that spans multiple cloud vendors (Amazon, Microsoft, Rackspace) and geographic regions.
Internationalization of the Jupyter Notebook, JupyterHub and Documentation.
Accessibility support across all our software.
Improvements to our parallel computing framework. There is significant interest in this area from federal agencies (NIH, NSF and DOE) in this area. Problems such as the use of our tools in High Performance Computing (parallel supercomputers) environments, or in domain-specific contexts like genome sequence analysis, present challenges that go far beyond the scope of the current proposal. We have already been approached by multiple scientists interested in pushing forward with ideas based on our architecture in directions like these and others.
Implementations of interactive widgets for languages other than Python, such as R.

We have already received preliminary interest from the following companies in funding these and other initiatives: Google, Microsoft, Bloomberg, Continuum, Quantopian, West Health, IBM. In the coming 3 years, much of the time and effort of the project leaders and PIs of this grant (Fernando Perez and Brian Granger) will focus on this fundraising work. Some of the industry money that we raise will also be used to build final versions of the deliverables of this grant that end up with a larger scope than currently envisioned.

Matlab, MathWorks, 2014 <http://www.mathworks.com/products/matlab> ↩︎
Mathematica, Wolfram, 2014 <http://www.wolfram.com/mathematica> ↩︎
SAS, 2014 <http://www.sas.com> ↩︎
SPSS, IBM, 2014 <http://www-01.ibm.com/software/analytics/spss> ↩︎
Sage Math Project, 2014 <http://www.sagemath.org> ↩︎
RStudio, 2014 <http://www.rstudio.com> ↩︎
Spyder IDE, 2014 <https://code.google.com/p/spyderlib> ↩︎
Beaker Notebook, 2014 <http://beakernotebook.com> ↩︎
Zeppelin Project, 2014 <http://zeppelin-project.org> ↩︎
“Account — SageMathCloud.” 2013. 30 Jan. 2015 <https://cloud.sagemath.com/> ↩︎
“Wakari — Web-based Python Data Analysis.” 2012. 30 Jan. 2015 <https://wakari.io/> ↩︎
Google Drive, Google, 2014 <https://www.google.com/drive> ↩︎
The git Project, 2014 <http://git-scm.com> ↩︎
GitHub, 2014 <https://github.com> ↩︎
“IPython interactive demo : Nature News & Comment.” 2014. 30 Jan. 2015 <http://www.nature.com/news/ipython-interactive-demo-7.21492> ↩︎
Our detailed governance policies are publicly available here: https://github.com/ipython/ipython/wiki/IPEP-29:-Project-Governance. ↩︎
http://en.wikipedia.org/wiki/Ubuntu_(operating_system)#Installed_base http://en.wikipedia.org/wiki/Linux_adoption#Measuring_desktop_adoption ↩︎
https://qa.debian.org/popcon.php?package=ipython ↩︎
“Learning IPython for Interactive Computing and Data …” 2014. 30 Jan. 2015 <https://www.packtpub.com/big-data-and-business-intelligence/learning-ipython-interactive-computing-and-data-visualization>, ↩︎
“IPython Notebook Essentials | Packt.” 2014. 30 Jan. 2015 <https://www.packtpub.com/application-development/ipython-notebook-essentials> ↩︎
Rossant, Cyrille. IPython Interactive Computing and Visualization Cookbook. Packt Publishing Ltd, 2014. ↩︎
“Python for Signal Processing — Featuring IPython Notebooks.” 2013. 30 Jan. 2015 <http://www.springer.com/engineering/signals/book/978-3-319-01341-1> ↩︎
“Mining the Social Web, 2nd Edition — O’Reilly Media.” 2013. 30 Jan. 2015 <http://shop.oreilly.com/product/0636920030195.do> ↩︎
“A gallery of interesting IPython Notebooks · ipython … - GitHub.” 2013. 30 Jan. 2015 <https://github.com/ipython/ipython/wiki/A-gallery-of-interesting-IPython-Notebooks> ↩︎
“Practical Numerical Methods with Python | GW Open edX.” 2014. 30 Jan. 2015 <http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about> ↩︎

Jupyter Blog - funding

2026 Jupyter Community Call For Funding Proposals

Announcing our first Jupyter Community Funded Proposals

Your Ideas, Our Support: Jupyter Community Call For Funding Proposals

Acknowledgements

JupyterGIS

Collaborative workflows in geosciences

From the desktop to the web

Real-world applications

Building collaborations in Europe and beyond

Affiliations

European Commission Funds Jupyter Bug Bounty Program

CZI awards three EOSS grants to Jupyter community members

Accessibility in Jupyter

Interactive documentation

Community strategic support for JupyterHub

The JupyterHub and Binder Contributor in Residence!

Jupyter meets the Earth

Geoscience use cases

Jupyter & Pangeo: empowering scientists

User-Centered Development

An opportunity for meaningful impact — join us!

In closing: a step toward sustainable open science

Acknowledgments

Affiliations

Continuum Analytics donates $100k to Project Jupyter

New funding for Jupyter

Project Jupyter: Computational Narratives as the Engine of Collaborative Data Science

Statement of the problem and audience

Related work and collaborations

Organizational background

Mission and Background

Core problems and constituency

Project organization

Project activities and highlights

Approach and strategy

Interactive Computing

Notebooks as interactive applications

Modular, reusable UI/UX

Software engineering with notebooks

Capstone

Computational Narratives

nbconvert

Element filtering

Documentation

Capstone

Collaboration

Real time collaboration

JupyterHub

Capstone

Sustainability

People

Events

Year by year output

Our approach to building software

Year-by-year plan of deliverables

Measuring effectiveness

Budget justification

Other sources of support