Jupyter Blog - HPC

Exploring Petabytes of the Night Sky — Jupyter Notebooks at NOIRLab’s Astro Data Lab Science Platform

2026-04-20T18:04:00+00:00

Imagine querying 420+ billion rows of astronomical catalog data — spanning 30 major sky surveys, observed over decades with telescopes on three continents — from a Jupyter notebook in your browser in seconds. No download. No HPC allocation request. No waiting.

That is what 4,800+ astronomers in over 90 countries can do every day at the Astro Data Lab science platform. Data Lab is operated by NSF NOIRLab, the National Optical-Infrared Astronomy Research Laboratory, headquartered in Tucson, Arizona, with observatories in Arizona, Hawai’i, and Chile. Since its public launch in June 2017, Astro Data Lab has quietly become one of the largest deployments of Jupyter notebooks in professional science — and a case study in what happens when you bring the compute to the data instead of the other way around.

World map of 2024 number of data queries at Astro Data Lab by country (log-scale color): 64.2 million queries from 72 countries that year.

The Data Problem Astronomy Had to Solve

Modern sky surveys are data machines. The Dark Energy Survey cataloged 690 million objects. Gaia measured positions and motions for 1.8 billion stars. The DESI Legacy Surveys cover 20,000 square degrees, nearly half of the full sky, in three optical bands. And the upcoming Rubin Observatory’s Legacy Survey of Space and Time (LSST) will generate roughly 10 million transient alerts per night starting later this year.

Traditional astronomy workflows begin with downloading relevant data to a local computer and to use locally installed specialized software tools to process and analyze the data. However, downloading these catalogs to a local machine is now often physically impossible. A single survey’s measurements table can exceed the combined disk space of an entire research group. And even if you could download it, the computing resources needed to query it efficiently at scale requires infrastructure most astronomers don’t have.

The answer the community converged on, like many industries dealing with big data: bring the compute to the data. Host the catalogs in databases, co-locate a computing environment next door, and give scientists a familiar interface to work in. That interface, increasingly, is a Jupyter notebook.

Astro Data Lab: Jupyter at the Observatory

Astro Data Lab was conceived in 2014 and went public in June 2017, originally built to support data releases from the Dark Energy Survey — a few terabytes of catalogs and tens of terabytes of images. We imagined a ceiling of roughly 500 users. We were wrong, in the best way.

Today the platform hosts:

About 420 billion catalog rows across 30+ major astronomical surveys (DES, Legacy Surveys, DESI, NOIRLab Source Catalog, SDSS, Gaia, unWISE, SMASH, S-PLUS, VHS, 2MASS, and dozens more)
31 million spectra via SPARCL, our spectral access service (DESI DR1+EDR, SDSS/BOSS DR17)
Petabytes of images, accessible through a Simple Image Access service and cutout API
Over 4,800 registered users from over 90 countries, who submit tens of millions of data queries each year

Every registered user gets a persistent JupyterHub environment with the full astronomy Python stack pre-loaded — Astropy, NumPy, SciPy, Matplotlib, Pandas, Scikit-learn — and our own astro-datalab client library. The library provides core services, for instance auth and DB queries:

from dl import authClient, queryClient
from getpass import getpass

# Log in
token = authClient.login(input("username: "),getpass("password: "))

# Query 10 objects from the NOIRLab Source Catalog near a sky position
# Right Ascension (RA) = 150.12 degrees
# Declination (Dec) = 2.21 degrees
# Search radius = 0.05 degrees
# q3c (Quad Tree Cube) is a spatial indexing scheme for Postgres
# gmag and rmag are the g-band and r-band magnitudes of objects
# in the NOIRLab Source Catalog Data Release 2, ‘object’ table.
result = queryClient.query(
    """SELECT ra, dec, gmag, rmag FROM nsc_dr2.object
       WHERE q3c_radial_query(ra, dec, 150.12, 2.21, 0.05) LIMIT 10""",
    fmt="pandas"
)

It’s as simple as that. The query runs on the database server next to the data; only the result set crosses the network.

Use Case 1 — Seeing the Sky Inside a Notebook with AladinLite

One of the most immediate joys of working with astronomical data is visualization: not just numbers in a table, but where things are in the sky, what the images look like, and how your query results relate to the underlying survey footprint.

We’ve integrated AladinLite v3 — the interactive sky atlas from Centre de Données Astronomiques de Strasbourg (CDS) — directly into the notebook environment via the ipyaladin widget. With a handful of lines, astronomers can embed a fully interactive sky viewer in a notebook cell or next to their notebook in a “sidecar”, and overlay their own data on top of real survey imagery:

import time
from astropy import units as u
from astropy.table import Table
from astropy.coordinates import SkyCoord
from ipyaladin import Aladin
from sidecar import Sidecar

# Instantiate the Aladin interactive sky viewer 
aladin = Aladin(full_screen=True)
with Sidecar(title="aladin_output",anchor='split-right'):
    display(aladin)

# globular cluster NGC 1851 (RA, Dec)
aladin.target = SkyCoord(78.52809*u.deg, -40.04656*u.deg)
aladin.coo_frame = "ICRSd"  # set coordinate frame to ICRS, angles in deg
time.sleep(1)  # race condition
aladin.fov = 0.4  # set field of view to 0.4 degrees

# Overlay catalog query results as circle markers
t = Table.from_pandas(df)  # e.g., from a previous query around NGC 1851
aladin.add_table(t,shape='circle',source_size=15,color='green')

The result is a pannable, zoomable sky viewer — right in the notebook — with your query results overlaid as green circles on the actual sky image of a globular cluster (see figure below). Users can overlay MOCs (Multi-Order Coverage maps, which encode survey footprints), user-generated catalogs from a prior query, or any Virtual Observatory-standard data source.

This capability turns what was once a static plot into an exploratory tool: zoom into a cluster, click on a source, cross-match on the fly. For students and scientists unfamiliar with a dataset, it is often the fastest path from “I have a list of objects” to “I understand where they are and what I’m looking at.”

AladinLite integration is now active in our notebook library, with full deployment into the new Data Lab Web Portal on the roadmap for later this year.

AladinLite v3 sky viewer inside a Jupyter notebook, showing a globular cluster with catalog query results in the outskirts of the cluster overlaid as green circles.

Use Case 2 — Stacking Galaxy Spectra with SPARCL

Spectroscopy — measuring how much light a star or a galaxy emits at each wavelength — is one of astronomy’s most powerful tools. But individual spectra are often noisy. The signal-to-noise ratio of a single optical spectrum for a faint galaxy can be too low to measure the emission lines that encode star formation rate, gas chemical content (Oxygen, Nitrogen, etc.) or the even more subtle absorption lines that create small wiggles in the shape of the spectrum, yet encapsulate crucial information such as the mass and age of the stars making up a galaxy.

One trick that astronomers have used for decades: combining or “stacking” spectra. Average hundreds of spectra together, and the noise level reduces while the signal builds up. What was invisible in a single spectrum becomes unmistakable in the stack. While the concept is simple, reading and manipulating large numbers of spectra can be time consuming or cumbersome.

SPARCL (SPectra Analysis and Retrievable Catalog Lab) makes this possible at scale directly in a notebook. With sparclclient, users can currently search 31 million spectra by redshift range, target type, and survey, then retrieve flux arrays and wavelength grids ready for stacking:

from sparcl.client import SparclClient

# Instantiate the SPARCL client (connected to production server)
client = SparclClient()

# Find SDSS spectra of galaxies in a redshift slice 0.1<z<0.3
found = client.find(
    outfields=['sparcl_id', 'ra', 'dec', 'redshift', 'spectype'],
    constraints={'spectype': ['GALAXY'],
                 'redshift': [0.1, 0.3],
                 'data_release': ['SDSS-DR17']},
)

# Retrieve flux, wavelength, and inverse-variance arrays
retrieved = client.retrieve(found.ids,
            include=['flux', 'wavelength', 'ivar'])

In our SpectralStacking_SDSS science example notebook, users first stack a small number of galaxy spectra (N=5) in eight bins of astrophysical color g−r (green and red filters), revealing trends from blue spectra with emission lines to red spectra with absorption lines but with noisy spectra. Then users stack hundreds of galaxy spectra for the same bins of color g−r and obtain much cleaner spectra where the small wiggles are now real astrophysical features and no longer buried in the noise.

The spectral rainbows below — N=5 then N=200 stacked galaxy spectra color-coded in bins of astrophysical color g−r — are each a single output cell from this notebook, generated entirely within the Data Lab environment.

SPARCL spectral rainbows: top panel shows N=5 galaxy spectra stacked and color-coded by g−r color, spanning wavelengths 3750–6450 Ångstrom. Bottom panel shows the same exercise but with N=200 galaxy spectra per bin, greatly enhancing the signal-to-noise ratio.

Use Case 3 — Variable Stars and the Coming Flood of Time-Domain Data

Not all astronomical data is a static snapshot of the sky. Many of the most scientifically rich phenomena — pulsating stars, transiting exoplanets, exploding supernovae, gravitational lensing events — reveal themselves through change over time.

Among the most useful calibration tools in astrophysics are RR Lyrae stars: old, low-mass stars that pulsate with periods of 0.2–1 day and a brightness variation that traces their distance. Finding and characterizing them across millions of square degrees of sky requires querying multi-epoch photometry catalogs, computing period statistics, and folding light curves — all tasks that fit naturally in a notebook workflow.

Our TimeSeriesAnalysisRrLyraeStar notebook demonstrates the full pipeline: query the SMASH DR2 catalog for stars with high photometric variability, run a Lomb-Scargle periodogram on the light curve, identify the dominant period, and phase-fold the observations to reveal the characteristic sawtooth pulsation profile:

from astropy.timeseries import LombScargle
import numpy as np

ls = LombScargle(t, y)  # time and magnitude from a previous query
frequency, power = ls.autopower()
period = 1./frequency # period is the inverse of frequency
best_period = period[np.argmax(power)]
phase = (t / best_period) % 1  # folded timeseries = light curve

The resulting phase-folded light curve shown in the figure below is clean, precise, and immediately recognizable to any variable-star astronomer — produced entirely from archival survey data without a single new observation.

This kind of workflow is also a proving ground for the upcoming Vera C. Rubin Observatory’s Legacy Survey of Space and Time (LSST). When Rubin begins operations and delivers 10 million nightly alerts, the only workflows that will scale are ones already designed to run against large databases or specialized file systems, in shared computing environments, with notebook-native tooling. Astro Data Lab users are building those workflows today.

Phase-folded RR Lyrae light curve from the TimeSeriesAnalysisRrLyraeStar notebook, showing characteristic sawtooth pulsation with a 0.65 day period.

The Notebook Ecosystem

The three use cases above are drawn from our library of 80+ open-source Jupyter notebooks at github.com/astro-datalab/notebooks-latest. The library is organized into six sections:

Directory	Contents
`01_GettingStartedWithDataLab/`	Authentication, dataset discovery, first queries
`02_DataAccessOverview/`	More advanced queries, image searches, etc.
`03_ScienceExamples/`	Many complete science cases (stellar streams, dwarf galaxies, large-scale structure, SED fitting, …)
`04_HowTos/`	Service-specific tutorials (SPARCL, SIA image cutouts, cross-matching, file storage)
`05_Contrib/`	Community-contributed notebooks (ANTARES alert broker, user science cases, etc.)
`06_EPO/`	Education & public outreach (Teen Astronomy Cafe, La Serena School for Data Science)

All notebooks are open-source and community contributions are welcome via pull request. We use them as living teaching materials in workshops at Astronomical Data Analysis Software & Systems (ADASS) and American Astronomical Society (AAS) conferences, summer schools, and university courses around the world. We have also recently translated most of our notebooks to the Spanish language.

In the coming months we will launch a tagged, searchable notebook gallery — filterable by science topic, Unified Astronomy Thesaurus (UAT) keywords, target audience, and difficulty level. The pilot framework was developed by two summer students working with the team.

Footprints of 24 sky survey datasets hosted at Astro Data Lab. This montage shows the wide variety of astronomical surveys, with some that cover the full sky, others focusing on the Milky Way (central plane), and yet others studying the extragalactic regions beyond the Milky Way. We ensure that each survey is represented in at least one of our example notebooks.

Looking Ahead

Nine years in, the Astro Data Lab science platform is evolving on several fronts simultaneously.

GPU computing. We are deploying a GPU node, which will be connected to the Jupyter notebook service. This opens deep learning and large-scale ML workflows in the same notebook environment where the data lives.

An AI assistant. Our first-ever user survey, conducted in September 2025, ranked an in-notebook AI assistant as one of the top requested features. We are actively exploring what responsible, science-aware AI assistance looks like in this context — helping users construct SQL/ADQL queries, navigate datasets, and debug notebook code, without hallucinating catalog column names. jupyter-ai might come in very handy here.

New integrated Web Portal. Our Data Explorer — an integrated web interface combining catalog browsing, query execution, image cutouts, spectral search, and job status monitoring — was rolled out last year. Some of the next milestones include integration of AladinLite into the portal, bringing the sky-visualization capability described above out of the notebook and into the browser-native interface, and a new integrated positional cross-matching service.

Try It

The full notebook library is open-source: github.com/astro-datalab/notebooks-latest. Community notebook contributions are welcome — see CONTRIBUTING in the repository. You can also run all notebooks locally, after installing the Data Lab command-line client and Python module: pip install astro-datalab

Astro Data Lab also offers a JupyterLab environment as a service to the broad astronomy community — students, researchers, educators, and citizen scientists. Registration takes just a moment at datalab.noirlab.edu.

Questions and feedback: datalab@noirlab.edu

Robert Nikutta is a scientist at NSF NOIRLab’s Community Science and Data Center, and lead of the Astro Data Lab science platform. Stéphanie Juneau is an associate astronomer at CSDC and lead of the SPARCL spectroscopy initiative. The platform is the work of the full Astro Data Lab team, past and present.

IPython Parallel in 2021

2021-11-30T07:55:00+00:00

Updates on IPython Parallel; new features, future direction

This post describes work funded by Bodo, Inc.

IPython Parallel’s a bit of an odd duck in the parallel computing space. In a world with dask, ray, bodo, pyspark, and other parallel computing tools, what is IPython Parallel’s role in 2021?

Context

IPython Parallel began (in 2006!) as a natural extension of the Jupyter messaging protocol(though it predates the Jupyter name by a few years): when you have a protocol for REPL-style remote code execution, what can you do with multiple remote execution environments? This question led to the development two basic models for parallel execution in IPython Parallel:

“multiplexed execution” — where you send code explicitly to different workers, and
“load-balanced execution” — where you tell IPython Parallel what to run, and a scheduler takes care of assigning each task to an available worker.

What IPython Parallel doesn’t do

To IPython, your tasks are black boxes (either Python functions or blocks of Python code as text) and you are in complete control of where and when your tasks run, as well as any dependencies or side effects they may have.

IPython Parallel specifically doesn’t and won’t do several things that other tools might:

infer dependencies or relationships between tasks (execution graph)
manage data distribution or locality
construct parallel algorithms or represent ‘natively’ parallel distributed data structures, such as distributed arrays or data frames

Everything is very explicit and implemented at the client-level in IPython Parallel, meaning that while it doesn’t do these things, you can. It also means that many of the things that have overhead costs associated with parallel computing (communication, data movement) have particularly high costs in IPython Parallel.

IPython Parallel doesn’t hide anything from you, for better and worse.

What IPython Parallel does

IPython Parallel makes explicit parallel computations interactive.

IPython Parallel is not just parallel Python, it’s parallel IPython. That means you have all the power of IPython’s inspection, interactivity, magics, and debugging available on all of your distributed workers. This makes it especially well-suited to prototyping and experimentation.

IPython Parallel also presents standard APIs such as Python Executors, compatible with many other implementations, to make it easy to migrate to and from IPython Parallel, enabling developers to write code that uses a single multicore laptop or a thousand cores on an HPC cluster or cloud.

IPython Parallel in 2021

Interactive progress across parallel engines. Those progress bars are interactive widgets running locally, and on each remote engine!

For a lot of today’s workloads, my default recommendation is: use dask or bodo or another modern tool. IPython even makes this easier if you already happen to have an IPython Parallel cluster, you can tell it to “become dask,” and off you go:

dask_client = rc.become_dask(ncores=1)

However, where IPython Parallel can shine, is making traditional SPMD (e.g., MPI) workloads interactive, especially for prototyping and debugging.

If you have an MPI simulation and you wish you could pause it in the middle, poke around and make plots and interact with just one node or all of them with all the interactive tools available to you in Jupyter, IPython Parallel may be the tool for you.

Recent developments

Because I see the main problem IPython Parallel solves well is adding interactivity to direct parallel execution, the focus of recent developments has been on improving that story, based on feedback from users who are often in traditional HPC environments like SLURM or PBS, or using MPI in the cloud.

Much of the feedback on challenges over the years have been around:

Scaling to larger numbers of engines (1,–10,000)
Better feedback and recovery when things go wrong
Security requirements

Thanks to Bodo, I’ve been able to spend a lot of time this year addressing several of those issues, now available as IPython Parallel 8.0.

Last year, Tom-Olav Bøyum developed a broadcast scheduler as part of his Master’s thesis, which vastly improves the efficiency of sending the same task to all engines (the main thing we do in IPP+MPI). This work is now mostly done and landed in IPython Parallel 7. We’ve also addressed longstanding issues with registering large numbers of engines (5–10,000) at once, a common source of failure on HPC clusters.

One of the most frustrating / challenging things when working with MPI or other parallel computations is dealing with hangs and restarting tasks. Through the new Cluster API, IPython Parallel now has one of its most requested features: the ability to send signals to one or all engines, and forcefully restart the cluster if it’s stuck.

We’ve also improved interactive feedback with progress bars, live streaming output, a new JupyterLab plugin based on dask-labextension, and support for the Jupyter widget protocol, so you can instantiate widgets on engines and interact with them directly from a notebook.

Security questions have also been raised, because IPython (and Jupyter’s) use of ZeroMQ assumes a ‘trusted network’, typically localhost or BSD sockets for notebooks. Since IPython Parallel usually runs across a network, security can be more of a concern (whereas notebooks typically only use standard HTTPS connections over a network), and the level of security in the Jupyter protocol alone may not be sufficient. In the past, we’ve resorted to tunneling TCP over SSH for more secure network traffic, but still always implicitly trusting localhost.

IPython Parallel 7.1 enables CurveZMQ for full authentication, encryption, and forward-secrecy at the transport level, solving a longstanding security shortcoming of IPython Parallel.

IPython Parallel in the future

Scaling is the biggest challenge for IPython Parallel due to its communication model. However, that scaling is on the order of IPython engines, which is not the same as the number of cores. You can benefit from this today if your tasks are already multi-threaded, e.g., through OpenMP threads in numpy, or your own Python threads or multiprocess breakdown of tasks.

Better support of multi-level parallelism, where each “engine” may represent a multicore node, as individual nodes get bigger and bigger would mean moving the bar where IPython Parallel communication is the bottleneck back two orders of magnitude on large machines, because one ‘engine’ could represent 128 cores or more. Good support for 1000 128-core nodes would give IPython Parallel some pretty comfortable headroom for our target use cases, scale-wise.

Also related to scaling, bringing the BroadcastView to maturity should allow us to completely replace the DirectView scheduler, as it should be able to match or beat it in almost every scenario with some further development. There are some features lacking, especially when it comes to error handling, but those can certainly be addressed in time.

Finally, I’ll invite you to get involved. Especially if you are interested in prototyping parallel code, or making traditional MPI-style code interactive, check out IPython Parallel 8 and let us know how it goes, or contribute your use case as an example.

pip install --upgrade ipyparallel

National Scale Interactive Computing

2019-08-22T19:06:00+00:00

This is an invited post from Jim Colliander, Professor of Mathematics at UBC and Director of the Pacific Institute for the Mathematical Sciences.^[1]

In 2017, the Pacific Institute for the Mathematical Sciences (PIMS), in partnership with Compute Canada and Cybera, launched Syzygy, a cloud-hosted interactive computing platform that delivers JupyterHub deployments for universities across Canada.

Syzygy has been used by over 16,000 students at 20 universities. The main results of the Syzygy experiment so far are:

Demand for interactive computing is ubiquitous^[2] and growing strongly at universities.
The Jupyter ecosystem is an effective way to deliver interactive computing.
A scalable, sustainable, and cost-effective interactive computing service for universities is needed as soon as possible.

Demand for interactive computing

Both research and teaching at universities are adapting to major societal changes driven by explosions in data and computational tools. New educational programs that prepare students to think computationally are emerging, while research strategies are changing in ways that are more open, reproducible, collaborative, and interdisciplinary. These transformations are inextricably linked and are accelerating demand for interactive computing. The Syzygy experiment has shown that using Jupyter in educational programs drives interest in using Jupyter for research (and vice versa). For example, students in mathematics, statistics, and computer science collaborated with a researcher from St. Paul’s Hospital in Vancouver using Syzygy to identify new pathways to prevent death from sepsis. Research communities typically need access to deeper computational resources and often span multiple universities, but the common thread is the need to expand access to interactive computing.

A map of JupyterHub deployments deployed by Syzygy.

Technical milestone achieved

The Syzygy project has demonstrated that it’s possible to deploy tools for interactive computation at a national scale rapidly and efficiently using an entirely open source technology stack. Students, faculty and staff across Canada use Syzygy to access Jupyter through their browsers with their university single-sign-on credentials. The JupyterHubs range from a “standard” configuration to bespoke environments with specially curated tools and data integrations. This richness is possible because of the architecture of the Syzygy and Jupyter projects and the flexibility of the underlying cloud resources. As a case-study, Syzygy demonstrates that the Jupyter community has achieved a significant technical milestone: interactive computing can be delivered at national scale using cloud technologies.

Service level requirements

The validation that interactive computing can be technically delivered at national scale prompts universities to ask a variety of questions. Can interactive computing service be delivered robustly? How will users be supported? What are the uptime expectations? What is the data security policy? How is privacy protected? Can the robustness of the service be clarified in a service level agreement? Syzygy, as an experimental service offered to universities at no charge and without a service level agreement, does not properly address these questions. To advance on their education-research-service mission and address growing demand, universities need a reliable interactive computing service with a service level agreement.

What’s next?

How should universities address their needs for interactive computing over the next five years? Right now, universities are following two primary approaches:

🙏 Ad hoc: faculty figure out how to meet their own needs for interactive computing; IT staff deploys JupyterHub on local or commercial cloud servers; this approach gives universities control over their deployments and hardware, though requires time and expertise that many may not have.
🎩 Use a cloud provider’s service: Google Colab, Amazon Sagemaker, Microsoft Azure Notebooks, IBM Watson Studio; this approach allows universities to quickly launch interactive computing services, with a loss of flexibility and some risks by becoming reliant upon a particular vendor’s closed-source and proprietary software.

These approaches are not sustainable over the long term. If universities all deploy their own JupyterHub services, many will need technical expertise they do not currently have and will involve a significant duplication of effort. If universities rely on hosted cloud notebook services, the reliance on proprietary technology will impair their ability to switch between different cloud vendors, change hardware, customize software, etc. Vendor lock-in will limit the ability of universities to respond to changes in price for the service. Universities will lose agility in responding to changes in faculty, staff, and student computing needs.

There is a third option that addresses the issues with these two approaches and generates other benefits for universities:

🤔 Form an interactive computing consortium: universities collaborate to build an interactive computing service provider aligned with their missions to better serve their students, facilitate research, and avoid risks associated with vendor lock-in.

To retain control over their interactive computing stacks, avoid dependence^[3] on cloud providers, and accelerate the emergence of new programs, universities should work together to deploy interactive computing environments in a vendor-agnostic manner. This might take the form of a consortium — an organization dedicated to serving the needs of universities through customized shared infrastructure for interactive computing. The consortium would also ensure that universities will continue to play a leadership role in the development of the interactive computing tools used for education and research.

The Syzygy experiment confirmed that growing demand for interactive computation within universities can be supplied with the available technologies advanced by the Jupyter open source community. In the coming year, we aim to build upon the success of the Syzygy experiment and seed an initial node of a consortium in Canada with the intention of fostering a global network of people invested in advanced interactive computing. If you are interesting in partnering, please get in touch! See this Jupyter Community Forum post to continue the discussion.

The author gratefully acknowledges feedback on this piece from Ian Allison, Lindsey Heagy, Chris Holdgraf, Fernando Perez, and Lindsay Sill. ↩︎
Interactive computing needs have been identified in agriculture, applied mathematics, astronomy, chemistry, climate science, computer science, data science, digital humanities, ecology, economics, engineering, genomics, geoscience, health sciences, K-12 education, neuroscience, political science, physics, pure mathematics, statistics, and sociology. ↩︎
Relying on commercial cloud vendors to provide the interactive computing service for universities risks recreating the problems associated with scientific publishing that emerged with the internet. ↩︎

Jupyter for Science User Facilities and High Performance Computing

2019-07-09T23:53:00+00:00

Jupyter is the “Google Docs” of data science. It provides that same kind of easy-to-use ecosystem, but for interactive data exploration, modeling, and analysis. Just as people have come to expect to be able to use Google Docs everywhere, scientists assume that Jupyter is there for them whenever and wherever they open their laptops.

But what if the data you want to interact with through Jupyter doesn’t fit on your laptop or is excruciating to move? What if the model you want to build and test requires more computing power and storage than you have right in front of you? As a scientist, you want the same interactive experience and all the benefits of Jupyter, but you also need to “reach out” to put something big into your science process: A supercomputer, a telescope data archive, a beam-line at a synchrotron. Can Jupyter help you do that big science? What efforts are in motion already to make this a reality, what work still needs to be done, and who needs to do it?

Doing this right will take a community: New collaborations between core Jupyter developers, engineers from high-performance computing (HPC) centers, staff from large-scale experimental and observational data (EOD) facilities, users and other stakeholders. Many facilities have figured out how to deploy, manage, and customize Jupyter, but have done it while focused on their unique requirements and capabilities. Still others are just taking their first steps and want to avoid reinventing the wheel. With some initial critical mass, we can start contributing what we’ve learned separately into a shared body of knowledge, patterns, tools, and best practices.

40+ participants from universities, national labs, industry, and science user facilities. Credit: Fernando Perez.

In June, a Jupyter Community Workshop held at the National Energy Research Scientific Computing Center (NERSC) and the Berkeley Institute for Data Science (BIDS) brought about 40 members of this community together to start distilling. Over three days in talks and breakout sessions, we addressed pain points and best practices in Jupyter deployment, infrastructure, and user support; securing Jupyter in multi-tenant environments; sharing notebooks; HPC/EOD-focused Jupyter extensions; and strategies for communication with stakeholders.

Here are just a few highlights from the meeting:

Michael Milligan from the Minnesota Supercomputing Center perfectly set the tone for the workshop with his keynote, “Jupyter is a One-Stop Shop for Interactive HPC Services.” Michael is the creator of BatchSpawner and WrapSpawner, JupyterHub Spawners that let HPC users run notebooks on compute nodes supporting a variety of batch queue systems. Contributors to both packages met in an afternoon-long breakout to build consensus around some technical issues, start managing development and support in a collaborative way, and gel as a team.
Securing Jupyter is a huge topic. Thomas Mendoza from Lawrence Livermore National Laboratory talked about his work to enable end-to-end SSL in JupyterHub and best practices for securing Jupyter. Outcomes from two breakouts on security include a plan to more prominently document security best practices, and a future meeting (perhaps another Jupyter Community Workshop?) focused specifically on security in Jupyter.
Speakers from Lawrence Livermore and Oak Ridge National Laboratories, the European Space Agency showed off a variety of beautiful JupyterLab extensions, integrations, and plug-ins for climate science, complex physical simulations, astronomical images and catalogs, and atmospheric monitoring. People at a variety of facilities are finding ways to adapt Jupyter to meet the specific needs of their scientists.

Really, there’s just too much to pack into a blog post so we encourage you to look at the talk slides and notes on Discourse — all the breakout notes have been posted there to this topic. We’re working on getting videos of the slide presentations up on the workshop website as well. Watch for announcements of future meeting opportunities and documentation on Discourse as well.

Finally we want to thank Project Jupyter, NumFOCUS, and Bloomberg for their help making this meeting happen. We all came away with a better sense of who is doing what in our community, and how we can work together on this new area of growth for the Jupyter community. The organizers also want to thank their respective institutions’ administrative staff (Seleste Rodriguez at NERSC, and Stacy Dorton at BIDS) for helping with workshop logistics.

Jupyter Community Workshop: Jupyter for Scientific User Facilities and High-Performance Computing

2019-01-29T19:35:00+00:00

We are excited to share more news about the Jupyter Community Workshop for Scientific User Facilities and High-Performance Computing! This is part of a series of Jupyter Community Workshops funded by Bloomberg to “bring together small groups of Jupyter community members and core contributors for high-impact strategic work and community engagement on focused topics.”

This workshop will be held in Berkeley, California, from Tuesday June 11 to Thursday June 13, 2019. The workshop is being jointly hosted at the National Energy Research Scientific Computing Center (NERSC, part of Lawrence Berkeley National Laboratory) and the Berkeley Institute for Data Science (BIDS, at the University of California). The workshop executive committee consists of Rollin Thomas (NERSC), Dan Allan (Brookhaven National Laboratory), and Chris Holdgraf (BIDS — UC Berkeley).

We, the organizers, invite your expression of interest in the effort through this Google form. Let us know whether you’d be interested potentially in attending, just want to be kept in the loop, or just want to express your support. Finding out who is doing what with Jupyter in this space is the first step in building our community.

Why? Advances in technology at experimental and observational science facilities (EOS facilities: telescopes, particle accelerators, light sources, genome sequencers and so on), in robust high-bandwidth global networks, and in high-performance computing (HPC) have resulted in an exponential growth of data for scientists to collect, manage, and understand. Interpreting these data streams requires computational and storage resources greatly exceeding those available on laptops, workstations, or university department clusters. Funding agencies increasingly look to HPC centers to address the growing and changing data needs of their scientists. These institutions are uniquely equipped to provide the resources needed for extreme scale science. At the same time, scientists seek new ways to seamlessly and transparently integrate HPC into their EOS workflows. That’s where we see Jupyter fitting in.

We know that scientists love Jupyter because it combines visualization, data analytics, text, and code into a document they can share, modify, and even publish. What about using Jupyter to control experiments in real-time, or steer complex simulations on a supercomputer, or even connect experiments to HPC for real-time feedback and decision making? How can users reach outside the notebook to corral external data and computational resources in a seamless, Jupyter-friendly manner?

These were the questions on our minds when we proposed a three-day workshop for Jupyter developers, HPC engineers, and staff from EOS facilities. We are looking to foster a new collaborative community that can make Jupyter the pre-eminent interface for managing EOS workflows and data analytics at HPC centers. EOS scientists need Jupyter to work well at their facilities and HPC centers, and this workshop will help us address the technical, sociological, and policy challenges involved.

The workshop itself will include presentations, posters, and a couple half-day hack-a-thon/breakout sessions for collaboration. We will identify best practices, share lessons learned, clarify gaps and challenges in supporting deployments, and work on new tools to make Jupyter easier to use for big science.

During the workshop, participants will be invited to collaborate on a survey white paper that documents the current state of the art in Jupyter deployments at various facilities and HPC centers. The document will include deployment descriptions, maintenance and user support strategies, security discussions, use cases, and lessons learned. A forward-looking summary provided at the end of the white paper will tie together common threads across various facilities and highlight areas for future research, development, and implementation. We will aim to have the paper completed and published to arXiv within three months of the end of the workshop. These ideas in a single document should help developers, maintainers, and researchers make the case they need to management and policymakers to drive the effort forward.

So let us know if you’re interested in the effort by filling out the Google form, even if you think you can’t make it. Part of what we’re doing is finding out who is doing what with Jupyter where in HPC and EOS. That’s the real first step in building our community!