Jupyter Blog - collaboration

JupyterGIS 0.16: New visualization capabilities, collaborative Story Maps, and more

2026-08-24T16:09:00+00:00

Read this article in Notebook.link, as a live story-map! https://notebook.link/@martinRenou/jupytergis-announcement

Earlier this year, we introduced STAC browsing and Story Maps in JupyterGIS, making it easier to discover geospatial datasets and communicate results without leaving Jupyter.

JupyterGIS 0.16 continues in the same direction. This release adds support for new geospatial formats, tighter integration with the scientific Python ecosystem, a redesigned Story Map editor, and a more expressive way to style geographic data.

Story Maps are getting a new look!

Story Maps in JupyterGIS let you build a scrollable presentation around your map. A Story Map is made up of a sequence of segments that can combine Markdown content with map views, so you can guide the reader through a geographic story as they scroll.

Each segment can define its own map state, including the current map location, visible layers, and layer styling. This means that the map can change as the reader moves through the story: layers can appear or disappear, the view can move to a new location, and symbology can change to highlight different aspects of the data.

Story Maps have received a significant update in this release.

Read this article in Notebook.link, as a live story-map! https://notebook.link/@martinRenou/jupytergis-announcement

The Story-map associated to this release annoucement. It contains Text, images, and map views with associated layer states.

A better Story Maps editing experience

We’ve also made substantial improvements to the Story Map editing experience.

The editor has been redesigned around Jupyter’s real-time collaboration infrastructure, allowing multiple people to edit the same Story Map simultaneously. Changes appear immediately for everyone, making it much easier to prepare presentations, reports, or educational material as a team.

Collaboratively edit the Story Map markdown.

The editor now gives you a much better sense of what the final story will look like while you are working on it. Markdown sections can be previewed directly in the editor, and a new Story Map preview makes it possible to see the complete presentation without leaving the editing workflow. This makes it easier to write, arrange, and refine a story while keeping an eye on the final result. We’ve also introduced a new layout that is better suited for long-form content. In addition to guided geographic narratives, Story Maps can now be used to create richer articles combining text, maps, images, and other interactive content.

New Story Map editor: Set story segment viewport, preview markdown, set layers properties for the story segment.

OpenEO layers

More and more geospatial workflows rely on remote processing instead of downloading datasets locally. openEO provides a common API to describe these processing pipelines as process graphs that are executed by a backend.

JupyterGIS can now display openEO process graphs directly as map layers. Instead of exporting intermediate results before visualizing them, you can connect an openEO backend and inspect the output of your processing pipeline directly in the map.

The visualization is tile-based and lazy: JupyterGIS only requests the data needed for the current map view and zoom level. This makes it possible to explore large remote sensing workflows interactively, without materializing the full result locally.

JupyterGIS can make use of any openEO server that supports tiling, such as titiler-openeo

import openeo
from jupytergis import GISDocument
from openeo.processes import process

connection = openeo.connect(SERVER_URL)
connection.authenticate_basic(username=BASIC_AUTH, password=BASIC_AUTH)

cube = connection.load_collection(
    "sentinel-2-global-mosaics",
    bands=["B03", "B08"],
    temporal_extent=["2022-04-15", "2022-12-31"],
)

cube = cube.reduce_dimension(
    dimension="t",
    reducer="first",
)

cube = cube / 10000.0

# NDWI = (GREEN - NIR) / (GREEN + NIR)
ndwi = cube.ndvi(nir="0", red="1")

ndwi_vis = (ndwi + 1) / 2

ndwi_png = ndwi_vis.linear_scale_range(
    input_min=0,
    input_max=1,
    output_min=0,
    output_max=255,
)

result = ndwi_png.save_result(format="PNG")

doc = GISDocument(latitude=40.75, longitude=-73.95, zoom=12)
await doc.ready()
doc.add_openeo_tile_layer(result)
doc

Define a process graph to compute a NDWI, using the Python API of OpenEO and JupyterGIS. It is then lazily evaluated on a per-tile basis while the user pans/zooms on the map.

In addition to defining openEO process graphs from the scripting Python API, JupyterGIS provides an advanced openEO process graph editor, allowing you to:

connect to an openEO tile server
define the graph graphically, with boxes and arrows
load data collections and define processes with a drag-and-drop UI
directly edit the JSON content

Editing an openEO process graph from the JupyterGIS front-end

Another interesting aspect of openEO is that process graphs have a well-defined, declarative JSON representation. Because of this structured format, they are a natural target for LLM-assisted workflows. Users can describe the analysis they want in natural language, have an LLM generate or refine the corresponding process graph (e.g. using jupyterlite-ai), and immediately visualize the result in JupyterGIS. Combined with the lazy, tile-based rendering, this makes it possible to quickly iterate on processing pipelines without waiting for complete datasets to be exported or downloaded.

Lazy visualization of Xarray datasets with jupyter-tiler

JupyterGIS now integrates with the new jupyter-tiler package, making it straightforward to visualize Xarray datasets from Python.

Datasets can come from anywhere: they may already exist in your notebook, or they can be loaded on demand from a STAC catalog using stackstac. Once you have an Xarray object, JupyterGIS can display it in the map without requiring an export to another format.

Rendering happens lazily, generating only the tiles needed for the current view. This makes it possible to explore datasets that are much larger than memory while keeping navigation responsive.

The result is a smoother workflow from data loading, to analysis, to visualization, all within the same notebook.

await doc.add_data_array_layer(
    name="NDSI Layer",
    data_array=ndsi,
    colormap_name="viridis",
    colormap_range=(-1, 1),
)

Visualizing an Xarray dataset in JupyterGIS.

This feature requires the optional dependency jupyter-tiler to be installed.

pip install jupyter-tiler

A more expressive symbology model

Styling geographic data often requires combining multiple visual properties to communicate patterns effectively.

JupyterGIS 0.16 introduces a new symbology model inspired by the Grammar of Graphics. Instead of relying on a fixed set of styling options, visual properties such as color, size, and opacity can be defined in a more flexible and composable way.

This makes it easier to build everything from simple thematic maps to more advanced visualizations while keeping styling definitions consistent and reproducible.

A symbology example: apply a Viridis color map to the circle colors, a linear scale to the radius of circles, and a fixed stroke color.

GeoZarr and GeoPackage support

This release also expands the range of formats that JupyterGIS can open directly.

Support for GeoZarr makes it possible to work with cloud-native multidimensional geospatial datasets, while GeoPackage support improves interoperability with existing GIS software and common data exchange workflows.

New Collaborative Editing Capabilities

JupyterGIS 0.16 also brings collaborative editing to vector layers. When working on a shared JupyterGIS document, multiple users can now edit the same vector data at the same time.

Features can be created, moved, and edited collaboratively, with changes synchronized in real time between users. This makes it possible to work together on tasks such as digitizing features, annotating a map, or refining a dataset without having to exchange files or manually merge changes.

Combined with the collaborative Story Map editor, this makes collaboration a more integral part of JupyterGIS: users can work together on the data itself, and then use the same shared document to explore and communicate their results.

A new R API

JupyterGIS 0.16 also introduces an R client, bringing JupyterGIS to R users through the new r-jupytergis package. The R client provides bindings for interacting with JupyterGIS widgets from an R notebook, using the same JavaScript front-end as the Python client.

The main interface is the GISDocument widget, which can be used to create and manipulate JupyterGIS documents directly from R. This makes it possible to build geospatial workflows in R while using the same interactive map interface available to Python users.

The R client also uses the same underlying collaborative infrastructure as the Python client, including the Yrs CRDT library. This means that R users can participate in the same collaborative JupyterGIS workflows rather than working in a separate environment.

doc <- GISDocument$new("france_hiking.jGIS")
layer <- doc$add_raster_layer(
    url = "https://mt1.google.com/vt/lyrs=y&x={x}&y={y}&z={z}",
    name = "Google Satellite",
    attribution = "Google",
    opacity = 0.6
)

Bug fixes and performance improvements

As usual, this release also includes many smaller improvements throughout the project.

We’ve fixed a number of bugs, improved performance in several parts of the application, and continued polishing both the user interface and the Python API.

JupyterGIS continues to evolve as a collaborative GIS environment that fits naturally within the Jupyter ecosystem. Whether your workflow starts from a notebook, a STAC catalog, an openEO backend, or a local dataset, the goal remains the same: make it easier to move between analysis, visualization, and communication without switching tools.

Acknowledgements

This work on JupyterGIS by QuantStack was funded by the European Space Agency (ESA) for the Story-maps development, the R API, openEO layers support and the collaborative labelling. Additionally, QuantStack was funded by the French National Centre for Space Studies (CNES) for the lazy visualization of xarray datasets.

Contributors to this release

By order of number of contributions:

Arjun Verma is a Developer at QuantStack. He worked on the server-side geoprocessing infrastructure and on the openEO editor in the JupyterGIS front-end.

Martin Renou is a Technical Director at QuantStack and a maintainer of JupyterGIS. For this release, Martin coordinated and guided much of the development, and worked on the integration of openEO layers.

Gregory Mooney is a Developer at QuantStack and a long-time contributor to JupyterGIS. He led much of the work on the new Story Map editor and its collaborative editing capabilities.

Matthias Meschede is Chief Operating Officer at QuantStack. He introduced the new Grammar of Graphics-inspired symbology model, bringing a more expressive and composable approach to styling geographic data.

Matt Fisher is the Community Manager of GeoJupyter. He contributed to many of the discussions around the release and helped shape several of the design decisions across the project.

Nakul Verma is an open-source contributor to JupyterGIS. He contributed numerous bug fixes and improvements throughout the release, and introduced support for Vega expressions in the new symbology system.

Antoine Prouvost is a Developer at QuantStack. He led the initial work on the R API for JupyterGIS, building its first skeleton and establishing the foundations for the r-jupytergis package.

Sandrine Pataut is a Developer at QuantStack. She worked on bringing many of JupyterGIS’s features to the R API, helping make the new client more complete and useful for R users.

We are grateful to everyone who contributed code, reviews, ideas, discussions, and feedback to this release. JupyterGIS continues to benefit from an increasingly diverse community of contributors, and we look forward to seeing what comes next!

Collaborative editing for GIS workflows with Jupyter and QGIS

2025-02-26T18:50:00+00:00

The QGIS open-source project is a cornerstone in the geosciences ecosystem, providing robust tools for spatial data analysis and visualisation. While QGIS excels in empowering researchers and engineers, the evolving landscape of geospatial research demands enhanced collaboration and accessibility, prompting a shift towards web-based technologies.

Collaboration in Jupyter

Collaboration has long been a key requirement for data science and scientific computing workflows, but Jupyter Notebooks historically lacked built-in real-time collaboration features.

To bridge this gap, the jupyter-collaboration project has focused on integrating multi-user editing capabilities directly into JupyterLab and Jupyter Notebook. This effort builds on foundational work in conflict-free replicated data types (CRDTs), leveraging technologies like Yjs, and required integration into the Notebook document model and WebSocket-based architecture.

Making Notebook code suggestions in collaboration mode with https://github.com/jupyterlab-contrib/jupyter-suggestions

This approach did not only enable collaborative editing of notebooks, but also opened the door for collaboration on other editing UIs. This was done e.g. in JupyterCAD, a 3D parametric modeler that allows multiple users to collaboratively design and manipulate CAD models within JupyterLab.

Read more about JupyterCAD here Announcing JupyterCAD 3.0

Collaboration in JupyterCAD

Introducing JupyterGIS

Today, we are excited to announce JupyterGIS, a web-based, collaborative, and extensible interface for GIS, leveraging the JupyterLab application framework and integrating seamlessly with the Jupyter notebook interface.

JupyterGIS provides the following key features:

a JupyterLab extension for collaborative editing of geospatial data files: with COG, Shapefile, GeoJSON data source support,
support for importing, exporting and collaborating with QGIS files,
advanced analysis and editing features with an identify tool and a symbology panel approaching what QGIS provides,
easily share and deploy GIS projects using JupyterLite.

Note: JupyterGIS is not an official Jupyter subproject.

GIS features

Raster and vector layers

JupyterGIS allows you to create multiple types of raster and vector layers, including local or remote GeoTIFFs, GeoJSON, shapefiles, and tile services.

You can select from a set of pre-defined tile services in our gallery, or you can provide your own source.

Identify tool and filtering

Akin to QGIS, we provide an identify tool and filtering capabilities, helping you to inspect your datasource.

Symbology

The symbology of a layer refers to its visual representation on the map, helping you to analyze your data comfortably. JupyterGIS provides a similar feature for its raster and vector layers.

Spatio-temporal data

You can easily animate and analyze your spatio-temporal data.

Python API

Even though you can do most operations from the UI, JupyterGIS also provides a Python API enabling scripting capabilities, which can be used to process data and operate on JupyterGIS documents programmatically from a Jupyter notebook.

Collaborative features

Building upon our previous work to enable collaborative editing in JupyterCAD, we developed a shared document model to support multi-user workflows in JupyterGIS.

Everything is shared between collaborators, from layers data to symbology and filtering rules.

We also provide some collaboration features like the ability to see other collaborators’ cursors, follow collaborators’ viewports, add geolocated Google Docs-like comment conversations, and see what layer features collaborators are identifying.

Try it!

You can directly try JupyterGIS in your browser, without installing anything, thanks to JupyterLite:

https://jupytergis.readthedocs.io/en/latest/lite/lab/index.html

Documentation

Check out the Python API reference and tutorials on how to get started in our documentation:

https://jupytergis.readthedocs.io

What’s next?

We are excited to welcome new contributors and see the JupyterGIS community grow. Here are some exciting developments on our radar for the future:

Support for more data source types.
Processing tools,
Support for suggestions by collaborators, similar to the inline suggestions feature of JupyterCAD,
An improved and more featureful Python API,
A better coverage of QGIS features.

How can you support our work?

Your support is important, if you would like to help us build more exciting tools around JupyterGIS, do not hesitate to contact us on the public geojupyter Zulip channel!

Additionally, you can join us for the bi-weekly GeoJupyter hackathon on Wednesday https://geojupyter.org/calendar.html

Acknowledgments

This work on JupyterGIS by QuantStack and Simula Research Lab (Simula) was funded by the European Space Agency (ESA) for our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS”.

We are also grateful to The Eric and Wendy Schmidt Center for Data Science & Environment (DSE) at UC Berkeley for funding the work of Matt Fisher on the project.

Finally, this wouldn’t have been possible without the foundational work on JupyterCAD initiated by Duc Trung Le.

About the JupyterGIS contributors

Meriem Ben Ismail is a Scientific Software Developer at QuantStack. She just completed her Software Engineering degree at INSAT (National Institute of Applied Sciences and Technology in Tunisia) and her six-month internship as an open-source scientific software engineer at QuantStack.

Nicolas Brichet is a Scientific Software Developer at QuantStack. He holds a master’s degree in robotics from the University of Montpellier. Prior to joining Quantstack, Nicolas worked for almost fourteen years in French research institutes (INRAE and CNRS), as software developer for scientific research. He also worked on different parts of the data lifecycle, such as data acquisition, analysis, and storage.

David Brochart is a Technical Director at QuantStack. David used to work in the semiconductor industry as a digital circuit designer, mostly using FPGAs. He studied telecommunications at the French “Telecom Bretagne” engineering school, with major in digital circuit design. David also worked as a researcher in hydrology at Irstea, a French scientific research institute, where he heavily used the Python scientific stack.

Matt Fisher is a Research Software Engineer and Community Manager at Schmidt Center for Data Science & Environment. Matt is passionate about open science, community-owned software, teaching and learning, accessibility, and inclusion. It is his treasured privilege to support researchers and educators in their missions by working on projects like QGreenland and JupyterGIS!

Anne Fouilloux, a Senior Research Engineer, specializes in Open Science, FAIR data, and Big Data analytics. With a PhD in Atmospheric Physics, she has contributed to Pangeo and EOSC (European Open Science Cloud), focusing on data management, Earth system modeling, and scalable computing. A strong advocate for Open Science, she advances data accessibility and interoperability to drive scientific innovation.

Greg Mooney is a Scientific Computing Developer at QuantStack. He received a Bachelors degree in Software Engineering from Arizona State University (ASU). Before joining QuantStack, he worked as an Aegis Computer Network Technician, working on the network infrastructure connecting various sensor systems onboard naval vessels.

Martin Renou is a Technical Director at QuantStack and a maintainer of Project Jupyter. Among other projects Martin is a core team member of the ipywidgets project and maintains many Jupyter widget packages such as ipyleaflet, ipydatagrid, ipygany, ipycanvas, and bqplot. He is a co-creator of the Voilà dashboarding system, and the xeus-python kernel.

Arjun Verma is a Scientific Software Development Intern at QuantStack. He is also a final year undergraduate student at Cluster Innovation Centre, University of Delhi. He loves 3D & GIS and also contributes to JupyterCAD and is one of the maintainers of PyBaMM where he has also been GSoC contributor and mentor.

Announcing JupyterCAD 3.0

2025-02-17T09:38:00+00:00

We are thrilled to introduce JupyterCAD 3.0, the newest version of the collaborative CAD modeler designed for JupyterLab.

JupyterCAD is an unofficial JupyterLab extension for 3D geometry modeling with collaborative editing support. It is designed to allow multiple people to work on the same file at the same time, and to facilitate discussion and collaboration around the 3D shapes being created.

JupyterCAD 3.0 represents a significant leap forward in usability and user experience compared to the previous release.

What’s new in JupyterCAD 3.0?

Color customization

JupyterCAD 3.0 allows you to customize object colors directly within the application, while we were limited to gray models in earlier versions. Adjust colors easily to suit your design needs or preferences.

Embedded Python console

The new embedded Python console offers advanced users direct access to JupyterCAD’s Python API, enabling on-the-fly scripting and automation for power users.

UX improvements

The new AxesHelper provides a clear visual reference for coordinate axes, making it easier to navigate and align models in 3D space.

We have also added copy-paste functionality using Ctrl+C and Ctrl+V shortcuts, making it faster and easier to duplicate objects within your models.

An improved toolbar now includes options to toggle wireframe, toggle transform controls, toggle clip plane and a toggleable console, giving users quick access to essential tools.

JupyterCAD 3.0 also introduces smooth camera tracking, where selecting an object from the object tree triggers a smooth transition of the camera to focus on the selected object, enhancing navigation and model exploration.

Mouse-based 3D controls

Move or rotate your 3D models effortlessly with mouse-based controls that bring fluidity and precision to model editing and interaction.

Suggestions support

With JupyterCAD 3.0, collaboration becomes even more seamless with suggestions. Collaborators can propose changes to a model, which can be reviewed and applied collaboratively, fostering a dynamic and efficient workflow.

Try it!

Screenshot of JupyterCAD GitHub repository, with a link to try JupyterCAD (circled in red) with JupyterLite. This is simply deployed as a static website on “GitHub Pages”.

Explore JupyterCAD 3.0 in action with our JupyterLite deployment — no installation required. Visit us at jupytercad.github.io/JupyterCAD and start modeling directly in your browser.

Installing JupyterCAD

You can easily install JupyterCAD with pip or mamba:

pip install jupytercad

or:

mamba install -c conda-forge jupytercad

Additionally, you can install FreeCAD along with the JupyterCAD-FreeCAD plugin to enable support for FCstd files:

mamba install -c conda-forge freecad jupytercad-freecad

What’s next?

JupyterCAD is evolving rapidly, with some exciting updates just around the corner and others in active development:

Upcoming Features

Enhanced User Experience:
- Smoother exploded view
- Support for renaming objects directly from object tree
- Incremental rotation controls for precise adjustments
- Enhanced 3D view settings for better customization.
Nested & Refined Side Panel: Navigate and organize objects more easily with a refined and nested side panel experience.
Support for more File Formats: Broaden compatibility with support for more file formats.

Features in Development

New CAD Operations: New tools for operations like linear and radial patterns to expand modeling capabilities.
Support for measuring distances: Tools to measure distances and dimensions directly in the viewport.
Improved 2D Sketcher: Enhanced features to make 2D sketching more intuitive and robust.

Acknowledgements

The development of the JupyterCAD open-source project by QuantStack was funded by a grant from DGAC in the framework of the French “Plan de Relance” (refinanced by Next generation EU) as part of a collaboration between Safran Aircraft Engines, INRIA, and Akkodis.

We also extend our gratitude to the external contributors whose efforts and feedback have been instrumental in shaping this release. Your support and collaboration drive the ongoing success of JupyterCAD.

This work was built upon the strong foundation laid by open-source software projects such as Three.js, OpenCascade, OpenCascade.js, FreeCAD, JupyterLab and JupyterLite.

About the authors

JupyterCAD is an open-source project resulting from contributors’ collective efforts. Below, we highlight the primary contributors for this release:

Le Duc Trung is a Scientific Software Developer at QuantStack. He works on several projects within the Jupyter ecosystem, from the main projects like JupyterLab, Voilà, and ipywidgets to various JupyterLab extensions and widgets.

Arjun Verma is a Scientific Software Development Intern at QuantStack. He is also a final year undergraduate student at Cluster Innovation Centre, University of Delhi. He loves 3D & GIS and also contributes to JupyterGIS and is one of the maintainers of PyBaMM where he has also been GSoC contributor and mentor.

Exploring a Document’s Timeline in JupyterLab

2024-09-13T19:15:00+00:00

In collaborative environments, keeping track of changes and understanding the evolution of a document is crucial. This is especially true in fields like scientific computing and data science, where multiple contributors work together on complex projects. To address this need, we’re excited to introduce a document timeline component, a new feature that ships with the Jupyter Collaboration extension and enhances how teams interact with shared documents.

The timeline component is designed to give users control over a document’s history. Positioned as a slider in the JupyterLab status bar, it allows users to navigate through the timeline of a document. You can explore past versions, compare changes over time without altering the original document, and even restore specific versions. This feature is compatible with all types of shared documents within Jupyter, including notebooks, text files, and even third-party document types like JupyterCAD models.

Exploring the history of a Jupyter Notebook with the timeline component

How Does it Work ?

Co-editing of notebooks and other documents was introduced in JupyterLab 3.1 and has been improved and consolidated in subsequent releases. The feature can be enabled by installing the Jupyter Collaboration extension, which is built upon the Yjs framework, an implementation of CRDTs (Conflict-free Replicated Data Types). An important byproduct enabling real-time collaboration was the introduction of a journal of updates capturing all changes made by collaborators, which we leveraged to implement document timeline navigation.

Two key changes to Jupyter collaboration underlie the timeline component:

The ability to fork the document history, which is required to visualize the past state of the document without altering it.
The support of Undo/Redo operations in the Python implementation of Yjs, PyCRDT, through the implementation of an undo manager. While this was already available in Yjs, it was missing in PyCRDT until now.

As you move the slider to go back in time, the first operation that takes place is forking the document. Forking creates a new branch of the document at the selected point in history. This approach ensures that the most recent version of the document remains unaltered while you explore earlier versions. The forked version essentially acts as a sandbox where you can navigate through the document’s history, test out changes, and decide on the best course of action without modifying the main document.

When you navigate through the document’s timeline using the slider, the undo manager allows you to reverse or reapply any series of changes, stepping back through the document’s history without losing any information. If you decide to move forward again, the undo manager reapplies those changes in the exact order they were originally made, preserving the document’s integrity.

After reviewing the timeline, you can choose to restore a particular version at a selected timestamp. This restored version becomes the new active state of the document, effectively merging the changes from the forked branch back into the main document.

The document timeline component in the status bar offers an intuitive, flexible way to manage document history. Users can confidently experiment with different versions of their work, knowing that every action is reversible and that no progress will be lost. Whether you’re refining a piece of code, iterating on a design, or reviewing past contributions, the timeline equips you to iterate efficiently and effectively.

Seamless Integration with JupyterCAD

The document timeline is not just limited to text-based documents or notebooks. It was designed in a document-agnostic way, allowing its usage for any collaborative document. Consequently, it integrates seamlessly with JupyterCAD, JupyterLab’s extension for creating and manipulating 3D models. For engineers and designers, this integration is a game-changer.

In complex engineering projects, design iterations are common, and the ability to track the evolution of a 3D model is crucial. With the document timeline, users can effortlessly navigate through different stages of their design process. If a recent change in the 3D model doesn’t yield the desired results, the document timeline allows you to slide back to a previous version and pick up your work from there.

By combining the power of JupyterCAD with the document timeline’s history management, a team can achieve a more dynamic and flexible design process, ensuring that every idea is captured and every change is reversible. This integration exemplifies how the Jupyter Collaboration extension is enhancing not just collaborative editing but also specialized workflows in engineering and design.

Exploring the history of a JupyterCAD document with the timeline component

A New Era of Collaboration in Jupyter

The document timeline is a significant step forward in our ongoing work to make Jupyter a user-friendly platform for collaborative technical computing. By leveraging the advanced capabilities of the PyCRDT undo manager, we’ve created a tool that simplifies document history management and enhances the overall collaborative experience.

As we continue to innovate and expand the Jupyter Collaboration extension, we invite the community to try out the new document timeline feature and share their feedback. Together, we can further refine these tools and ensure that Jupyter remains at the forefront of collaborative computing, empowering users across all disciplines to achieve more.

Acknowledgments

The development of the document timeline and its integration into the Jupyter Collaboration extension has been a true team effort, made possible by the dedication and expertise of many individuals. Special thanks are extended to the following team members for their invaluable contributions:

David Brochart : For their innovative approach to integrating the undo manager within PyCRDT.
Afshin T. Darian : For their supervision and feedback, which have been crucial in refining the functionality of the timeline component.
Duc Trung Le : For their expertise in JupyterCAD and for ensuring that the timeline component seamlessly integrates with 3D model workflows.

About the Author

Meriem Ben Ismail just completed her Software Engineering degree at INSAT (National Institute of Applied Sciences and Technology in Tunisia) and her six-month internship as an open-source scientific software engineer at QuantStack.

JupyterGIS

2024-06-12T17:09:00+00:00

We are thrilled to announce that the European Space Agency (ESA) is funding our proposal “Real-time collaboration and collaborative editing for GIS workflows with Jupyter and QGIS.”

The consortium spearheading this project comprises QuantStack and Simula Research Lab (Simula), two organizations with a long history of contributions to the Jupyter project and the broader open-source scientific computing ecosystem.

The goal of the project is to build solid foundations for a versatile web-based user interface for Geographic Information Systems (GIS) workflows. It will comprise several components, including a JupyterLab extension for collaboratively editing QGIS project files and the integration of these APIs in the Jupyter Notebook. We will then explore integration with platforms like the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC).

Collaborative workflows in geosciences

Collaborative editing of documents has become an integral part of our digital lives and has made us collectively more productive. Gone are the days of cumbersome email exchanges with documents shuttling back and forth.

Looking ahead, the potential of co-editing extends far beyond text documents and will apply to all authoring UIs, from CAD to image processing. We think that the shift to collaborative editing will be even more transformative for larger and more complex projects, which are inherently social and require the concerted effort of large teams. Whether you are designing a stadium, a plane, or an ocean liner, you need to coordinate a diverse expertise to build a unified model. In geosciences, it may range from climate modeling to agriculture, ecology, urban planning, and many more areas of expertise. For such endeavors, we must embrace tools favoring collaboration, and this applies to the future web-based user interfaces for geoscience research.

We have been working on collaborative editing in the core of JupyterLab for the past three years. Our approach is based on the Yjs framework, an implementation of CRDT data structures (Conflict-free Replicated Data Type). We have learned that since it is so tied to the data model, retrofitting these features into an existing application is considerably more arduous than building the initial data model on the appropriate paradigm from inception. This is why JupyterGIS will be built from the ground up with collaborative editing in mind.

JupyterGIS will be the first open-source GIS tool to provide collaborative editing features.

Importantly, JupyterGIS is part of a broader mission to enable collaborative workflows in open-source technical computing. The JupyterCAD project, a collaborative CAD modeler, is another example of this endeavor. By sharing knowledge and resources, both JupyterGIS and JupyterCAD will benefit from and contribute to each other’s progress, ultimately driving innovation and productivity in their respective fields.

From the desktop to the web

Advanced authoring tools, including IDEs, CAD modelers, image processing software, and GIS applications, are essential for professionals who rely on them for extended periods. These users have high expectations and demand key features to optimize their productivity and workflow. Among these features are extensibility with plugins, configurable keyboard shortcuts, themability, internationalization, scriptability, a unified settings system, and the ability to operate across multiple browser windows and devices.

Developing a new application from the ground up that meets all these requirements is a formidable challenge. This is where the JupyterLab application framework proves invaluable. By leveraging this framework, developers can build custom, feature-rich authoring tools that incorporate these essential features from the outset, making it an ideal solution for creating modern, user-centric applications.

Beyond the pure authoring user interface, for which JupyterLab is a great tool, it is also the perfect tool for integration with notebook-based workflows. As part of this project, we will develop an advanced Python API to manage JupyterGIS sessions, leveraging Jupyter’s robust display system to incorporate sophisticated GIS features inline in Jupyter notebooks and consoles. This integration will further enhance the capabilities and versatility of our application. This feature will follow the same architecture as that of JupyterCAD for the integration with the notebook user interface.

Real-world applications

JupyterGIS is meant to become a general-purpose tool. However, we will work with teams of practitioners on specific real-world applications to ensure that it addresses their needs.

One such project concerns the use of digital and GIS solutions for emergency management. While practitioners involved in emergency management already have experience with such tools, we are convinced that web-based applications built with collaboration in mind from the start can foster improved coordination and collaboration, and therefore the effectiveness of the response.

Response teams comprise different organizations, including the affected municipalities, police, emergency response organizations like the Red Cross, and advisers such as the U.S. Army Corps of Engineers (USACE) in the United States or the Norwegian Water Resources and Energy Directorate (NVE) in Norway. Even when using the same incident management system, miscommunications and misunderstandings can occur between these stakeholders, resulting in delays or improper use of resources. An improved integration of collaborative GIS software can improve upon the existing tooling.

Building collaborations in Europe and beyond

We will partner with key practitioners and make sure JupyterGIS addresses their use cases. Integration in the Copernicus Data Space Ecosystem (CDSE) and the European Open Science Cloud (EOSC) will be key to the adoption of JupyterGIS and demonstrate its deployment in such environments.

Beyond Europe, we are currently working on including this project into a broader scope. We are excited to be partnering with the “LIVE-Env” project, an open-source initiative spearheaded by Alyssa Goodman at Harvard University. Furthermore, we are establishing a key collaboration with the Berkeley Institute for Data Sciences (BIDS) and the Schmidt Center for Data Science and Environment (DSE) at UC Berkeley and 2i2c, with a focus on Jupyter-based geosciences.

Affiliations

Sylvain Corlay: QuantStack
Anne Fouilloux: Simula
Monika Weissschnur: Simula

Collaborative CAD in JupyterLab

2023-06-02T15:15:00+00:00

We are thrilled to introduce JupyterCAD, a tool that integrates Computer-Aided Design (CAD) capabilities into JupyterLab. With its JupyterLab extension, dedicated JupyterCAD application, and Python API for CAD operations, JupyterCAD allows users to effortlessly create, edit and share 3D designs without leaving the Jupyter ecosystem.

This blog post provides an overview of JupyterCAD’s features and highlights its collaborative editing capabilities which enable teamwork in the realm of CAD.

Everything that is shown below can be tested by installing JupyterCAD from PyPI:

# From PyPI
pip install jupytercad

One extension, three interfaces

Built using the latest JupyterLab 4 release, JupyterCAD combines CAD functionalities with the enhanced performance and accessibility improvements introduced in JupyterLab 4. The flexibility of JupyterLab allows us to create the extension once but deploy it in multiple ways to target three groups of audiences: users who prefer working in the JupyterLab environment, users who favor dedicated applications, and those advanced users who do everything programmatically.

JupyterLab Extension

Experience CAD features directly within JupyterLab with the JupyterCAD extension. Open and edit FreeCAD files, perform CAD operations, including creating 3D primitives, applying boolean operators, and exploding the view. JupyterCAD’s extension integrates seamlessly into JupyterLab, providing an intuitive environment for CAD enthusiasts and data scientists alike.

JupyterCAD extension inside JupyterLab

JupyterCAD Application

For a streamlined CAD experience, JupyterCAD offers a dedicated application that eliminates distractions and emphasizes CAD features.

The standalone JupyterCAD Application.

Starting the dedicated JupyterCAD application is as simple as starting JupyterLab or Jupyter Notebook:

jupyter cad

This dedicated application has been built in a similar way to the coming Jupyter Notebook 7. It’s a JupyterLab “remix” built from the ground up using JupyterLab core components and custom extensions. Much like JupyterLab, it offers theming and localization support, and more planned features on the horizon.

Python API for CAD Operations

Advanced users can unlock the full potential of CAD operations with JupyterCAD’s Python API. JupyterCAD allows users to programmatically visualize, create, and manipulate the shapes from a Jupyter Notebook. For example, one can open a FreeCAD file and start modifying it from the notebook with:

from jupytercad import CadDocument
doc = CadDocument('example.FCStd')
# Create a cone, a sphere then cut the cone with the sphere.
doc.add_cone().add_sphere(radius=0.8).cut()
display(doc)

Adding object to FreeCAD file from Jupyter Notebook.

JupyterCAD API also integrates nicely with the OpenCascade Python API, providing expanded capabilities for working with shapes.

from jupytercad import CadDocument
...
# Create a prism shape with OpenCascade
prism = BRepPrimAPI_MakePrism(profile, vec).Shape()
doc = CadDocument()
doc.add_occ_shape(prism)
display(doc)

Visualize OpenCascade shape object with JupyterCAD.

Unlocking teamwork with real-time collaboration

Collaborative Editing

One of the standout features of JupyterCAD is its shared editing functionality, which seamlessly connects users across different interfaces within the JupyterCAD ecosystem. Whether collaborators are using the dedicated JupyterCAD application, the JupyterLab extension, or working with the Python API in a Notebook, any changes made to a shared document are instantly reflected for all users.

Real-time collaboration allows individuals working in the Python API to make modifications to the CAD document, while simultaneously providing a synchronized view for collaborators using the JupyterLab extension or the JupyterCAD application. This ensures that all participants have access to the most up-to-date version of the design, fostering efficient communication and eliminating the need for manual synchronization or file exchange.

Real-time collaborative editing in JupyterCAD.

Follow Mode

With JupyterCAD’s follow-mode feature, collaboration becomes even more fluid. You can follow another user’s camera movements and view adjustments in real-time. This mode allows you to gain insights into the design process, and enhance communication among team members.

Follow Mode in action.

Annotations System

The annotations system in JupyterCAD adds an interactive layer to 3D designs. Now you can add annotations to specific shapes within a CAD file, give context, provide feedback, or write instructions.

Adding annotation at precise positions

What goes on under the hood?

All the above features are made possible thanks to two major open-source components: OpenCascade.js, the WebAssembly build of OpenCascade, and jupyter_collaboration, the real-time collaboration framework of JupyterLab.

In-browser geometric modeling kernel

To execute all geometric operations, JupyterCAD uses a custom build of OpenCascade.js, which is a port of the OpenCascade library to JavaScript and WebAssembly via Emscripten. Running on a separate thread, the CAD Kernel of JupyterCAD allows users to perform complex operations at near-native speed. It also helps to lower the load of the server while serving JupyterCAD to many users.

JupyterLab Real-Time Collaboration framework

In JupyterLab 4, Real-Time Collaboration (RTC) is not only about editing notebooks but it has become a framework for building collaborative applications. Using the components of jupyter_collaboration helps us accelerate the development of the RTC features in JupyterCAD. Collaborative Editing is built upon a shared data model where jupyter_collaboration handles all the transportation and conflict resolution, while Follow Mode makes use of Awareness, a lightweight notification system provided by the library.

Future work

The current state of JupyterCAD illustrates the capabilities of JupyterLab as a foundation to build complex applications, not just for scientific computing but also for more general technical computing applications.

We will continue to improve JupyterCAD on two axes:

Enhancing the 3D viewer with features to help users interact with the objects
Enriching the Python API to improve interoperability with other CAD libraries

We plan on upstreaming some of the features of JupyterCAD, like the annotations system and the Follow Mode, to jupyter_collaboration to enable it in other file contexts in JupyterLab (Notebooks, text files, etc).

User feedback from the community also plays a big role in the project roadmap. Try JupyterCAD and share your feedback with us using the project’s GitHub issues!

About the Authors

Martin Renou is a Technical Director at QuantStack and a maintainer of Project Jupyter. Among other projects Martin is a core team member of the ipywidgets project and maintains many Jupyter widget packages such as ipyleaflet, ipydatagrid, ipygany, ipycanvas, and bqplot. He is a co-creator of the Voilà dashboarding system, and the xeus-python kernel.

How we made Jupyter Notebooks collaborative with Yjs

2021-06-04T18:46:00+00:00

Collaborative editing — à la Google Docs — is a feature that you still rarely find in applications. One of the few good things that came out of this pandemic is that more people seem to care about making their applications fit for remote collaboration. Of course, they always cared about real-time collaboration. It’s just a very hard feature to add to your application. It took the Jupyter project several years to land this feature. Finally, they ended up with a solution that is based on the Yjs Framework which I authored. This article gives an overview of all the work that was put into Jupyter Notebooks and finally describes how we want to make even more components collaborative.

Jupyter Notebooks started in 2011 (back then IPython Notebook) as an effort to make data science reproducible and more accessible by visualizing data dynamically in a notebook. Basically, it allows you to write markdown and code directly in a document. The code can be executed interactively directly from the notebook and shows the result immediately below the code. This is great for data scientists that want to share their research with others. But it’s also a great tool for learning programming because it doesn’t require setting up a programming environment.

Right from the beginning, collaborative editing was on the agenda for Jupyter Notebooks. In 2012, core Jupyter contributor and creator/lead of JupyterHub, @minrk, wrote in the GitHub issue tracker:

[…] This will finally make decent live collaboration feasible, which is our single most-requested and highest-priority new feature. (source)

Back then, everyone tried to replicate Google Docs’ collaborative editing functionality. Google Docs was released in 2006 and was the first web application that supported collaborative editing on rich text. In many ways, Google Docs was ahead of its time. It will take years for others to reproduce this functionality. Even today, collaborative editing is far from being ubiquitously available, even though the technology has been available since the ‘80s.

No wonder, that the first collaborative Jupyter Notebook implementation, Colaboratory (or Colab), was created by Google engineers. They rewrote the UI for Jupyter Notebooks and gave it a collaborative notebook model via Google’s Realtime API, which was deprecated in 2017. This history underscores how challenging it is to build real-time collaboration into applications: when the Google Realtime API was deprecated, Colab lost its real-time collaboration capabilities, a gap which continues to this day.

In 2013 William Stein launched CoCalc, a Jupyter notebook service with collaborative editing support right from the beginning. Like Colaboratory, CoCalc wrote a new UI for Jupyter Notebooks, while reusing other parts of the Jupyter architecture. They made different choices and implemented a custom solution for conflict resolution. I highly recommend watching the below talk where William Stein shared his experience about the state-of-the-art solutions for shared editing back then.

Exploration of different solutions for collaborative editing by CoCalc.

Still, the open-source JupyterLab project didn’t include collaborative editing. What follows is a series of discussions about the best shared-editing solution to integrate into Jupyterlab to make collaborative editing available to the users of the open-source project. In 2017 Brian Granger, Chris Colbert, and Ian Rose shared their work that integrated the Google Realtime API into the existing JupyterLab project, where they showed an awesome demo of what collaborative editing could become.

During my research, I found several PRs by Ian Rose that separated the view (how the Jupyter editor is rendered) from the model (how the data is represented). I haven’t talked to him, but I assume what he found is that it is helpful to have observable data structures that can be synced using some framework for conflict resolution. In this case, he just happened to use the Google Realtime API for synchronization. His work included the abstract factory IModelDB for creating observable data structures that are used to this day. In theory, one just needs to implement the IModelDB interface with observable data structures that synchronize automatically through some real-time API.

But the provided solution was still based on a proprietary API that requires you to hand over your data to Google services. So the Jupyter community was looking into implementing their own solution for conflict resolution. Different people started to explore Concurrent Replicated Data Types (“CRDTs”) for automatic conflict resolution on their observable data structures. This technology has become very popular in recent years as a solution to synchronize data that can be manipulated by many peers at the same time. If you are interested in the topic, I recommend reading some introductory material on https://crdt.tech/.

Lumino (formerly PhosphorJS) is a JS toolkit that underlies the JupyterLab IDE, by providing a rich toolkit of widgets, layouts, events, data structures, and a plugin system at the foundation of the JupyterLab extension system. Specifically for our use-case, it provides observable data structures that are used as a model for all Jupyter packages. In 2017, Chris Colbert started the ambitious endeavor to build high-performance CRDT data structures that can be used as an observable data model. In theory, we could have used that to make any application, that is based on Jupyter data structures, collaborative. Although the Lumino CRDT is little known, to this day it remains the second-fastest CRDT implementation that works on the web.

jupyterlab/lumino

In 2019, Vidar Tonaas Fauske, Ian Rose, and Saul Shanabrook started work to integrate the Lumino CRDT into JupyterLab. Their work lived for a time in JupyterLab#6871 and has later been moved to a separate repository JupyterLab/rtc.

This is basically where I come in. While the Lumino CRDT is pretty awesome, in 2020 Brian Granger created a Lumino CRDT performance benchmark that revealed critical performance and algorithm issues (such as the so-called interleaving anomoly). In the process, Brian discovered my CRDT implementation Yjs and the two of us began to discuss CRDT implementations and Yjs in particular.

To add collaborative editing functionality rivaling Google Docs, we need a bunch of features aside from automatic conflict resolution. For example, we expect that we never revert changes from other users when we hit the undo button. So we need a selective undo-manager that somehow ignores changes from remote users. Something like this is really hard to implement correctly on top of a CRDT.

Brian eventually asked me to work with QuantStack to bring collaborative editing to JupyterLab. Yjs has ready-to-use solutions for most problems related to building collaborative applications and is the only CRDT implementation that beats the Lumino CRDT in performance.

yjs/yjs

When I heard that JupyterLab was pushing for collaborative editing for 8 years, I was determined to produce results as fast as possible. I appreciate all the work that came before me because the codebase was already clearly separating the view from the model. My work was just to exchange the existing observable data structures with Yjs’ shared types (which is a fairly similar concept). After one month of work, I was able to produce the first prototype.

[WIP] Collaborative editing using Yjs by dmonad · Pull Request #9785 · jupyterlab/jupyterlab

But there was a problem. Another group, led by Eric Charles, also acquired funding to work on collaborative editing and they chose another approach. While I simply replaced the existing observable data structures, they were trying to reuse the existing data structures. I wanted to make full use of Yjs’ features and didn’t want to build extra abstraction layers just to be able to switch to another CRDT implementation. For some time, it seemed we could not reconcile our approaches.

After many discussions with Eric, we finally came up with a compromise that I’m now really excited about. Yjs and ModelDB only provide raw data structures to build collaborative applications. Our plan was to build a notebook model with an easy-to-use API to manipulate, observe, and synchronize changes on the notebook. This would make it possible for other applications to keep compatibility with JupyterLab without forking the whole JupyterLab repository. My hope is that other notebook-related products like CoCalc or VSCode will eventually use this collaborative model to provide cross-compatibility with other Jupyter services. Everything collaborative, of course.

Since February Eric Charles, Carlos Herrero, Jeremy Tuloup, and I have been working on designing and integrating this collaborative model into JupyterLab. Others can use the @jupyterlab/shared-models package from the npm registry to build their own interfaces for Jupyter Notebooks using the same shared editing technology. Our changes have finally been merged into JupyterLab and are already available in the alpha releases of JupyterLab v3.1.0. Simply start JupyterLab with the --collaborative flag to enable collaborative editing.

@ProjectJupyter is collaborative! JupyterLab just merged our collaborative-editing branch that @echarles @CarlosHerreroB @jtpio and me have been working on.https://t.co/7Tm0WXnfkS

— Kevin Jahns (@kevin_jahns), May 10, 2021

Making the separation between model and shared data structure has been quite a revelation for me. Yjs’ shared types are very powerful and allow you to make any kind of application collaborative. But shared models that define an application-specific API make it easier for developers to manipulate the data without understanding how the data is represented in the CRDT. This is particularly relevant because CRDT implementations are almost always schemaless (Cambria being the exception). A well-maintained model could ensure that the model is compatible with previous versions. In the future, I want to define more shared models for things that are not trivial to represent in Yjs like calendars, contacts, drawings, and graphs.

Next steps

I’m currently working with Bartosz Sypytkowski on a Rust implementation of Yjs. The Rust implementation will be the baseline for all other ports of the Yjs CRDT to other languages. Thanks to Pierre-Olivier Simonard and PyO3 we already have the template to create language bindings from the Yrs CRDT to a Python package “y-py”. In the coming months, we will implement a Python CRDT that is fully compatible with the web-based CRDT that is used in JupyterLab. This will allow for backends and frontends to efficiently exchange data that can be manipulated simultaneously.

yjs/y-crdt

Currently, we still send HTTP PUT requests to save a document to the file system via the Jupyter Server. Once we have the Yjs CRDT working in Python, we will create a Python implementation of the shared notebook model which will allow the Jupyter Server to directly access the collaborative state and synchronize that with the file system. Moreover, this will finally resolve a long-awaited issue that allows the kernel to write the output directly to the notebook without first connecting to a client. Why is that important? This will enable you to run a notebook and then close the browser to run your computations overnight. The kernel will compute the output in the background, and write it to the shared model which is then saved to the filesystem by Jupyter Server.

We want to provide more than a collaborative text editing experience. Yjs provides the data structures to make any kind of application collaborative. Users of Yjs use it to build collaborative drawing-, and diagraming- solutions. Relm.us, for example, uses the collaborative data structures that Yjs provides for modeling a 3D world that thousands of users can visit to work together. We are exploring heavily how we can make Jupyter widgets collaborative using the same technology.

Carlos Herrero is working on a drawio widget for JupyterLab that automatically synchronizes using a custom shared model that he developed. We want to produce documentation on how you can create custom collaborative widgets for JupyterLab. Jupyter widgets will be able to leverage the rich ecosystem of the Yjs CRDT to create widgets. You want, for example, to add a WYSIWYG rich-text editor widget? Sure, just add the awesome TipTap editor to your widget, as it already uses Yjs as the default shared editing technology.

Jeremy Tuloup is working on a WASM-powered JupyterLab distribution that I’m particularly excited about. JupyterLite runs entirely in the browser using only static assets. The code cells are executed using a WebAssembly-based Python runtime. This is already awesome mad science. Now he integrated our collaborative editing approach but uses WebRTC to synchronize peers without the need to set up a central server for conflict resolution. This gives you an offline-ready, collaborative editing experience without setting up any server. Pretty cool!

Real time collaboration in JupyterLite coming soon: https://t.co/rbEvEPMv17

— Jeremy Tuloup (@jtpio), May 28, 2021

I hope this got you excited about the future of Jupyter. For sure, I am.

At last, I hope we can appreciate the amazing open-source work that so many people put into this project. I’m looking forward to a future where people take collaborative editing for granted. It has been a rough road and we all needed to learn our lessons to arrive at a solution that works.

Funding acknowledgement: My work on this effort at QuantStack has been funded by Schmidt Futures and the Alfred P. Sloan Foundation through grants to Cal Poly San Luis Obispo.