Robot development with Jupyter

This post shows the available tools in the Jupyter ecosystem to build advanced visualizations in Jupyter Notebooks and standalone web apps using Voilà, and how to deploy those apps to the robotics cloud.

5 min readDec 31, 2019

Jupyter and Voilà are two popular packages in the data science ecosystem. But the Jupyter plugins jupyter-ros and jupyter-amphion make them more interesting than ever for robotics applications. This blog post shows how to use them and how to create your first cloud-deployed application using rapyuta.io, the robotics cloud.

Creating a interactive Jupyter Notebook

Using Jupyter and Jupyter widgets, we can easily create a widget to display and control a robot. For example, we might want to be able to send speed and orientation to the robot in order to make it to go to some place in space, or set the joint coordinates for a robotic arm.

With ipywidgets we can create simple widgets like sliders, text or number inputs, and create layouts using containers, accordions or tabs (you can check out the widget list here).

For example, below you find the code to create a interface to accept three floating point numbers, and send them to a robot using ROS.

A simple widget for sending x and theta values

Now, we haven’t implemented any callbacks just yet. This can be done using the on_click event on the buttons. On the click, we just ask for the .value property of the text input fields and take action! We will see this in a short while — first let’s get some robot visualizations going.

Robot visualizations in the web browser

In order to create some robot visualizations we can use jupyter-ros’s 3D widgets, or the new and fancy jupyter-amphion widgets.

For this we need to create a websocket connection to the rosbridge (usually ws://localhost:9090 ). Then creating a TfViewer widget on top of this connection, with the odom frame as the fixed frame.

In the following cell, we see how the layout can be influenced by passing a couple of CSS parameters to the layout trait of the widget. Things like border thickness and color, or widget height can be set directly.

Then we add some objects to the viewer, in particular a robot model (the specific robot model is picked up automatically from the /robot_description parameter, but we need to serve the mesh file packages through a URL, in this case the Jupyter server also serves mesh files under the rospkg endpoint (a feature of the jupyter-ros notebook extension), and we display a odometry visualization (as in RViz, it’s the red arrow).

A turtlebot visualization using jupyter-amphion

Adding some callbacks

Now, if we want to interact with the robot, we can connect the previous widget for x and theta values to the real robot and publish ROS messages to the appropriate endpoints.

The callback will create a Twist ROS message that takes the x and theta value from the widget on click and publish to the /cmd_vel topic. This is a very simple function, but in theory one could do any sort of complex computation using the full power of the Python data science ecosystem in the callback —in some way it really is an ad-hoc way to create a backend for a “website”.

Now clicking on the “Go” button will move the robot!

Creating a standalone Voilà application

Voilà is a package that, in the basic sense, takes a Jupyter Notebook, removes all source code, and only leaves you with the widgets. This means, you end up with a web application with a Python backend (the backend are the callbacks from ROS or from the widgets!). As with the widgets in the Jupyter Notebook, the widgets in Voilá automatically sync the state between the front-and backend.

To run this Notebook as a Voilà application, just execute voila --enable_nbextensions=True mynotebook.ipynb from the command line. This will fire up a webserver (by default running on port 8866). After Voilà finishes executing all your cells, you should be left with a view of the widgets, just like in the notebook, and all buttons and callbacks will work just like they do in the notebook.

This is extremely powerful because you could run Voilà from a shell script as part of your launch file to run a specialized web-app for your robot environment (similar to RViz). The big difference is that this web-app will be accessible from any computer that has a web-browser installed (such as Windows computers).

You can also deploy Voilà applications to a cloud provider, or embed it side-by-side on your robotics cloud (for example, a Voilà deployment should be straightforward using the rapyuta.io robotics cloud!). If you deploy a Voilà app in the rapyuta.io cloud, it will be able to communicate with your robots using regular rospy, as if you were running robot and application layer locally.

If you want to go further, you can create beautiful, customized robot apps using the Voilà templating mechanism. You can find some advanced template examples under the following links:

About the author

Wolf Vollprecht lives and works in Paris and Berlin for QuantStack, a small software consultancy. We specialize in scientific and robotic application development. Among other things, we work on high performance array computing, 2- and 3D visualization in the web-browser and applications using the Project Jupyter framework. And we help small and large companies at moving robotic applications to the cloud.