2019 in Review
Another year has passed at QuantStack— let’s look at what we have achieved! We’ve spent an awesome year working on a ton of open source projects: C++ libraries for array computing and low-level SIMD accelerators, JavaScript libraries and Jupyter widgets for scientific data visualization and finally a visual debugger for JupyterLab!
The JupyterLab Debugger, xeus and xeus-python
This year has seen major changes in xeus and xeus-python, driven by the requirements for developing a visual debugger. We have implemented a prototype that supports the Debug Adapter Protocol from Microsoft in xeus-python, an alternative kernel for Python based on Xeus. We also collaborated with other core developers of Jupyter, Two Sigma (Afshin and Borys) to implement an awesome debugger frontend in JupyterLab that will give you an IDE-like experience right in your web browser (you can try it out by clicking the binder link in this repo).
Xeus and related projects have been accepted as Jupyter projects, and have been moved to the jupyter-xeus organization and under the Jupyter governance.
Mamba, the fast conda alternative
In 2019, we started mamba, a faster drop-in replacement for the conda package manager, and turned our prototype into a working product. Mamba has received a lot of attention because many people (like us) are getting frustrated by the speed and reliability of conda. In 2020 we are getting significant funding which will allow us to move mamba further and incorporate more interesting ideas from the Linux packaging world. We will experiment with better error reporting and tools for analyzing conda environments (asking for reverse dependencies, etc.).
xtensor-stack: xtensor and xsimd
xtensor, the NumPy for C++ library, and xsimd are moving forward as well! Both packages have gathered a lot of stars on GitHub and xtensor looks like it’s getting more and more adoption in the real world. We have the first core-contributor that is not part of QuantStack (welcome Tom de Geus and thanks for all your contributions!) and we are always happy to see more community engagement and code contributions in the xtensor stack! On an organizational level we moved xtensor and friends to a new umbrella-organization (xtensor-stack) with its own little website, and Johan started an awesome, technical blog series about the xtensor design: https://xtensor-stack.github.io/blogs
While we had big plans for GPU support in 2019, we had to move the target to 2020. It will surely come to xtensor, it’s just a question of finding enough time and funding. If you are interested in helping the xtensor & C++ world out by sponsoring GPU development (we are also extremely happy about code-contributions!) then let us know!
It’s especially cool to see xtensor being picked up by the robotics community in projects such as xtensor-quaternion.
xsimd also earned a lot of GitHub stars — unsurprisingly — as it is one of the nicest C++ SIMD wrappers out there with support for SSE, AVX2, AVX512, and ARM NEON and very fast mathematical approximation functions.
Voilà, turning Jupyter Notebooks into Dashboards
The fastest rising star of all QuantStack projects is without a doubt the Voilà toolkit that turns Jupyter notebooks into dashboards (standalone web applications). A Voilà app takes the notebook, strips out all code and display only the results of the notebook as a standalone website. The widget functionality, however, remains, so e.g. clicking a button, selecting an area on a map, etc. will still trigger callbacks that can update what’s displayed on the website.
Voilà has been moved to its own organization this year, and we are hosting the voila-gallery.org to showcase some cool applications (the organization already contains a couple of awesome custom templates!).
jupyter-ros, jupyter-amphion and jupyterlab-zethus
2019 was the year jupyter-ros was started. Thanks to jupyter-ros, one can now easily communicate with robots from a Jupyter Notebook. We just released version 0.3.0 of jupyter-ros which has functions to quickly create widgets from message definitions and publish them with the click of a button, has a 3D robotics viewer right in the notebook and comes with a rosbag player.
Some folks in the community are already working on support for ROS2 in jupyter-ros, and hopefully, this will make it to the main package in 2020!
We have also integrated the same ideas in jupyter-amphion (a modern take on ROS3D.js), and jupyterlab-zethus integrates an RViz-like visualization tool into a JupyterLab tab.
For 2020 we have exciting ideas around rosbag analysis inside JupyterLab, combining the strengths of tools that we know and love from the data science community with the robotics world: for example, we’re thinking about using vaex to analyze gigabytes of data and plot the results in JupyterLab.
Jupyter widgets: ipygany, ipysheet, ipycanvas, bqplot, …
The ipywidgets eco-system is still going very strong! QuantStack’s Martin Renou has done some incredible work: with ipygany (a 3D scientific visualization widget based on a new TypeScript library called GanyJS), ipysheet (a spreadsheet widget for tabular data such as data frames), ipycanvas (a powerful widget to interact directly with the HTML5 canvas to draw raw pixels on the screen), and ipywebrtc (stream video live from a webcam, or a file to the notebook, for example to create a chat program or test OpenCV filters live).
And of course, we are still improving bqplot. In a massive undertaking 2019, the bqplot JavaScript was ported to modern TypeScript, and work is ongoing at refactoring bqplot to be the best interactive plotting library out there. Next year we will have lots of opportunities to work on bqplot again, thanks to Bloomberg’s continued interest at pushing the envelope of interactive plotting in the notebook.
Personal Side Projects
suggestr, a typo corrector
A while ago I extracted the C++ source code for the Android Open Source Project (AOSP) keyboard. That source code contains a pretty nice implementation of a trie based typo correction and word prediction engine that takes into account the distance of the keys on a keyboard. I’ve called the project suggestr. It is going to be useful for on-screen keyboards (I am having my hopes up that the project will be used by the new Purism soft-keyboard for the upcoming Librem 5 phone), but can also be used very well on Linux computers or as a standalone Python library!
This year I have added Python bindings using pybind11 so users can create new custom dictionaries, and match them against arbitrary strings to get the closest matches. This could be the backbone of some future search engine!
Adjacent: a 2D constraint solver
Lately, I have been curious about constraint solving in 2D projects. It’s a technique that is commonly implemented in CAD programs (such as FreeCAD or SolveSpace). However, I haven’t been able to find a standalone implementation of such a constraint solver. I believe a standalone implementation can have a lot of interesting applications: for example, in Inkscape to say that you want two lines that are always parallel and in a certain distance to each other, or for OpenStreetMaps editors to integrate more CAD-like features where angles can be properly constrained.
While looking at this problem, Alexey Egorov contacted me and told me about his impressive NoteCAD project where he has implemented a really clean constraint solver. Together, we have ported the solver to C++, and for the moment it’s called Adjacent.
Soon I’d like to add Python bindings and integrate the facilities to add constraints to geometric primitives using Python so that this solver becomes even more useful.
scikit-geometry
Another side project that we’ve spent a couple of cycles on in 2019 is scikit-geometry. The Python scientific ecosystem doesn’t really have a package that implements the basic geometric types and interactions thereof (such as Point, Segment, Line, Ray, Polygon, …). The scikit-geometry project attempts to fix that. For the moment, it builds on top of the CGAL project (a very powerful C++ library that implements a lot of algorithms for computational geometry).
Right now, the basic functionality is there, but to make scikit-geometry a proper “scikit” package, vectorization needs to be added (so that one can easily intersect two NumPy arrays of lines, for example). It would be interesting to explore this using xtensor, of course!
conda-forge packages for ROS
One other major side project (and one of the early motivations to create mamba) was to bring ROS packages to conda-forge. ROS is the Robot Operating System, a collection of over 2000 packages. We’ve successfully brought the core hundred or so packages to conda-forge. But there are still a couple of outstanding questions that we can hopefully solve in 2020: should we use conda-forge, or would it be better if ROS had its own channel and build platform for the distros? How do we solve the problem of binary compatibility?
The great news is that Sean Yen from Microsoft has started an experiment to have a custom channel with ROS2 packages for Windows and this shows that conda is an awesome cross-platform package manager. If everything goes well in 2020 we might get support from OSRF for ROS packages on conda!
2020!
We’ve got a lot on our plate for 2020! If you are interested in working with us, we are hiring an intern in Berlin (or Paris). Just drop me a line at wolf.vollprecht@quantstack.net
About the author
Wolf Vollprecht lives in Paris and Berlin where he works for QuantStack. QuantStack is a small software consultancy focusing on scientific software development in Europe. We are experts in Python, JavaScript and C++. Among other things, we specialize on high-performance array computing, 2- and 3-D visualization in the web browser and applications using the Project Jupyter framework.
To stay up to date, you should follow me on twitter here, and the QuantStack twitter.
