Introducing plugins for loading any file to Rerun
Developers in robotics and AI use the Rerun SDK to log and visualize multimodal data that changes over time, and that would otherwise be stuck inside their running code.
Rerun also opens many common file types (.jpg, .obj, .md etc) directly.
Still, developers should be able to open any file containing data they want to see, whether it's highly complex, rare, or proprietary to their team.
With the new plugin system for loading arbitrary files in Rerun's 0.12 release that's now starting to become possible.
Here are a couple examples for inspiration and potential starting points for your own plugins.
Example: Open a Tensorboard log file (TFRecord of Events)
This example uses an external plugin, written in Python, to open Tensorboard log files (TFRecord containing Events) directly in Rerun. Check out the code here.
Example: Open a docx file
This example also uses an external Python plugin to open docx files directly in Rerun. Check out the code here
Unified file loading with DataLoaders
The Rerun Viewer can load files in 3 different ways:
- via CLI arguments (e.g.
rerun myfile.jpeg), - using drag-and-drop,
- using the open dialog in the Rerun Viewer.
All these file loading methods support loading a single file, many files at once (e.g. rerun myfiles/*), or even folders.
How does it work?
All internal data-loaders implement the DataLoader trait and are registered dynamically at runtime. This means you can easily add your own custom data-loaders if you are extending the viewer in Rust.
When a user attempts to open a file in the Viewer, all known DataLoaders are notified of the path to be opened, unconditionally.
This gives DataLoaders maximum flexibility to decide what files they are interested in, as opposed to e.g. only being able to look at a file's extension.
Once notified, a DataLoader can return a DataLoaderError::Incompatible error to indicate that it doesn't support a given file type.
If all loaders known to the Viewer return an Incompatible error code, then an error message is shown to the user indicating that this file type is not (yet) supported. Multiple DataLoaders may send data from the same file to the Data Store.
High level overview of how file contents get into the Rerun Viewer's Data Store.
User defined external data-loaders
The easiest way to create your own DataLoader is by implementing what we call an external data-loader; a stand alone executable written in any language that the Rerun SDK ships for. Any executable on your $PATH with a name that starts with rerun-loader- will be treated as an external data-loader.
This executable takes a file path as a command line argument and outputs Rerun logs on stdout.
It will be called by the Rerun Viewer when the user opens a file, and be passed the path to that file.
From there, it can log data as usual, using the stdout logging sink.
The Rerun Viewer will then automatically load the data streamed to the external loader's standard output.
High level overview of how file contents get into the Rerun Viewer using an external data-loader.
How do they get called?
Under the hood, the system for finding and calling external data-loaders is implemented as just another built-in DataLoader called ExternalLoader. It keeps track of all executables with names that start with rerun-loader- and passes on the file path and recording_id (if present) when a user tries to open a file. If all external loaders respond with the error code rr.EXTERNAL_DATA_LOADER_INCOMPATIBLE_EXIT_CODE, the ExternalLoader will itself return the DataLoaderError::Incompatible error. If one or more external data-loaders respond with streams of data it will route those streams to the correct place.
Example view with more detail, showing how file contents get into the Rerun Viewer using an external data-loader.
We're looking forward to see what you build
We hope this new plugin system for loading arbitrary files will help make Rerun even more useful for you and your team. We'd love to share any useful data-loaders you build with the broader community so if you have anything you'd like to share please let us know. As usual, we'd also love to hear any thoughts you might have on how to improve this feature further!
Join us on Github or Discord and let us know what you think and if you have anything we can help share with the community.