Bootstrapping a Personal Computing Environment

I firmly believe that the best creative software is an operating system (in the smalltalk sense, vehemently not in the UNIX sense 😉). This is the only approach I know of that allows us to build software that is malleable enough for artists to take full control of their computational tools and push them to their extremes. One great example of what is possible is the 100 rabbits UXN project where they implement their creative tools on top of a custom virtual machine. My dream is to one day have a cohesive substrate that can replace my current “application software” with a soup of actors passing messages between each other using a standardized messaging format. In order to accomplish this dream I’ve identified four core actor archetypes that need to be implemented well for such a system to succeed:

• Graphics
• Audio
• Networking
• Persistence

Graphics

Our modern “operating systems” have completely failed us in providing easy to use, real-time modifiable, graphics stacks. It is sad that I can’t open a text editor and in a few lines of any programming language, draw a rectangle on the screen, like I can with opening a file. As a result, writing modern graphical applications is often much more challenging than it should be.

Modern compositors/window managers deal with similar issues. The obvious examples of this is the re-implementation of simple windowing themes like tabbing and auto-tiling inside of many “standalone” applications. Imagine if our operating systems offered compositional window managers.

My ideal personal computing environment should make it as easy as possible to efficiently display the internal state of our actors. Some ideas for implementing this are presented on the Klee project page.

I want to stress that in the most yak-shaving way possible, getting this aspect of the environment working properly is of utmost importance, I believe more than anything that providing a high-performance, malleable way of building UIs in an “actor model way” will open a lot of interesting possibilities for creative computational tooling.

Audio

This isn’t nearly as important as graphics for the average user, but I am specifically interested in creating musical instruments and audio effects, so this is important for me. Details about my approach to this area of the environment will eventually be covered extensively on the Flume project page

One other quick note is that I believe audio can play an important role in building interfaces for “computational spaces” like Folk.

Networking

I have a number of different computing devices, which collectively form my “personal computing environment”. This personal mesh of devices should be able to easily communicate with each other and form a single, cohesive environment. The actor model and object capabilities makes this much easier to reason about compared to more traditional methods, but this is still a hard task. The Horton approach for managing identities on capability systems seems like a solid approach towards implementing this.

Ambient-Aware

More than anything, I want my devices to work together when they physically are together, all of my devices have bluetooth radios, so no matter where I am or what the network conditions are, I should be able to at least send small-to-medium sized messages between my devices. KDE Connect gives a good open-source attempt at this idea, but following the theme of this idealized personal computing environment, we should have a service that provides nearby-aware networking capabilities to arbitrary actors. This system could also use horton for handling authentication etc. My approach to this specific subset of this environment’s networking is presented in Ambient Dataspaces

Peripheral Coordination

Besides communication between my full-fledged computing devices (laptop, desktop, mobile device, tablet), my ideal computing environment would make it simple to work with sensors & actuators that extend the physical capabilities of our environment. From off the shelf MIDI controllers to custom ESP32-based sensor arrays, this system should make it possible for its user to construct reactive systems that communicate between these hardware components.

Would be nice if this environment offers an IDE for doing microcontroller development in an actor-y way.

Persistence

To make all of the previous points possible, we need a native way to store and retrieve persistent data. Orthogonal Persistence and/or Orthogonal Sync might be an approach to this but I have no clue.

This is still the major question mark. Maybe projects like Willow will fill this gap?