In traditional Syndicate implementations, entities respond to patterns that get asserted to dataspaces. In the Rust implementation of Syndicate, there are two levels of state management, Actors, which have fields that are globally accessible to all Entities within that actor.
Syndicate-rs also allows individual Entities to hold internally-managed state. This means for each single Entity, there can be one associated Rust struct that can hold data.
For UI purposes, I’ve found this to be slightly cumbersome. To see what I mean, let’s take write something basic, like a circle-drawing entity. The general approach we usually take is to create an entity that respond to assertions in this method:
(during "<MyCircle@idany@xint@yint@radiusint>; check if we have seen id before; if we have, update our previous draw assertion to the new info and update our internal state to map new assertion handle to this id; if we have not, start asserting new draw circle, and save mapping of assertion handle to this id; ; on retraction, get associated circle id from the handle, and retract draw assertion for that id)
Notice the id, we need this so that we can differentiate between assertions representing unique circles. The traditional approach for actually implementing this entity is to have this single entity manage all of the existing circles on the screen. This is fine I guess, but I’m worried if it will be too complicated when we have more complicated widgets, for example, I want to create widgets that model multi-stage-envelope generators and managing all the existing MSEG’s and their draw assertions inside one callback seems messy.
Instead of having 2 level’s of state, actor-level and entity-level. I’m curious about adding an optional third: per-identifier. Funny enough, I realized that this sort of thing can be implemented as a hierarchical URI system:
Actor/ would contain the globally accessible fields
Actor/$entity/ (where $entity is a per-entity identifier) would contain entity-local state
Actor/$entity/$id (where $id is the per-assertion identifier) would contain assertion-local state
This also has interesting side-effects for entities spawned in sub-facets, they would have a local uri of the form Actor/$parentEntity/$childEntity so entities spawned inside of sub-facets can access their parents fields sort of like a prototype-object system?
Revisiting the previous example, a version with per-id state management could look like this:
(during"<MyCircle @self any @x int @y int @radius int>" (fieldentity-idself'xx)
(fieldentity-idself'yy)
(fieldentity-idself'radiusradius)
(dataflow/beginself'draw-circle (circle (getentity-idself'x)
(getentity-idself'y)
(getentity-idself'radius))))
there’s a bit of magic going on here, field writes to a key-value store, and does so using a URI path composed of all the passed parameters except the last, so the first field call is setting the value of x, to the path Actor/$entity-id/$self/x where entity-id and self are both id’s passed at runtime.
The dataflow block has a slightly different path, this is because dataflow blocks are already per-entity, so we just need a path to show that we want to create a new dataflow block for each record with a unique self value we see.
I think there will need to be some ways to clean up the verbosity of typing out the full paths, but I quite like this approach for my GUI needs.
One issue with the design of Klee that I was stuck on until just now, was how to separate draw commands, from draw instructions. Draw commands meaning expressions that directly create the side-effect of drawing to the screen, and draw instructions meaning sets of fill and stroke calls that aren’t meant to be evaluated, but merely passed as data to other functions. There is a need for both in Klee because we want to be able to create reusable draw functions that can be passed other sets of draw commands as an argument. A simple example of this would be some function we can call draw-red-bbox that takes a list of draw commands l and draws a red Rect that bounds all the included draw commands.
Because stroke and fill create side effects, I was completely stumped on how to do this, the best thing I could muster up was having seperate stroke, stroke! and fill, fill! commands, where only the ! variants would create the side effect. This sucks though because you cant immediately reuse code.
Like many things, the answer was right in front of me! Code as data! Remember!!! One of the main ideas of lisp!!! Quoted expressions of draw commands can be passed as instructions to other functions which can manipulate the data and then evaluate it when ready. So in practice our draw-red-bbox function will look like this:
To make this actually work we will need to have a bbox impl that will dispatch on list data inputs by eval-ing the list, inspecting what draw side effects were generated, and calculating the bbox of those collective side effects.
All in all I’m very happy with this, it also gives us a fantastic way of defining growable shapes, meaning shapes that fill remaining space in the X and/or Y directions based on the defined layout. Users can define the abstract shape in terms of draw instructions, then pass the instructions as data to a growable function, which will do all the layout calculations and then evaluate the abstract shape in terms of actual float values.
For a quick example let’s say you wanted to have a horizontal auto-layout where you have a hard-coded red square, then a blue rect that fills the rest of the width. Using this draw instruction idea the code definition could look like this:
(flex-row (fill'red (rect002020))
(grow-row; grow-row will evaluate this in an environment where 100%; is evaluated to the remaining width of the screen in pixels.;; also note that this is passed as data'(fill'blue (rect00100%20))))
If you run this program on the current Klee system, it will re-run the program from scratch, every frame! Wasting thousands of CPU calls in order to redraw something it already fully rendered & unchanged on the previous frame. If you look closely at the program, you can see that there are no dependencies in the code that will change due to external factors. Let us compare the above static program with it’s real-time animated counterpart:
Inspecting the source code shows there are two uses of (now). eval-ing this symbol at runtime will return an integer timestamp of the current time, so at the beginning of each frame, this value is different from the previous frame. In other words, this program depends on the value of now and needs to be re-evaluated after every time now is updated.
Propagator Parallels
For those familiar with Progagators, maybe you can start to see the overlapping ideas. The animating Klee program shown above is similar to a propagator that observes a now cell, and is rerun each time additional information is added to the now cell. You can even take this a step further and model the Klee program as something which observes some set of input cells, and as a result of it’s computation, write’s to an output cell that represents the Wayland surface. As an aside: Imagining the wayland surface as a propagator cell is quite interesting because similar to cell behavior, we ideally want wayland to only propagate pixels that receive new information between frames. For more info see: https://wayland-book.com/surfaces-in-depth/damaging-surfaces.html
…With a Syndicate Twist
Ok, so in order to efficiently render Klee programs, we can try modeling our little drawing programs as propagators and track their dependencies so we only rerun them when their dependencies change… cool. But doesn’t syndicate also have it’s own ways of structuring dependency management?
In Syndicate we can “scope” execution using facets, which provide “Structured Concurrency”. Because we already want to use syndicate keywords like during to scope our drawings based on state shared by actors in a dataspace, we can try blending during scopes with propagator ideas:
(atds (during (record"KleeWindow"@w'int@h'int)
(let ((w (create-cell$w))
(h (create-cell$h)))
; write behaviors that depend only on $w and $h (during (record'pointer@w'int@h'int)
(let ((mouse-w (create-cell$w))
(mouse-h (create-cell$h))))
; write behaviors that depend on both window size and mouse pos;; use assertions to pass side effects (if (and (mouse-x>50) (mouse-x<100))
(assertds (record"inside range!")))))))
In this example, the high-level wiring of the propagator circuit is changed depending on dataspace assertions, in essence it’s a way to use Syndicate assertions as a way of doing propagator meta-programming. When the pointer leaves the wayland client, the 'pointer assertion is retracted, and the mouse-w and mouse-h cells are destroyed until the pointer returns. Fun! Also worth noting that assertions and messages to/from the corresponding dataspace can serve as a wonderful way for propagators to pass side-effects, which is still a relatively unexplored area of the research.
All-in-all I’m very excited to see where this is all heading, I think that it’s interesting enough as a computing platform that I think I will eventually splice out the “Steel VM + propagator implementation” and put it in it’s own stand-alone actor to act as a more general-purpose networked computing environment (imagine a scheme repl that you can ssh into basically).
Implementation Blues
This all sounds great but unfortunately I do not have a great way of actually implementing it currently, I will need to consult the syndicate rkt impl and the wonderful Alexey Radul thesis to start getting to the implementation details.
Initial Window Open/Close Support + During Entity for Pointer Movements
Managed to get construction/destruction of the Wayland thread to be contained by facets owned by an Actor watching for <KleeWindow @title string @width int @height int> assertions, so now external actors can open/close the window. Wayland-based exit actions, for example closing the window via client-side-decorations, will also kill the wayland facet and cause the Actor to message the dataspace that a wayland exit was made. This allows for external processes to cleanup and exit when the window is closed by the user.
I also added a sorta hacky (during-pointer l) function that applys l if the mouse cursor is focused on the window. Any use of the mouse_x and mouse_y symbols inside the function passed to during-pointer will be converted to the float value of the mouse position.
Here’s a quick demo of these two new functionalities:
Here is the during-pointer call that the blue rect program is asserting:
I quickly started to run into performance issues with my initial lisp interpreter forked from lisp-rs so I ported the Klee vm struct to use Steel instead and the results have so far been awesome.
We are now able to render the Ronin benchmark program at a cool 60fps on my framework laptop with an integrated GPU. We can even animate it and it still is able to compute the 2000 draw commands efficient enough to finish the frame callback in time.
Here’s Klee rendering the Ronin benchmark, before the Steel port this would cause a stack overflow:
And here is the corresponding Klee program. Note that rect-wh is needed instead of Klee’s normal rect because the default rect in Klee follows a x0 y0 x1 y1 signature, rather than using width and height.
EDIT: Update from late tonight, I ended up getting basic assertion-tracking working. A window is created by asserting the following form to a dataspace with an observing Klee actor: <KleeWindow @title string @width int @height int>. Klee expressions are added to the Klee interpreter via assertions of this form: <KleeExpr @label string @expr string>.
The following demo shows two python programs being run, next to an already-running klee window. The red-rect.py program is asserting the following Klee statement:
(stroke1 (rgba1001) (rect5050200200))
and blue_rect.py is asserting this klee expr:
(stroke1 (rgba0011) (rect300300400400))
Notice that the runtime retracts the drawings when the programs are killed. Super excited to see how this keeps evolving.
When you load this program into a Klee interpreter, and evaluate it, it produces a series of draw commands that when encoded and passed to the GPU, evaluate to the following image:
Next steps will be to make it responsive to dataspace assertions by adding the during, except, and collect keywords! I already have most of this work wired up on the interpreter side, I now need to hook it up to a sufficiently sophisticated Syndicate actor. After this I’ll need to spend some time trying to figure out how to handle user input, and responsive layouts (initial layout system will be based off of Clay UI).
Once these building blocks are finished, I suspect I will then have to spend a lot of time on performance and adding more QoL functions for programmers. Maybe also an object system? 👀
Got my first custom object to evaluate properly which is exciting!
Here is the first eval test that passed:
#[test]fntest_rel_len_with_view_dim() {
let v = KleeView { w: 200.0, h: 200.0 };
letmut env = Rc::new(RefCell::new(Env::new_view(v)));
let obj = Object::List(Rc::new(vec![
Object::BinaryOp("+".to_string()),
Object::ParentHeight(1.05),
Object::ParentWidth(2.00),
]));
let result = eval(&obj, &mut env).unwrap();
assert_eq!(result, Object::Float(6.10));
}
The ParentHeight and ParentWidth objects correctly get evaluated to be 1.05% and 2% of the containing environments dimensions, which in this case is 200x200. They get eval’d to floats, which then get added together to reach the intended output of 6.10.
One important feature of traditional scheme implementations is their use of Numerical Towers, and Klee is no different.
Eventually, I would love to implement the full scheme numerical tower on top of the Klee-specific values, but for now this is the current Rust description of Scalars (e.g. raw values without direction) and 2d vectors:
The major takeaway is the inclusion of relative units, and also the Polynomial definition. Relative units, as mentioned briefly above, allow for the definition of shapes that adapt to changes in their parent container’s shape. They work by “snapping into” absolute values when executed inside an eval environment that has ParentWidth, ParentHeight, ViewWidth and ViewHeight values. The top-level environment always has these value set based on the current underlying wayland surface dimensions.
Polynomials represent combinations of these various units via +, -, *, / operations. What is exciting about this is that you can use this to set extremely precise responsive dimensions. For example, you could define the height of a rectangle to be a responsive value of (+ 10vh 20px) which is equivalent to 20px more than 10% of the current view height. All of this is very similar to the current calc() function available in modern CSS.
The idea of RootUnit and Unit also pull from CSS, these are equivalent to the rem and em units, but freed from their inherent ties to font-size. A good example use could be for defining the size, in pixels, of a grid unit for a grid layout, and then defining your layout in terms of these RootUnits so that everything sits on this grid.
As a note, I would really love to have cm be the base absolute unit users think in terms of over pixels, especially as higher DPI screens continue to improve. Getting 1cm to equal 1 IRL cm will be a challenge to get working reliably but I believe we can figure out a decent system for calculating this using wayland to gather display info.
“This is an important nuance of the klee system, you cannot for example send the expression (+ 1 2) to the dataspace and expect for the system to assert a response, the only expression which is ever evaluated at the top level is the view expression.”
We have this wonderful substrate of a dataspace that allows us to communicate between processes really well, limiting the control of the entire display surface to a single assertion greatly hampers the collaborative possibilities. For large swarms of actors contributing to a single klee dataspace, I think this might become an issue where actors might have to “fight” for the single view assertion to make large changes to the client.
I think the better option is to allow for arbitrary top-level execution, and execute each top-level function with the environment of the client surface. So for example:
Will execute my-rect with the ParentWidth and ParentHeight environment variables set to 100% of the window’s dimensions. Thus this will draw a red rectangle that covers the entire window no matter how it is resized!
I think the view function will still be useful, it’ll still provide the first start point for each frame execution, and it will also be used to set the screen dimensions and other important wayland client properties.
One other thing to consider is that loose top-level evals ruin any concept of proper ordering, so this may create issues with draw ops happening out of intended order. To compensate for this, we can create a surface object that will have a z-index.
Let’s say the surface function takes properties x, y, w, h, and z-index in that order, so the surface function call above will draw with a z-index of 1. The default z-index is the “sea-level” of 0. This full set of assertions will end up drawing a full red background that resizes with the window, and then a blue rectangle on top which will take up the top-left quadrant of the window, and responsively resize.
Surfaces can potentially map nicely to wayland subsurfaces under the hood, and we can probably optimize the interpreter’s interaction with the compositor to efficiently manage redraws.
Originally, the Klee interpreter plan was a very traditional Scheme-like language. The main issue with this is that it misses out on all the lovely features of the Syndicated Actor Model that are particularly useful for rendering responsive graphics. The best example of this is the during behavior which is a common theme in syndicate implementations. This behavior frames an actor’s action to when a value is being asserted. Adding this type of behavior to Klee would allow for shape definitions like this:
This shape labelled with the symbol my-square takes a name parameter, and only draws the square while a record of name square with the same name value is being asserted to the Klee dataspace.
I also think a keyword like collect based on the folk computer “collecting matches” phrase would be useful for rendering things based on dynamic sets of assertions. For example rendering a dynamic point cloud could look like this:
When a mouse down event occurs inside the area defined by the square call, the controller will assert a mousedown event with the associated symbol c-square. Any observing actor in the dataspace can then react to this assertion and possibly update the view.
Raw assertions are exactly what it sounds like, the ability to make assertions using preserves format. These can be combined with the previous keywords.
Here, when both the mouspressed and shift-pressed records are asserted in the dataspace, the c-square procedure asserts a hello record to the space. I’m not exactly sure of great uses for this yet, but it might allow for some interesting extensions to the system. In general, once I have the core system working, I want to spend a lot of time working on making the system growable. A big part of this effort might be giving this system some sort of metaobject protocol, something to make it so I’m not having to hard-code every new object I want to add on the rust side, as it feels very fragile.
