[Cuis-dev] Redrawing on events and "immediate mode" UIs

[Mariano Montone]

Hi Paul,

El 31/5/23 a las 11:02, Paul D. Fernhout via Cuis-dev escribió:
> From links in your work it seems that it may be based on or inspired 
> by Johannes Norneby's previous IMGUI work?

Yes. The only difference is that I instead of redrawing on every frame, 
I redraw only when an event happens or status changes. But it is still 
Immediate Mode, there's no persistent tree of visual components to maintain.

> While I have heard the term "immediate mode" long ago (first in 
> relation to 3270 graphics adapter sin the 1980s probably, and then 
> OpenGL), I had not thought of it in terms of this UI approach or 
> redrawing on events. Following that up, I see now though:
> https://en.wikipedia.org/wiki/Immediate_mode_GUI
> "Immediate mode GUI widget toolkits are a good choice for those who 
> prefer a simple, easily changeable, and extendable GUI toolkit. They 
> are usually generic, open-source, and cross-platform. One way to have 
> the flexibility and composability of an immediate mode GUI without the 
> disadvantages of keeping the widget tree only in function calls, with 
> the lack of direct control of how the GUI is drawn in the rendering 
> engine would be to use a virtual widget tree, just like React uses a 
> virtual DOM."
>
> Would be curious to hear more about any more about pros and cons and 
> outstanding challenges from your Cuis experiments. And also on 
> whatever inspirations perhaps from other libraries lead you to create 
> this. And how you felt about the results of immediate mode vs. the 
> morphic version. And also why you paused in pursuing this idea further 
> in the Cuis context.

Perhaps my current implementation is not good enough, so I should not 
judge the approach in general. I think it makes some things easier and 
some others more difficult. The good part is not having to maintain a 
persistent tree of visual components, and so it is very good for dynamic 
interfaces. Also, more control of events and status, since it becomes 
more centralized. And it is more natural to implement some dynamic parts 
of the interface.

The cons are:

1) Performance issues. It is more expensive than the normal persistent 
tree approach.

2) Layout is more difficult, since you don't have many passes, like when 
you build a tree, then layout observing how the tree is composed, like 
in the persistent approach.

3) Handling of events can get tricky some times, specially if you need 
to have overlapping GUI elements. As events are handled at the same time 
that the drawing occurs, sometimes this is not good enough, as there may 
be another gui element going to be drawn next that should have handled 
that event. In a way it is a similar problem that we have with layout. 
As there are no persistent tree of widgets to observe, it is difficult 
to apply some operations (layout, events) in a precise way some times.

> It looks like you used some this though in Mold? That's where I found 
> the above link, where you wrote: "IMGUI Immediate mode GUI inside 
> Morphic. Used by StructurEd and Dia."
> https://bitbucket.org/mmontone/mold/src/master/

Yes, those are interfaces that I thought could benefit from the 
Immediate Mode approach. I had some success with that. Only some :)

> I've gotten sidetracked by something else higher priority for now in 
> my spare time, but hope to pick up these explorations again later in 
> the summer. Perhaps I can build on your immediate mode drawOn: 
> approach now that you've done the heavy lifting to get a basic 
> framework in place for Cuis.
You can load the IMGUI package and have a look at some of the examples. 
Some work, others don't. Needs adjustments.
> Al that said, from exploring this idea just a little in Cuis, I can 
> definitely see value of the Morphic "changed" message approach in that 
> it forces people to think about custom events (aspects that change) in 
> designing their components which then -- in theory -- create points 
> for plugability. I am seeing how this alternative immediate-mode 
> approach loses something there related to reuse and composability -- 
> especially for developing graphical tools (like say VisualAge does). 
> And from looking at previous work by the original Smalltalk team 
> (especially Dan Ingalls) it seems like moving in a pluggable UI 
> direction using a visual programmign paradigm was an aspiration 
> resulting in various implementations, or which Morphic was one result.
>
> But for me, I think that plugability is a lesser issue compared to 
> quicker development in code and potentially easier maintenance -- 
> although others may feel differently about all that. And that point by 
> the IMGUI author resonates with my own experience that reuse of UI 
> components does not happen as often as one might expect (at least of 
> custom components beyond basic ones like a text entry widget or button).
In my opinion, different approaches are good for different things. I 
think they can all coexist. For instance, it doesn't make sense to 
change the whole Cuis interface to something completely different, but 
it does to have specific morphs use something different, like the IMGUI 
or the ReactiveMorphs. And that's how my current experiments work, you 
can choose to use an alternative GUI approach for specific morphs.
> Still, I want to acknowledge that visual programming remains 
> intriguing (like Mold itself supports to an extent with the diagram 
> editor). I don't fully understand the various situations where 
> text-based programming or visual programming is "better" for various 
> users. And I don't know how they can interoperate best. Or if there is 
> some unification possible there sometimes with possibly flipping 
> between equivalent representations (like in VisualAge or similar 
> systems)? Probably a PhD dissertation in there somewhere (not for me) 
> -- and maybe there are some already written on that?
Part of my motivation was to try some more visual approach to code 
authoring. But once you move away from text, it gets extremely difficult 
in my experience.
> In any case, I continue to believe it's plausible that an "immediate 
> mode" UI that redraws on events may overall be simpler to understand, 
> use, and maintain in Cuis than Morphic. But, as you imply, it would 
> still be a lot of (re)implementation work to get to that point if not 
> reusing Morphs as in your "Reactive" example.
>
> And if we then want to build plugability on top of an immediate mode 
> somehow (like with Mold or any other 2D or 3D graphics system for 
> linking components like, say, Stormworks programming), it is an an 
> open question if that combined result would be "simpler" that what we 
> have now. I have not looked into the Mold code to see how that worked 
> out as far as linking elements visually (and how far you used Morphic 
> in that situation versus an immediate mode), but would be curious how 
> you felt about the results in Mold however you achieved them.

I'm not sure how I feel, it is unfinished work :| .

But to your original point about Mithil.js, I would like to suggest to 
you to have a look at SolidJs approach, that is what my ReactiveMorphs 
package uses. It can be an eye-opener. It makes it possible to compose 
GUIs declaratively and without the need of an intermediate "virtual 
dom", diff and reconciliation algorithms. And can be made to work with a 
normal persistent GUI toolkit like Morphic. I think it can be a good 
trade-off.

> Thanks again for the inspirational examples.
>
> P.S. The Mold meta-modeling workbench looks really cool!

Hehe. Yes, I wish I could finish that. I'm a machine of unfinished 
works. The only good part is that some times I can transform parts of 
the whole project into more or less usable libraries or tools.

Cheers,

     Mariano