[Cuis-dev] Redrawing on events and "immediate mode" UIs

[Paul D. Fernhout]

Thank you for these great examples, Mariano!

Appreciate you sharing these. And amazing that you did it four years 
ago. And to see that you also explored the tension between doing this as 
just redraw (including text editing) versus trying to reuse morphs.

Doing an entire redraw approach seemed to me like too much work (at 
first_ to reinvent everything already implemented in Morphic, but I see 
(from the gifs) that you made the drawOn: basics work well already in Cuis!

 From links in your work it seems that it may be based on or inspired by 
Johannes Norneby's previous IMGUI work?
https://www.johno.se/book/imgui.html
"Programming user interfaces has a reputation of being difficult. This 
is perhaps much due to the fact that user interface toolkits tend to be 
large and complex software systems. They often have a steep learning 
curve and are often cumbersome to use, typically involving quite a bit 
of application specific implementation in order to integrate. When you 
next log onto Poker dk, or Twitter, take into account the multifaceted 
nuances of the UI, as you'll be surprised at how many components they 
really have. Designing such a toolkit is harder still, and it is my 
experience (when it comes to games) that even a toolkit that is 
explicitly designed for re-use doesn't get re-used very much in 
practice. Due in large part to the advances in dedicated graphics 
hardware (GPUs) during the past 10+ years, it is now entirely feasible 
to approach user interfaces in a novel way. Immediate Mode Graphical 
User Interface (IMGUI) represents a paradigm where user interfaces are 
simpler to create (for the client application) and simpler to implement 
(for the toolkit designer). ... There is a dominant paradigm within 
programming since (forever?), and that simply: The user interface and / 
or visualization of any program is inherently stateful. I maintain that 
this is a broken paradigm. Not that such things CANNOT be stateful; the 
current state of various software technlogies are indeed based upon this 
paradigm. I will however argue that avoiding such statefulness 
significantly simplifies software."

While I have heard the term "immediate mode" long ago (first in relation 
to 3270 graphics adapter sin the 1980s probably, and then OpenGL), I had 
not thought of it in terms of this UI approach or redrawing on events. 
Following that up, I see now though:
https://en.wikipedia.org/wiki/Immediate_mode_GUI
"Immediate mode GUI widget toolkits are a good choice for those who 
prefer a simple, easily changeable, and extendable GUI toolkit. They are 
usually generic, open-source, and cross-platform. One way to have the 
flexibility and composability of an immediate mode GUI without the 
disadvantages of keeping the widget tree only in function calls, with 
the lack of direct control of how the GUI is drawn in the rendering 
engine would be to use a virtual widget tree, just like React uses a 
virtual DOM."

Would be curious to hear more about any more about pros and cons and 
outstanding challenges from your Cuis experiments. And also on whatever 
inspirations perhaps from other libraries lead you to create this. And 
how you felt about the results of immediate mode vs. the morphic 
version. And also why you paused in pursuing this idea further in the 
Cuis context.

It looks like you used some this though in Mold? That's where I found 
the above link, where you wrote: "IMGUI Immediate mode GUI inside 
Morphic. Used by StructurEd and Dia."
https://bitbucket.org/mmontone/mold/src/master/

I've gotten sidetracked by something else higher priority for now in my 
spare time, but hope to pick up these explorations again later in the 
summer. Perhaps I can build on your immediate mode drawOn: approach now 
that you've done the heavy lifting to get a basic framework in place for 
Cuis.

Al that said, from exploring this idea just a little in Cuis, I can 
definitely see value of the Morphic "changed" message approach in that 
it forces people to think about custom events (aspects that change) in 
designing their components which then -- in theory -- create points for 
plugability. I am seeing how this alternative immediate-mode approach 
loses something there related to reuse and composability -- especially 
for developing graphical tools (like say VisualAge does). And from 
looking at previous work by the original Smalltalk team (especially Dan 
Ingalls) it seems like moving in a pluggable UI direction using a visual 
programmign paradigm was an aspiration resulting in various 
implementations, or which Morphic was one result.

But for me, I think that plugability is a lesser issue compared to 
quicker development in code and potentially easier maintenance -- 
although others may feel differently about all that. And that point by 
the IMGUI author resonates with my own experience that reuse of UI 
components does not happen as often as one might expect (at least of 
custom components beyond basic ones like a text entry widget or button).

Still, I want to acknowledge that visual programming remains intriguing 
(like Mold itself supports to an extent with the diagram editor). I 
don't fully understand the various situations where text-based 
programming or visual programming is "better" for various users. And I 
don't know how they can interoperate best. Or if there is some 
unification possible there sometimes with possibly flipping between 
equivalent representations (like in VisualAge or similar systems)? 
Probably a PhD dissertation in there somewhere (not for me) -- and maybe 
there are some already written on that?

In any case, I continue to believe it's plausible that an "immediate 
mode" UI that redraws on events may overall be simpler to understand, 
use, and maintain in Cuis than Morphic. But, as you imply, it would 
still be a lot of (re)implementation work to get to that point if not 
reusing Morphs as in your "Reactive" example.

And if we then want to build plugability on top of an immediate mode 
somehow (like with Mold or any other 2D or 3D graphics system for 
linking components like, say, Stormworks programming), it is an an open 
question if that combined result would be "simpler" that what we have 
now. I have not looked into the Mold code to see how that worked out as 
far as linking elements visually (and how far you used Morphic in that 
situation versus an immediate mode), but would be curious how you felt 
about the results in Mold however you achieved them.

Thanks again for the inspirational examples.

P.S. The Mold meta-modeling workbench looks really cool!

--Paul Fernhout (pdfernhout.net)
"The biggest challenge of the 21st century is the irony of technologies 
of abundance in the hands of those still thinking in terms of scarcity."

On 5/30/23 12:12, Mariano Montone via Cuis-dev wrote:
> I've got another example, an editable TODO list.
> 
> This is the kind of dynamic GUI that is difficult to compose without an 
> extra reactivity layer or similar because components need to be added, 
> removed and replaced dynamically. If the interface is more static, then 
> using normal Morphs is just fine.

> ...

>> I've tried a couple of GUI experiments, one that does Immediate Mode, 
>> and another that does Reactive updates.
>>
>> Here are my IMGUI experiments: 
>> https://bitbucket.org/mmontone/mold/src/master/IMGUI.pck.st.
>> Instead of rebuilding morphs like in yours, there's no Morph tree 
>> maitained (that's why it is not a persistent GUI, and it is 
>> immediate). Everything happens in the drawOn: method, logic, 
>> rendering, rendering of subcomponents, etc.
>> I would qualify them as half-failed and a bit excessive, because I was 
>> trying to do some extremely difficult thing (for me, at least) with 
>> GUI, some structuring editing, and wanted to have a very dynamic 
>> funcional oriented gui (view = f (model)).
>> Unlike normal Immediate GUI frameworks, it doesn't redraw on every 
>> frame, but only when an event occurs.
>>
>> My other experiment is a reactive GUI: 
>> https://bitbucket.org/mmontone/mold/src/master/ReactiveMorphs.pck.st.
>> Like your approach it rebuilds the morphs automatically, but there's 
>> some plugging involved, as it doesn't every submorph blindly, it 
>> rebuilds the affected parts of the gui based on what reactive 
>> variables changed.
>>
>> The comparison in Javascript world would be SolidJs library, I believe