[Cuis-dev] Porting State of Magma

[Gerald Klix]

On 6/7/21 5:32 PM, Juan Vuletich via Cuis-dev wrote:
Hi Juan, Hi folks,

Juan, thanks for taking the time to read my post and answer
my questions.
> Hi Gerald,
> 
> On 5/22/2021 6:02 AM, Gerald Klix via Cuis-dev wrote:
>> Hoi Folks,
>>
>> It's bit hard for me to answer without starting
>> to rant. OTH, staying quiet isn't an option:
>> I am to much interested in a DB.
>>
>> So please bear with me, when I leave
>> the DB topic for a while.
>>
>>
>> Philip's story of porting Magma
>> and my first real encounter
>> with the opensmalltalk VM taught
>> me a lesson, alas it left a serious open
>> question without proper answer.
>>
>> A famous book, we probably all know,
>> states:
>>
>> "Personal Mastery. If a system is to serve the creative spirit, it 
>> must be entirely comprehensible to a single individual."
>>
>> (See 
>> https://cuis-smalltalk.github.io/TheCuisBook/Historical-Context.html)
>>
>> Personal Mastery can be achieved with Cuis,
>> Haver is a fine example of that.
>>
>> But once you start with Magma or
>> the OSVM, you can make the transition from
>> "Personal Mastery" to "Cargo-Cult" programming in less than 30 minutes.
> 
> I agree. These are the real limits of the Smalltalk approach. There are 
> other examples too. Avoiding this (at least in the base image!) is the 
> measure of success of the Cuis project.
You did well with Cuis. Disregarding the VectorEnginePlugin stuff,
I understand Cuis.
> 
>> My observation is:
>> Simple and approachable systems tend to grow,
>> sometimes their community of users grows with
>> them. In this case an interested newcomer
>> will find a critical mass of people,
>> that have achieved personal mastery and (sic!)
>> also can explain their insights to a newbie.
> 
> And then, the system and the community are alive.
Yes, I forgot to mention code examples one can copy and
modify.
> 
>> The big question is:
>> *How can an individual be a personal master
>> in a certain topic and form a community
>> that grows with him?* (I intentionally
>> avoid writing "her", I have the gut feeling, that women do not have 
>> that sort of problem).
> 
> The answers I know are surprise to no one here:
> - Avoid complexity at all costs.
Yes, I completely agree!

Next best thing is: Provide an abundance of examples and demos.
I don't know how to make Haver's implementation simpler to understand,
so for the next two weeks I concentrate on tools for demoing
Haver and the development of PlanE, the database.
Gifted with perfect hindsight, I dare say: Having
those demoing tools available before doing the environments
stuff would have been wise.

Some details:
Currently I can record primitive events and replay them.
I want to add text-bubble- and arrow-morphs to enable the
demo author to explain things during demo replay.

I now believe these tools are more useful
than all the modules and environments stuff.

> - Attract and help people with real interest in learning about the stuff.
> Too much complexity will take its toll. No doubt of that. WRT bringing 
> people to a project, I'm not a fan of pushing too much. Unfortunately, 
> giving incentives like Google Summer of Code, or a M.Sc. thesis, tend to 
> attract people that, while capable, don't have enough interest in the 
> project to continue attached to it in the long term, after the incentive 
> has ended. People need to find the stuff interesting, fun and rewarding 
> in itself, or really need it to achieve other objectives (such as 
> building a commercial product, or supporting another project). And I 
> think long term involvement is needed to build community, and not just 
> software.
Oh yes. Python lost much of its appeal (to me), once Google "confessed"
using it and all these "me-too/wannabe Googlers" started pouring
out of the bushes.
Sista is another such thing ...
> 
>> Now back to the database issue:
>>
>> In February and March 2021 I tried
>> to solve this DB problem, by doing
>> it right from scratch. I cast all
>> the good ideas, I had accumulated over
>> the years, into software.
>> You may have guessed it, it soon
>> grow over my head; thus I abandoned it.
>> This was Plan A.
>>
>> Plan B was porting Magma to Cuis.
>> Philip did a great job, he
>> has good reasons to give
>> up on that one.
>>
>> I had my doubts from the beginning,
>> therefore I started Plan C,
>> writing a wrapper for LMBD:
>> http://www.lmdb.tech/doc/
>>
>> This did not work and I did not
>> want to debug this using gdb.
>> So I quickly gave up and
>> started Plan D:
>>
>> Porting Image Segments to Haver.
>> It started well, after 20 minutes
>> it could load the Squeak code
>> into a Haver module, with
>> duplicate classes neatly encapsulated
>> into a module. It then took
>> 2 hours to remove all the obscure
>> dependencies on Projects and other
>> Etoys stuff. Finally it produced
>> a binary dump of a few test instances.
>> Then I spent another to hours to
>> find out on how to load that image
>> segment back into a my image.
>> I still don't know how.
>>
>> Again these image segments are
>> nice example for someone with
>> personal mastery,, who has lost
>> everyone else to his climb
>> to his personal Mount Everest.
>> (Probably it was more than one person).
> 
> Not having used ImageSegments in decades, I still believe that making 
> them work in Cuis is entirely possible, and not too much work.
I would agree, if I understood how they work in Squeak.
> 
> But I'm not sure if that would, for example, suit your needs. It was a 
> good solution to give users comfortable access to content that was much 
> larger than available RAM. In these days, I don't see much need for that 
> use case. 
Even SAP's monster software uses an in-memory-database these days,
so if that was the problem these beasts solved,
image segments can be considered obsolete.
> And it doesn't offer most of what a database offers. In many 
> cases, SmartRefStream is a good substitute.
Yes to some extent. As far as I understand, one can not
update a single object.
AFIR I once had problems to persist some
very simple information, I ended up using chunks
on a stream.

(Note to myself: Publish that baseliner-package, it useful
for everyone, who patches Cuis.)

> 
>> Again I gave up and did the simplest
>> thing that might possible work.
>> I used an record format that supports
>> updates and at the evening of the 13th of
>> May I could persist some simple classes.
>>
>> For obvious reasons I called that object store
>> "PlanE". You can find it here:
>> https://hg.sr.ht/~cy-de-fect/HaverOnCuis/browse/haver/db/PlanE.pck.st?rev=tip 
>>
>>
>> Currently it can't be called a database,
>> because it does no recovery. It can't collect
>> garbage, but *is still in a state, where it
>> can be understood by others".
>>
>> Therefore everyone is invited to climb
>> the pass to personal mastery with me
>> and contribute to that project.
>> If there is demand I will remove all
>> the dependencies to Haver's modules
>> and make it a simple Cuis project.
>>
>> *I want to avoid another Smalltalk
>> project, that is only understood
>> by its author*, we have enough of those!
>>
>>
>> Thanks for following me up to here and
>>
>>
>> Best Regards,
>>
>> Gerald
> 
> Thanks Gerald for sharing all this. My main question is: What are your 
> requirements? There are a lot of possible requirements for a persistence 
> scheme, and some can be satisfied with a solution that can't be called a 
> database. Some possible requirements could be:
> 
> A1- Persist objects outside the image, meaning that the image might 
> never be saved, and the stuff can be recovered, updated, and persisted 
> again, only from the image that generated it.
> A2- Same as A1, but making it possible to use it from different images 
> (meaning that class definitions are also persisted, or at least validated).
> A3- Same as A2, but isolating it from the rest of the image, meaning 
> that objects of different versions of the same classes can coexist in 
> the same running image.
> A4- Same as A3, but allowing migration and conversions between these 
> different definitions
> 
> B1- Limited to "all the stuff must fit in a single memory space" and 
> "stuff is small enough that the performance cost of loading it all at 
> once is not a problem"
> B2- Has means of querying the "database", and load into memory only what 
> is required. Database might be larger than actual memory.
> 
> C1- Single user. At least, single client image at a time.
> C2- Multi user. Simple locking. Several users can access the database, 
> but an optimist locking scheme makes concurrent modifications to related 
> objects fail.
> C3- Multi user with more advanced concurrency control.
> C4- Supports flexible distribution and replication of objects across a 
> network of servers and clients.
> C5- Supports dynamic balancing of load. An efficient implementation 
> makes very good use of server resources (in terms of simultaneous 
> transactions, server load, cost of replications, etc).
> 
> D1- No consistency guarantees. A crash can corrupt data. In same simple 
> situations, saving to a new file each time might suffice.
> D2- ACID transactions. A crash can not corrupt data, but a crash or 
> network disruption can generate data inconsistencies over distributed 
> object graphs.
> D3- ACID transactions and crash recovery. Consistency guarantees over an 
> object graph distributed over many servers and clients.
> 
> I have just sketched four dimensions of the top of my head.
Your sketch of the different dimension of database requirements is
pretty complete. See below.
> I don't know 
> where Magma or other projects stand on each of these dimensions. But 
> Cuis already includes SmartRefStream, possibly A1-B1-C1-D1, the minimum 
> possible feature set. On the other extreme there is Gemstone, that I 
> guess could be A4-B2-C5-D3, and also provide a lot of the features I 
> didn't remember today or I'm not aware of. Gemstone is of course the way 
> to go if you need a really good and solid object database. I believe 
> they have non-commercial alternatives, and they are startup friendly, 
> meaning that they will charge you only when you are making big money.
> 
> So, do you need something like Gemstone?
No.
  Like Magma?
Also no.
Like
> SmartRefStream.
No updates no recovery.
  (btw, what have I overlooked in this mail?)
I don't think so.
> 
> Maybe this kind of discussion could help enough really interested people 
> get together and make it happen.
> 
> Cheers,
> 
I want Haver to be a tool for single-developer/small-team
projects that deliver single-user offline-applications.


Therefore:
- There should be modules, using other's code should be simple.
- It should be easy to persist application data, just by:
   `PlanE commit`.
- Of course there should be transactions.
- And recovery.
- The database should support versioning.
- The database should support schema migration.
- The database should be thread-safe; it should support Cuis' processes.
- The programmer should only need to press one button, to
   either create a single file (Linux) application or a windows
installer. (Trivial on MacOS, as long as you ignore
Apple's AppStore). (Which reminds of this beautiful
piece of software:
http://www.openwatcom.org/download.php)
- It should be easy to demo an application written with Haver.

There is no immediate need to persist code, other than
by reference.

There is no need to support different users.
(I can not remember a single project using an SQL-database,
that uses more than one database-user during normal operation).
There is no immediate need for a server-process.

However, it should not be impossible to extend the database
with some means of peer to peer replication,
If someone want's to write an application for a team
or a small bushiness.
(Which raises
a plethora of semantic and security issues)

I still have the idea of a peer too peer based
AppStore for Cuis/Haver.

Magma is a solution, but also overkill.
Gemstone I don't know, I presume it needs some
installation steps and perhaps an admin.


Thanks a lot and good night,

Gerald