[Cuis-dev] Porting State of Magma

[Juan Vuletich]

Hi Gerald,

On 5/22/2021 6:02 AM, Gerald Klix via Cuis-dev wrote:
> Hoi Folks,
>
> It's bit hard for me to answer without starting
> to rant. OTH, staying quiet isn't an option:
> I am to much interested in a DB.
>
> So please bear with me, when I leave
> the DB topic for a while.
>
>
> Philip's story of porting Magma
> and my first real encounter
> with the opensmalltalk VM taught
> me a lesson, alas it left a serious open
> question without proper answer.
>
> A famous book, we probably all know,
> states:
>
> "Personal Mastery. If a system is to serve the creative spirit, it 
> must be entirely comprehensible to a single individual."
>
> (See 
> https://cuis-smalltalk.github.io/TheCuisBook/Historical-Context.html)
>
> Personal Mastery can be achieved with Cuis,
> Haver is a fine example of that.
>
> But once you start with Magma or
> the OSVM, you can make the transition from
> "Personal Mastery" to "Cargo-Cult" programming in less than 30 minutes.

I agree. These are the real limits of the Smalltalk approach. There are 
other examples too. Avoiding this (at least in the base image!) is the 
measure of success of the Cuis project.

> My observation is:
> Simple and approachable systems tend to grow,
> sometimes their community of users grows with
> them. In this case an interested newcomer
> will find a critical mass of people,
> that have achieved personal mastery and (sic!)
> also can explain their insights to a newbie.

And then, the system and the community are alive.

> The big question is:
> *How can an individual be a personal master
> in a certain topic and form a community
> that grows with him?* (I intentionally
> avoid writing "her", I have the gut feeling, that women do not have 
> that sort of problem).

The answers I know are surprise to no one here:
- Avoid complexity at all costs.
- Attract and help people with real interest in learning about the stuff.
Too much complexity will take its toll. No doubt of that. WRT bringing 
people to a project, I'm not a fan of pushing too much. Unfortunately, 
giving incentives like Google Summer of Code, or a M.Sc. thesis, tend to 
attract people that, while capable, don't have enough interest in the 
project to continue attached to it in the long term, after the incentive 
has ended. People need to find the stuff interesting, fun and rewarding 
in itself, or really need it to achieve other objectives (such as 
building a commercial product, or supporting another project). And I 
think long term involvement is needed to build community, and not just 
software.

> Now back to the database issue:
>
> In February and March 2021 I tried
> to solve this DB problem, by doing
> it right from scratch. I cast all
> the good ideas, I had accumulated over
> the years, into software.
> You may have guessed it, it soon
> grow over my head; thus I abandoned it.
> This was Plan A.
>
> Plan B was porting Magma to Cuis.
> Philip did a great job, he
> has good reasons to give
> up on that one.
>
> I had my doubts from the beginning,
> therefore I started Plan C,
> writing a wrapper for LMBD:
> http://www.lmdb.tech/doc/
>
> This did not work and I did not
> want to debug this using gdb.
> So I quickly gave up and
> started Plan D:
>
> Porting Image Segments to Haver.
> It started well, after 20 minutes
> it could load the Squeak code
> into a Haver module, with
> duplicate classes neatly encapsulated
> into a module. It then took
> 2 hours to remove all the obscure
> dependencies on Projects and other
> Etoys stuff. Finally it produced
> a binary dump of a few test instances.
> Then I spent another to hours to
> find out on how to load that image
> segment back into a my image.
> I still don't know how.
>
> Again these image segments are
> nice example for someone with
> personal mastery,, who has lost
> everyone else to his climb
> to his personal Mount Everest.
> (Probably it was more than one person).

Not having used ImageSegments in decades, I still believe that making 
them work in Cuis is entirely possible, and not too much work.

But I'm not sure if that would, for example, suit your needs. It was a 
good solution to give users comfortable access to content that was much 
larger than available RAM. In these days, I don't see much need for that 
use case. And it doesn't offer most of what a database offers. In many 
cases, SmartRefStream is a good substitute.

> Again I gave up and did the simplest
> thing that might possible work.
> I used an record format that supports
> updates and at the evening of the 13th of
> May I could persist some simple classes.
>
> For obvious reasons I called that object store
> "PlanE". You can find it here:
> https://hg.sr.ht/~cy-de-fect/HaverOnCuis/browse/haver/db/PlanE.pck.st?rev=tip 
>
>
> Currently it can't be called a database,
> because it does no recovery. It can't collect
> garbage, but *is still in a state, where it
> can be understood by others".
>
> Therefore everyone is invited to climb
> the pass to personal mastery with me
> and contribute to that project.
> If there is demand I will remove all
> the dependencies to Haver's modules
> and make it a simple Cuis project.
>
> *I want to avoid another Smalltalk
> project, that is only understood
> by its author*, we have enough of those!
>
>
> Thanks for following me up to here and
>
>
> Best Regards,
>
> Gerald

Thanks Gerald for sharing all this. My main question is: What are your 
requirements? There are a lot of possible requirements for a persistence 
scheme, and some can be satisfied with a solution that can't be called a 
database. Some possible requirements could be:

A1- Persist objects outside the image, meaning that the image might 
never be saved, and the stuff can be recovered, updated, and persisted 
again, only from the image that generated it.
A2- Same as A1, but making it possible to use it from different images 
(meaning that class definitions are also persisted, or at least validated).
A3- Same as A2, but isolating it from the rest of the image, meaning 
that objects of different versions of the same classes can coexist in 
the same running image.
A4- Same as A3, but allowing migration and conversions between these 
different definitions

B1- Limited to "all the stuff must fit in a single memory space" and 
"stuff is small enough that the performance cost of loading it all at 
once is not a problem"
B2- Has means of querying the "database", and load into memory only what 
is required. Database might be larger than actual memory.

C1- Single user. At least, single client image at a time.
C2- Multi user. Simple locking. Several users can access the database, 
but an optimist locking scheme makes concurrent modifications to related 
objects fail.
C3- Multi user with more advanced concurrency control.
C4- Supports flexible distribution and replication of objects across a 
network of servers and clients.
C5- Supports dynamic balancing of load. An efficient implementation 
makes very good use of server resources (in terms of simultaneous 
transactions, server load, cost of replications, etc).

D1- No consistency guarantees. A crash can corrupt data. In same simple 
situations, saving to a new file each time might suffice.
D2- ACID transactions. A crash can not corrupt data, but a crash or 
network disruption can generate data inconsistencies over distributed 
object graphs.
D3- ACID transactions and crash recovery. Consistency guarantees over an 
object graph distributed over many servers and clients.

I have just sketched four dimensions of the top of my head. I don't know 
where Magma or other projects stand on each of these dimensions. But 
Cuis already includes SmartRefStream, possibly A1-B1-C1-D1, the minimum 
possible feature set. On the other extreme there is Gemstone, that I 
guess could be A4-B2-C5-D3, and also provide a lot of the features I 
didn't remember today or I'm not aware of. Gemstone is of course the way 
to go if you need a really good and solid object database. I believe 
they have non-commercial alternatives, and they are startup friendly, 
meaning that they will charge you only when you are making big money.

So, do you need something like Gemstone? Like Magma? Like 
SmartRefStream. (btw, what have I overlooked in this mail?)

Maybe this kind of discussion could help enough really interested people 
get together and make it happen.

Cheers,

-- 
Juan Vuletich
www.cuis-smalltalk.org
https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev
https://github.com/jvuletich
https://www.linkedin.com/in/juan-vuletich-75611b3
@JuanVuletich