<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=Windows-1252"> <meta name="Generator" content="Microsoft Word 15 (filtered medium)"> <style></style> </head> <body lang="EN-US" link="blue" vlink="#954F72" style="word-wrap:break-word"> <div class="WordSection1"> Hi Juan, <o:p> </o:p> ><o:p> </o:p> > Hi Jaromir, hi all, ><o:p> </o:p> > This is a most impressive work! <o:p> </o:p> Many thanks for your encouragement - I'm really glad you like it :) <o:p> </o:p> ><o:p> </o:p> > It made me remember this talk given by Martin McClure in Smalltalks 2019: https://www.youtube.com/watch?v=AvM5YrjK9AE . At minute 22:00, Martin states the behavior he expects from #terminate, and shows a test. I tried his test in Cuis, and sometimes (but not always) it failed. Jaromir, with your code, Martin's test now pass! In addition, I ran the existing Process tests, and none is made to fail. <o:p> </o:p> Very interesting: I haven't touched the part for releasing critical sections (I thought it was an independent part I wanted to revisit later) but I'm relieved to hear it works properly with the proposed changes :) <o:p> </o:p> In the talk Martin McClure also mentioned #valueUninterruptably. I've noticed Cuis doesn't implement it but I think with the suggested #terminate it could be implemented safely (fingers crossed). <o:p> </o:p> BlockClosure >> valueUninterruptably ^ self ifCurtailed: [^ self] <o:p> </o:p> At the moment however, Squeak and Pharo suffer a horrible damage to the image if you try: <o:p> </o:p> [ self error ] valueUninterruptably <o:p> </o:p> and later e.g.: <o:p> </o:p> Semaphore new waitTimeoutMSecs: 50 <o:p> </o:p> The worst part is the user won't notice anything and a simple code will run as if nothing happened - until an error or a crash happens. One of only a few ways to see something's off is e.g. Squeak's `Processor activeProcess = Project current uiProcess` which all of a sudden answers false (!) <o:p> </o:p> But that's just Squeak/Pharo; Cuis is not affected as badly because it doesn't use the `effectiveProcess` mechanism. <o:p> </o:p> Martin McClure also mentioned #valueUnpreemptively; funny thing is it doesn't work as designed because there's a bug in Process>>#priority which will leave the elevated process run at highest priority until it reaches a suspension point instead of bumping the priority back lower before returning from #valueUnpreemptively. It's discussed in this thread: http://forum.world.st/The-Inbox-Kernel-jar-1368-mcz-tp5126894p5126982.html . The conclusion is a new primitive for #priority is needed. <o:p> </o:p> ><o:p> </o:p> > I also tried all your tests and examples. They cover a really broad and exhaustive set of scenarios! A few comments: > - Many examples don't work in Cuis, because we made #fork answer nil (and not the forked process). The reason for this is explained in the comment in #fork. The pattern is instead of "p := [stuff] fork." do "p := [stuff] newProcess. p resume." With this change, all the examples work as expected. <o:p> </o:p> Oops, apologies, I noticed the fork pattern (very reasonable), I modified my #fork to return the process to avoid the need to modify all the examples I created for Squeak :) Same for 'self error' unfortunately - for Cuis <o:p> </o:p> self error <o:p> </o:p> in the examples has to change to: <o:p> </o:p> self error: 'unwind error' <o:p> </o:p> sorry... <o:p> </o:p> <o:p> </o:p> > - The notes for a few examples speak of crashes in Cuis. I pushed some fixes a few days ago, and now cmd-. works as expected. The examples behave the same as in Squeak. Please pull your repos. ><o:p> </o:p> <o:p> </o:p> I will, thanks! <o:p> </o:p> <o:p> </o:p> > Additionally, this remark of you (Jaromir) I don't understand: > ///// > ad 1) The #isTerminated condition `suspendedContext pc > suspendedContext startpc` is always met after executing the first instruction of the bottom context � generally not a sign of a terminated process. > \\\\\\ > I don't see that condition neither in Squeak (#19435) nor Cuis (#4567). ><o:p> </o:p> <o:p> </o:p> My bad explaining; I was talking about #isTerminated implementation in Cuis (and Pharo) using this old condition - on the last line of: <o:p> </o:p> Process >> isTerminated "Answer true if terminated, i.e. can never be resumed again, because have nothing to run." self isRunning ifTrue: [^ false]. ^suspendedContext isNil or: [ suspendedContext pc isNil or: ["If the suspendedContext is the bottomContext it is the block in Process>>newProcess. If so, and the pc is greater than the startpc, the bock has alrteady sent and returned from value and there is nothing more to do." suspendedContext isBottomContext and: [ suspendedContext pc > suspendedContext startpc]]] <o:p> </o:p> Squeak improved this condition a while ago but it's still insufficient :) It's discussed here: http://forum.world.st/The-Inbox-Kernel-jar-1376-mcz-tp5127335p5127341.html <o:p> </o:p> <o:p> </o:p> > In any case, I think this is a great contribution, It perfectly fixes a lot of broken behavior, and doesn't seem to bring problems. I'm all for integrating it. ><o:p> </o:p> > Jaromir, do you intend to turn the rest of your examples into tests? I think we should also do that, so we integrate all your experiments and what you learnt. ><o:p> </o:p> <o:p> </o:p> I'd love to but I realized I don't know how to build tests involving raising errors and pressing Abandon on the Debugger :D If you could point me in the right direction it'd save me a lot of trying :) <o:p> </o:p> Thanks a lot for all your comments. I'm happy you like the suggestion and would be delighted to see it integrated. <o:p> </o:p> > Thanks! ><o:p> </o:p> <o:p> </o:p> Best regards, <o:p> </o:p> Jaromir <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div style="mso-element:para-border-div;border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"> From: <a href="mailto:juan@jvuletich.org">Juan Vuletich</a> Sent: Tuesday, April 13, 2021 20:45 To: <a href="mailto:cuis-dev@lists.cuis.st">Discussion of Cuis Smalltalk</a> Cc: <a href="mailto:hernan.wilkinson@10pines.com">Hernan Wilkinson</a>; <a href="mailto:mail@jaromir.net"> Jaromir Matas</a> Subject: Re: [Cuis-dev] Unwind mechanism during termination is broken and inconsistent </div> <o:p> </o:p> Hi Jaromir, hi all, This is a most impressive work! It made me remember this talk given by Martin McClure in Smalltalks 2019: <a href="https://www.youtube.com/watch?v=AvM5YrjK9AE"> https://www.youtube.com/watch?v=AvM5YrjK9AE</a> . At minute 22:00, Martin states the behavior he expects from #terminate, and shows a test. I tried his test in Cuis, and sometimes (but not always) it failed. Jaromir, with your code, Martin's test now pass! In addition, I ran the existing Process tests, and none is made to fail. I also tried all your tests and examples. They cover a really broad and exhaustive set of scenarios! A few comments: - Many examples don't work in Cuis, because we made #fork answer nil (and not the forked process). The reason for this is explained in the comment in #fork. The pattern is instead of "p := [stuff] fork." do "p := [stuff] newProcess. p resume." With this change, all the examples work as expected. - The notes for a few examples speak of crashes in Cuis. I pushed some fixes a few days ago, and now cmd-. works as expected. The examples behave the same as in Squeak. Please pull your repos. Additionally, this remark of you (Jaromir) I don't understand: ///// ad 1) The #isTerminated condition `suspendedContext pc > suspendedContext startpc` is always met after executing the first instruction of the bottom context � generally not a sign of a terminated process. \\\\\\ I don't see that condition neither in Squeak (#19435) nor Cuis (#4567). In any case, I think this is a great contribution, It perfectly fixes a lot of broken behavior, and doesn't seem to bring problems. I'm all for integrating it. Jaromir, do you intend to turn the rest of your examples into tests? I think we should also do that, so we integrate all your experiments and what you learnt. Thanks! <o:p></o:p> <pre>-- </pre> <pre>Juan Vuletich</pre> <pre><a href="http://www.cuis-smalltalk.org">www.cuis-smalltalk.org</a></pre> <pre><a href="https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev">https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev</a></pre> <pre><a href="https://github.com/jvuletich">https://github.com/jvuletich</a></pre> <pre><a href="https://www.linkedin.com/in/juan-vuletich-75611b3">https://www.linkedin.com/in/juan-vuletich-75611b3</a></pre> <pre>@JuanVuletich</pre> On 4/13/2021 6:02 AM, Jaromir Matas via Cuis-dev wrote: <o:p></o:p> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> Hi Hernan, hi all,<o:p></o:p> <o:p></o:p> I'm enclosing a slightly improved version of #terminate along with a few semantic tests covering basic nested unwind scenarios (terminating an active, suspended, blocked and ready processes in nested unwind blocks). More tests can be based on examples in the enclosed collection - for scenarios involving non-local returns and errors during termination and especially their combinations. A few examples explore resiliency of the code exposed to pathologic situations (nested termination or errors).<o:p></o:p> <o:p></o:p> Thanks again for your time. I look forward to your response.<o:p></o:p> best,<o:p></o:p> Jaromir<o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:rgb(225, 225, 225) -moz-use-text-color -moz-use-text-color"> From: <a href="mailto:cuis-dev@lists.cuis.st">Jaromir Matas via Cuis-dev</a> Sent: Sunday, April 11, 2021 10:51 To: <a href="mailto:hernan.wilkinson@10pines.com">Hernan Wilkinson</a>; <a href="mailto:cuis-dev@lists.cuis.st"> Discussion of Cuis Smalltalk</a> Cc: <a href="mailto:mail@jaromir.net">Jaromir Matas</a> Subject: Re: [Cuis-dev] Unwind mechanism during termination is broken and inconsistent<o:p></o:p> </div> <o:p></o:p> Hi Hernan,<o:p></o:p> <o:p></o:p> Thank you very much for your immediate response.<o:p></o:p> <o:p></o:p> > Hi Jaromir,<o:p></o:p> > thank you for sharing this with us! It looks very interesting.<o:p></o:p> > As you can imagine a change of this magnitude and impact has to be analyzed in detail and tested rigorously, and that will take time. <o:p></o:p> <o:p></o:p> Oh absolutely, I'm aware of that and wouldn't expect anything less :)<o:p></o:p> <o:p></o:p> > It would help us deeply if there are tests that reproduce the errors, do you have them? <o:p></o:p> <o:p></o:p> Yes, I have a set of Squeak tests I'm planning to extend and make presentable ASAP. I'll "translate" them to Cuis too.<o:p></o:p> <o:p></o:p> > If not, could you write them? I know it is a difficult problem to test but if we could have automated tests for this it would help a lot. <o:p></o:p> > Also, it would help to know how you tested it. <o:p></o:p> <o:p></o:p> That's important: I tested solely for 'correct' semantics in a series of simple scenarios I'll send you.<o:p></o:p> <o:p></o:p> > For example, have you tried in stress conditions? <o:p></o:p> <o:p></o:p> Nope, I'm not that experienced unfortunately :)<o:p></o:p> <o:p></o:p> > for example with an app that uses many processes (ie. seaside or any web framework). Please, do not take this question as disrespectful, but again,<o:p></o:p> > it is a change with a big impact and we need to be sure it will keep the current behavior (but the bugs of course) and it will not introduce new bugs.<o:p></o:p> > I'm also wondering if there could be systems that are implemented as these bugs you mention were "features" ...<o:p></o:p> <o:p></o:p> That's an interesting question. I realized the sooner bugs this deep in the system are fixed the less chance someone learns to 'live' with them, crippling their code, or even uses them as 'features'. I just hope it's not the case. How would we know? Well... actually here's one: while testing it on Pharo I found two tests that sort of 'legitimize' or use the Unwind error (bug) to achieve certain behavior... (if I interpret the tests correctly).<o:p></o:p> <o:p></o:p> Thanks again for your questions!<o:p></o:p> <o:p></o:p> Jaromir<o:p></o:p> <o:p></o:p> > Thanks!<o:p></o:p> > Hernan.<o:p></o:p> <o:p></o:p> <div style="border:none;border-top:solid windowtext 1.0pt;padding:3.0pt 0in 0in 0in;border-color:rgb(225, 225, 225) -moz-use-text-color -moz-use-text-color"> From: <a href="mailto:hernan.wilkinson@10pines.com">Hernan Wilkinson</a> Sent: Sunday, April 11, 2021 1:47 To: <a href="mailto:cuis-dev@lists.cuis.st">Discussion of Cuis Smalltalk</a> Cc: <a href="mailto:mail@jaromir.net">Jaromir Matas</a> Subject: Re: [Cuis-dev] Unwind mechanism during termination is broken and inconsistent<o:p></o:p> </div> <o:p></o:p> <div> Hi Jaromir,<o:p></o:p> <div> thank you for sharing this with us! It looks very interesting.<o:p></o:p> </div> <div> As you can imagine a change of this magnitude and impact has to be analyzed in detail and tested rigorously, and that will take time. (Sadly Juan and me are really busy at this time so it will even take more time :-) )<o:p></o:p> </div> <div> It would help us deeply if there are tests that reproduce the errors, do you have them? If not, could you write them? I know it is a difficult problem to test but if we could have automated tests for this it would help a lot. <o:p></o:p> </div> <div> Also, it would help to know how you tested it. For example, have you tried in stress conditions? for example with an app that uses many processes (ie. seaside or any web framework). Please, do not take this question as disrespectful, but again, it is a change with a big impact and we need to be sure it will keep the current behavior (but the bugs of course) and it will not introduce new bugs.<o:p></o:p> </div> <div> I'm also wondering if there could be systems that are implemented as these bugs you mention were "features" ...<o:p></o:p> </div> <div> <o:p></o:p> </div> <div> Thanks!<o:p></o:p> </div> <div> Hernan.<o:p></o:p> </div> <div> <o:p></o:p> </div> </div> <o:p></o:p> <div> <div> On Sat, Apr 10, 2021 at 4:20 PM Jaromir Matas via Cuis-dev <<a href="mailto:cuis-dev@lists.cuis.st">cuis-dev@lists.cuis.st</a>> wrote:<o:p></o:p> </div> <blockquote style="border:none;border-left:solid windowtext 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"> <div> <div> Hi All,<o:p></o:p> <o:p></o:p> I'd like to present to you a rewrite of #terminate and #isTerminated that fixes a few bugs and inconsistencies in process termination. I hoped it might interest you.<o:p></o:p> <o:p></o:p> I'm a Smalltalk enthusiast with education background in math/CS. I've been experimenting with processes in Squeak lately and discovered a few bugs (or at least inconsistencies) in process termination and would like to offer and discuss a solution for Cuis.<o:p></o:p> <o:p></o:p> The bugs are not unique to Cuis; Squeak/Pharo inherited them too and to a degree even Visual Works and VA are affected. The proposal presented here doesn't copy any VW or VA solution but rather implements a different approach :)<o:p></o:p> <o:p></o:p> Before boring you to death I'll list the bugs:<o:p></o:p> <o:p></o:p> 1. #isTerminated falsely reports almost any bottom context as terminated<o:p></o:p> <o:p></o:p> 2. an active process termination bug causes an image freeze <o:p></o:p> <o:p></o:p> 3. a nested unwind bug: ensure blocks may get skipped during unwind<o:p></o:p> <o:p></o:p> 4. a failure to complete nested unwind blocks halfway thru execution <o:p></o:p> <o:p></o:p> 5. a failure to correctly execute a non-local return or an error in an unwind block <o:p></o:p> <o:p></o:p> 6. inconsistent semantics of protected blocks unwind during active vs. suspended process termination<o:p></o:p> <o:p></o:p> The current implementation of #terminate uses three different approaches to terminate a process:<o:p></o:p> - the active process is terminated via a 'standard' unwind algorithm used in context #unwindTo: or #resume:,<o:p></o:p> - a suspended process termination attempts completing unwind blocks halfway through their execution first using #runUntilErrorOrReturnFrom:<o:p></o:p> - and after that the termination continues unwinding the remaining unwind blocks using the simulation algorithm of #popTo:<o:p></o:p> <o:p></o:p> This approach looks inconsistent and indeed leads to inconsistencies, undesirable behavior and an instability mentioned above.<o:p></o:p> <o:p></o:p> The Idea: In my view the easiest and most consistent solution is to simply extend the existing mechanism for completing halfway-through unwind blocks and let it deal with all unwind blocks. To make this approach applicable to terminating the active process, we suspend it first and then terminate it as a suspended process.<o:p></o:p> <o:p></o:p> A commented code is enclosed. <o:p></o:p> <o:p></o:p> I know it's pretty difficult to get a feedback on an ancient code; I'll be all the more grateful for your inputs.<o:p></o:p> <o:p></o:p> Best regards,<o:p></o:p> <o:p></o:p> Jaromir<o:p></o:p> <o:p></o:p> <o:p></o:p> PS:<o:p></o:p> Note the change in newProcess - `Processor activeProcess terminate` is replaced by `Processor activeProcess suspend` because there's no need for `terminate` at the bottom context ("terminate = unwind + suspend") and because it would lead to an infinite loop combined with my proposed changes to #terminate.<o:p></o:p> <o:p></o:p> PPS:<o:p></o:p> More on the bugs:<o:p></o:p> ad 1) The #isTerminated condition `suspendedContext pc > suspendedContext startpc` is always met after executing the first instruction of the bottom context � generally not a sign of a terminated process.<o:p></o:p> <o:p></o:p> ad 2) Explained in [1]. Concerns e.g. examples like <o:p></o:p> [ [ Processor activeProcess terminate ] ensure: [ Processor activeProcess terminate ] ] fork. <o:p></o:p> <o:p></o:p> ad 3-4) Explained in detail in [2].<o:p></o:p> <o:p></o:p> One example to illustrate the bug:<o:p></o:p> <o:p></o:p> | p |<o:p></o:p> p := [<o:p></o:p> [<o:p></o:p> [ ] ensure: [<o:p></o:p> [ ] ensure: [<o:p></o:p> Processor activeProcess suspend. <o:p></o:p> Transcript show: 'x1']. <o:p></o:p> Transcript show: 'x2']<o:p></o:p> ] ensure: [<o:p></o:p> Transcript show: 'x3']<o:p></o:p> ] newProcess.<o:p></o:p> p resume.<o:p></o:p> Processor yield.<o:p></o:p> p terminate<o:p></o:p> <o:p></o:p> ===> x1<o:p></o:p> <o:p></o:p> The unwind procedure prints just x1 and skips not only x2 but x3 as well ! You'd like to see them all.<o:p></o:p> <o:p></o:p> ad 5) This happens in cases like (better save your image before trying this):<o:p></o:p> [self error: 'e1'] ensure: [^2] "discovered by Christoph Thiede"<o:p></o:p> and<o:p></o:p> [self error: 'e1'] ensure: [self error: 'e2']<o:p></o:p> These are generally the types of situations causing the Unwind errors. The root cause is that the unwinding is done via simulation (#popTo) rather than 'directly'; the problem is during the simulated execution of unwind blocks a non-local return forwards the execution into a wrong stack - resulting in the Unwind errors. <o:p></o:p> <o:p></o:p> ad 6) I just prefer a unified approach... unless I somehow overlooked a reason for two different approaches (I hope not).<o:p></o:p> <o:p></o:p> <o:p></o:p> [1] <a href="http://forum.world.st/A-bug-in-active-process-termination-crashing-image-td5128186.html" target="_blank"> http://forum.world.st/A-bug-in-active-process-termination-crashing-image-td5128186.html</a><o:p></o:p> [2] <a href="http://forum.world.st/Another-bug-in-Process-gt-gt-terminate-in-unwinding-contexts-td5128171.html#a5128178" target="_blank"> http://forum.world.st/Another-bug-in-Process-gt-gt-terminate-in-unwinding-contexts-td5128171.html#a5128178</a><o:p></o:p> <o:p></o:p> </div> </div> -- Cuis-dev mailing list <a href="mailto:Cuis-dev@lists.cuis.st" target="_blank">Cuis-dev@lists.cuis.st</a> <a href="https://lists.cuis.st/mailman/listinfo/cuis-dev" target="_blank">https://lists.cuis.st/mailman/listinfo/cuis-dev</a><o:p></o:p> </blockquote> </div> <o:p></o:p> <div> <o:p></o:p> </div> -- <o:p></o:p> <div> <div> <div> <div> <div> <div> <div> <div> <div> <div> <a href="https://10pines.com/" target="_blank"><img border="0" width="108" height="34" style="width:1.125in;height:.35in" id="_x0000_i1025" src="https://10pines.github.io/email-signature/10pines-firma@2x.png"></a><o:p></o:p> <h1 style="margin:0in">Hern�n Wilkinson<o:p></o:p></h1> <h2 style="margin-bottom:12.0pt">Software Developer, Teacher & Coach<o:p></o:p></h2> Alem 896, Floor 6, Buenos Aires, Argentina +54 11 6091 3125 </div> </div> </div> </div> </div> </div> </div> </div> </div> </div> @HernanWilkinson </blockquote> <o:p></o:p> <o:p> </o:p> </div> </body> </html>