<div dir="ltr"><div dir="ltr">Turns out the failing tests were expected in OSX, that's why the exit code is indeed 0.<div><br></div><div>It looks like there is a feature that allows us to mark some tests as "broken" (didn't know about this one). See:</div><div><font face="monospace"><br></font></div><div><font face="monospace">FloatTest expectedFailures<br> Smalltalk platformName = 'Mac OS' ifTrue: [<br> ^#(testE testExp testPrimExpAndLn) ].<br> ^#().</font></div><div><font face="monospace"><br></font></div>What is confusing is that these tests are reported as failures in the console:<div><font face="monospace"><br></font></div><div><div style="box-sizing:border-box;color:rgb(240,246,252);font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;background-color:rgb(1,4,9)"><div class="gmail-js-check-step-line gmail-CheckStep-line gmail-d-flex gmail-log-line-plain" style="box-sizing:border-box;line-height:20px;display:flex"><span class="gmail-CheckStep-line-content gmail-d-inline-block gmail-flex-auto gmail-ml-3 gmail-js-check-line-content" style="box-sizing:border-box;overflow-x:auto;display:inline-block"><span class="gmail-ansifg-r" style="box-sizing:border-box;color:rgb(255,123,114)">--------
</span></span></div></div><div style="box-sizing:border-box;color:rgb(240,246,252);font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;background-color:rgb(1,4,9)"><div class="gmail-js-check-step-line gmail-CheckStep-line gmail-d-flex gmail-log-line-plain" style="box-sizing:border-box;line-height:20px;display:flex"><a class="gmail-CheckStep-line-number gmail-color-fg-muted gmail-d-inline-block gmail-text-mono gmail-text-normal gmail-flex-shrink-0" href="https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2053" style="box-sizing:border-box;background-color:transparent;text-decoration-line:none;width:48px;overflow:hidden;text-align:right;text-overflow:ellipsis;display:inline-block">2052</a><span class="gmail-CheckStep-line-content gmail-d-inline-block gmail-flex-auto gmail-ml-3 gmail-js-check-line-content" style="box-sizing:border-box;overflow-x:auto;display:inline-block"><span class="gmail-ansifg-r" style="box-sizing:border-box;color:rgb(255,123,114)">3 FAILURE(S)
</span></span></div></div><div style="box-sizing:border-box;color:rgb(240,246,252);font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;background-color:rgb(1,4,9)"><div class="gmail-js-check-step-line gmail-CheckStep-line gmail-d-flex gmail-log-line-plain" style="box-sizing:border-box;line-height:20px;display:flex"><a class="gmail-CheckStep-line-number gmail-color-fg-muted gmail-d-inline-block gmail-text-mono gmail-text-normal gmail-flex-shrink-0" href="https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2054" style="box-sizing:border-box;background-color:transparent;text-decoration-line:none;width:48px;overflow:hidden;text-align:right;text-overflow:ellipsis;display:inline-block">2053</a><span class="gmail-CheckStep-line-content gmail-d-inline-block gmail-flex-auto gmail-ml-3 gmail-js-check-line-content" style="box-sizing:border-box;overflow-x:auto;display:inline-block"><span class="gmail-ansifg-r" style="box-sizing:border-box;color:rgb(255,123,114)">--------
</span></span></div></div><div style="box-sizing:border-box;color:rgb(240,246,252);font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;background-color:rgb(1,4,9)"><div class="gmail-js-check-step-line gmail-CheckStep-line gmail-d-flex gmail-log-line-plain" style="box-sizing:border-box;line-height:20px;display:flex"><a class="gmail-CheckStep-line-number gmail-color-fg-muted gmail-d-inline-block gmail-text-mono gmail-text-normal gmail-flex-shrink-0" href="https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2055" style="box-sizing:border-box;background-color:transparent;text-decoration-line:none;width:48px;overflow:hidden;text-align:right;text-overflow:ellipsis;display:inline-block">2054</a><span class="gmail-CheckStep-line-content gmail-d-inline-block gmail-flex-auto gmail-ml-3 gmail-js-check-line-content" style="box-sizing:border-box;overflow-x:auto;display:inline-block"><span class="gmail-ansifg-r" style="box-sizing:border-box;color:rgb(255,123,114)">r274C; FloatTest>>#testE has failed because: Assertion (3) failed.
</span></span></div></div><div style="box-sizing:border-box;color:rgb(240,246,252);font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;background-color:rgb(1,4,9)"><div class="gmail-js-check-step-line gmail-CheckStep-line gmail-d-flex gmail-log-line-plain" style="box-sizing:border-box;line-height:20px;display:flex"><a class="gmail-CheckStep-line-number gmail-color-fg-muted gmail-d-inline-block gmail-text-mono gmail-text-normal gmail-flex-shrink-0" href="https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2056" style="box-sizing:border-box;background-color:transparent;text-decoration-line:none;width:48px;overflow:hidden;text-align:right;text-overflow:ellipsis;display:inline-block">2055</a><span class="gmail-CheckStep-line-content gmail-d-inline-block gmail-flex-auto gmail-ml-3 gmail-js-check-line-content" style="box-sizing:border-box;overflow-x:auto;display:inline-block"><span class="gmail-ansifg-r" style="box-sizing:border-box;color:rgb(255,123,114)">r274C; FloatTest>>#testExp has failed because: Assertion (2) failed.
</span></span></div></div><div style="box-sizing:border-box;color:rgb(240,246,252);font-family:ui-monospace,SFMono-Regular,"SF Mono",Menlo,Consolas,"Liberation Mono",monospace;font-size:12px;background-color:rgb(1,4,9)"><div class="gmail-js-check-step-line gmail-CheckStep-line gmail-d-flex gmail-log-line-plain" style="box-sizing:border-box;line-height:20px;display:flex"><a class="gmail-CheckStep-line-number gmail-color-fg-muted gmail-d-inline-block gmail-text-mono gmail-text-normal gmail-flex-shrink-0" href="https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2057" style="box-sizing:border-box;background-color:transparent;text-decoration-line:none;width:48px;overflow:hidden;text-align:right;text-overflow:ellipsis;display:inline-block">2056</a><span class="gmail-CheckStep-line-content gmail-d-inline-block gmail-flex-auto gmail-ml-3 gmail-js-check-line-content" style="box-sizing:border-box;overflow-x:auto;display:inline-block"><span class="gmail-ansifg-r" style="box-sizing:border-box;color:rgb(255,123,114)">r274C; FloatTest>>#testPrimExpAndLn has failed because: Assertion (1) failed. </span></span></div></div></div></div><div><br></div><div>Maybe we should use another color/category to let users know these were expected failures?</div><div><br></div><div>Best,</div><div>Nico PM</div><br><div class="gmail_quote gmail_quote_container"><div dir="ltr" class="gmail_attr">On Sun, Dec 29, 2024 at 3:00 PM Nicolás Papagna Maldonado <<a href="mailto:nicolas.papagna@gmail.com">nicolas.papagna@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi folks!<div><br></div><div>I was checking out how Cuis runs tests, and It looks like GitHub jobs are passing even when some tests fail:</div><div><ul><li><a href="https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/actions/runs/12520458151" target="_blank">Build jobs passing</a></li><li><a href="https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/actions/runs/12520458151/job/34926155084" target="_blank">Tests log with failed tests</a></li></ul></div><div><br></div>I ran runTests.sh locally (OSX, adjusting VM path) and got exit code 1 (which should have caused the GitHub job to fail).<div><br><div>Anyone else noticed this?</div><div><br></div><div>Best,</div><div>Nico PM</div></div></div>
</blockquote></div><div><br clear="all"></div><div><br></div><span class="gmail_signature_prefix">-- </span><br><div dir="ltr" class="gmail_signature"><br>Nicolás Papagna</div></div>