[Cuis-dev] GitHub jobs passing even when tests fail

[Nicolás Papagna Maldonado]

Turns out the failing tests were expected in OSX, that's why the exit code
is indeed 0.

It looks like there is a feature that allows us to mark some tests as
"broken" (didn't know about this one). See:

FloatTest expectedFailures
  Smalltalk platformName = 'Mac OS' ifTrue: [
    ^#(testE testExp testPrimExpAndLn) ].
  ^#().

What is confusing is that these tests are reported as failures in the
console:

--------
2052
<https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2053>3
FAILURE(S)
2053
<https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2054>--------

2054
<https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2055>&#16r274C;
FloatTest>>#testE has failed because: Assertion (3) failed.
2055
<https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2056>&#16r274C;
FloatTest>>#testExp has failed because: Assertion (2) failed.
2056
<https://github.com/npapagna/Cuis-Smalltalk-Dev-Fix-CI-Tests/actions/runs/12538106004/job/34962890559#step:5:2057>&#16r274C;
FloatTest>>#testPrimExpAndLn has failed because: Assertion (1) failed.

Maybe we should use another color/category to let users know these were
expected failures?

Best,
Nico PM

On Sun, Dec 29, 2024 at 3:00 PM Nicolás Papagna Maldonado <
nicolas.papagna at gmail.com> wrote:

> Hi folks!
>
> I was checking out how Cuis runs tests, and It looks like GitHub jobs are
> passing even when some tests fail:
>
>    - Build jobs passing
>    <https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/actions/runs/12520458151>
>    - Tests log with failed tests
>    <https://github.com/Cuis-Smalltalk/Cuis-Smalltalk-Dev/actions/runs/12520458151/job/34926155084>
>
>
> I ran runTests.sh locally (OSX, adjusting VM path) and got exit code 1
> (which should have caused the GitHub job to fail).
>
> Anyone else noticed this?
>
> Best,
> Nico PM
>


-- 

Nicolás Papagna
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.cuis.st/mailman/archives/cuis-dev/attachments/20241229/50eae2f8/attachment-0001.htm>