[Cuis-dev] BaseImageTests for 5302

[Andres Valloud]

These tests should not be written in terms of hardcoded milliseconds. 
What matters is the relationship between the time these take and the 
time it takes to do a number of very simple operations, like

	1 to: 1000000 do: [:x | x + 1]

or the factorial of a reasonably small number (so that it's somewhat 
resistant to future optimization).  Define a standard unit of 
measurement in terms of such an operation.  Then, the tests should say 
something like this.

	self
		itTakesNoMoreThan: anInteger
		timeUnitsToRun: ["this stuff"]

Then you actually compare the two.  It takes a bit more to run the test 
because now the measurement is relative, but you no longer care how fast 
or slow the machine is.  This carries the assumption that the system is 
reasonably fair with regards to time allocation, of course.

If you do not do something like this, you will be periodically 
increasing and decreasing hardcoded times which will look magical and 
dangerous.  Of course such times will increase to the point that the 
test never fails on *any* machine (because otherwise you get more 
reports along the lines of the ones you got now).  And now that means 
that grotesque performance can still pass the test because it could be 
that what you're measuring should take 1ms on a fast machine, but since 
it takes 90ms instead the test passes because the threshold is 100ms.

On 6/23/22 5:48 PM, Juan Vuletich via Cuis-dev wrote:
> On 6/23/2022 6:57 PM, ken.dickey--- via Cuis-dev wrote:
>> On 2022-06-23 08:31, Juan Vuletich wrote:
>> ..
>>> Nope. #valueWithin:onTimeout: should work correctly if processes are
>>> preempted. It actually uses process preemption to do its job (see the
>>> implementation).
>>>
>>> The only (non-) problem here is that on the Sipeed Lichee RV,
>>> #valueWithin:onTimeout: takes a bit longer to do its job. I just want
>>> to give it this little extra time. That's why I'm asking you for an
>>> appropriate timeout value.
>>
>> In a workspace [LicheeRV Dock]:
>>
>>  [1000 milliseconds asDelay wait] valueWithin: 100 milliSeconds 
>> onTimeout: [] ]
>>    durationToRun.
>>   -> 0:00:00:00:102   [102 milliseconds]
>>
>>  [ [ [5 seconds asDelay wait] valueWithin:100 milliseconds onTimeout: 
>> [] ]
>>    valueWithIn: 500 milliseconds onTimeOut: [] ] durationToRun.
>>   -> 0:00:00:00:104   [104 milliseconds]
>>
>> The preemptions can only happen during #durationToRun, right?  Between 
>> getting the start time, running the block, and getting the end time.
> 
> No. Preemption can happen almost anytime. For instance, for sure 
> preemption will occur inside #valueWithin:onTimeout:, as I said before. 
> Without preemption, there would be no way to stop the delay before it ends!
> 
>> This is why running the tests uninterruptedly (e.g. Cmd-t in a 
>> browser) pass and running in background fails.
> 
> Ok. I see now. It looks like in slow hardware Morphic takes all CPU and 
> doesn't leave enough time to run the tests quickly. Please try this 
> version of the test:
> 
> testValueWithinTimingBasic
>      "Test timing of valueWithin:onTimeout:"
>      | time |
>      [
>          time := [
>              [1000 milliSeconds asDelay wait]
>                  valueWithin: 100 milliSeconds onTimeout: []
>          ] durationToRun.
>          self assert: time < 200 milliSeconds.
>      ] forkAt: Processor userInterruptPriority.
> 
> I believe this could do the trick. And actually that something like 110 
> or 120 milliseconds could be enough, never requiring 200 milliseconds.
> 
>> I think that giving extra time for #durationToRun in background is 
>> testing the wrong thing.
>>
>> $0.02,
>> -KenD
> 
> BTW, can you post the results you get on `0 tinyBenchmarks`? I'd like to 
> add them to the comment in the method.
> 
> Thanks,
>