Hi, On 15-10-2023 14:41, Christian Kastner wrote:
I hope this doesn't derail the thread too much, but regarding this point:
Thanks for the suggestion.
On 2023-10-14 07:31, Paul Gevers wrote:* ensure you can change the timeouts of autopkgtest [...] o ideally per source package, as some tests run successfully, but just take long, or just take long to unpack (so maybe even per timeout type). E.g. we have had to block libreoffice on nearly all our architectures because it's too big to startup on some hosts (also too slow disks). Ubuntu has a big_package list for this [1]. [...]Having run into this issue myself, I wonder whether the autopkgtest maintainers would be interested in extending the spec to support hints for the CI infra, so that maintainers could specify e.g.: test-timeout: 30m So basically, instead of (or in addition to) CI infra operators to maintain their own lists like [1] below, for package maintainers to be able to proactively suggest performance constraints. The benefit would be that being part of the package, other testers (eg: teams with their own CI; CI of derivative distributions) would also have access to this information, whereas lists such as [1] are operator-specific.
To be honest, I think time isn't going to be very useful. One thing is that reasonable timeouts also depend a lot on the infra (like architecture, I/O setup, network speed, parallel other jobs, amount of CPU's, amount of memory, ...) and isn't only determined by the test. I've seen test run in minutes on one arch, taking hours on another. I've seen tests taking minutes on a beefy host (lots of memory and cores), while taking hours on a regular host. I have filed quite some bugs already where the internal test time outs are too strict for running on our ci while they look OK from upstream point of view (running on amd64 on their own laptop).
This would need to be much finer grained to work. Maybe a test could hint that default timeout should be extended or reduced, so maybe a scaling factor might work.
Paul
Attachment:
OpenPGP_signature.asc
Description: OpenPGP digital signature