RFC: (ab)using autopkgtest for benchmarking

[Christian Kastner <ckk@debian.org>]

I would like to initial a discussion on -devel on the possible merits of
packages shipping benchmark tests, specifically in the form of autopkgtests.

Such tests being beyond the scope of autopkgtest, I would like to
solicit the maintainers' feedback before doing so.

Specifically, I would propose using the following new restrictions
(unknown to autopkgtest, and therefore skipped by default):

   benchmark-task       A typical task for which the package is used
   benchmark-io         An I/O intensive task
   benchmark-network    A task that requires connectivity

I see at least three use cases for this:
  (1) On the same platform: comparison of the effects of build
      parameters, eg:
        * compiler optimization flags (various ISAs)
        * effects of features (eg: spectre/meltdown prevention)
  (2) Comparing platforms to each other
  (3) Baselining the current system, which could be interesting
      especially in the cloud computing context, eg:
        * Local storage speed
        * Network throughput, latency

As a concrete example: The maintainer of gzip could a "benchmark-task"
test for a typical compression scenario.

Do you think that pursuing this would be worthwhile? Would you have any
objections or concerns regarding the suggestion of using autopkgtest to
this end?

Christian