kernel defconfig, debugging, preemption, and very noticeable speedups/ debugging

[Werner Almesberger]

Gennady Kupava wrote:
> so you state that if we mesure T({}) - T({i} for each i and compare to
> epsilon, we can judge can that option be included or not: if T({}) -
> T({i} < epsilon so lets keep option.

Yes, or T(S)-T(S+i) if you think the cumulative effects dominate.

The single option tests would show where an option spends an
inordinate amount of CPU time (or has an excessive cache impact
already on its own). Comparing the sum of T({i}) with T(union i)
would also let us quantify how much the cumulative effects
contribute.

The cumulative test would also show at what point we start
thrashing. So where there's a big step in the series of results,
one would have to further examine whether it's caused by CPU
time or by cache thrashing.

Considering that our CPU has quite small caches, I would expect
them to mainly affect relatively small loops, so cumulative effects
should be small, but I may very well be wrong on this.

> so, for example if i'll later propose to change 2 options not related to
> debug later and cumulative effect will be > epsilon, I'll have hard time
> to prove that, instead of simple change, move on and measure all options
> in sum.

Ah, but I'm not saying that we should use the results like a
restaurant menu from which we pick items until we've reached the
budget in our wallets.

What I'm looking for is an indication of which options are
particularly bad and which look friendly. So if anyone at some
point in time proposes to enable one of the nasty ones, we'll
already know that this will cause problems.

The apparently friendly ones would not raise an immediate red
flag. But of course, if your results show a large cumulative
effect, then it would be reasonable to also ask the proponent
to provide a performance comparison.

With your results in hand, it will be easy to convince future
contributors that such a performance evaluation is not just "the
kernel guys" being difficult, but that there could be unexpected
consequences that need to be examined.

> problem 1: we can't measure C({i}) for all tasks, we can only do some
> systetetic testing.

Yes, that's the crux of all benchmarks ;-) I would pick boot to
GUI time as an important synthetic test, because of its high
visibility (even though we shouldn't have to boot very often ...)

It would also be nice to have a test of interactive responsiveness,
e.g., application launch time. Do we have something that could be
set up without too much effort ?

> problem 2: options depends on each other you can't measure some options
> 'alone'

Fortunately, this one is easy :-) If option B depends on A, you
could first measure { A, B } and if it has a major impact, then
measure { A } too.

> problem 3: people don't know if they need or not particual options,

Yes, this one is a bit more work and I agree that we have to do
this as well. I'm also curious if any of the options we consider
indispensable might turn up being troublesome. In this case, the
question would change from "what is the minimum set of options we
need ?" to "how can we make the options we need behave better ?".
Quite a difference.

> so, all the tasks or measuring 'options one by one' reminds me russian
> national fairytale where Ivan-the-fool was sent to bring 'something I do
> not know particulary what'. it brings nothing but sends me to the
> distant travel.

Heh, I very much hope that it's not a fool's errand :-) What I'm
hoping for is a map that allows us to gain a deeper understanding
of how the debug options affect performance and how tightly our
choices are constrained.

I still expect some options to stick out as particularly expensive.
It will be good to know which ones they are.

> so, we'll never agree to keep some options, as i'll be
> always for performance except reason to keep (like backtraces) is strong
> enought from my pov,

You see, you already agree to keep at least something :-)
Now let's find out how expensive it is.

> also, i think you assumption that some one particular option is
> 'responsible' for slowdown is wrong, sorry, but soon be sure in that.

Perhaps I wasn't clear on this. I'm not looking for the "one
culprit" that has to be removed and then everything will look
fine again and we can continue ignoring the problem.

Quite to the contrary. Now that it's obvious that we have a
problem, I want to know more about it. Maybe there is indeed
just one option that causes all the misery. But I wouldn't be
surprised if there are several with a non-trivial impact. The
things is, without experiments, we can't be sure which they are.

Thanks,
- Werner