If you have the option, changing execution priority can help a lot, as can moving to a different scheduling engine. Using a Real-Time scheduler and flagging the cache-process as such will do a lot for maintaining a steady maximum throughput.

To be frank, I have never have much trusted fair-share style scheduling systems – while they work well for the common workloads, they really only work well for the common workloads.

Give them a workload that has a feeder component, or one that is heavily DMA and/or IO bound and things get strange.

Additionally, most of the development and modelling that has been done for Scheduler performance is based on Multiple-Process workloads, not on the newer Multiple-Thread environment.

The tuning paramaters (and the associated bottlenecks and scheduler (and dont forget the memory manager) prioritisation quirks) change subtly (but significantly) when you are looking at using inter-process communication (pipes, sockets & the like) rather than intra-process communication (shared memory regions, mutex-locks, etc).

Part of this will be to do with Process vs. Process prioritisation as opposed to Process vs. (Thread vs. Thread vs. Thread) prioritisation.

Marshall K McKusick has some very interesting things to say about thread scheduling in his ACMpresentation from 2004.

MKK’s scheduling work has been the basis for that done by many other people. In particular, he has the following to say about how the legacy queuing systems work:

The system tailors this short-term scheduling algorithm to favor interactive
jobs by raising the scheduling priority of threads that are blocked waiting for
I/O for one or more seconds and by lowering the priority of threads that
accumulate significant amounts of CPU time.

Your post is wonderfully composed, and does a very good job of exposing a rarely approached portion of the system to a much wider audience – thankyou.

That’s what generates the staccato behaviour – the system starves, causing the whole pipeline to stop, the ends eventually block, leaving idle cycles, which then bring the pipeline to life. All the parts start working again, and then scheduler starvation stalls the components in the middle again, bringing everything to an eventual halt. Rinse and repeat.