Can you think of a way to validate if this metric is accurate and useful?

Currently, we post at most 1000 samples every second. Since there's no particular logic to decide which samples should proceed, I fear that the information might be skewed towards certain extension point.

It will not be skewed because we either sample the whole cycle or not. We can increase the sampling rate if we wish to make it more accurate, but in the end it depends on the number of instances we are sampling from (so number of scheduling cycles), and that depends on the cluster.

I don't think it provides useful information fleet-wide, but when debugging a specific cluster that we know uses heavily a specific feature, this metric can provide useful information on how much time that feature is adding overhead on the scheduling cycle.

It will not be skewed because we either sample the whole cycle or not.

Why do you say that?

Because we don't sample based on the extension point, we either time for all extension points or for none of them in a scheduling cycle.

Also, the sampling is random

to recap from offline discussion: There are 2 things controlling what gets recorded:

• a random sampling per scheduling cycle

  kubernetes/pkg/scheduler/scheduler.go

  Line 441 in 6ac2d8e

  state.SetRecordPluginMetrics(rand.Intn(100) < pluginMetricsSamplePercent)

• a buffer

  kubernetes/pkg/scheduler/framework/runtime/metrics_recorder.go

  Line 54 in 6ac2d8e

  bufferCh: make(chan *frameworkMetric, bufferSize),

My only concern is that the buffer might cause certain extension points to not be recorded because they are not called often. I think the most important one is the preemption PostFilter plugin. I would be more comfortable graduating this metric if we can ensure that the buffer is enough to fit an entire scheduling cycle for a 5k cluster with a default percentageOfNodesToScore.

ok, I think we should just drop the graduation of this one for now, for the filter plugin, it would be ideal if we can measure for all nodes, not per node.

+1 that would reduce a lot of measuring points.

Thanks!