Add a new webhook request total metric and add a 1.0s bucket boundary to the webhook latency metric #103162

rmoriar1 · 2021-06-24T18:27:05Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

We have a webhook which is configured to fail open and we would like to use metrics to detect if it is working properly.

The existing admission_webhook_rejection_count metric doesn't track requests that failed open (because they are not considered rejections) and the admission_webhook_admission_duration_seconds only includes a bool indicating whether the request was rejected or not. As a result, we are unable to track the number of requests that have failed open.

This PR adds a new admission_webhook_request_total metric which includes the webhook's HTTP response as a label. This will allow us to identify requests that have failed open.

This also adds a new bucket boundary of 1.0s to the latency metric, which will enable us to get more accurate latency data from the webhook.

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Introduces a new metric: admission_webhook_request_total with the following labels: name (string) - the webhook name, type (string) - the admission type, operation (string) - the requested verb, code (int) - the HTTP status code, rejected (bool) - whether the request was rejected, namespace (string) - the namespace of the requested resource.

k8s-ci-robot · 2021-06-24T18:27:08Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2021-06-24T18:27:13Z

Welcome @rmoriar1!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

k8s-ci-robot · 2021-06-24T18:27:13Z

Hi @rmoriar1. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

alexcope

nevermind

staging/src/k8s.io/apiserver/pkg/admission/metrics/metrics_test.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go

caesarxuchao · 2021-06-24T20:09:07Z

/cc @roycaihw
/sig instrumentation
/remove-sig api-machinery

staging/src/k8s.io/apiserver/pkg/admission/metrics/metrics.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/mutating/dispatcher.go

staging/src/k8s.io/apiserver/pkg/admission/plugin/webhook/validating/dispatcher.go

lavalamp · 2021-07-08T22:24:57Z

looks generally good, some suggestions though.

lavalamp · 2021-07-09T17:38:15Z

/lgtm
/approve

k8s-ci-robot · 2021-07-09T17:38:39Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lavalamp, logicalhan, rmoriar1

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/apiserver/OWNERS~~ [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rmoriar1 · 2021-07-09T18:29:37Z

/retest

dims · 2021-08-05T10:30:53Z

@rmoriar1 needs rebase

Also add a 1.0s bucket boundary to the webhook latency metric.

rmoriar1 · 2021-08-05T18:48:18Z

@rmoriar1 needs rebase

Done.

rmoriar1 · 2021-08-05T20:08:55Z

@lavalamp Mind re-reviewing after rebase?

lavalamp · 2021-08-13T18:50:48Z

/lgtm

k8s-triage-robot · 2021-08-13T22:04:23Z

The Kubernetes project has merge-blocking tests that are currently too flaky to consistently pass.

This bot retests PRs for certain kubernetes repos according to the following rules:

The PR does have any do-not-merge/* labels
The PR does not have the needs-ok-to-test label
The PR is mergeable (does not have a needs-rebase label)
The PR is approved (has cncf-cla: yes, lgtm, approved labels)
The PR is failing tests required for merge

You can:

Review the full test history for this PR
Prevent this bot from retesting with /lgtm cancel or /hold
Help make our tests less flaky by following our Flaky Tests Guide

/retest

k8s-ci-robot added release-note-none Denotes a PR that doesn't merit a release note. kind/feature Categorizes issue or PR as related to a new feature. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 24, 2021

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Jun 24, 2021

k8s-ci-robot requested review from apelisse and sttts June 24, 2021 18:27

k8s-ci-robot added area/apiserver sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. labels Jun 24, 2021

rmoriar1 force-pushed the rmoriar1-add-http-code-to-webhook-metric branch from d85a012 to 5a9ed6d Compare June 24, 2021 18:58

alexcope suggested changes Jun 24, 2021

View reviewed changes

k8s-ci-robot requested a review from roycaihw June 24, 2021 20:09

k8s-ci-robot removed the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Jun 24, 2021

rmoriar1 force-pushed the rmoriar1-add-http-code-to-webhook-metric branch from 7e41658 to 709279c Compare June 24, 2021 20:34

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Jun 24, 2021

rmoriar1 force-pushed the rmoriar1-add-http-code-to-webhook-metric branch from 709279c to c3afa46 Compare June 24, 2021 21:01

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 24, 2021

rmoriar1 force-pushed the rmoriar1-add-http-code-to-webhook-metric branch from c3afa46 to 83bfc00 Compare June 24, 2021 21:04

k8s-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jun 24, 2021