Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrated scheduler files comparer.go, dumper.go, node_tree.go to structured logging #105968

Merged
merged 1 commit into from Nov 9, 2021

Conversation

shiva1333
Copy link
Contributor

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Migrated pkg/scheduler/internal/cache/debugger/comparer.go, pkg/scheduler/internal/cache/debugger/dumper.go, and pkg/scheduler/internal/cache/node_tree.go to structured logging

Which issue(s) this PR fixes:

Ref #105841

Special notes for your reviewer:

Added missing migration points, fixed previously migrated logs

Does this PR introduce a user-facing change?

Migrated scheduler files `comparer.go`, `dumper.go`, `node_tree.go` to structured logging

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:


@k8s-ci-robot k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 28, 2021
@shiva1333
Copy link
Contributor Author

/sig instrumentation
/wg structured-logging
/area logging
/priority important-soon
/kind cleanup
/cc @kubernetes/wg-structured-logging-reviews

@k8s-ci-robot k8s-ci-robot added sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging. area/logging priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Oct 28, 2021
@shiva1333 shiva1333 added this to Waiting on Reviewer in WG Structured Logging - 1.23 Migration Oct 28, 2021
@@ -57,7 +57,7 @@ func (d *CacheDumper) dumpSchedulingQueue() {
for _, p := range pendingPods {
podData.WriteString(printPod(p))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please don't use custom object serialization for pods. Please either use klog.KObjs(pendingPods) to get basic information like pod name and namespace or propose custom struct that will include fields that should be printed.

cc @pohly
This is interesting case where we want to log slice of K8s object but with additional information, maybe you have suggestion how to best handle it.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's possible to write a custom formatter like this:

type formatter *v1.Pod

func (f formatter) String() string {
    return fmt.Sprintf("%s/%s ...", f.Namespace, f.Name, ....)
}

func (f formatter) MarshalLog interface{} {
    return struct {
        Namespace, Name string
        <other data>
    } {
       Namespace: f.Namespace,
       Name: f.Name,
       <other data>
   }
}

var _ logr.Marshaler = formatter{}

klog.Infos("some data about a pod", "pod", formatter(pod))

Text output via klog will call Stringer and JSON output will call MarshalLog, then render the struct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This example is for a single pod. The same works with []*v1.Pod or any other type, it just gets a bit more complex.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So before this new helper lib is available, we keep the current code as-is?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this PR needs a replacement for podData.WriteString and printPod, then this PR also would be the right place to implement this custom formatter.

It could be local to the pkg/scheduler/internal/cache/debugger package. Or is the same code also needed somewhere else?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If this PR needs a replacement for podData.WriteString and printPod

I don't have a strong option on this as the debugger package is not extensively used, esp. not by the regular end-users.

Or is the same code also needed somewhere else?

I think the pattern is only used in the debugger pkg.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@shivanshu1333: do you intend to implement such a formatter as part of this PR?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, actually I've some doubts I need your help to figure out the right way to do this, I had posted a query in slack for the same https://kubernetes.slack.com/archives/C020CCMUEAX/p1636304943054700

@dashpole
Copy link
Contributor

dashpole commented Nov 4, 2021

/assign @serathius
/triage accepted

@k8s-ci-robot k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Nov 4, 2021
@serathius
Copy link
Contributor

/unassign
/assign @pohly

@k8s-ci-robot k8s-ci-robot assigned pohly and unassigned serathius Nov 8, 2021
@serathius
Copy link
Contributor

This PR is needed to finish scheduler structured logging migration planned for 1.23
/milestone v1.23

@k8s-ci-robot k8s-ci-robot added this to the v1.23 milestone Nov 8, 2021
@shiva1333
Copy link
Contributor Author

/retest

@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Nov 9, 2021
@shiva1333
Copy link
Contributor Author

The unit test to check the behaviour for formatter.go is https://gist.github.com/shivanshu1333/3e7b7cda9fad55c2065900cd30f75781
If needed, I can add the unit test in the project itself

@shiva1333
Copy link
Contributor Author

cc @pohly @serathius @damemi

@shiva1333
Copy link
Contributor Author

Please let me know if any change is required, and also let’s think about adding the above unit test as well, please guide me about possibilities of introducing a unit test for formatter.go

// printPod writes parts of a Pod object to a string.
func printPod(p *v1.Pod) string {
return fmt.Sprintf("name: %v, namespace: %v, uid: %v, phase: %v, nominated node: %v\n", p.Name, p.Namespace, p.UID, p.Status.Phase, p.Status.NominatedNodeName)
klog.InfoS("Dump of scheduling queue", "podData", formatPod{pendingPods})
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
klog.InfoS("Dump of scheduling queue", "podData", formatPod{pendingPods})
klog.InfoS("Dump of scheduling queue", "podData", formatPods{pendingPods})

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This of course then also needs to be updated in formatter.go.

func (f formatPod) String() string {
var podInfo strings.Builder
for _, v := range f.pods {
podInfo.WriteString(fmt.Sprintf("Name: %v, Namespace: %v, UID: %v, Phase: %v, NominatedNode: %v\n", v.Name, v.Namespace, v.UID, v.Status.Phase, v.Status.NominatedNodeName))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In #104868 it was pointed out that multi-line strings do not actually get printed as multiple lines, making the text output fairly hard to read.

I wonder whether the same happens here... yes, it does, as your unit test in https://gist.github.com/shivanshu1333/3e7b7cda9fad55c2065900cd30f75781 shows.

I suspect the scheduler developers will consider this as a regression because it makes the text output less useful to them.

Your formatter looks (mostly) good to me and we should come back to it once we know how to emit multi-line strings in text format, but for now we should keep the original klog.Info(d.printNodeInfo(name, nodeInfo)) and
klog.Infof("Dump of scheduling queue:\n%s", podData.String()).

})
}

if nodeInfo.NominatedPods != 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if check can be removed.

nodeInfo.WriteString(fmt.Sprintf("Name: %v, Namespace: %v, UID: %v, Phase: %v, NominatedNode: %v\n", v.Pod.Name, v.Pod.Namespace, v.Pod.UID, v.Pod.Status.Phase, v.Pod.Status.NominatedNodeName))
}

if len(f.nominatedPodInfo) != 0 {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This if check can be removed.

@shiva1333
Copy link
Contributor Author

Your formatter looks (mostly) good to me and we should come back to it once we know how to emit multi-line strings in text format, but for now we should keep the original klog.Info(d.printNodeInfo(name, nodeInfo)) and
klog.Infof("Dump of scheduling queue:\n%s", podData.String())

@pohly Does this mean that for the scope of this migration PR, let's not worry about the formatter in this PR and keep the original things as it is, and start new efforts (PR) to continue formatting work?

@pohly
Copy link
Contributor

pohly commented Nov 9, 2021

Yes. It means the migration won't be complete, but that's better than migrating and producing output that is worse than what it is without the migration.

@k8s-ci-robot k8s-ci-robot added size/S Denotes a PR that changes 10-29 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Nov 9, 2021
@shiva1333
Copy link
Contributor Author

Created #106276 to take care of custom formatter, I think now this PR is unblocked and we can work on the mentioned issue separately.
Looking for LGTM and APPROVED label on this

pkg/scheduler/internal/cache/debugger/dumper.go Outdated Show resolved Hide resolved
pkg/scheduler/internal/cache/debugger/dumper.go Outdated Show resolved Hide resolved
@Huang-Wei
Copy link
Member

/approve

Leaving /lgtm to @pohly.

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Nov 9, 2021
Copy link
Contributor

@pohly pohly left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Nov 9, 2021
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Huang-Wei, pohly, shivanshu1333

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@shiva1333
Copy link
Contributor Author

/retest

@k8s-ci-robot k8s-ci-robot merged commit 8eda02c into kubernetes:master Nov 9, 2021
WG Structured Logging - 1.23 Migration automation moved this from Waiting on Reviewer to Done Nov 9, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. area/logging cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. kind/cleanup Categorizes issue or PR as related to cleaning up code, process, or technical debt. lgtm "Looks good to me", indicates that a PR is ready to be merged. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/instrumentation Categorizes an issue or PR as relevant to SIG Instrumentation. sig/scheduling Categorizes an issue or PR as relevant to SIG Scheduling. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. triage/accepted Indicates an issue or PR is ready to be actively worked on. wg/structured-logging Categorizes an issue or PR as relevant to WG Structured Logging.
Development

Successfully merging this pull request may close these issues.

None yet

6 participants