Hello, I noticed a discrepancy between what cAdvisor reports and what the kubelet /stats/summary endpoint reports, pertaining to memory (working_set_bytes), when using cri-dockerd.
I found that cri-dockerd is seemingly reporting total usage (dockerStats.MemoryStats.Usage) as CRI's WorkingSetBytes. On Linux, this includes inactive file cache and differs from the working-set bytes reported by the Docker CLI, cAdvisor, and containerd.
And in particular for my usecase, it causes the kubelet /stats/summary endpoint to report a higher working-set memory for docker-backed nodes.
I believe the necessary keys exist already in .Stats to subtract the inactive file cache (total_inactive_file on cgroup v1, inactive_file on cgroup v2).
I am happy to provide any further information if necessary!
Hello, I noticed a discrepancy between what cAdvisor reports and what the kubelet /stats/summary endpoint reports, pertaining to memory (working_set_bytes), when using cri-dockerd.
I found that cri-dockerd is seemingly reporting total usage (
dockerStats.MemoryStats.Usage) as CRI's WorkingSetBytes. On Linux, this includes inactive file cache and differs from the working-set bytes reported by the Docker CLI, cAdvisor, and containerd.And in particular for my usecase, it causes the kubelet /stats/summary endpoint to report a higher working-set memory for docker-backed nodes.
I believe the necessary keys exist already in .Stats to subtract the inactive file cache (
total_inactive_fileon cgroup v1,inactive_fileon cgroup v2).I am happy to provide any further information if necessary!