Solveddatadog agent Tagger error on Kubernetes

Describe what happened:
I am trying to set up log collection for Docker containers on a Kubernetes cluster.

Snippet from my config:

    logs:
    - type: docker
      image: "datadog/agent"
      servie: datadog
      source: datadog

Logs are not being sent to Datadog, and I see a lot of these errors in the DD container log:

2018-01-10 16:47:50 UTC | WARN | (tagger.go:248 in Tag) | error collecting from kubelet: container docker://3cacb7a1688be922adcb89f39950d4526cafb1c0e92ee050f630fe0e90132b0b not found in podlist

Describe what you expected:
Working log collection.

Steps to reproduce the issue:
Deploy DD agent on Kuberentes with Daemonset.

Additional environment details (Operating System, Cloud provider, etc):

Kubernetes 1.8.6
Docker 0.13

agent status ===================== Agent (v6.0.0-beta.7) =====================

Status date: 2018-01-10 16:49:09.760419 UTC
Pid: 7
Python Version: 2.7.14
Logs:
Check Runners: 10
Log Level: info

Paths

Config File: /etc/datadog-agent/datadog.yaml
conf.d: /etc/datadog-agent/conf.d
checks.d: /etc/datadog-agent/checks.d

Clocks

NTP offset: 0.00326513 s
System UTC time: 2018-01-10 16:49:09.760419 UTC

Host Info

bootTime: 2018-01-09 20:04:23.000000 UTC
kernelVersion: 4.4.65-k8s
os: linux
platform: debian
platformFamily: debian
platformVersion: 9.3
procs: 63
uptime: 74247
virtualizationRole: guest
virtualizationSystem: xen

Hostnames

ec2-hostname: ip-172-20-174-85.ec2.internal
hostname: i-0d84be79a0113f36b
instance-id: i-0d84be79a0113f36b
socket-fqdn: dd-agent-j6b4h
socket-hostname: dd-agent-j6b4h

=========
Collector

Running Checks

cpu
---
  Total Runs: 29
  Metrics: 6, Total Metrics: 168
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

disk
----
  Total Runs: 29
  Metrics: 160, Total Metrics: 4640
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

docker
------
  Total Runs: 29
  Metrics: 266, Total Metrics: 7342
  Events: 0, Total Events: 5
  Service Checks: 1, Total Service Checks: 29

file_handle
-----------
  Total Runs: 29
  Metrics: 1, Total Metrics: 29
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

io
--
  Total Runs: 29
  Metrics: 52, Total Metrics: 1472
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

kube_dns
--------
  Total Runs: 29
  Metrics: 41, Total Metrics: 1189
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

load
----
  Total Runs: 29
  Metrics: 6, Total Metrics: 174
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

memory
------
  Total Runs: 29
  Metrics: 14, Total Metrics: 406
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

network
-------
  Total Runs: 29
  Metrics: 20, Total Metrics: 580
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

ntp
---
  Total Runs: 29
  Metrics: 1, Total Metrics: 28
  Events: 0, Total Events: 0
  Service Checks: 1, Total Service Checks: 29

uptime
------
  Total Runs: 29
  Metrics: 1, Total Metrics: 29
  Events: 0, Total Events: 0
  Service Checks: 0, Total Service Checks: 0

Loading Errors

docker_daemon
-------------
  Core Check Loader:
    Check docker_daemon not found in Catalog
    
  JMX Check Loader:
    check is not a jmx check, or unable to determine if it's so
    
  Python Check Loader:
    No module named docker_daemon

========
JMXFetch

Initialized checks

no checks

Failed checks

no checks

=========
Forwarder

CheckRunsV1: 29
IntakeV1: 6
RetryQueueSize: 0
Success: 64
TimeseriesV1: 29

API Keys status

https://6-0-0-app.agent.datadoghq.com,*************************4aa08: API Key valid

=========
DogStatsD

Checks Metric Sample: 16694
Event: 6
Events Flushed: 6
Number Of Flushes: 29
Series Flushed: 12074
Service Check: 377
Service Checks Flushed: 393
Dogstatsd Metric Sample: 697

18 Answers

✔️Accepted Answer

I got the same error.

[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://320fd8ad981a5b62d67d7d89b287af341f0e62581fc4b56e866ad774d78b540f not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://320fd8ad981a5b62d67d7d89b287af341f0e62581fc4b56e866ad774d78b540f not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://7e4d3e55e060ce3077bdc675526ab1562677c9d0e072596d275de46ed7c3d64c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://7e4d3e55e060ce3077bdc675526ab1562677c9d0e072596d275de46ed7c3d64c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://330e0acf6f5f8715f9df19711cf5874835c793e22e355424a306df1c0348a31c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://330e0acf6f5f8715f9df19711cf5874835c793e22e355424a306df1c0348a31c not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://f31c94bd9762573cf96bfbc8a85c2d699285102eec6ef47af0bdca809b289a1e not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://f31c94bd9762573cf96bfbc8a85c2d699285102eec6ef47af0bdca809b289a1e not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://6355e733d9f4d06267ff7ed2c156344516502e9986692921a69a6fadb4c1df46 not found in podList
[ AGENT ] 2018-03-08 10:15:13 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kube-service-collector: container docker://6355e733d9f4d06267ff7ed2c156344516502e9986692921a69a6fadb4c1df46 not found in podList
[ AGENT ] 2018-03-08 10:15:14 UTC | WARN | (tagger.go:246 in Tag) | error collecting from kubelet: container docker://3ab71114c06ebd2ed0b641f604ef7e13034399c86de2bf38d323139e3db86735 not found in podList
  • Kubernetes v1.8.7-gke
  • Datadog Agent v6.0.2
  • Install agent using Helm (stable/datadog)
values.yaml
image:
  repository: datadog/agent               # Agent6
  tag: 6.0.2  # Use 6.0.0-jmx to enable jmx fetch collection
  pullPolicy: IfNotPresent

daemonset:
  enabled: true
  updateStrategy: RollingUpdate

deployment:
  enabled: false
  replicas: 1

kubeStateMetrics:
  enabled: true

datadog:
  apiKey: xxxxx
  name: dd-agent
  logLevel: WARNING
  collectEvents: false
  env:
    - name: DD_LOGS_ENABLED
      value: "true"
  leaderLeaseDuration: 600s
  confd:
    kubernetes.yaml: |-
      init_config:
      instances:
        - port: 4194
          collect_events: True
          namespace_name_regexp: .*
    docker_daemon.yaml: |-
      logs:
        - type: docker
          service: docker
          source: docker
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 256m
      memory: 512Mi

rbac:
  create: true
  serviceAccountName: default

tolerations: []

kube-state-metrics:
  rbac:
    create: true
    serviceAccountName: default

Other Answers:

Hi everyone, we have a fix #1345 that should resolve the logging issues and will be included in the next 6.1 release.

We're aware of an issue on kubernetes side where static pods are not correctly updated in the kubelet podlist (#1447) but we'll keep an eye on this to see if there could be other issues.

@mfpierre when will the 6.1.1 release be available as a Helm chart?
Why doesn't every new release result in a new version of the Helm chart?
Related to #1447

Related Issues:

11
datadog agent Tagger error on Kubernetes
I got the same error Kubernetes v1.8.7-gke Datadog Agent v6.0.2 Install agent using Helm (stable/dat...
3
datadog agent Logs with datadog agent and ECS with excluded containers
Hello @amundra2016 Sorry to hear that Describe what happened: We are running tasks on ECS so on a ty...
3
datadog agent Agent v6.5.2 broken logs from docker
Also seeing a similar problem with v6.5.2 Output of the info page (if this is a bug) Describe what h...
523
kubernetes deleting namespace stuck at "Terminating" state
@ManifoldFR I had the same issue as yours and I managed to make it work by making an API call with j...
457
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
447
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
381
git lfs Encountered 1 file(s) that should have been pointers, but weren't
Run git lfs uninstall and then git reset --hard (assuming there are no changes you want to keep!) On...
371
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
369
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
337
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
297
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the docker-entrypoint.sh an executable before building the image: otherwise it cant b...
279
kubernetes PV is stuck at terminating after PVC is deleted
I got rid of this issue by performing the following actions: Then I manually edited the pv individua...
271
kubernetes x509 cert issues after kubeadm init
do you have $KUBECONFIG pointing to /etc/kubernetes/kubelet.conf? BUG REPORT: (I think?) What happen...
264
kubernetes Ingress: Allow for multiple hosts
I also would like to see this feature but as a workaround I use YAML ids Here is how it would look f...
257
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
230
git lfs Simple steps to uninstall Git LFS from your repository
Thanx that helped me a lot while I was migrating from BitBucket to AWS CodeCommit ...
226
git lfs git-lfs aborts with "LFS: Git credentials for https://XXX not found." even for repositories without passwords
In my case on Windows using gitbash I had to use: git config --global credential.helper wincred ...
225
kubernetes The connection to the server localhost:8080 was refused - did you specify the right host or port?
Run these commands solved this issue: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HO...
210
minikube minikube start - Error starting host, machine does not exist
On macOS Sierra rm -rf ~/.minikube fixed it for me After that minikube start worked as expected mini...
208
kubernetes Force pods to re-pull an image without changing the image tag
@yujuhong Sometimes it's very useful to be able to do this For instance Problem A frequent question ...
205
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb http://cz.archive.ubuntu.com/ubuntu trusty main in /etc/...
183
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
182
kubernetes 'unknown revision v0.0.0' errors, seemingly due to 'require k8s.io/foo v0.0.0'
For anyone else who hits this issue after much weeping and gnashing of teeth this is the little scri...
169
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
163
go question: "go: cannot find main module; see 'go help modules'"
After upgrade to go1.16 try to set : go env -w GO111MODULE=auto https://blog.golang.org/go116-module...
161
go Proposal: A built-in Go error check function, "try"
I actually really like this proposal However I do have one criticism The exit point of functions in ...
150
protobuf protoc-gen-go: program not found or is not executable
I had this problem as well recently [using ubuntu 18.04] I found a long-lasting solution by adding t...
149
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
148
minikube kube-proxy configmap update: timed out (unknown root cause)
I had this error when upgrading from 0.25 to 0.26.1 Simply performing minikube delete and then re-cr...
147
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
142
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
138
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
137
gin Got "ambiguous import: found github.com/ugorji/go/codec in multiple modules" after upgraded to latest (1.3.1)
I find a way to fix this problem you can use replace command to set package alias I added the follow...
127
kubernetes JSONpath fails to return keys containing dots in a map
Escaping dots works now To revisit the example in my original message: Closing 🎉 ...
127
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
126
nvm windows nvm installs node but fails to deliver npm
Have the same problem My Environment Windows 10 I'm using NVM4W version: 1.1.7 I have already.. ...
123
kubernetes "Failed to setup network for pod \ using network plugins \"cni\": no IP addresses available in network: podnet; Skipping pod"
I had a simliar issue while testing kubernetes with kubeadm This started to happen after I did a kub...
119
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...
119
compose Release docker/compose image for armv7 / arm64v8
Last time I checked compose wants python 3 while raspbian's global python is still 2 apt-get install...
119
go proposal: Go 2: simplify error handling with || err suffix
A plain idea with support for error decoration There have been many proposals for how to simplify er...
119
kubernetes Kubectl cp gives "tar: removing leading '/' from member names" warning
Something I found is that if I do not put a / at the beginning of my path following : in <pod>:<path...
116
compose I am having this "Encountered errors while bringing up the project." error
Does running docker-compose down solve the issue? when $ docker-compose up it returns Docker Version...
114
moby docker.service Failed with result 'start-limit-hit'.
I had the same error message once because of an empty /etc/docker/daemon.json file Delete it if you ...
112
kubernetes no kind "Deployment" is registered for version "apps/v1beta2"
Small tip: To find out what exact apps/xxx api version your cluster supports use kubectl api-version...
111
minikube Can't pull images from an insecure registry in Minikube VM
I just tried this with minikube v0.10.0 and --insecure-registry='docker-registry.example.com:443' wa...
106
compose "Can't find a suitable configuration file" on Ubuntu 18.04.1
I ran into the same problem with a fresh Ubuntu 18.04 install where I selected Docker from install o...
103
compose How to make a Systemd Unit for docker-compose?
Place following content to the /etc/systemd/system/docker-compose@.service Place you docker-compose....
103
moby containerd: start container" error="oci runtime error: fork/exec /usr/bin/docker (deleted): no such file or directory: "
Not sure if this is the same cause but this might help someone Steps to reproduce the issue: Install...
102
moby Error response from daemon: Get https://registry-1.docker.io/v2/
may be you are behind a firewall/proxy server i was also behind my office firewall so i tried below ...
101
grpc go mustEmbedUnimplemented*** method appear in grpc-server
@josegonzalez I believe is what you are looking for correct me if I am wrong @dfawley ...