Solveddatadog agent Agent v6.5.2 broken logs from docker

Output of the info page (if this is a bug)

==============
Agent (v6.5.2)
==============

  Status date: 2018-09-28 09:34:53.577571 UTC
  Pid: 808
  Python Version: 2.7.15
  Logs: 
  Check Runners: 4
  Log Level: info

  Paths
  =====
    Config File: /etc/datadog-agent/datadog.yaml
    conf.d: /etc/datadog-agent/conf.d
    checks.d: /etc/datadog-agent/checks.d

  Clocks
  ======
    NTP offset: -346µs
    System UTC time: 2018-09-28 09:34:53.577571 UTC

  Host Info
  =========
    bootTime: 2018-09-27 19:59:30.000000 UTC
    kernelVersion: 3.10.0-693.2.2.el7.x86_64
    os: linux
    platform: centos
    platformFamily: rhel
    platformVersion: 7.4.1708
    procs: 148
    uptime: 11s

  Hostnames
  =========
    hostname: k***s.com
    socket-fqdn: h***2.hostwindsdns.com.
    socket-hostname: h***2.hostwindsdns.com
    hostname provider: configuration

=========
Collector
=========

  Running Checks
  ==============
    
    cpu
    ---
        Instance ID: cpu [OK]
        Total Runs: 3,261
        Metric Samples: 6, Total: 19,560
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 0s
        
    
    disk (1.3.0)
    ------------
        Instance ID: disk:e5dffb8bef24336f [OK]
        Total Runs: 3,261
        Metric Samples: 58, Total: 175,170
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 40ms
        
    
    file_handle
    -----------
        Instance ID: file_handle [OK]
        Total Runs: 3,260
        Metric Samples: 5, Total: 16,300
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 0s
        
    
    io
    --
        Instance ID: io [OK]
        Total Runs: 3,261
        Metric Samples: 26, Total: 84,768
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 3ms
        
    
    load
    ----
        Instance ID: load [OK]
        Total Runs: 3,260
        Metric Samples: 6, Total: 19,560
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 0s
        
    
    memory
    ------
        Instance ID: memory [OK]
        Total Runs: 3,261
        Metric Samples: 17, Total: 55,437
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 0s
        
    
    network (1.6.1)
    ---------------
        Instance ID: network:2a218184ebe03606 [OK]
        Total Runs: 3,261
        Metric Samples: 32, Total: 104,340
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 1ms
        
    
    ntp
    ---
        Instance ID: ntp:b4579e02d1981c12 [OK]
        Total Runs: 3,261
        Metric Samples: 1, Total: 3,261
        Events: 0, Total: 0
        Service Checks: 1, Total: 3,261
        Average Execution Time : 31ms
        
    
    uptime
    ------
        Instance ID: uptime [OK]
        Total Runs: 3,261
        Metric Samples: 1, Total: 3,261
        Events: 0, Total: 0
        Service Checks: 0, Total: 0
        Average Execution Time : 0s
        
========
JMXFetch
========

  Initialized checks
  ==================
    no checks
    
  Failed checks
  =============
    no checks
    
=========
Forwarder
=========

  CheckRunsV1: 3,260
  Dropped: 0
  DroppedOnInput: 0
  Errors: 81
  Events: 0
  HostMetadata: 0
  IntakeV1: 249
  Metadata: 0
  Requeued: 87
  Retried: 82
  RetryQueueSize: 0
  Series: 0
  ServiceChecks: 0
  SketchSeries: 0
  Success: 6,769
  TimeseriesV1: 3,260

  API Keys status
  ===============
    API key ending in 24fb5 for endpoint https://app.datadoghq.com: API Key valid

==========
Logs Agent
==========

  custom
  ------
    Type: docker
    Name: front-container
    Status: Pending
  
=========
DogStatsD
=========

  Checks Metric Sample: 533,936
  Event: 1
  Events Flushed: 1
  Number Of Flushes: 3,260
  Series Flushed: 420,505
  Service Check: 29,401
  Service Checks Flushed: 32,652

Describe what happened:
Logs Agent dont collect logs, just pending.

Describe what you expected:
Agent (v6.4.2) works as expected (installed on same server 4/9/2018 13:08)

  custom
  ------
    Type: docker
    Name: front-container
    Status: OK
    Inputs: 99b00c7d6467b686ce83333dfb86e5297cd20cd1810b99e0ac32dd218cadade1 

Steps to reproduce the issue:
Agent (v6.4.2), Docker version 18.06.1-ce, build e68fc7a - working
Agent (v6.5.2), Docker version 18.06.1-ce, build e68fc7a - not working

/etc/datadog-agent/conf.d/custom.yaml

logs:
  - type: docker
    name: front-container
    source: nginx
    service: docker

datadog.yaml

dd_url: https://app.datadoghq.com

api_key: a***5

hostname: k***s.com

tags:
  - role:shop-front

logs_enabled: true

Additional environment details (Operating System, Cloud provider, etc):
Centos 7, Docker version 18.06.1-ce
Digital Ocean/Hostwindsdns - same problem

22 Answers

✔️Accepted Answer

Also seeing a similar problem with v6.5.2. We are using Docker labels on our app containers to configure the Datadog container agent on hosts running Docker version 18.06.1-ce. Seems like Datadog is having a problem with parsing container labels, which didn't change when we upgraded Datadog

[ AGENT ] 2018-09-27 22:54:20 UTC | INFO | (file.go:70 in Collect) | File: searching for configuration files at: /etc/datadog-agent/conf.d
[ AGENT ] 2018-09-27 22:54:20 UTC | INFO | (tailer.go:86 in Start) | Start tailing container: e***1
[ AGENT ] 2018-09-27 22:54:20 UTC | INFO | (file.go:70 in Collect) | File: searching for configuration files at: /opt/datadog-agent/bin/agent/dist/conf.d
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (file.go:74 in Collect) | Skipping, open /opt/datadog-agent/bin/agent/dist/conf.d: no such file or directory
[ AGENT ] 2018-09-27 22:54:20 UTC | INFO | (file.go:70 in Collect) | File: searching for configuration files at:
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (file.go:74 in Collect) | Skipping, open : no such file or directory
[ AGENT ] 2018-09-27 22:54:20 UTC | ERROR | (docker.go:126 in parseDockerLabels) | Can't parse template for container c***9: missing instances key
[ AGENT ] 2018-09-27 22:54:20 UTC | ERROR | (docker.go:126 in parseDockerLabels) | Can't parse template for container d***f: missing instances key
[ AGENT ] 2018-09-27 22:54:20 UTC | ERROR | (docker.go:126 in parseDockerLabels) | Can't parse template for container 8***5: missing instances key
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (check.go:250 in Configure) | could not get a check instance with the new api: __init__() takes at least 4 arguments (4 given)
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (check.go:251 in Configure) | trying to instantiate the check with the old api, passing agentConfig to the constructor
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (check.go:276 in Configure) | passing `agentConfig` to the constructor is deprecated, please use the `get_config` function from the 'datadog_agent' package (disk).
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (check.go:250 in Configure) | could not get a check instance with the new api: __init__() takes at least 4 arguments (4 given)
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (check.go:251 in Configure) | trying to instantiate the check with the old api, passing agentConfig to the constructor
[ AGENT ] 2018-09-27 22:54:20 UTC | WARN | (check.go:276 in Configure) | passing `agentConfig` to the constructor is deprecated, please use the `get_config` function from the 'datadog_agent' package (network).

2018/10/05 additional info:

We are running our agent containers with the following environment variables:

DD_API_KEY=8***d
DD_DOGSTATSD_NON_LOCAL_TRAFFIC=true //to collect statsd from containers on host
DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL=true
DD_LOGS_ENABLED=true
SD_BACKEND=docker

Example of a container service with Docker labels:

com.datadoghq.ad.check_names=["nginx"]
com.datadoghq.ad.init_configs=[{}]
com.datadoghq.ad.logs=[{"source": "nginx", "service": "my-service"}]

Related Issues:

11
datadog agent Tagger error on Kubernetes
I got the same error Kubernetes v1.8.7-gke Datadog Agent v6.0.2 Install agent using Helm (stable/dat...
3
datadog agent Logs with datadog agent and ECS with excluded containers
Hello @amundra2016 Sorry to hear that Describe what happened: We are running tasks on ECS so on a ty...
3
datadog agent Agent v6.5.2 broken logs from docker
Also seeing a similar problem with v6.5.2 Output of the info page (if this is a bug) Describe what h...
523
kubernetes deleting namespace stuck at "Terminating" state
@ManifoldFR I had the same issue as yours and I managed to make it work by making an API call with j...
457
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
447
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
381
git lfs Encountered 1 file(s) that should have been pointers, but weren't
Run git lfs uninstall and then git reset --hard (assuming there are no changes you want to keep!) On...
371
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
369
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
337
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
297
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the docker-entrypoint.sh an executable before building the image: otherwise it cant b...
279
kubernetes PV is stuck at terminating after PVC is deleted
I got rid of this issue by performing the following actions: Then I manually edited the pv individua...
271
kubernetes x509 cert issues after kubeadm init
do you have $KUBECONFIG pointing to /etc/kubernetes/kubelet.conf? BUG REPORT: (I think?) What happen...
264
kubernetes Ingress: Allow for multiple hosts
I also would like to see this feature but as a workaround I use YAML ids Here is how it would look f...
257
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
230
git lfs Simple steps to uninstall Git LFS from your repository
Thanx that helped me a lot while I was migrating from BitBucket to AWS CodeCommit ...
226
git lfs git-lfs aborts with "LFS: Git credentials for https://XXX not found." even for repositories without passwords
In my case on Windows using gitbash I had to use: git config --global credential.helper wincred ...
225
kubernetes The connection to the server localhost:8080 was refused - did you specify the right host or port?
Run these commands solved this issue: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HO...
210
minikube minikube start - Error starting host, machine does not exist
On macOS Sierra rm -rf ~/.minikube fixed it for me After that minikube start worked as expected mini...
208
kubernetes Force pods to re-pull an image without changing the image tag
@yujuhong Sometimes it's very useful to be able to do this For instance Problem A frequent question ...
205
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb http://cz.archive.ubuntu.com/ubuntu trusty main in /etc/...
183
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
182
kubernetes 'unknown revision v0.0.0' errors, seemingly due to 'require k8s.io/foo v0.0.0'
For anyone else who hits this issue after much weeping and gnashing of teeth this is the little scri...
169
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
163
go question: "go: cannot find main module; see 'go help modules'"
After upgrade to go1.16 try to set : go env -w GO111MODULE=auto https://blog.golang.org/go116-module...
161
go Proposal: A built-in Go error check function, "try"
I actually really like this proposal However I do have one criticism The exit point of functions in ...
150
protobuf protoc-gen-go: program not found or is not executable
I had this problem as well recently [using ubuntu 18.04] I found a long-lasting solution by adding t...
149
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
148
minikube kube-proxy configmap update: timed out (unknown root cause)
I had this error when upgrading from 0.25 to 0.26.1 Simply performing minikube delete and then re-cr...
147
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
142
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
138
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
137
gin Got "ambiguous import: found github.com/ugorji/go/codec in multiple modules" after upgraded to latest (1.3.1)
I find a way to fix this problem you can use replace command to set package alias I added the follow...
127
kubernetes JSONpath fails to return keys containing dots in a map
Escaping dots works now To revisit the example in my original message: Closing 🎉 ...
127
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
126
nvm windows nvm installs node but fails to deliver npm
Have the same problem My Environment Windows 10 I'm using NVM4W version: 1.1.7 I have already.. ...
123
kubernetes "Failed to setup network for pod \ using network plugins \"cni\": no IP addresses available in network: podnet; Skipping pod"
I had a simliar issue while testing kubernetes with kubeadm This started to happen after I did a kub...
119
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...
119
compose Release docker/compose image for armv7 / arm64v8
Last time I checked compose wants python 3 while raspbian's global python is still 2 apt-get install...
119
go proposal: Go 2: simplify error handling with || err suffix
A plain idea with support for error decoration There have been many proposals for how to simplify er...
119
kubernetes Kubectl cp gives "tar: removing leading '/' from member names" warning
Something I found is that if I do not put a / at the beginning of my path following : in <pod>:<path...
116
compose I am having this "Encountered errors while bringing up the project." error
Does running docker-compose down solve the issue? when $ docker-compose up it returns Docker Version...
114
moby docker.service Failed with result 'start-limit-hit'.
I had the same error message once because of an empty /etc/docker/daemon.json file Delete it if you ...
112
kubernetes no kind "Deployment" is registered for version "apps/v1beta2"
Small tip: To find out what exact apps/xxx api version your cluster supports use kubectl api-version...
111
minikube Can't pull images from an insecure registry in Minikube VM
I just tried this with minikube v0.10.0 and --insecure-registry='docker-registry.example.com:443' wa...
106
compose "Can't find a suitable configuration file" on Ubuntu 18.04.1
I ran into the same problem with a fresh Ubuntu 18.04 install where I selected Docker from install o...
103
compose How to make a Systemd Unit for docker-compose?
Place following content to the /etc/systemd/system/docker-compose@.service Place you docker-compose....
103
moby containerd: start container" error="oci runtime error: fork/exec /usr/bin/docker (deleted): no such file or directory: "
Not sure if this is the same cause but this might help someone Steps to reproduce the issue: Install...
102
moby Error response from daemon: Get https://registry-1.docker.io/v2/
may be you are behind a firewall/proxy server i was also behind my office firewall so i tried below ...
101
grpc go mustEmbedUnimplemented*** method appear in grpc-server
@josegonzalez I believe is what you are looking for correct me if I am wrong @dfawley ...