Solveddatadog agent Logs with datadog agent and ECS with excluded containers

Describe what happened:

We are running tasks on ECS so on a typical machine we have at least one container named: ecs-agent from image amazon/amazon-ecs-agent:latest running at all time.
We also launch the datadog agent with these options:

docker run -d --name datadog-agent \
   -e DD_API_KEY=<REDACTED> \
   -e DD_LOGS_ENABLED=true \
   -e DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL="true" \
   -e DD_AC_EXCLUDE="image:.*" \
   -e DD_AC_INCLUDE="image:<REDACTED>.dkr.ecr.eu-west-2.amazonaws.com/.*" \
   -v /var/run/docker.sock:/var/run/docker.sock:ro \
   -v /proc/:/host/proc/:ro \
   -v /opt/datadog-agent/run:/opt/datadog-agent/run:rw \
   -v /cgroup/:/host/sys/fs/cgroup:ro \
   datadog/agent:latest

We have a task where we want to collect logs for. The image for this task matches the .dkr.ecr.eu-west-2.amazonaws.com/.* regex. We use these labels for autodiscovery on the container:

com.datadoghq.ad.check_names=[]
com.datadoghq.ad.init_configs=[{}]
com.datadoghq.ad.logs=[{"source":"twitter","service":"api","log_processing_rules":{"type":"include_at_match","name":"include_only_warning_or_error","pattern":"warn|WARN|Warn|error|ERROR|Error"}}]
com.datadoghq.ad.instances=[{}]

Another fact: removing the DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL option stops the agent at startup with the line [ AGENT ] 2018-05-28 09:15:25 UTC | ERROR | (start.go:228 in StartAgent) | Could not start logs-agent: could not find any valid logs configuration even when DD_LOGS_ENABLED=true and with the correct autodiscover labels
So autodiscover is not really an autodiscovery for logs.

Describe what you expected:

I expected to only see logs from my application and not from datadog-agent or the ecs-agent. I do not need these logs on datadog.

How can we exclude these logs ?

Steps to reproduce the issue:

Launch the datadog agent docker with the options specified above.
Launch two container, one with the labels above and one without.
You will see logs from both containers.

Additional environment details (Operating System, Cloud provider, etc):

$ cat /etc/os-release
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
$ docker --version
Docker version 17.12.1-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3

With ecs_agent_version: 1.18.0

(but this should not change anything)

34 Answers

✔️Accepted Answer

Hello @amundra2016,

Sorry to hear that.
Could you confirm that you did remove (or set to false) the DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL env variable?

As the environment variable behaves like a configuration file that ask to collect logs from all containers.

If you did, would it be possible to have more information about the configuration you tried?

Related Issues:

11
datadog agent Tagger error on Kubernetes
I got the same error Kubernetes v1.8.7-gke Datadog Agent v6.0.2 Install agent using Helm (stable/dat...
3
datadog agent Logs with datadog agent and ECS with excluded containers
Hello @amundra2016 Sorry to hear that Describe what happened: We are running tasks on ECS so on a ty...
3
datadog agent Agent v6.5.2 broken logs from docker
Also seeing a similar problem with v6.5.2 Output of the info page (if this is a bug) Describe what h...
523
kubernetes deleting namespace stuck at "Terminating" state
@ManifoldFR I had the same issue as yours and I managed to make it work by making an API call with j...
457
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
447
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
381
git lfs Encountered 1 file(s) that should have been pointers, but weren't
Run git lfs uninstall and then git reset --hard (assuming there are no changes you want to keep!) On...
371
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
369
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
337
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
297
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the docker-entrypoint.sh an executable before building the image: otherwise it cant b...
279
kubernetes PV is stuck at terminating after PVC is deleted
I got rid of this issue by performing the following actions: Then I manually edited the pv individua...
271
kubernetes x509 cert issues after kubeadm init
do you have $KUBECONFIG pointing to /etc/kubernetes/kubelet.conf? BUG REPORT: (I think?) What happen...
264
kubernetes Ingress: Allow for multiple hosts
I also would like to see this feature but as a workaround I use YAML ids Here is how it would look f...
257
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
230
git lfs Simple steps to uninstall Git LFS from your repository
Thanx that helped me a lot while I was migrating from BitBucket to AWS CodeCommit ...
226
git lfs git-lfs aborts with "LFS: Git credentials for https://XXX not found." even for repositories without passwords
In my case on Windows using gitbash I had to use: git config --global credential.helper wincred ...
225
kubernetes The connection to the server localhost:8080 was refused - did you specify the right host or port?
Run these commands solved this issue: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HO...
210
minikube minikube start - Error starting host, machine does not exist
On macOS Sierra rm -rf ~/.minikube fixed it for me After that minikube start worked as expected mini...
208
kubernetes Force pods to re-pull an image without changing the image tag
@yujuhong Sometimes it's very useful to be able to do this For instance Problem A frequent question ...
205
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb http://cz.archive.ubuntu.com/ubuntu trusty main in /etc/...
183
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
182
kubernetes 'unknown revision v0.0.0' errors, seemingly due to 'require k8s.io/foo v0.0.0'
For anyone else who hits this issue after much weeping and gnashing of teeth this is the little scri...
169
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
163
go question: "go: cannot find main module; see 'go help modules'"
After upgrade to go1.16 try to set : go env -w GO111MODULE=auto https://blog.golang.org/go116-module...
161
go Proposal: A built-in Go error check function, "try"
I actually really like this proposal However I do have one criticism The exit point of functions in ...
150
protobuf protoc-gen-go: program not found or is not executable
I had this problem as well recently [using ubuntu 18.04] I found a long-lasting solution by adding t...
149
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
148
minikube kube-proxy configmap update: timed out (unknown root cause)
I had this error when upgrading from 0.25 to 0.26.1 Simply performing minikube delete and then re-cr...
147
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
142
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
138
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
137
gin Got "ambiguous import: found github.com/ugorji/go/codec in multiple modules" after upgraded to latest (1.3.1)
I find a way to fix this problem you can use replace command to set package alias I added the follow...
127
kubernetes JSONpath fails to return keys containing dots in a map
Escaping dots works now To revisit the example in my original message: Closing 🎉 ...
127
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
126
nvm windows nvm installs node but fails to deliver npm
Have the same problem My Environment Windows 10 I'm using NVM4W version: 1.1.7 I have already.. ...
123
kubernetes "Failed to setup network for pod \ using network plugins \"cni\": no IP addresses available in network: podnet; Skipping pod"
I had a simliar issue while testing kubernetes with kubeadm This started to happen after I did a kub...
119
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...
119
compose Release docker/compose image for armv7 / arm64v8
Last time I checked compose wants python 3 while raspbian's global python is still 2 apt-get install...
119
go proposal: Go 2: simplify error handling with || err suffix
A plain idea with support for error decoration There have been many proposals for how to simplify er...
119
kubernetes Kubectl cp gives "tar: removing leading '/' from member names" warning
Something I found is that if I do not put a / at the beginning of my path following : in <pod>:<path...
116
compose I am having this "Encountered errors while bringing up the project." error
Does running docker-compose down solve the issue? when $ docker-compose up it returns Docker Version...
114
moby docker.service Failed with result 'start-limit-hit'.
I had the same error message once because of an empty /etc/docker/daemon.json file Delete it if you ...
112
kubernetes no kind "Deployment" is registered for version "apps/v1beta2"
Small tip: To find out what exact apps/xxx api version your cluster supports use kubectl api-version...
111
minikube Can't pull images from an insecure registry in Minikube VM
I just tried this with minikube v0.10.0 and --insecure-registry='docker-registry.example.com:443' wa...
106
compose "Can't find a suitable configuration file" on Ubuntu 18.04.1
I ran into the same problem with a fresh Ubuntu 18.04 install where I selected Docker from install o...
103
compose How to make a Systemd Unit for docker-compose?
Place following content to the /etc/systemd/system/docker-compose@.service Place you docker-compose....
103
moby containerd: start container" error="oci runtime error: fork/exec /usr/bin/docker (deleted): no such file or directory: "
Not sure if this is the same cause but this might help someone Steps to reproduce the issue: Install...
102
moby Error response from daemon: Get https://registry-1.docker.io/v2/
may be you are behind a firewall/proxy server i was also behind my office firewall so i tried below ...
101
grpc go mustEmbedUnimplemented*** method appear in grpc-server
@josegonzalez I believe is what you are looking for correct me if I am wrong @dfawley ...