Solveddatadog agent Logs with datadog agent and ECS with excluded containers

Describe what happened:

We are running tasks on ECS so on a typical machine we have at least one container named: ecs-agent from image amazon/amazon-ecs-agent:latest running at all time.
We also launch the datadog agent with these options:

docker run -d --name datadog-agent \
   -e DD_LOGS_ENABLED=true \
   -e DD_AC_EXCLUDE="image:.*" \
   -e DD_AC_INCLUDE="image:<REDACTED>*" \
   -v /var/run/docker.sock:/var/run/docker.sock:ro \
   -v /proc/:/host/proc/:ro \
   -v /opt/datadog-agent/run:/opt/datadog-agent/run:rw \
   -v /cgroup/:/host/sys/fs/cgroup:ro \

We have a task where we want to collect logs for. The image for this task matches the* regex. We use these labels for autodiscovery on the container:[][{}][{"source":"twitter","service":"api","log_processing_rules":{"type":"include_at_match","name":"include_only_warning_or_error","pattern":"warn|WARN|Warn|error|ERROR|Error"}}][{}]

Another fact: removing the DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL option stops the agent at startup with the line [ AGENT ] 2018-05-28 09:15:25 UTC | ERROR | (start.go:228 in StartAgent) | Could not start logs-agent: could not find any valid logs configuration even when DD_LOGS_ENABLED=true and with the correct autodiscover labels
So autodiscover is not really an autodiscovery for logs.

Describe what you expected:

I expected to only see logs from my application and not from datadog-agent or the ecs-agent. I do not need these logs on datadog.

How can we exclude these logs ?

Steps to reproduce the issue:

Launch the datadog agent docker with the options specified above.
Launch two container, one with the labels above and one without.
You will see logs from both containers.

Additional environment details (Operating System, Cloud provider, etc):

$ cat /etc/os-release
NAME="Amazon Linux AMI"
ID_LIKE="rhel fedora"
PRETTY_NAME="Amazon Linux AMI 2018.03"
$ docker --version
Docker version 17.12.1-ce, build 3dfb8343b139d6342acfd9975d7f1068b5b1c3d3

With ecs_agent_version: 1.18.0

(but this should not change anything)

34 Answers

✔️Accepted Answer

Hello @amundra2016,

Sorry to hear that.
Could you confirm that you did remove (or set to false) the DD_LOGS_CONFIG_CONTAINER_COLLECT_ALL env variable?

As the environment variable behaves like a configuration file that ask to collect logs from all containers.

If you did, would it be possible to have more information about the configuration you tried?

Related Issues:

datadog agent Tagger error on Kubernetes
I got the same error Kubernetes v1.8.7-gke Datadog Agent v6.0.2 Install agent using Helm (stable/dat...
datadog agent Logs with datadog agent and ECS with excluded containers
Hello @amundra2016 Sorry to hear that Describe what happened: We are running tasks on ECS so on a ty...
datadog agent Agent v6.5.2 broken logs from docker
Also seeing a similar problem with v6.5.2 Output of the info page (if this is a bug) Describe what h...
kubernetes deleting namespace stuck at "Terminating" state
@ManifoldFR I had the same issue as yours and I managed to make it work by making an API call with j...
compose Docker-compose up failing because "port is already allocated"
I ran into the same issue today (with a postgres container) and despite having tried docker-compose ...
moby The name "/data-container-name" is already used by container <hash>. You have to remove (or rename) that container to be able to reuse that name.
I have a helper function to nuke everything so that our Continuous blah cycle can be tested erm.. co...
git lfs Encountered 1 file(s) that should have been pointers, but weren't
Run git lfs uninstall and then git reset --hard (assuming there are no changes you want to keep!) On...
compose Compose error "HTTP request took too long to complete"
By simply restarting the docker service via sudo service docker restart I was able to get the aforem...
compose error on launching docker-compose by piping to sh ( echo 'docker-compose ... ' | sh )
I could get it to work by adding the -T parameter to not create a Pseudo-TTY docker-compose exec -T ...
compose docker-compose up fails if network attached to container is removed
Thanks for the report! I think there are several things to note here: First and foremost ...
compose Error when trying to run docker-compose up. "oci runtime error: container_linux.go:247..."
you gotta make the an executable before building the image: otherwise it cant b...
kubernetes PV is stuck at terminating after PVC is deleted
I got rid of this issue by performing the following actions: Then I manually edited the pv individua...
kubernetes x509 cert issues after kubeadm init
do you have $KUBECONFIG pointing to /etc/kubernetes/kubelet.conf? BUG REPORT: (I think?) What happen...
kubernetes Ingress: Allow for multiple hosts
I also would like to see this feature but as a workaround I use YAML ids Here is how it would look f...
compose docker-compose up doesn't pull down latest image if the image exists locally
Imagine that git didn't have pull because git fetch && git merge origin/master is functionally ident...
git lfs Simple steps to uninstall Git LFS from your repository
Thanx that helped me a lot while I was migrating from BitBucket to AWS CodeCommit ...
git lfs git-lfs aborts with "LFS: Git credentials for https://XXX not found." even for repositories without passwords
In my case on Windows using gitbash I had to use: git config --global credential.helper wincred ...
kubernetes The connection to the server localhost:8080 was refused - did you specify the right host or port?
Run these commands solved this issue: mkdir -p $HOME/.kube sudo cp -i /etc/kubernetes/admin.conf $HO...
minikube minikube start - Error starting host, machine does not exist
On macOS Sierra rm -rf ~/.minikube fixed it for me After that minikube start worked as expected mini...
kubernetes Force pods to re-pull an image without changing the image tag
@yujuhong Sometimes it's very useful to be able to do this For instance Problem A frequent question ...
moby docker-engine 1.10.2-0~trusty can't install on clean Ubuntu 64-bit 14.04.3
I seem to have resolved this by putting deb trusty main in /etc/...
moby Docker service update --image "could not accessed on a registry to record its digest"
When updating services that need credentials to pull the image you need to pass --with-registry-auth...
kubernetes 'unknown revision v0.0.0' errors, seemingly due to 'require v0.0.0'
For anyone else who hits this issue after much weeping and gnashing of teeth this is the little scri...
compose Docker Compose mounts named volumes as 'root' exclusively
Actually I come here with news it seems what I am trying to achieve is doable but I don't know if th...
go question: "go: cannot find main module; see 'go help modules'"
After upgrade to go1.16 try to set : go env -w GO111MODULE=auto
go Proposal: A built-in Go error check function, "try"
I actually really like this proposal However I do have one criticism The exit point of functions in ...
protobuf protoc-gen-go: program not found or is not executable
I had this problem as well recently [using ubuntu 18.04] I found a long-lasting solution by adding t...
compose INTERNAL ERROR: cannot create temporary directory!
Confirming this happened to me Today Was running low on space: After removing a container.. it works...
minikube kube-proxy configmap update: timed out (unknown root cause)
I had this error when upgrading from 0.25 to 0.26.1 Simply performing minikube delete and then re-cr...
compose docker-compose up -d doesn't expose ports when defined with build directive
oh you didn't specify but I'm assuming you're using run instead of up? If so you need --service-port...
compose How does compose chooses subnet for default network?
I'm also running into this issue Another way around this is to set the default-address-pools in your...
kubernetes ingress 413 Request Entity Too Large
FYI the annotation has changed and is now: Also I had to restart the nginx pod for the effect to tak...
gin Got "ambiguous import: found in multiple modules" after upgraded to latest (1.3.1)
I find a way to fix this problem you can use replace command to set package alias I added the follow...
kubernetes JSONpath fails to return keys containing dots in a map
Escaping dots works now To revisit the example in my original message: Closing 🎉 ...
moby docker daemon unable to access registry - Client.Timeout exceeded while awaiting headers
I found out that the problem might be in /etc/resolv.conf I had: but moving the non-working (yet) 10...
nvm windows nvm installs node but fails to deliver npm
Have the same problem My Environment Windows 10 I'm using NVM4W version: 1.1.7 I have already.. ...
kubernetes "Failed to setup network for pod \ using network plugins \"cni\": no IP addresses available in network: podnet; Skipping pod"
I had a simliar issue while testing kubernetes with kubeadm This started to happen after I did a kub...
compose ERROR: for db Cannot start service db: driver failed programming external connectivity on endpoint ltg_db_1
The following worked for me when i do the following : ± docker-compose up Starting ltg_db_1 ERROR: f...
compose Release docker/compose image for armv7 / arm64v8
Last time I checked compose wants python 3 while raspbian's global python is still 2 apt-get install...
go proposal: Go 2: simplify error handling with || err suffix
A plain idea with support for error decoration There have been many proposals for how to simplify er...
kubernetes Kubectl cp gives "tar: removing leading '/' from member names" warning
Something I found is that if I do not put a / at the beginning of my path following : in <pod>:<path...
compose I am having this "Encountered errors while bringing up the project." error
Does running docker-compose down solve the issue? when $ docker-compose up it returns Docker Version...
moby docker.service Failed with result 'start-limit-hit'.
I had the same error message once because of an empty /etc/docker/daemon.json file Delete it if you ...
kubernetes no kind "Deployment" is registered for version "apps/v1beta2"
Small tip: To find out what exact apps/xxx api version your cluster supports use kubectl api-version...
minikube Can't pull images from an insecure registry in Minikube VM
I just tried this with minikube v0.10.0 and --insecure-registry='' wa...
compose "Can't find a suitable configuration file" on Ubuntu 18.04.1
I ran into the same problem with a fresh Ubuntu 18.04 install where I selected Docker from install o...
compose How to make a Systemd Unit for docker-compose?
Place following content to the /etc/systemd/system/docker-compose@.service Place you docker-compose....
moby containerd: start container" error="oci runtime error: fork/exec /usr/bin/docker (deleted): no such file or directory: "
Not sure if this is the same cause but this might help someone Steps to reproduce the issue: Install...
moby Error response from daemon: Get
may be you are behind a firewall/proxy server i was also behind my office firewall so i tried below ...
grpc go mustEmbedUnimplemented*** method appear in grpc-server
@josegonzalez I believe is what you are looking for correct me if I am wrong @dfawley ...