[BUG] Datadog Agent stops processing traces and spans over time

[ngoduykhanh]

Agent Environment

• ECS Fargate
• Datadog agent deployed as a sidecar
• Use latest version (7.61.0) via image public.ecr.aws/datadog/agent:latest

Notable Env vars:

• DD_APM_ENABLED: true
• ECS_FARGATE: true
• DD_APM_MAX_TPS: 100
• DD_APM_ERROR_TPS: 1000

Describe what happened:

The number of traces sent to Datadog from the Agent has significantly reduced over time. Whenever this happened, I redeployed the ECS task and it started working again. The number of ingested/indexed traces increased. But after 2-3 hours, the issue happened again.

Looking at the graphs, it seems to me, that for some reason, the Datadog agent stopped processing the traces although it received the traces from the application.

express.request trace metric

Describe what you expected:

Datadog agent processes all the traces it receives and I don't see a drop in traces on the Datadog dashboard.

Steps to reproduce the issue:

Additional environment details (Operating System, Cloud provider, etc):

• AWS ECS Fargate.
• The task's container resource usage looks good. There was no issue with CPU/memory allocation.

[ajgajg1134]

Hello! Thanks for opening an issue, to better investigate this for your specific setup / traffic pattern could you please open a support ticket? This will allow us to gather more specific information and determine what's going on here.