Missing agent logs in Azure DevOps Container App Jobs

Recently we ran into pipeline jobs that kept timing out after an hour, with no clear pattern. Different agents, different steps in the pipeline, different times of day. It took some digging to find the cause, and along the way we changed how we handle diagnostics for our self-hosted agents.

Situation

For one of our customers, we run a sizeable Azure environment, including an Azure Landing Zone. One of their pipelines deploys around 40 function apps through Azure DevOps. To run these deployments, we use self-hosted agent pools, hosted in Azure as Container App Jobs, with the option to run up to 5 agents in parallel.

Every so often, one of these agent jobs would time out after an hour. There was no obvious trigger, and it wasn’t tied to a specific agent or a specific step.

Investigation

The normal place to start is the agent’s diagnostic logs. Through the Azure Pipelines, we found out that the logs are written to a folder called _diag. These logs record what the agent and its worker process were doing, which is usually enough to pinpoint where a job went wrong. Except the logs weren’t there, which turned out to make total sense.

The reason is that Container App Jobs are transient. Once a job finishes, whether it succeeds, fails, or times out, the container is torn down, including its filesystem. Anything the agent wrote to _diag disappeared before we ever got a chance to look at it.

How we fixed it

The fix was to move the diagnostics folder off the container’s local disk and onto storage that survives after the container is gone. We added a volume to the Container App Job:

• A volume named diagnostics, backed by an Azure Files share on a storage account.
• Mounted at /azp/_diag, the same path the agent already writes to, so no agent configuration changes were needed.
• Since 5 agents run in parallel, each run’s logs land in the share without overwriting each other, so a specific failed job can still be traced back to its own agent and worker log.
• A retention policy on the share automatically removes older logs, so it doesn’t just keep growing with files we’ll never look at again.

In Bicep, this comes down to two things: a storage definition on the Container Apps environment, and a volume mount on the job itself.

resource agentDiagnostics 'Microsoft.App/managedEnvironments/storages@2023-05-01' = {
  parent: environment
  name: 'agent-diagnostics'
  properties: {
    azureFile: {
      accountName: storageAccountName
      accountKey: storageAccountKey
      shareName: 'agent-diagnostics'
      accessMode: 'ReadWrite'
    }
  }
}

resource agentJob 'Microsoft.App/jobs@2023-05-01' = {
  // ...
  properties: {
    template: {
      containers: [
        {
          name: 'devops-agent'
          // ...
          volumeMounts: [
            {
              volumeName: 'diagnostics'
              mountPath: '/azp/_diag'
            }
          ]
        }
      ]
      volumes: [
        {
          name: 'diagnostics'
          storageType: 'AzureFile'
          storageName: 'agent-diagnostics'
        }
      ]
    }
  }
}

Finding the right log

With multiple agents running in parallel, the share fills up relatively quickly, and nothing in the pipeline points you straight to the right file. Each agent writes its own Agent_<timestamp>-utc.log and Worker_<timestamp>-utc.log, for example Agent_20260101-080000-utc.log. What worked for us was matching on time: we took the start and failure time of the job from the Azure DevOps run, then looked for the log files whose timestamps fell in that same window.

Because several agents can be active at once, this narrows it down to a handful of files rather than one specific file. In that case, digging a little deeper and checking the job or agent name logged inside the file is enough to confirm which one belongs to the failed run.

Conclusion

Now, when a job times out, we can open the agent and worker logs for that run and see exactly where it stopped, instead of guessing. If you’re running self-hosted Azure DevOps agents on Container App Jobs, it’s worth mounting the diagnostics folder to persistent storage before you need it, because by default, those logs disappear along with the container.

As a separate note: around the same time, the agents were upgraded from .NET 8 to .NET 10, and the timeouts have not reappeared since. We consider that the likely fix, but we are still actively monitoring just to be sure. Either way, if the issue comes back, we now have the agent and worker logs to figure out why the agents are failing.