Seen this pattern a few times. Dev evals pass because the harness is clean, but prod telemetry gets polluted by unlabeled test runs, synthetic traces, and “implied” events that never actually fired. Three things that fixed it for us:
1. Tag every synthetic row in the row itself
Add a source: "synthetic-demo" (or eval, replay, canary) field directly on the log line. Not in a separate index, not in docs — on the row. Unlabeled fake data is the fastest way to kill trust; once someone spots one, they assume all of it is suspect.
2. Log what happened, not what you think happened
If the agent attempted a tool call but the runtime swallowed the error, log attempted_tool_call with the error code. Do not log tool_call_succeeded: true with a note. Normalize at write time; downstream consumers shouldn’t have to guess which fields are factual.
3. Every dashboard number must resolve to a queryable row
“95% success rate” is noise until you can click through to the exact run_ids that compose it. If a metric can’t be traced to source rows with a single query, treat it as unverified.
We wrote up the pattern in the free MIT-licensed Governed Agent Mesh Playbook (Section 6, hash-linked and claim-evidenced audit trails) if you want the reference.