refactor(gooddata-eval): centralize agentic outcome fields; fix exit_reason and tool_calls gaps

[Tomkess]

Summary

Follow-up to #1845. After shipping best_run_latency_s, I ran a comprehensive
comparison across all 11 agentic evaluators (alert_skill, anomaly_detection,
conversation, dashboard_skill, general_question, guardrail, kda_skill,
metric_skill, search_tool, visualization, what_if) and live production JSON
output to check whether the same field-is-missing-from-some-kinds pattern recurs
elsewhere. It does, twice more, both confirmed in real result files before this PR:

1. exit_reason (the LoopExit enum) — invented specifically so a run that hit
   max_iterations stops being indistinguishable from a genuine refusal (every
   downstream check is produced_output and <check>, so an exhausted run reports
   every field as False either way). Wired into 5 of 8 multi-turn kinds. Missing
   from anomaly_detection, dashboard_skill, what_if.
2. detail's tool_calls key — timeline_detail() builds both
   latency_breakdown and tool_calls from the same events so they stay
   index-aligned, but roughly half the evaluators called build_latency_breakdown()
   directly instead and silently never got a tool_calls key.

Root cause of both (and of best_run_latency_s before them, and of .timings
before that — see AgenticAssertionError's own docstring, which documents this
happening once already): every evaluator hand-writes the same ~7-10 line block
copying reasoning_steps/conversation_id/response_id/detail/runs_passed/
runs_effective/(timings)/best_run_latency_s onto either a raised exception or
the returned AgenticEvalOutcome. 11 independent copies of the same logic — a new
universal field needs 11 edits, and nothing fails loudly when one is skipped. There's
a test asserting every kind is dispatched (test_agentic_runner.py's
assert covered == set(AGENTIC_TEST_KINDS)), but none asserting every kind's result
carries the same fields.

What changed

• New core/agentic/_outcome.py:
  • agentic_detail(tool_call_events, reasoning_step_events, **kind_specific) — every
    evaluator's detail dict now goes through this, so the tool_calls gap can't
    recur by omission.
  • raise_agentic_failure(exception_cls, message, ...) / agentic_success(...) —
    the shared tail that used to be hand-rolled. exception_cls is typed
    type[AgenticAssertionError], which is what makes the bare-annotation fields
    (reasoning_steps, detail, etc.) type-check without # type: ignore.
• All 11 evaluators migrated to call these instead of constructing the
  exception/outcome by hand. Per-evaluator duplicate-field-assignment count: ~85
  lines total before this PR, 3 after — the 3 remaining are general_question's
  and guardrail's "no judge verdict for any run" branches, which raise a bare
  JudgeResponseError (a RuntimeError, not an AgenticAssertionError subclass) and
  so can't route through raise_agentic_failure; left as direct attribute sets,
  unchanged from before.
• anomaly_detection.py, dashboard_skill.py, what_if.py each gained an
  exit_reason: LoopExit field on their per-run result dataclass, set at every break
  point in their run loop. dashboard_skill.py's loop has no send_message
  try/except and no simulated-user call, so it only ever produces SUCCESS,
  AGENT_SILENT, or the BUDGET_EXHAUSTED default — never CHAT_ERROR or
  SIMULATED_USER_FAILED, unlike the other two.

Explicitly NOT in this PR

• agentic_dashboard_summary not existing on master at all. Checked: your pin
  is on integration/eval-rebuild, which carries 2,180+ lines of unmerged diffs
  across 19 files (a whole separate forecasting.py evaluator, a new
  _failed_runs.py module, and substantial rewrites already in flight) — a
  pre-existing release effort of its own. Porting just dashboard_summary.py here
  would duplicate or conflict with that work rather than fix anything.
• A ~9% gap in agentic_guardrail's .reasoning.json sidecar capture, where runs
  that genuinely had reasoning steps (per detail.latency_breakdown) still got no
  sidecar written. Investigated as far as: ruled out a --no-reasoning config
  difference (same run_id, mixed result within it); ruled out SSE-level divergence
  (ChatResult.reasoning_steps and .reasoning_step_events are built from the exact
  same accumulator list in sse_client.py, so they can't disagree there). The
  remaining cause needs a live repro with actual reasoning content, not static source
  reading — not fixed here rather than guessed at.

Test plan

• Full gooddata-eval suite: 1382 passed
• ruff check / ruff format --check: clean
• ty check: clean (one real issue found and fixed along the way — the
  exception_cls parameter needed type[AgenticAssertionError], not
  type[Exception], for the bare-annotation attributes to type-check at all;
  and conversation_id needed to stay non-optional str to match
  AgenticAssertionError's own declared type, distinct from
  AgenticEvalOutcome's optional one)
• Extended tests for anomaly_detection, dashboard_skill, what_if assert the
  actual exit_reason value at each new break point (SUCCESS, AGENT_SILENT,
  BUDGET_EXHAUSTED, CHAT_ERROR) — not just that existing tests still pass,
  which they already did even before exit_reason was wired in correctly, since
  none of them asserted on it
• Extended tests for general_question, kda_skill, search_tool assert the
  new tool_calls key is present in detail

🤖 Generated with Claude Code

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

• Configuration used: Organization UI
• Review profile: CHILL
• Plan: Advanced
• Run ID: 0dda85a8-144f-4583-ae01-068fb7a98d90

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

• Autopilot · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

---

_{Comment @coderabbitai help to get the list of available commands.}