AWR for LLMs: a 30-year Oracle DBA's framing for pipeline profiling

Every Oracle tuning engagement I have run since 1996 starts with the same question: WHERE is the time going? Not "is it slow?" — slow is a symptom. The question is which subsystem is the bottleneck. You pull an Automatic Workload Repository (AWR) report — Oracle's built-in performance profiler — look at the Top 5 Timed Foreground Events, and the wait class answers in one glance. User I/O means storage. Application means lock contention. Scheduler means resource manager is throttling sessions. You do not guess; you rank by share of DB Time and follow the evidence.

After thirty years doing that with Oracle databases — at banks, at Amazon, at telecom carriers — I spent a year watching engineers debug LLM pipeline performance the old way: guessing, logging individual call durations, hunting for the slow span by hand. The wait-interface discipline transfers directly. The question is the same. The substrate is new.

Oracle's AWR captures a sixty-minute snapshot and ranks where database sessions spent their time. The key is not the list — it is the ranking. DB Time is finite. If 68% of it sits in User I/O, you fix storage first and ignore everything else. The same logic applies to LLM pipeline latency: rank by share of total time and attack the top class before touching anything else.

The money shot: same grammar, new domain

The two reports below ask the same question — WHERE is the time going? — against two very different systems.

Oracle AWR — Top 5 Timed Foreground Events Synthetic OLTP workload, 60-minute snapshot, DB Time: 216.8 s

                                                            Avg      % DB
Event                                        Waits   Time(s)  Wait(ms)  Time Wait Class
-------------------------------------------- ------- -------  --------  ---- ----------
db file sequential read                        8,240   148.3      18.0  68.4 User I/O
SQL*Net message from dblink                    1,052    42.1      40.0  19.4 Network
parse time elapsed †                              62    16.8     271.0   7.7 [Time Model]
enq: TX - row lock contention                    124     6.2      50.0   2.9 Application
resmgr: cpu quantum                              481     2.0       4.1   0.9 Scheduler

† parse time elapsed is a Time Model metric, not a foreground wait event — shown for structural symmetry with context-assembly-wait below.

LLM pipeline — Top 5 Wait Classes Synthetic multi-step agent run, 48 pipeline executions, total pipeline time: 216.8 s

                                                            Avg      % Pipeline
Class                                        Calls   Time(s)  Wait(ms)   Time Wait Class
-------------------------------------------- ------- -------  ---------  ---- ----------
model-inference-wait                              48   148.3      3,090  68.4 Inference
tool-call-wait                                   124    42.1        340  19.4 External
context-assembly-wait                             48    16.8        350   7.7 Prep
retry-overhead                                     4     6.2      1,550   2.9 Contention
queue-wait                                        48     2.0         41   0.9 Scheduler

Read them row by row. In the Oracle case, 68.4% of DB Time accumulates in db file sequential read. Tuning target: storage — caching, faster I/O, fewer physical reads per execution. Everything else is noise until that class is addressed.

In the LLM case, 68.4% of total pipeline time accumulates in model-inference-wait. Tuning target: the inference call — reduce prompt length, switch to a smaller model for subtasks where quality allows, enable streaming to overlap generation with downstream processing.

Now look at retry-overhead. It accounts for just 2.9% of total time — easy to dismiss. But the average wait is 1,550 milliseconds across only four occurrences. Without per-call normalisation, four retries are invisible. With it, they surface immediately as a spike pointing at rate-limit provisioning, not model performance. Oracle surfaces the same pattern through its elapsed-time-per-execution column: a query that ran once but consumed ten seconds of DB Time ranks alongside a query that ran ten thousand times at one millisecond each. Both lenses matter.

Wait-class crosswalk

The six-class taxonomy, with Oracle analog and one-line diagnostic meaning:

Each class points at a specific tuning lever — rank first, then pull the lever the dominant class names.

The gap: why no tool ships this yet

Per-span latency is already captured everywhere in this category. Langfuse, Datadog LLM Observability, Helicone, Arize Phoenix, LangSmith — every tool records how long each span took. That is not the gap.

The missing layer is classification and ranking. As of mid-2026, no tool ships an application-layer wait-class taxonomy with a ranked Top-N report. Three specific gaps, each traceable to publicly available documentation: Datadog LLM Observability ranks individual call instances by cost and latency — knowing that call forty-seven is slow does not tell you whether it is slow because of a tool call, a large context, or a retry loop; the wait-class layer is what is missing. MLflow's decomposition by model and retrieval step is a dimension filter, useful for attribution but not a ranked diagnostic taxonomy across all span types. The OpenTelemetry GenAI Semantic Conventions define the data schema for LLM spans with precision; they define no analysis or ranking layer on top of that data.

The diagnostic question — WHERE is the time going? — is the gap. Not the data collection.

Does this resonate?

This is the first time I have written this framing publicly. The Oracle analogy is not a metaphor I reached for because it sounded clever. It is the methodology I used for thirty years to answer the WHERE question before touching a single configuration file. The discipline transferred to LLM pipelines the first time I applied it, and I want to know whether it maps to a problem real teams are running into.

If you build or operate multi-step agent pipelines — five or more spans, two or more tool calls — and you recognise the triage gap described above, I would welcome a conversation. A tool that automates this report follows if the framing resonates. No roadmap, no pricing, no commitment. Just whether this is worth building.