TL;DR

We compared Onyx against ChatGPT Enterprise, Claude Enterprise, and Notion AI on a set of real-world enterprise-style questions spanning our company’s internal and web data across various complexity types. Overall, Onyx outperformed the other vendors by a ~2:1 margin.

Onyx achieved averaged win rates of 64.0% : 36.0% vs ChatGPT Enterprise, 68.1% : 31.9% vs Claude Enterprise, and 73.9% : 26.1% vs Notion AI (on a reduced question set), derived from multiple distinct LLM-as-a-Judge methodologies.

Table 1: Overall Win Rates [ties are counted as ½ wins; N=99 except for comparisons with Notion AI (N=74)]

Onyx also showed the lowest time-to-last-token, with a 34.7s average response time vs 36.2s (Claude Enterprise), 45.4s (ChatGPT Enterprise), and 46.7s (Notion AI).

These results complement and line up well with Onyx’ recent excellent Deep Research benchmark results using DeepResearch Bench.

We share the benchmark questions (with some redactions), our LLM-as-a-judge prompts, and scores for all questions in our repo.

1. Introduction

Evaluating modern Agentic RAG systems is as interesting as it is important. And it is rather challenging.

Answer quality directly affects whether users trust and adopt the tool. A bad or incomplete answer is often worse than no answer at all. At the same time, a more thorough answer that takes minutes to compile - versus tens of seconds - may not be preferred either. Lastly, "quality" depends heavily on the types of questions asked, the documents available to the system, and who is asking.

RAG also has evolved substantially since the early days. Initially, systems largely retrieved relevant chunks from internal documents and had an LLM generate an answer from them. Today, agentic systems decompose answering into multiple steps: issuing sub-questions, using multiple tools, identifying gaps in retrieved information, asking follow-up questions behind the scenes, etc., and synthesizing a final response. At the extreme end, Deep Research approaches explore multiple substantial research threads in parallel and/or iteratively, reasoning over sources for 10+ minutes before producing a final report.

Unfortunately, no public benchmarks that we have found adequately represent what enterprise customers actually need as most of them are set up to only use web data. These results, while interesting, have limited relevance for enterprise teams deploying (or evaluating) an agentic search and chat tool over the relevant knowledge sources, which crucially include both internal data and web information.

At Onyx, we've revisited this problem multiple times over the past couple of years and have run multiple internal benchmarks. Some focused more narrowly on search quality using synthetic questions over a largely synthetic document corpus. Another test compared our Deep Research implementation against a competitor’s solution using questions generated by the Onyx team, evaluated via blind comparisons by team members. However, especially for the search benchmark tests, we never felt that the results were as realistic and customer-relevant as we’d like.

We therefore decided to create a new internal benchmark test that we believe is much more representative of actual customer needs: it is based on real data and informed by actual customer questions. It covers questions of varying complexity over a corpus of internal and web data. We share details, to the degree possible, here. We also provide summary metrics and a clear methodology below.

A note on fairness and limitations

To address the main question right on: yes, Onyx designed this benchmark, wrote the questions, selected the data sources, and defined the evaluation criteria. While we believe our approach provides the most realistic comparison we have available, we certainly recognize the potential for an inherent conflict of interest. To mitigate this to the extent possible:

• We employed four independent judging methodologies and two different judging models (GPT-5.2 and Opus 4.), and reported mean win rates from the combinations plus standard deviations for key metrics.
• The two judging LLMs evaluate answers without knowing which vendor produced them
• We publish the full question set (with some redactions), rubrics, and some answers so others can inspect some of our our work [link]
• We conducted human spot-checks across questions and judge outputs to verify alignment
• We are in the process of building a fully synthetic, open dataset so the community can independently test systems and evaluate results

In general, it would be desirable to also conduct human double-blind comparisons, ideally with external human judges. As we are using internal data, we opted in this case against this additional comparison.

We also acknowledge that our question count (N=99) introduces statistical uncertainty, and that our internal dataset is smaller than that of many of our customers.

With that context, let's get into the specifics.

2. Complexities of Agentic RAG Benchmarking

The problem of creating a suitable benchmark that reflects customer experiences is more complicated than it may first appear. We'll focus here on key design decisions.

What is a "suitable" set of questions?

Given the rapidly increasing abilities of AI-based systems, users often don't realize what's actually possible, habitually asking for far less than the technology can deliver.

A benchmark based purely on questions modeled after "current user questions" would therefore underestimate actual user intent over the near future. Instead, a good benchmark should approximate what users would realistically want to ask once they discover and get used to the full range of what's possible. We designed a mix of questions spanning a broad range of complexities: from simple fact lookups like "Who founded Onyx?" to complex multi-step analytical questions like "how to improve sales opportunity with customer xyz?". This led to 99 questions that, in our view, reflect what customers could realistically expect to be answered well. See repo for details.

What is a "good" answer?

We define answer quality as a combination of correctness, completeness, faithfulness, ability to handle uncertainty, and clarity. A solid assessment must consider all of these and attach realistic weights to them. Better yet, one should use something more nuanced than a simple weighted sum. We balanced these dimensions across multiple judging approaches; more details can be found in Section 3.

Another important aspect for agentic systems is the trajectory the system takes to arrive at an answer, which we call the ‘flow’. A strong flow is generally better aligned with the user's expectations (and hence easier to verify), tends to be more token efficient, and generally leads to better answers. As the flows are not always available however, we judge flow alignment through how the answers reflect it.

What are "good" tools and data sources?

Many RAG-type benchmarks focus only on web search. While convenient, open, and replicable, benchmarks based on web data alone do not reflect what matters most to enterprise users. For most RAG deployments, internal data is the primary target and it is significantly more complex. Organizations store information across multiple systems and document structures vary widely (a Standard Operating Procedure is very different from a call transcript, for example). For this benchmark, we used both Web and Internal Searches, where Internal Searches included only document types that all compared systems support for a given question.

What is a "good" internal dataset?

This is a key question. It is essential to cover data sources customers rely on to do their jobs — common examples include call transcripts, email, cloud drives, etc. Due to the lack of suitable public data that simulates this environment, we decided to use for now a subset of our own internal data, spanning about 220k documents. Consequently, we cannot release this dataset.

3. Benchmark Setup

Below we illustrate the high-level benchmark setup and provide more details.

The evaluated systems were:

• Onyx's most recent version using GPT-5.2 as a base model
• ChatGPT Enterprise (GPT-5.2)
• Claude Enterprise (Opus 4.5)
• Notion AI (Auto)

The tests were implemented through browser automation using Playwright.

What is not included in this benchmark is:

• Our more advanced Deep Research and Craft modes
• Memory and personalization — for any vendor's solution

While both of these are expected to improve answer quality, we deliberately excluded them to evaluate core agentic search performance.

3.1 Tools & Sources

We compared the vendors with both web search and access to a subset of our internal documents enabled via connectors.

The common connectors used for internal sources were:

• GitHub
• Gmail
• Google Drive
• Slack
• Linear

In addition, for all vendors except Notion, we also leveraged:

• Fireflies
• HubSpot

Call transcripts and sales data are essential to generating actionable, high-value insights.

3.2 Questions

We created a test set of 99 questions that we framed after real customer questions and questions we anticipate they will want to pose to the system.

Examples include:

• “I'm using version 2.11.0, how do I ingest documents programmatically?"
• “How do I upgrade [customer xyz]’s instance?”
• “How should I prioritize my first 10 enterprise companies for outreach?”
• “How do enterprise buyers evaluate AI knowledge search solutions?”
• “What common pains usually come up in discovery calls with prospects?”
• “Tell me what customers said about us last month”
• “How to improve sales opportunity with [customer xyz]?”

Overall, the question set has the following distribution across source categories:

The `Undefined` category was included to see how well the system deals with substantially under-defined questions, like “What did we decide?”. Generally, users would not want the Assistant to report on some decision taken about something, but rather return something like “I do not have enough information to answer this question.”, or “Decision about what?”.

In addition, each question carries one or more characteristic tags:

3.3 Judgments

Judging answer quality is inherently difficult and subjective beyond a certain point. Our approach is to employ four independent LLM-as-a-Judges to verify robustness. Across all of them:

• Vendor names are not provided to the judges and the order was randomized
• Citations were removed as different vendors handle citations differently.
• Where applicable, scores are on a 1–10 scale (normalized for Judge 2; see below).
• All judges were evaluated with two models, GPT-5.2 and Claude Opus 4.5
• For Judges 1 - 3, final win rates were computed by first averaging each vendor's scores across all three judges and both judging models for a given question, then comparing the averaged scores pairwise.

Here are the details for the four judges:

Judge 1: Gold Facts & Behaviors, Graded on Scale

Rather than providing full "gold answers”, we defined for each question where meaningful:

• A Gold Flow — how we expect the agentic RAG system to approach the question
• Gold Facts — directional facts we would expect a good answer to include

Each vendor's answer was graded independently against these expectations.

Judge 2: Gold Facts & Behaviors, Pass/Fail

This approach also judged answers against the gold information, but used a binary pass/fail criterion applied independently to each rubric item. Scores were then normalized to the 1–10 range.

Judge 3: Relative Judging

Rather than comparing against a gold standard, this judge viewed all vendor responses at once (anonymized and randomly ordered) and scored each based on its quality relative to the others. The underlying assumption: facts shared across multiple responses are likely accurate, so the collective serves as a proxy for ground truth.

Judge 4: Pairwise Preference Judging [excluding Notion AI]

Similar to Judges 1 and 2, gold flows and facts were provided as well as general target metrics. The judging LLMs were asked to rank the Onyx answer vs the competitor’s answer.

These approaches certainly do not fully guard against hallucinations or missing critical information, but we believe, supported by our human spot-checks, that collectively they produce sufficiently robust results that are both reasonably grounded and not overly dependent on any single judge's specifics.

Our overall metric categories with corresponding weights applied across all judges were:

• Correctness: 40%
• Completeness: 30%
• Uncertainty Handling: 20%
• Faithfulness to Context: 5%
• Clarity and Format: 5%

Note that we are not explicitly considering retrieval quality as an independent metric in the benchmark. We view retrieval as a backend process that will inform the answer, and it will be implicitly scored through the existing metrics.

The detailed prompts and metrics for the judges can be found in the repo.

3.4 The Dataset

We are using a subset of our internal data for this benchmark test. It includes about 220k documents across all sources: 2750 Fireflies call transcripts, 8k+ Github PRs and Issues, about 190k emails, 7k+ Google Drive documents, 2k+ Linear tickets, 10k+ Slack messages, and 12k+ HubSpot companies, deals, and contacts). We cannot make this critical component of the tests public.

4. Results

Overall, we found that Onyx outperformed our competitors approximately by a 2:1 margin.

Table 4: Overall Win Rates (N=99 for Onyx, ChatGPT, Claude, and *N=74 and Judges 1-3 for Notion AI comparisons)

Note that ties are not uncommon; we accounted for them by giving each of the compared systems ½ of a win. We should state that the four judges were reasonably consistent. Their standard deviation for the Onyx win rate was 7.0%/3.4% for the ChatGPT/Claude comparisons respectively. Details can be found here.

It is also worth examining results across categories and tags:

Latency (Time to Last Token, All Questions Available for Given System)

Lastly, largely just to demonstrate that all systems used a similar approach and not a Deep Research or another notably different framework, we also report on the average Time-to-Last-Token times:

Onyx delivered the fastest average response time while also producing more comprehensive answers as reflected in the win rates above.

Discussion and Observations

Overall, these results show Onyx shows strong performance across all but one category under our evaluation approach. The exception is the "Undefined" category (very loosely defined or under-specified queries), where our agentic flow tends to attempt an answer rather than decline to answer. Whether this is preferable depends on user expectations.

We believe that the overall superior results are largely driven by two architectural choices:

1. Direct data indexing

Onyx indexes all connected data directly, enabling us to apply our hybrid search approach consistently across diverse data sources. Some competitors rely on MCP (Model Context Protocol) connections for at least some of their data connectors. While we have not isolated this variable in a controlled test, we believe direct indexing contributes meaningfully to retrieval consistency — particularly across heterogeneous document types. Whereas the win rates on first sight do not directly support this assertion, we find the average weighted score (across all judges) per category most telling in this case: 

We also see very good evidence for this in the two examples below where we compare the answers of Onyx, ChatGPT, and Claude. Particularly the second one clearly demonstrates that the systems using MCP connections were not able to retrieve highly relevant internal documentation.

2. Agentic orchestration

Our system's approach to query decomposition, tool selection, and iterative refinement appears well-suited to the kinds of complex, multi-source questions that enterprises care about. We also separate searching from reading for both, external and internal sources. That gives the system the ability to read the full document if needed to get the full context. Consider for example call transcripts: if only chunks are retrieved, a lot of critical context can get lost.

Some Examples

Below we share a few answer examples that we find instructive:

Question 1: "What are our guarantees on patching discovered vulnerabilities?"

Question 2: "does [COMPANY] have a preference for a more search-like interface?"

For the first question we believe that Onyx’ answer is the best as it very clearly finds and articulates the proper information, providing a holistic answer to the user’s question. ChatGPT’s answer is not wrong, but not as pointed and precise.

For the second question, Onyx’ answer is certainly much more specific and informative than ChatGPT’s. Onyx clearly found call information, apparently missed by both ChatGPT and Claude, which is clearly highly relevant to inform the user. In fact, Claude’s answer suggests the system did not even attempt any internal searches to construct the answer. Maybe a follow-up interaction with the user would have provided more relevant information, but the first interaction certainly fell short.

Finally, a couple of notable observations regarding the gap between ChatGPT Enterprise and Claude Enterprise - as Assistants and as Judges:

• We traced some of the answer quality/win rate differences between Claude and ChatGPT (vs. Onyx and vs. each other) to Claude frequently asking for clarifications rather than leveraging available context to attempt an answer directly. This is admittedly a somewhat subjective distinction. Some users may prefer clarification over assumptions. In our assessment however, it occurred more often than most users would likely find useful. This is reflected in the scores. We attempted to compensate by appending an explicit instruction to each question for Claude discouraging this behavior, but results did not improve under our evaluation criteria, and response times increased by approximately 10 seconds. Therefore, we report the results here without this modification.
• It is also interesting to note that the two models GPT-5.2 and Claude Opus 4.5 behaved rather differently when used as the models powering the judges, with each one being bias towards answers generated "by themselves".
  We are not commenting on which model we consider to be the better judging model as these results inherently also depend on the prompts and other choices, but we consider the difference notable and interesting in its own right.

Overall, these results line up well with the success of our recent Deep Research benchmark performance using DeepResearch Bench, where we had achieved #1 ranking.

They are also supported by our customers, as evidenced by this quote:

“Onyx is answering thousands of questions a week at Ramp. We tried a variety of other AI tools but none had the same answer reliability as Onyx. It’s been a huge productivity boost as we continue to scale.” (Tony Rios, Director of Product Ops, Ramp)

5. Conclusions & Outlook

Our key finding is that Onyx outperformed the other evaluated vendors by approximately 2:1 in terms of pairwise win rates, with some variation across question categories.

The results of this benchmark helped us evaluate our solution relative to other vendors, identify where we excel and where we can rethink our approach, and better understand how answer quality varies across question types.

Areas we would like to improve include:

• More questions. N=99 provides directional signals but limits statistical power, particularly for smaller category slices. We aim to expand the set substantially in future tests.
• Broader data. Our internal dataset is smaller than that of many current and prospective customers. Patterns in larger, more diverse knowledge bases may differ.
• Independent verification. While we publish questions, rubrics, and some other information, the underlying internal documents and many of the answers cannot be shared. We recognize the limitations of this approach.

To address these gaps, we are planning on building a synthetic dataset that models the types of documents a larger enterprise would have.