AI-Powered Enterprise Search: How to Evaluate It, and Who Is Credible

Every product in this category demos well. That is not a criticism of the products; it is a property of demos. They are run against a curated corpus, by someone who knows the answer, asking a question the system has been tuned for.

Your corpus is not curated. It contains four versions of the expense policy, a Confluence space nobody has touched since a 2023 reorganization, and a SharePoint site whose permissions were configured by someone who has left.

So the useful discussion is not a feature ranking. It is the set of criteria that predict behaviour once the demo conditions are gone.

The five criteria that matter

1. Permission-aware retrieval

The one most often assumed and least often verified.

Enterprise search indexes everything and answers from what it indexed. If the permission model is not enforced at retrieval time, per user, per document, you have built an extremely efficient tool for leaking compensation data, severance agreements, unannounced reorganizations and legal correspondence.

Post-filtering is not sufficient. If the system retrieves a document the user cannot see and then declines to cite it, the content has already influenced the generated answer. The answer leaks the substance without ever showing the source.

Ask: is permission enforced at retrieval or after it? What happens when permissions change. Is it reindexed, and how quickly? How are inherited and group-based permissions resolved? Then test it: have someone search for something they should not be able to find.

2. Freshness and the correction loop

The practical test of grounding: when you fix the source document, how long until the answer changes?

If the answer is "next full reindex, weekly," then for a week the system will confidently give the old answer. For a policy change, that is a compliance problem, not a latency problem.

Related and equally important: does the system prefer the current version when the corpus contains four? Recency is a weak signal, a 2019 document can be current and a 2026 draft can be wrong. Products differ substantially in how they handle document lineage, and most handle it poorly.

3. Citation discipline

Every claim in an answer should point at a source, and the source should actually support the claim.

The failure mode is subtle and common: an answer that is broadly correct, with citations attached, where the cited passage does not quite say what the answer says. It survives casual inspection because the citation exists and the answer is plausible.

DRBench is instructive here. It evaluates agents on 100 open-ended research tasks across ten business domains, spanning public web and private organizational data, scoring insight recall, factual accuracy and proper citation, with deliberately planted distractor content the agent must ignore. Those criteria are the right lens for any enterprise search evaluation, as we covered in detail.

4. Refusal

What happens when the answer is not in the corpus?

The correct behaviour is to say so. The common behaviour is to produce a fluent, plausible, unsupported answer: which is the single most damaging thing an enterprise search product can do, because it is indistinguishable from a correct answer until someone acts on it.

Ask for the refusal path in the demo. Ask a question you know the corpus cannot answer, and watch.

5. Does it do anything?

Search that returns an answer is useful. Search that completes the task is a different product.

"Your laptop refresh eligibility is 36 months and you are at 41" is an answer. Opening the request, pre-filled, against the right approver, is a resolution. The gap between those two is where most of the value sits, and it is the boundary between enterprise search and an agentic service desk.

The credible categories

Products rather than a ranking, because the right answer depends heavily on where your content lives.

Suite-native search: Microsoft (Copilot and Microsoft Search), Google (Vertex AI Search), Atlassian (Rovo). Deep, native access to their own ecosystem, permission models that are already correct because they are the same permission model, and pricing that is often incremental to what you already buy. Weaker across the boundary, Microsoft's view of your Confluence estate will never match its view of SharePoint.

Dedicated enterprise search platforms: Glean, Coveo, Sinequa, Lucidworks, Dashworks. Built for the heterogeneous case: many connectors, permission mirroring across systems, relevance tuning as a first-class discipline. Stronger where content is genuinely scattered. The tradeoff is another platform to own, another permission model to keep synchronized, and connector maintenance as a permanent obligation.

Search infrastructure: Elastic, OpenSearch, vector databases. You are building rather than buying. Correct when search is part of your product or your requirements are genuinely unusual; expensive when chosen because it looked cheaper.

Employee-service platforms with search inside, including Rezolve.ai, which Everest Group named a Major Contender in its 2026 Enterprise Search PEAK Matrix assessment. The distinguishing property is that retrieval is in service of resolution rather than an end in itself: the answer is a step toward the action, not the deliverable.

Run the evaluation on your own corpus

The single most useful thing you can do, and it costs a week.

Assemble 50 real questions from your actual desk: support tickets, HR queries, the things people ask in Slack. Not questions you invented for the test.

Include ten with no answer in the corpus. This is the most informative subset and the one everybody omits.

Include five where the corpus contradicts itself. Three versions of the travel policy. See what the system does: pick one silently, surface the conflict, or hedge.

Include five that require a permission boundary to be respected. Run them as a user who should not see the answer.

Score for grounding, not fluency. Every answer will read well. The question is whether it is right, whether the citation supports it, and whether it declined when it should have.

The uncomfortable finding

Most enterprise search evaluations end up as knowledge management projects, because the corpus turns out to be in worse shape than anyone believed.

That discovery is genuinely valuable and consistently unwelcome. Search exposes knowledge debt; it does not repay it. No product ranks its way out of four contradictory expense policies, and no vendor can write your documentation.

The teams that get the most from this treat the evaluation as an audit of their content as much as a comparison of products, and budget accordingly. See building an AI-ready knowledge base for what that work involves, and why AI agents fail on enterprise data for what the benchmarks say about the ceiling.