In a groundbreaking independent benchmarking study released this April, Vecflow's Oliver has emerged as the top-performing AI assistant for in-house legal teams, outranking competitors including GC AI, ChatGPT, Microsoft Copilot, Google's NotebookLM, and DeepSeek AI.

The comprehensive evaluation, conducted by practicing attorneys Anna Guo and Arthur Souza Rodrigues, tested six AI tools on real-world information extraction tasks submitted by in-house counsel across the United States, United Kingdom, Singapore, and China. The results clearly demonstrate that purpose-built legal AI solutions deliver superior performance in the metrics that matter most to legal professionals.

Oliver Achieved the Highest Overall Score

While Google's NotebookLM narrowly edged out Oliver in raw accuracy (77.8% vs. 66.7%), Oliver secured the highest composite score when factoring in both accuracy and usability factors like helpfulness, adequate response length, and feature support. This comprehensive assessment reflects the real-world value Oliver delivers to legal teams beyond just providing correct answers.

Purpose-Built Legal AI Delivers Superior Usability

The study found that while general-purpose AI tools can sometimes match legal-specific tools in raw accuracy, purpose-built legal AI assistants like Oliver excel in the usability features that streamline legal workflows. Oliver stood out by offering source-linked answers, multi-document support, and structured outputs tailored for legal review.

General-Purpose AI Tools Often Miss the Mark

The evaluation revealed concerning performance issues with mainstream AI tools:

• Microsoft Copilot demonstrated the lowest accuracy at just 38.9%

• ChatGPT and DeepSeek, while occasionally accurate, produced outputs that were generic, overly long, and frequently missed important legal nuances

Why These Results Matter for Legal Teams

At Vecflow, we've always believed that legal work requires purpose-built AI solutions that understand the specific workflows, terminology, and requirements of legal professionals. This independent study validates our approach of building AI agents specifically designed to produce end-to-end legal work.

The findings align perfectly with what we hear from our users every day: legal professionals need AI tools that not only extract accurate information but present it in a way that integrates seamlessly into their existing workflows. General-purpose AI simply can't match the specialized capabilities of platforms like Oliver when it comes to legal-specific tasks.

Common AI Failure Modes Identified

The study identified six common failure modes where AI assistants often stumble with legal tasks:

1. Struggling with open-ended questions - Most AI tools provided incomplete answers when prompts lacked clear boundaries

2. Hallucinating missing information - Many tools fabricated answers rather than admitting uncertainty

3. Errors when handling multiple documents - Few tools could properly analyze information across multiple files

4. Mirroring user assumptions - AI tools often reinforced false premises in queries rather than verifying them

5. Technical limitations preventing content access - Some tools failed due to file format issues or upload limits

6. Difficulty with contradictory information - AI assistants frequently failed to recognize or flag discrepancies in source documents

Notably, Oliver was one of the few tools that successfully avoided many of these pitfalls, particularly excelling in identifying when information was missing rather than fabricating answers.

The Future of Legal AI

As the study authors note: "As accuracy becomes a baseline, the real differentiators will shift to usability, workflow integration, and support. Features like an intuitive interface, integration with email or document systems, strong data security, and responsive support will increasingly define which tools deliver real value to legal teams."

At Vecflow, we're proud that Oliver is leading the way in this new paradigm of legal AI. We continue to refine and enhance our platform based on real user feedback and rigorous testing like this independent benchmark.

The complete study offers invaluable insights for legal teams navigating the rapidly evolving AI landscape. It provides a transparent, practical assessment of how these tools actually perform in real-world legal tasks and how much human oversight they still require.