Greatest Native LLM for RAG: 15 Open Fashions Examined

“Our findings present that selecting an LLM for RAG is just not merely a query of mannequin dimension or reply high quality. A dependable enterprise assistant should retrieve the appropriate info, comply with source-priority guidelines, and use instruments as meant. These behaviors must be examined explicitly and strengthened by the system structure, fairly than left to the mannequin to determine.”

— Vadzim Tashlikovich, Head of ERP Options Division, SCAND

The smallest mannequin that may summarize a doc is just not essentially the smallest mannequin you’ll be able to belief along with your firm data base. To search out the very best native LLM for RAG on actual firm paperwork, SCAND’s R&D staff examined 15 open-weight mannequin configurations on primary operations: summarization, knowledge-base retrieval, instruction following, and spreadsheet calculations.

Whereas summarization is a comparatively simple activity, answering questions over inner paperwork requires a mannequin to retrieve the appropriate info, comply with source-selection directions, and supply dependable solutions grounded within the out there context.

In SCAND’s R&D checks, most open-weight LLMs dealt with primary doc duties, however solely two efficiently discovered info in inner paperwork whereas following the required source-selection directions.

The outcomes spotlight why selecting a mannequin for personal RAG requires greater than evaluating parameter counts or benchmark scores.

The Problem: Working AI on Personal Firm Information

Enterprises need AI assistants that may reply questions on inner insurance policies, analyze enterprise paperwork, summarize reviews, and retrieve info from company data bases. But many organizations can not ship confidential information to public AI APIs due to safety, privateness, or regulatory necessities.

Self-hosted, open-weight massive language fashions (LLMs) provide another. They’ll run on infrastructure managed by the group, permitting firms to retain higher management over their information and AI workflows.

A extra complete deployment might require organizations to construct a non-public LLM answer round their very own information, infrastructure, and safety necessities. Relying on the use case, smaller on-device LLMs may additionally be appropriate when native processing, lowered information switch, or restricted connectivity is a precedence.

Nevertheless, non-public deployment comes with a price. GPU reminiscence necessities, {hardware} availability, inference efficiency, and common upkeep all affect the full price of possession. A bigger mannequin might provide stronger capabilities, however it may possibly additionally make deployment far more costly.

This results in a sensible query for enterprise know-how leaders: what’s the smallest mannequin that may carry out on a regular basis duties over an organization data base with out compromising safety or accuracy?

At first look, the reply may appear simple. Most fashionable LLMs can summarize textual content, extract info, and reply questions on paperwork they obtain immediately.

However a knowledge-base assistant should do greater than course of provided textual content. It should determine the place to search for info, choose the appropriate instruments, comply with directions, and distinguish verified inner data from basic data.

These are examples of low-agency AI duties: operations that contain restricted decision-making and a small variety of software calls fairly than lengthy, autonomous plans. Typical duties embrace:

• Looking inner paperwork and retrieving related info.
• Summarizing reviews and analyzing insurance policies.
• Evaluating tables and checking enterprise guidelines.
• Studying spreadsheets and calculating balances.
• Drafting directions and answering worker assist questions.
• Monitoring adjustments and retrieving historic info.

These duties don’t require a completely autonomous AI agent. They do, nevertheless, require a mannequin that follows the appliance’s guidelines. The encompassing workflow is simply as essential because the mannequin itself, which is why AI agent growth entails designing software entry, directions, and safeguards alongside mannequin choice.

Our R&D staff evaluated open-weight LLMs on consultant knowledge-base operations to grasp the place smaller fashions are enough, the place bigger fashions make a distinction, and which failure modes matter most in a non-public enterprise surroundings.

One discovering stood out: the principle threat was not merely an incorrect reply. It was a mannequin that silently skipped the interior data base, relied on its basic data, or tried to make use of an exterior search software when the duty required inner sources.

How We Examined the Fashions

We in contrast open-weight fashions utilizing the identical set of inner company paperwork, seven on a regular basis questions, and a unified infrastructure for brokers geared up with auxiliary instruments.

Every mannequin was instructed to prioritize inner paperwork. Questions had been posed a number of instances to evaluate how persistently the fashions adhered to the required workflow. This issues as a result of two AI brokers with the identical LLM can provide totally different outcomes when their prompts, instruments, runtime settings, or surrounding workflows differ.

Our LLM tool-calling benchmark centered on whether or not fashions may use out there instruments appropriately, retrieve proof from the right supply, and comply with the appliance’s directions fairly than merely generate believable solutions.

The agent harness supplied 5 capabilities:

• Inside knowledge-base search.
• File-system instructions.
• Web and Wikipedia search.
• Calculations.
• Excel file studying.

We grouped the seven questions into 4 classes.

Every query (a number of runs) was scored as a cross or fail, producing a most rating of seven. The analysis intentionally coated greater than the standard of the ultimate response.

We additionally examined the mannequin’s habits: Did it use the suitable software? Did it retrieve proof from the right supply? Did it comply with the standing instruction, or did it take a shortcut?

This distinction is essential. A solution may be factually right and nonetheless be unsuitable for an enterprise utility if it was obtained via an unauthorized workflow.

The analysis is a task-specific check fairly than a common rating of each mannequin’s tool-use capabilities. A broader software calling benchmark LLM comparability would want to cowl further instruments, activity varieties, and runtime configurations.

Fashions and {Hardware} Necessities

We evaluated fashions of various sizes and architectures, together with dense fashions and mixture-of-experts (MoE) fashions. We additionally in contrast totally different quantization settings for chosen fashions.

Quantization reduces the precision used to symbolize mannequin weights, typically reducing reminiscence necessities and making native deployment extra accessible. Nevertheless, it was decided that 4-bit quantization has the very best steadiness between high quality and dimension.

When a mannequin struggles with domain-specific terminology or directions, adaptation strategies corresponding to parameter-efficient fine-tuning (PEFT) may additionally be price evaluating. Superb-tuning is a separate consideration from the checks reported right here, nevertheless, and needs to be assessed towards the identical task-specific standards as the unique mannequin.

The next desk summarizes the mannequin configurations and reminiscence necessities recorded throughout our checks.

Precise deployment necessities depend upon the inference runtime, context size, KV cache, concurrent requests, and {hardware} configuration.

The reminiscence comparability highlights an essential consideration: parameter rely alone doesn’t decide the sensible price of working a mannequin. Structure, quantization, lively parameters, and implementation all matter.

For enterprise deployments, the related query is just not merely how massive a mannequin is. It’s how a lot {hardware} it must ship dependable efficiency on the meant workload.

Key Outcomes: Summarization Is Simple; Dependable Retrieval Is Not

The outcomes revealed a major distinction between processing info that’s already out there and deciding the place to retrieve info from.

Fourteen of the 15 fashions within the scored comparability handed each summarization questions. In distinction, solely two fashions handed each knowledge-base retrieval questions.

Discovering 1: Summarization Is Largely Solved for This Workload

Fourteen fashions handed each summarization questions. Even comparatively small fashions may condense provided paperwork into helpful summaries.

This makes summarization a poor standalone benchmark for choosing a non-public knowledge-base assistant. A mannequin can carry out effectively when the related textual content is positioned immediately in its context and nonetheless fail when it should retrieve that textual content independently.

For organizations that want solely fixed-prompt doc processing, a small mannequin could also be enough. For an assistant that should search throughout inner repositories, extra intensive analysis is critical.

Discovering 2: Data-Base Retrieval Is the Actual Dividing Line

Solely Qwen 3.8 27B at 4-bit quantization (small mannequin) and DeepSeek V4.1 Flash (medium-sized mannequin) handed each retrieval questions.

The opposite fashions continuously did not retrieve the anticipated info from inner paperwork, regardless of gaining access to a knowledge-base search software and being instructed to prioritize it.

That is the distinction between understanding find out how to reply a query and understanding find out how to acquire the proof required to reply it.

In a company surroundings, an assistant should retrieve info from the right supply, not merely produce a believable response. A solution drawn from basic coaching information could also be outdated, incomplete, or inconsistent with the corporate’s present insurance policies. A purpose-built non-public RAG chatbot for an inner data base illustrates the type of workflow by which reliable retrieval is central to the outcome.

Discovering 3: Multilingual Retrieval Can Work Throughout Language Boundaries

Qwen 3.8 27B efficiently retrieved details about Poland from a doc written in one other language.

For worldwide organizations, this functionality may be notably beneficial. Company data bases typically include paperwork created by totally different regional workplaces, native authorized groups, or enterprise items working in several languages.

Language protection, doc high quality, retrieval configuration, and the complexity of the query ought to all be included in a manufacturing analysis.

Discovering 4: Following Directions Was Tougher Than Answering Questions

Eleven of the 15 fashions handed neither instruction-following query. No mannequin handed each instruction questions cleanly.

The fashions struggled once they had been anticipated to make use of inner sources even for questions they appeared able to answering from basic data.

This exposes a limitation of counting on system prompts alone. A standing instruction corresponding to “At all times search inner paperwork first” might affect habits, nevertheless it doesn’t assure that the mannequin will comply with the required workflow each time.

For enterprise methods, the appliance ought to implement vital source-selection guidelines wherever potential fairly than treating them as elective mannequin habits.

Discovering 5: Spreadsheet Calculations Profit From Devoted Instruments

Ten of the 15 fashions reportedly calculated the spreadsheet steadiness accurately, typically by utilizing a calculator.

It is a reminder that language fashions don’t have to carry out each operation themselves. For arithmetic, spreadsheet processing, and different deterministic duties, specialised instruments may be extra dependable than asking a mannequin to purpose via uncooked values in textual content.

The mannequin’s position is to determine the related information and invoke the suitable operation. The software ought to carry out the calculation, and the appliance ought to validate the outcome the place accuracy is essential.

Discovering 6: Extra Parameters Do Not Assure Higher Outcomes

Qwen 3.8 27B at 4-bit quantization handed six of seven duties, whereas gpt-oss 120B (the golden business commonplace) handed 4.

Equally, Gemma 4 26B MoE and Qwen 3.8 27B at 4-bit quantization had broadly comparable reported reminiscence necessities however considerably totally different scores.

The sensible lesson is simple: parameter rely is just not an alternative to task-specific analysis.

A mannequin with fewer parameters would be the better option if it follows directions extra reliably, makes use of instruments appropriately, and produces higher outcomes on the corporate’s precise workload.

Commerce-Offs: Accuracy, Privateness, and Infrastructure Value

The outcomes recommend 4 broad teams of fashions for the examined workload. These tiers assist slim down the very best open supply LLM for RAG for a given workload, however they need to be handled as evaluation-specific steering fairly than a common rating.

These tiers describe efficiency in our analysis, not a common rating of mannequin high quality. When evaluating the very best LLM fashions for RAG, the appropriate alternative will depend on whether or not the appliance prioritizes retrieval, summarization, software use, or a mixture of duties.

Privateness Is an Utility-Degree Duty

Probably the most consequential findings considerations exterior search. If a mannequin can name an web search software and chooses to make use of it for a query about inner firm information, the question might depart the group’s managed surroundings.

Relying on the search supplier, the question may reveal confidential enterprise context, buyer info, or particulars about inner operations. In fact for manufacturing methods, web search instruments should be turned off.

For a corporation with strict data-handling necessities, that may be a compliance or safety incident even when the ultimate reply is right.

This threat is just not inherent to each open-weight mannequin or each RAG structure. It will depend on how the appliance, instruments, permissions, and community entry are configured.

The necessities may be particularly strict in regulated settings: for instance, non-public AI for clinics should account for the confidentiality of affected person info, whereas non-public AI for regulation companies should shield privileged communications and delicate case supplies.

A safer design makes the appliance liable for supply choice:

• Search the approved inner data base first.
• Prohibit or disable exterior seek for delicate workflows.
• Apply entry controls earlier than retrieving paperwork.
• Preserve software permissions slim and auditable.
• Require express authorization earlier than any motion that would disclose delicate info.
• Log retrieval choices and gear requires analysis and incident investigation.

If the appliance can not discover enough proof internally, it ought to comply with an outlined fallback coverage: ask for clarification, report that the knowledge is unavailable, or request a licensed exterior lookup. It mustn’t silently substitute public info for inner proof.

Quantization and Configuration Matter

The identical Qwen 3.8 27B mannequin achieved six passes at 4-bit quantization and three at 8-bit quantization within the check outcomes.

It is a noteworthy distinction, nevertheless it shouldn’t be interpreted as proof that 4-bit quantization is inherently superior. Quantization alone doesn’t clarify the end result. Runtime settings, prompts, sampling parameters, software integration, and check variability may additionally affect outcomes.

The proper method is to benchmark the precise configuration meant for manufacturing, together with its quantization methodology, inference engine, context size, and agent harness.

{Hardware} Prices Can Outweigh the Advantages

Qwen 3.8 27B at 4-bit quantization was the smaller of the 2 top-scoring fashions on this analysis. DeepSeek V4.1 Flash achieved the identical complete rating however had a a lot increased reported reminiscence requirement.

That distinction issues when designing a non-public AI deployment. A mannequin that requires lots of of gigabytes of reminiscence could also be troublesome to justify for a workload involving doc retrieval and easy calculations.

Infrastructure choices must also account for inference latency, concurrent customers, context size, availability necessities, and prices. The smallest mannequin that passes the checks will not be the one viable possibility, nevertheless it is a vital place to begin. And it’s worthy to say that for the price of DeepSeek infrastructure, you’ll be able to afford a number of Qwen fashions.

Suggestions: Select a Personal LLM

Figuring out how to decide on LLM for RAG begins with defining the duties the mannequin should carry out and the implications of failure. Moderately than deciding on a mannequin by its parameter rely, consider retrieval high quality, instruction following, software use, {hardware} necessities, and privateness controls collectively.

1. Begin With Qwen 3.8 27B at 4-Bit Quantization for Analysis

Of the examined configurations, Qwen 3.8 27B at 4-bit quantization provided the strongest steadiness between the reported activity rating and reminiscence necessities.

DeepSeek V4.1 Flash achieved the identical rating, however its reported reminiscence requirement was considerably increased. It might be price evaluating when its explicit capabilities justify the infrastructure price.

gpt-oss 120B is also thought-about for workloads the place its strengths are related, however its retrieval and instruction-following habits needs to be examined and the agent harness configured to match its capabilities.

These are beginning factors for analysis, not common manufacturing suggestions. If you happen to’re selecting LLM for RAG, benchmark the mannequin with your individual paperwork and deployment configuration earlier than making a ultimate resolution.

2. Use Smaller Fashions for Less complicated Doc Operations

For summarization and chosen duties involving a clearly recognized doc, Granite 4.2 8B and Qwen 3.5 9B are cheap candidates to judge.

Smaller fashions could also be enough when the appliance provides the related textual content, limits the out there instruments, and doesn’t depend upon the mannequin independently navigating a big data base.

This may scale back infrastructure prices with out paying for capabilities the workload doesn’t want.

3. Consider the Greatest Native LLM for Device Calling

One of the best native LLM for software calling is just not essentially the most important or probably the most correct on strange question-answering benchmarks. It should reliably choose the appropriate software, present legitimate inputs, interpret the software’s output, and comply with the appliance’s guidelines.

In our checks, the strongest retrieval outcomes didn’t translate into good instruction following. This means that software choice and source-selection habits needs to be evaluated individually from reply high quality.

For a manufacturing agent, take into account limiting the out there instruments, validating arguments earlier than execution, checking software outputs, and recording every software name for auditing.

4. Implement Supply Choice within the Utility

Don’t rely solely on a mannequin’s promise to make use of inner paperwork. For delicate workflows, the appliance ought to management retrieval order, entry permissions, and exterior connectivity. It may be achieved by compelled graph AI-agent structure.

If exterior search is prohibited, the mannequin mustn’t have entry to that software within the first place. This architectural safeguard is extra reliable than a immediate alone.

5. Use Devoted Instruments for Calculations

For balances, monetary totals, spreadsheet comparisons, and different numerical operations, let a deterministic software carry out the calculation.

The mannequin ought to determine the related rows and columns, invoke the suitable operation, and clarify the outcome. The applying ought to validate inputs and outputs when errors may have materials penalties.

6. Deal with Very Small Fashions Cautiously in Agent Workflows

On this check, the smallest fashions struggled with software use, retrieval, or calculations. Fashions round 8B parameters and above had been typically extra helpful for doc operations, however dimension alone didn’t assure dependable agent habits.

Moderately than adopting a common parameter threshold, consider fashions towards the precise operations required by the appliance. A smaller mannequin with restricted tasks could also be preferable to a bigger mannequin with poorly managed entry to instruments.

7. Check With Actual Firm Paperwork Earlier than Deployment

Public benchmarks and general-purpose evaluations can not totally symbolize an organization’s terminology, doc construction, permissions, languages, or insurance policies. A helpful analysis set ought to embrace:

• Questions whose solutions are current in inner paperwork.
• Questions that require looking out throughout a number of information.
• Paperwork written in several languages.
• Questions the mannequin is more likely to consider it may possibly reply from basic data.
• Conflicting or outdated paperwork.
• Spreadsheet calculations and information extraction.
• Questions for which the right response is that the proof is inadequate.
• Makes an attempt to entry info the person is just not approved to see.

Consider not solely the ultimate reply but additionally the retrieval path, software calls, supply citations, access-control habits, and consistency throughout repeated runs. This type of validation is a vital step when transferring an AI pilot to manufacturing, as a result of a promising demonstration doesn’t by itself set up manufacturing readiness.

How SCAND Approaches Personal LLM Choice

At SCAND, deciding on an LLM for an enterprise answer is just not merely a matter of selecting the most important mannequin that matches the out there {hardware}.

By our non-public LLM growth companies, we consider candidate fashions towards the shopper’s paperwork, goal duties, and infrastructure constraints. For retrieval-driven functions, this consists of testing the structure and retrieval pipeline as a part of RAG growth, not treating the language mannequin as the one issue that determines high quality.

The aim is to determine a configuration that gives the appropriate steadiness of reply high quality, retrieval reliability, privateness, efficiency, and value. This analysis helps reply sensible questions earlier than manufacturing deployment:

• Can the mannequin retrieve the appropriate info from the shopper’s data base?
• Does it comply with the appliance’s source-selection and safety guidelines?
• Can it deal with the shopper’s paperwork and terminology?
• Which duties require specialised instruments fairly than language-model reasoning?
• What {hardware} is required to fulfill efficiency and concurrency expectations?

These concerns additionally apply to specialised use instances, corresponding to a non-public LLM for supply code documentation, the place the assistant should retrieve correct technical info from inner repositories fairly than depend on generic solutions.

The outcomes then inform the structure, mannequin configuration, and safeguards wanted for the particular use case. Organizations contemplating an analogous method can run this analysis in your information to grasp which mannequin and deployment configuration greatest match their paperwork, safety necessities, and infrastructure.

Conclusion: The Smallest Mannequin That Works Is the One That Follows the Guidelines

Our analysis reveals that summarization and dependable knowledge-base help are totally different capabilities.

Most examined fashions may summarize provided textual content, however solely two achieved the strongest total rating of six out of seven duties. Neither handed each instruction-following questions cleanly. Even the strongest candidates subsequently require application-level safeguards and additional validation earlier than manufacturing use.

For enterprises, the aim shouldn’t be to search out the smallest LLM that may produce a convincing reply. It needs to be to search out the smallest sensible configuration that retrieves the appropriate proof, follows the required workflow, makes use of instruments accurately, and respects information boundaries.

That’s the reason non-public LLM choice ought to start with the workload and the dangers, not the parameter rely. A rigorously evaluated smaller mannequin, supported by a well-designed RAG structure, could be a extra sensible enterprise answer than a a lot bigger mannequin deployed with out ample controls.

One of the best non-public AI assistant is just not the mannequin that is aware of probably the most. It’s the one which is aware of the place to look, follows the principles, and stays throughout the boundaries of the system.

Regularly Requested Questions (FAQs)

Which LLM is greatest for RAG?

In our checks, Qwen 3.8 27B at 4-bit quantization and DeepSeek V4.1 Flash achieved the very best complete rating, with six out of seven duties handed. Neither handed each instruction-following questions cleanly, so your best option will depend on the workload, {hardware}, and application-level safeguards.

What’s the greatest LLM for a RAG chatbot?

One of the best LLM for RAG chatbot functions is one which retrieves the appropriate proof persistently, follows source-selection guidelines, and works throughout the out there {hardware} price range. Qwen 3.8 27B at 4-bit quantization is a helpful place to begin from our analysis, nevertheless it needs to be examined towards the chatbot’s precise paperwork and questions.

What’s the smallest LLM for RAG?

The smallest LLM for RAG will depend on the complexity of the paperwork, the retrieval pipeline, and whether or not the mannequin should use instruments independently. In our analysis, smaller fashions dealt with summarization higher than knowledge-base retrieval, so a compact mannequin may match for fixed-prompt duties however require extra validation for interactive RAG. In our checks 27B+ fashions is the appropriate parameter dimension begin.

What’s the greatest native LLM for brokers or agentic duties?

One of the best native LLM for brokers will depend on the instruments out there, the complexity of the workflow, and the way vital instruction following is. Our outcomes present that robust retrieval alone is just not sufficient: brokers additionally want managed software entry, clear source-selection guidelines, and analysis throughout repeated runs.

Can a smaller native mannequin outperform a bigger mannequin in RAG?

Sure. In our analysis, Qwen 3.8 27B at 4-bit quantization achieved a better rating than gpt-oss 120B whereas requiring considerably much less reported reminiscence. This doesn’t set up a common rating, nevertheless it demonstrates why actual activity efficiency issues greater than parameter rely alone.

How do you cease a RAG assistant from sending queries to the web?

Take away the online search software from delicate workflows as an alternative of counting on a immediate. In our checks, fashions generally selected exterior search even when advised to make use of inner paperwork first. The applying ought to management retrieval order, community entry, and gear permissions, and log each software name for evaluation.