The AI firm Galileo has simply introduced its newest Hallucination Index, which is a framework that evaluates 22 main generative AI fashions.
Fashions are examined utilizing a metric referred to as context adherence, which measures “closed-domain hallucinations: circumstances the place your mannequin mentioned issues that weren’t offered within the context.”
The most effective performing mannequin total for RAG, in keeping with the rating, is Claude 3.5 Sonnet from Anthropic. Galileo mentioned that this mannequin and Anthropic’s different mannequin Claude 3 Opus had close to good scores, beating out OpenAI’s fashions, which received final yr.
From a price perspective, the perfect performing mannequin was Google’s Gemini 1.5 Flash. And Alibaba’s Qwen2-72B-Instruct was total the perfect performing open supply mannequin, although briefly context RAG exams, Meta’s llama-3-60b-instruct was the perfect.
Damaged down by context size, the perfect closed-source mannequin briefly context RAG was Claude 3.5 Sonnet, in medium context RAG was Google’s Gemini-1.5-flash-001 (with price being the tiebreaker with different fashions that additionally scored an ideal rating), and in massive context RAG was once more Claude 3.5 Sonnet.
“In as we speak’s quickly evolving AI panorama, builders and enterprises face a crucial problem: easy methods to harness the ability of generative AI whereas balancing price, accuracy, and reliability. Present benchmarks are sometimes based mostly on tutorial use-cases, quite than real-world functions. Our new Index seeks to deal with this by testing fashions in real-world use circumstances that require the LLMs to retrieve knowledge, a typical follow in enterprise AI implementations,” says Vikram Chatterji, CEO and co-founder of Galileo. “As hallucinations proceed to be a significant hurdle, our aim wasn’t to simply rank fashions, however quite give AI groups and leaders the real-world knowledge they should undertake the fitting mannequin, for the fitting activity, on the proper worth.”
You may additionally like…
Meta’s new Llama 3.1 mannequin competes with GPT-4o and Claude 3.5 Sonnet