The July launch of Android Bench — its LLM benchmark leaderboard for Android improvement duties — is now primarily based upon the Harbor framework for evaluating fashions, Google has introduced.
In its weblog submit saying the updates, the corporate famous that when Android Bench was launched in March, the purpose was to supply insights into mannequin capabilities, supply AI choices and to induce bettering the fashions. On this launch, eight fashions have been added to the leaderboard — Claude Fable 5, Sonnet 5 and Opus 4.8; GLM 5.2; Kimi K2.7; MiniMax M3; and Qwen 3.7 Plus and three.7 Max.
“From the start, we’ve valued an open and clear strategy, which is why we made our authentic methodology and take a look at harness publicly obtainable on GitHub,” the corporate wrote within the weblog. “Now we’re taking collaboration a step additional by providing you with, the Android developer group, an opportunity to form Android Bench.”
Builders can now design and submit Android improvement duties and see how the fashions deal with the eventualities vital to them, and now can really run benchmark evaluations to check the fashions they use in opposition to Android’s dataset of duties. These duties could be discovered on this GitHub repository and builders can submit evaluations or discover Android’s knowledge at Harbor Hub.
“With increasingly more choices for agentic improvement,” the weblog concluded, “sustaining a cutting-edge benchmark ensures that the AI help you depend on retains getting smarter, extra useful, and simpler.”
