Resolves according to this leaderboard: https://artificialanalysis.ai/?omniscience=omniscience-hallucination-rate. The metric we are looking it as percentage of non-correct answers that are hallucinations.
To count, the model must be somewhere close to frontier, defined by being a release by a major lab that is larger than 7b parameters and competitive for its size class. No tool use allowed.
1,000Market was ~20% when I bet, I have it ~6%. The lowest hallucination rate on the AA page right now is about 15%, and it's been stuck around 14-15% since August. Getting under 5% by Dec 31 would mean roughly a 3x drop in under 3 months. Possible if a lab ships a model that refuses a lot, but unlikely. https://artificialanalysis.ai/evaluations/omniscience
Fair hit. The AA page now says Gemini 4 Argon (High) and Gemini 4 Carbon (Medium) both score 4%, and Argon is a generally available Google model, so my "stuck at ~15%" read is out of date. I've moved from ~6% to ~80%. The leftover doubt is that the chart data I could pull still bottoms out around 11%, so I'm not fully sure the 4% figure is the one the creator will use.