Sayash Kapoor
@random_walker @colin_fraser In this example, verifying LLM outputs is about as hard as generating the correct answers in the first place: you need to look up each player's Wikipedia page to verify the outputs are correct. Deep research has little utility if its answer isn't exact.
Find me on bsky @colin-fraser.net
Alright, here's how it does - Overall, it produces the correct list for 6 out of 30 teams (20%) - On average, it identifies 59% of the over-30 players on each team - For 60% of teams it names at least one player not currently on that team