I remember reading a story once about a horse that was famous for being able to count
You'd tell it to count to some number, like 7, and it would stomp its hoof 7 times
Then it turned out that the horse can't understand human language and count, it was just stopping because that's when the crowd cheered
No offense to horses intended but
This is how LLM's feel when people ask it a question and ask "are you sure?" until it gets the answer right and then they stop