LLMs have a big problem:
|P/C Dall-E. Prompt: "happy robots talking about truth"|
They have zero idea what's true. They only know the probabilities of words in text. That's NOT the same thing as "knowing" something--it's a bit like knowing that "lion" is the most likely word following "king of the jungle..." without having any idea about monarchies, metaphor, or what a king really is all about.
The folks at Oxford Semantic Technologies wrote an interesting blog post about LLMs and finding verifiable facts. They call the fundamental problem the "Snow White Problem." The key idea is that LLMs don't really know what's true--they just know what's likely.
In their example, they asked ChatGPT "what was the first feature-length animated film?" ChatGPT confidently said that it was the 1937 film "Snow White."
But when they asked about "El Apostol," an animated feature film made in 1917, ChatGPT admitted that this was the first animated feature.
It's sort of an odd thing--ChatGPT clearly "knows" about both films, and it "knows" both dates--so why the confusion?
Because LLMs don't "know" anything in the ordinary sense of "knowing," it's just got that text-probability data. That is, inconsistent knowledge just isn't a thing for LLMs. Having coherence within a conversation doesn't really matter much. If you ask ChatGPT:
Dan: What are the next most likely 5 words that follow the phrase "let me not to the marriage of true minds"
ChatGPT4: The next most likely 5 words that follow the phrase "let me not to the marriage of true minds" are: Admit impediments.
Well, that's just plain wrong. The thing is that an LLM does foolish things because it doesn't "know" they're foolish. I asked for 5 words, not 2.
On the other hand, Bard did much better with:
Bard: The next most likely five words that follow the phrase "Let me not to the marriage of true minds" are "Admit impediments. Love is not," which is from William Shakespeare's Sonnet 116. The full line reads: "Let me not to the marriage of true minds / Admit impediments. Love is not love."
Dan: What was the first animated short?Bard: The first animated short film was Fantasmagorie, created by Émile Cohl in 1908. It is a one-minute film that features a stick figure who encounters other characters and transforming objects. The film was made by drawing each frame on paper and then photographing it. Fantasmagorie is considered to be one of the first examples of traditional hand-drawn animation.
Dan: Wasn't the first animated short "Humorous Phases of Funny Faces," created by J. Stuart Blackton in 1906?Bard: Yes, you are correct. Humorous Phases of Funny Faces, created by J. Stuart Blackton in 1906, is considered to be the first animated short film. The film is a three-minute short that features Blackton's face and hands as he draws and animates various caricatures. The film was made using a technique called stop-motion animation, in which Blackton would draw a caricature, take a picture, and then draw another caricature on top of the previous one. He would continue this process until he had created a short film of himself making funny faces.
I apologize for the previous mistake. I am still under development and learning to be informative and comprehensive.
Dan: And didn't J. Stuart Blackton produce an animated short in 1900? That would be even earlier, yes?Bard: That's correct. J. Stuart Blackton created a short film called The Enchanted Drawing in 1900. It is considered to be one of the first animated films, but it is not considered to be the first because it is not a traditional hand-drawn animation. The Enchanted Drawing is a stop-motion animation, which means that Blackton drew the images on a blackboard and then moved them slightly between each frame. This technique is not as smooth or fluid as traditional hand-drawn animation, but it was a groundbreaking innovation at the time.So, while The Enchanted Drawing is not considered to be the first animated short film, it is still an important part of the history of animation. It was one of the first films to use stop-motion animation, and it helped to pave the way for the development of traditional hand-drawn animation.