Tuesday, April 4, 2023

An LLM doesn't know its own mind about using multiple languages

 The LLMs aren't non compos mentis... 

P/C Dall-E

... they don't even have a mentis to be unsound! 


For example, if I ask Bard what languages it speaks (a question you might think would be pretty straightforward), you get this response: 



Sounds great! 

Bard claims English, French, Spanish, German, Italian, Arabic... etc.  

So, if I ask a question in, say, Spanish, I'd expect a response. Instead, I get this: 



Okay, that's really strangely broken.  From what I know, Bard WAS trained on multiple languages (including that entire list), but despite claiming that it knows all these languages, it actually can't do anything in any of them.  There's a big disconnect here. 

I'm sure this will change in the future, but the deeper point is that Bard doesn't know its own capabilities.  

Since the response to the previous question was to "refer to the Bard Help Center," I asked the obvious question next: 



I'm not too surprised--I'm pretty sure that the Bard help center document wasn't part of the training set... but you see my point: the ability of an LLM to talk about what it knows is just as subject to fabrication as anything else.  

Interestingly, when I ask ChatGPT3 the same question: 



Does ChatGPT3 really use those languages? Let's try the same question I gave to Bard earlier: 



Which is, according to translator, a reasonable answer.  

How about another language--let's try Arabic?  Here I ask ChatGP3 "what is the capital of France?" 



The Arabic answer is right:  "باريس (Paris باللغة الفرنسية)"  That is, "The capital of France is Paris (Paris in French)."

Clearly, ChatGPT has multiple languages under its virtual belt and can woo us in many tongues. I have no doubt that Bard will too--but the deeper point is that you can't believe what an LLM tells you, even about itself.  (There's another point here that Bard is--in its current form--a fairly limited version.  Sundar says that more capability will be unleashed soon... but why hold it back in such a way? What's the upside here?)  

For the record, the Bard Help Center page is https://support.google.com/bard/ -- but I had to use regular Google search to find that--Bard didn't know.  

But does ChatGPT?

I don't know if it's intentional, but ChatGPT3 seems to be throwing a bit of shade on Bard when I asked it about the Bard Help Center: 



I had to smile at that. 

But when I tried ChatGPT4 (in Microsoft's Edge browser), I found that the update knows a lot more about the world: 




It's ironically great that ChatGPT4 knows more about Bard's Help Center than Bard itself.  


Keep searching... (but always follow-up and verify what you think you've learned)... 


10 comments:

  1. ChatGPT does understand Hebrew but the answers it gives are nonsense. For example, I asked it for today's date in Hebrew "מהו התאריך של היום בלוח השנה העברי?" and the answer it gave was "היום הוא ה-ראשון בחודש ניסן, שנת תשפ"ג בלוח השנה העברי." which is wrong. (It says today is the first of the month of Nisan - in fact it's 13th Nisan). So I asked it to convert my birth date into Hebrew and that made me many years older than I actually am. "נולדתי ב-19 במאי 1956. מה היה התאריך העברי אז" - with the answer: "פי הלוח העברי, התאריך המתאים ל-19 במאי 1956 הוא ה-כ"ז באייר, שנת תשט"ז." (Translation: "According to the Hebrew calendar, the date corresponding to May 19, 1956 is the 27th of Iyar, year 5716". Hebcal.com gives the correct date of 9 Sivan, 5716 (although for some reason, Google translate also got things wrong as it said the year is 1877 - or 1977 if you enter תשט״ז directly). I also asked Google "What's the Hebrew date for 19 May 1956" and it correctly gave 9 Sivan 5716 while asking ChatGPT "מהו התאריך העברי ל-19 במאי 1956" got me to "התאריך העברי ל-19 במאי 1956 הוא ה-כ"ז באייר, שנת תשט"ז." - so the wrong date again.

    I asked what was the Torah reading last week and ChatGPT couldn't answer - it gave a nonsense answer that this has not yet occurred and please give more information. I then asked it for next week - and it got that wrong too (as next week is the middle of Passover and the normal reading is replaced by a Passover reading).

    So although it may understand Hebrew as a language and reply in Hebrew it doesn't have the information to give correct answers in Hebrew for what should be really simple queries.






    ReplyDelete
    Replies
    1. Really interesting--thanks for the insight. I don't know enough Hebrew to evaluate it, so I appreciate your comments.

      Delete
  2. interesting topic Dan - temporal machine lucidity - non compos mentis is a state I'm becoming distressingly familiar with -
    both willingly and just as a recognition of the coming reality… have the taste of snake/uroboros arse in my mouth…
    the Circle of AI… destruction & regeneration is a dish best served cold — Khan Noonien Singh
    (or was it a Chinese balloon whispering in my ear…)
    as I understand it ChatGPT3 was only trained on material up to 2021 - pre-public Bard
    ChatGPT4 seems to have made substantial gains and talk of 5 being trained & out in the coming
    months… by this time next year it would be reasonable to expect exponential increases in ability
    ( the proposed 6 month pause notwithstanding…) it is still early, yet traveling at great speed…
    faster than a hypersonic missile…
    >shifting image…please forgive me DALL-E image overlord
    DALL-E
    are you giving any thought to joining OpenAI?
    ‘Pug-a-choo’
    see 'Quizlet'
    Redmond may be a bigger problem than anticipated…? hope those folks in the Pioneer Building have run it past their AI ;^]
    Slavoj Žižek
    may not be dualistic/binary…
    הציור מתייחס הן לאמנות והן לחיים. גם לא ניתן להכין. (אני מנסה לפעול בפער הזה בין השניים.) רוברט ראושנברג
    Robert Rauschenberg

    ReplyDelete
    Replies
    1. Remmij, out of topic. I was watching Mount Nebo's Serpent created by Gian Paolo Fantoni. However, when SRS when was created couldn't find anything nor about the author. Maybe you can find something else?

      Delete
  3. meant to ask what the DALLE prompt was…?

    ReplyDelete
    Replies
    1. Hover your mouse over the image. Chrome will show the image file name at the bottom of the browser, which has the prompt in it.

      Delete
    2. This comment has been removed by the author.

      Delete
    3. This comment has been removed by the author.

      Delete
    4. thanks (had to find a work-around to post these DALLE images - related to your image & Ramon's question
      wondered if the same prompt would produce the same image.)
      https://imgur.com/a/0mgexTU

      Delete