Friday, June 16, 2023

Another cautionary note: Why you need to fact-check everything an LLM-based system tells you

 It was annoying, but necessary.. 



.. I needed a new laptop, so I hied myself down to the local Apple shop and picked up a fancy new laptop.  Not a problem--I've used Macs for years (except when I worked at IBM, when the IBM PC was the axe of choice), so I know the drill.  I know how to shift all my stuff from old-Mac to new-Mac.

But I'd forgotten how many settings needed to be copied over as well. There's the wifi, the default font size, the settings for folder appearances, etc etc.  

Then I noticed a new behavior that was driving me crazy.  Whenever I was working in Chrome, the system would add an extra period after a closing parenthesis.  It looked like this: 

 (blah blah blah.). 


Why would it add an extra period?  Don't know.  After fooling around for a bit, I figured out that the problem was not Chrome adding a period after the parenthesis, but after I typed two spaces.. then it would add an extra period  (I learned to type back in an era when two-spaces were the default.)  

So I did the obvious thing and asked Bard:  

  [ how do I get Chrome to stop adding an extra period after two spaces? ] 

Here's what it told me.  See that line with the big red arrow in the image below?  Yeah... that's bogus.  That line and everything after it is just purely made-up.  There IS no "Advanced" option under Chrome settings, so everything after line 3 is just hallucinated and utterly bogus.    


If you follow these instructions, you'll see this after you select "Settings": 


Note that there's no "Advanced" option here.  

Of course, this could be advice about a previous version of Chrome (but wouldn't that be relevant information to include here?).  

So it was with no little surprise that I learned today that Google is telling its employees to be cautious in their use of Bard.  They're more concerned about people leaking sensitive information, but the general policy of Fact Check Everything still applies here.  

After a little more conventional poking around, I found that the actual way to fix this problem is by changing a setting in the Mac OS!  (See this article for how to do it.)  That is, the problem had nothing to do with Chrome at all!  

Instead, the problem really is that the systems we use are composite systems.  An app like Chrome sits on top of the Mac OS, which in turn uses more systems below it.  The behavior you, the user, sees is the compilation of everything below. 

In this case, the "adding a period after two spaces" thing is part of the text handling system of MacOS.  Chrome can layer its own behaviors on top of that (such as redefining how Control-F works, but that's another story).  

What this means for you as a SearchResearcher is: 

1. Behaviors you see in your computer might be caused by any of a number of settings.  Don't be quick to blame the system you see on top (Chrome, in this case).  The troubles might be caused deeper in the stack. 

2. Once again, Fact Check everything.  Even things that look like simple documentation might be pure fabrications.  Check before wasting your time.  


Keep searching.  


7 comments:

  1. I was wondering if anyone would notice that I switched the background screen image to the Big Sur pic. I ALMOST went all the way back to Catalina because I've spent a lot of my life diving on Catalina. But Big Sur is closer to home now, and I've run up and down that road, over that bridge a few times.

    ReplyDelete
  2. And, FWIW, I grew up close to the Ventura Highway--that song is like old home week.

    ReplyDelete
  3. dang — I wish I had saved that album cover… unique…
    …it was a harbinger for America…㋡
    1872 seems like just yesterday — I should Google_check that…
    "The yellow smiley icon was born in 1963 in Worcester, Massachusetts, when the graphic designer Harvey Ball was approached by State Mutual Life Assurance Company to create a morale booster for employees."
    "designer Shigetaka Kurita is considered to be the founding father of today's emojis. In 1999, NTT DOCOMO, a Japanese cell phone company, released a set of 176 emojis for mobile phones and pagers. Emoji is the blend of two Japanese words: picture and letter."

    ReplyDelete
  4. Serious question: why would anyone ask Bard this question?
    Type exactly the same question (i.e. [ how do I get Chrome to stop adding an extra period after two spaces? ]) into Google Search and the first result is the correct answer!

    Also, a couple of small typos: "I learned to type back in an era when two-periods where the default." (You meant "spaces" not "periods" and "were" not "where").

    ReplyDelete
    Replies
    1. We're exploring the space of possibilities. Is Bard good enough to answer this question now? Not really. But at some point it might be, and we should understand what works, what doesn't work, and why. This is about exploration!

      Thanks for the correction info. Will fix it up.

      Delete
    2. Fair enough; I see.
      It seems to me that what everyone says about LLMs is that anything they tell you has to be checked, and if you are going to have to verify with info from elsewhere, you may as well just start elsewere. And even if LLMs acheive 99.9% reliability, that 0.1% might be mission-critical.

      Delete