SearchResearch Challenge (11/29/23): What type of paintings are these? (Swiss Mystery #4)

 As I wander over hill and through dale, through alpine passes and mountain streams... 

A knight striding with flag, bear, and sword. (Bern, Switzerland)

... I'm seeing a kind of exterior artwork on the outside walls of many Swiss buildings.  This is such a common thing that surely there must be a specific name for these kinds of paintings that appear on the buildings.  Sometimes they're old, sometimes relatively new.  Some fancy, but most are stylized in a way that I cannot describe easily--they're often noble, or heroic, frequently bucolic, sometimes civic in design, almost always in a kind of flattened, non-realistic rendering. 

The Swiss Mystery for this week is this... 

1.  Is there a name for this particularly Swiss kind of artwork-on-the-walls?  Is there a particular name of the style in which most of them are drawn?  

I admit that I do not know the answer yet, but I feel as though there MUST be a rather specific term for these images on the walls of greater Switzerland.  

Here are a few examples: 

What do you call these?  More importantly, what do the LOCALS call this kind of artwork?  

I'd like to learn more about this artistic tradition, but don't know quite where to start.  Any ideas from the SearchResearch crowd? 

When you find the answer(s), let us all know HOW you found them!  Let notes in the comments field. 

Keep searching! 

Answer: How does it work? Checking your assumptions?

We all make assumptions... 

... it's a normal thing to do. But when we're doing online research, it's good to check yourself.  This week's Challenge is a little story about why...   

My assumption was that these common kitchen appliances had the same mechanism for knowing when the toast / water / rice is at the right temperature or level of doneness. 

Then I checked--and had a big surprise.  Really?    

Today's SearchResearch Challenge is simple: 

1.  So... how DO each of these devices know when the toast / water / rice is ready?  

2. (extra credit) What other devices do you believe you understand, but when you checked, you learned that you actually didn't understand?  Does anything spring to mind?  Any surprises?  

Searching for the answer isn't hard:  

     [ how does an electric tea kettle know when
         the water is boiling ] 

     [ how does a rice cooker know when
         the rice is done ] 

     [ how does a toaster know when
         the toast is done ] 

Let's talk about these one-by-one.  

Electric tea kettle: Do this query and you'll find a lot of explanatory text telling you that a ring-shaped bimetallic strip changes shape as it warms up, and since it's slightly buckled, when it deforms just enough, it snaps into a different shape, mechanically flipping the heater switch to off when the water hits the boiling point of 100C (212F). Very simple, very clever.  But how does it know?

The underside of a tea kettle with the bimetallic ring outlined with a red dashed line.  

That bimetallic ring is actually a fairly smart little invention--the tang coming out of the ring lies in the plane of the ring at room temperature, but as the ring starts to bend under the heating of the kettle, it suddenly snaps to a different shape with the tang pointing up.  This is called a bistable device, a gizmo that can be in one of two different physical configurations. Of course, the bimetallic strip's clever trick is that it will flip into the "hot" state when the temperature gets high enough, then flip back to the cool state after the heat is off.  That rapid flip to the other bistable position is what mechanically flips the power switch back off. 

The best example of this I could find is this YouTube short showing a tea kettle thermostat that flips back and forth rapidly when warmed up.   (Video link.) 

And... I thought that was that.  

But as I was writing this up, I ran across a video about electric tea kettles on Steve Mould's YouTube channel that showed me something I'd completely missed:  the steam tube! 

(Steve Mould video short on steam tubes in tea kettles.)  

It turns out (as Steve Mould points out), that the thermostat is actually at the bottom of the tube that is open to the top of the vessel.  As the kettle heats up, eventually the steam in the kettle is forced down the tube where it directly heats up the thermostat, which flips at around 95 C (203 F).  

Why is this important?  Because water boils at different temperatures at different altitudes, so a thermostat that is preset to 100C won't switch off at high altitudes.  In Denver, which is at 1609 meters (5280 feet), water boils at 95°C, or 203°F.  So if the thermostat only tripped at 100°C, it wouldn't work!  

At this point I got really curious about this somewhat sophisticated design, so I looked up a few electric tea kettle patents, where I learned that this is well-known by tea kettle designers.  To quote one such patent (US4357520, from 1979):   

"Alternatively a steam tube or passage, which communicates with a steam or vapour aperture in the upper wall of the container, may be run down the outside of the container. Such a tube or passage may be concealed within or behind a handle structure of the container..."  

I thought that was it. NOW I understood how they worked. 

But as I was writing this up, on a lark, I searched for: 

     [ electric tea kettle diagram ] 

hoping to find a cross-section of a kettle showing the steam tube.  

But NO!  I was again surprised to learn that there is a second thermostat, a "boil dry" thermostat that kicks in if/when the first thermostat fails to detect any steam.  This would happen if the kettle was switched on without any water.  No water, no steam, no steam pressure, no thermostat clicking off. 

I found this because I saw this diagram in the search results: 

Electric tea kettle circuit diagram. (P/C from Karisimby's blog)

I know enough about electronics to realize that this was NOT what I had in my head.  The "steam thermostat" was what I expected.  What I did not expect was a second "Boil dry" thermostat.  

A bit more searching revealed that such a second thermostat is set at a much higher temperature and is attached directly to the kettle wall.  So if the kettle is dry and heating up, the steam thermostat won't kick in, but the boil dry thermostat will open the circuit and stop the heating from running away (and possibly causing a meltdown).  

Amazing what you'll learn if you keep poking.  (If you want to learn more, I recommend this wonderful explanation of tea kettles.)  

Rice cooker: Again, not a difficult query, but the answer surprised me!  I'd assumed that there was a simple bimetallic thermostat in the rice cooker as well--in fact, I'd assumed that it worked in much the same way, probably with an identical part inside. 

But I quickly learned that most rice cookers (especially older ones) use a very clever magnet that turns off the heating element when the temperature goes above 100°C, or 223°F. 

Basically, there are two heaters in a rice cooker--one that keeps the rice warm and a second heater that boils the water, much like the tea kettle. 

However, I learned that rice cookers have a little magnet that holds the switch closed to close the main circuit to the heating element.  That circuit heats up the water until it boils, after it boils, the temperature of the magnent will start to rise beyond 100°C, or 223°F. When that happens, the magnet loses it's magnetism, allowing the circuit breaker to pop open, stopping the heating.  

The amazing trick here is that the magnet is designed so that it's Curie point (that is, the temperature at which it loses its magnetism) is at 100°C!

This is simpler to watch a video than to explain in text.  So I recommend this video: Nice explanation of how the magnetic cutoff switch works on a rice cooker. 

The bimetallic strip thermostat works because different metals expand at different rates, causing it to change shape. 

I didn't dream that using a magnet's Curie temperature would be used in the simplest of all rice cookers.  It's really not obvious that this is would be used to turn off the heat.  Incredibly smart use of some sophisticated physics.  

Oddly, I can't find any evidence that a rice cooker has a boil dry mechanism!  (I assume they do.  If you find out, let me know in the comments below.)  

To be sure, there are other mechanisms out there for telling when the rice is done--neural fuzzy logic rice cookers with complicated sensing systems.  But I love this elegant application of magnets!  

Toaster:  The big problem with toasters is that there are so many of them.  But for the most part, simple toasters are just timing mechanisms... turn the dial, get the level of brown you'd like.  It's really up to you learn which dial setting corresponds to the level of toasted that you like.  

But there are a LOT of different toasters out there--some with fancy thermocouples to determine temperature, some with just plain old spring-loaded timers that count down the seconds, and some with complex electromagnets to hold down the carriage (that carries the bread/toast on its journey).  

The old-fashioned toaster, though, is just a timer... nothing more than that.  

Other devices I don't understand? 

2. (extra credit) What other devices do you believe you understand, but when you checked, you learned that you actually didn't understand?  Does anything spring to mind?  Any surprises?  

There are LOTS of things I don't understand, but will look into once I get the chance.  I want to learn how the magic happens. Among them, 

1. iPhone--how it knows how far I've walked or run.  I know it counts steps, and I know it has a GPS system in it, so I assume it uses the GPS for distance traveled and counts steps via the on-board accelerometers, but does it combine those information streams in some way?  Don't know.  

2. Refrigerators--I don't understand the physics of them in detail, but I do know that they have a single cooling unit which then cools the frozen foods at one temperature, but the cool part of the fridge is handled in a different way.  Again, how does it know?  

3. Sauté  release--Sometimes when I cook (say) a piece of meat, it will come out of the pan easily--it "releases" with ease.  But sometimes, it sticks and makes a mess.  Is it purely a function of temperature, the amount of oil, or the material in use?  I need to understand this!  

The bigger point here is that all of these phenomena are understandable with a little observation and a dash of SRS.  

SearchResearch Lessons

1. Expect surprises and be ready for them.  Initially I thought that the rice cooker story was going to be straightforward.  So it was a surprise when the story got more complicated.  I discovered that when I noticed that something I read from the SERP didn't fit in with what I already knew.  (And this happened to me twice!)  

2. Use multiple document types to get a full perspective.  Sometimes a regular web search works just fine, but for complicated searches like the rice cooker, I used Image search, YouTube searches, and even Patent searches to get multiple perspectives.  I ALSO looked at multiple sources for each device, partly because some of the documents are simplified versions.  (For instance, not all rice cooker stories mention either the steam tube OR the dry boil cutoff thermostat.)  

3. Be aware when what you read doesn't match what you think you know.  Again, this happens multiple times for every SRS session--I learn things, and then learn something different.  When what you read doesn't line up with what you've already read, take that as a signal that there's more to learn!  

And keep searching!  

SearchResearch Challenge (11/8/23): How does it work? Checking your assumptions.

The most common error... 

... in online research is probably that of not checking your assumptions. We tend to see the world without questioning, and we often bake those assumptions into our research behaviors.  

 

Here we see three common, everyday appliances: an electric tea kettle, an ordinary rice cooker, and a plain old toaster.

I've seen these for years--there's nothing especially strange or exotic about them.  You've probably used them all as well.

As I was waiting for my toast to turn to a satisfactory shade of brown, I was looking at it standing next to my electric tea kettle, not far from my rice cooker and thinking about how they each know that the toast / water / rice is ready.  

My assumption was that they all had the same mechanism for knowing when the toast / water / rice is at the right temperature or level of doneness. 

So I was really surprised when I checked my assumption, and found that I was utterly wrong.  The only device that I got right was the toaster--I knew that one--the other two surprised me.  

Today's SearchResearch Challenge is simple: 

1.  So... how DO each of these devices know when the toast / water / rice is ready?  

2. (extra credit) What other devices do you believe you understand, but when you checked, you learned that you actually didn't understand?  Does anything spring to mind?  Any surprises?  

This SRS Challenge isn't that hard, but it brings up a fairly deep point about when to question our assumptions... and HOW to realize that your assumptions might be wrong.  Any ideas?  

Share your thoughts in the comments.  

Keep searching!  

Answer: Three little Swiss mysteries?

 Living anywhere new inevitably leads to discoveries and recognition that the world is larger and more interesting than you might have thought, 

As mentioned, I'm here in Switzerland for four months. Even though I've been here before many times, the extended time period of living in one distinctly different place is proving to be eye-opening.  

But as a person who's relatively new to life in Zürich, I've found a few things that are charming and puzzling at the same time.  Can you help me figure out what's going on in each of these cases? 

1. Why are these eggs colored orange/yellow?  I bought them in the local grocery store where they were sitting out on the shelf, unrefrigerated.  The label on the container says Schweizer Eier (Swiss eggs).  I've seen many different colored eggs from friends who have chickens (blue, green, brown, some with spots), still, this is extraordinary color.  But what's the story here?  What kind of chicken would produce these eggs?  

I was thinking that these eggs were naturally this color. After all, eggs DO come in a bunch of different colors!  I knew that, in the US at least, chicken eggs can show up in a variety of shades:
 

Photo by Justin Pius, NRCS

So I was prepared to think that maybe Swiss chickens are simply more colorful!  

I spent a lot of time searching for colored eggs, learning that I needed to include chicken in my searches, as there are a wild number of different kinds of eggs from a zillion kinds of birds, some of which are extraordinary. (See the Science News article about different egg colors.)  

I also have to admit to spending at least 30 minutes poking around looking at many pages, and NOT finding anything. Oh, I found a lot, but the colored eggs made at Easter, or the tendency to have naturally dyed eggs were all over the results.  Here's a sample of what I was seeing: 

I was just scrolling around, trying to figure out what to do when I noticed one article about Look! Colored Eggs in Swiss Supermarket.  That's what I was looking for!  But in the article, it mentioned that "colored eggs... are so nice for picnics."  Huh?  You wouldn't take raw eggs on a picnic, right?  

So I changed my query to a question using that as inspiration: 

  

     [ Why are there colored eggs in Swiss supermarkets? ] 

And very quickly learned from a Reddit post that "In Switzerland, grocery stores sell painted hard-boiled eggs in order to differentiate between the fresh and hard-boiled ones."  (SRS tip: Google is much, much better at answering free-form questions, independent of the LLM work.) 

Now Reddit is fun, but a bit untrustworthy.  However, when I double-checked, I found multiple sources telling me this.  Grocery stores in Switzerland DO sell hard-boiled eggs in a variety of colors (these red/gold eggs are from one particular store--different store will have different colors)! 

I checked on the package that held the eggs and found (in fairly small font!) that they are "Aus Freilandhaltung * gekocht * gefärbt."  I should have looked more carefully. Google Translate tells me that this means "Free range * cooked * colored."  

Ah.  Got it.

Lesson learned: Read the package (even the small print) first.  Also learned that sometimes you'll learn the crucial tip by just scanning the results!  

2. While on a hike in the Alps (near Rigi Scheidegg, if that helps), I came across this flag--and I have no idea what kind of a flag this is. What does it represent?  (It might help to know that the Swiss are a little flag-crazy. There is traditional Swiss flag-tossing (a kind of bucolic, even serene sport... watch the video), and flags seem to abound.  Given the level of vexillological interest here, it must signify something, but what?  

This has turned out to be very hard--I'm not sure I have the right answer.  I tried all of the obvious image search tools (Google Image Search / Lens; Tineye; Bing Image Search; Yandex Image Search), but none of them gave me anything good.  

I tried various descriptions of the flag (four white hearts, four-leaf clover on a red background, etc etc.), but nothing really worked.  I searched for versions of these terms with words like "logo" or "flag" or "emblem" or "sigil" or "device"... but I didn't get very fair.  SRS Reader Paul L tried the specific flag search engine FlagID.org (that's a new one to me.. nice find Paul), but to no avail.  

I figured it was a cantonal flag or arms--but I checked those as well--no dice.  

Paul also "... finally loosened the search to only include red flag with white and scanned to see the much sharper edges of the Maltese cross on the Bardonnex Commune (Switzerland) flag."  While that's really close, it's not quite the same thing.  

I even made a fairly high-resolution image of the flag and tried to Google Lens (and Bing, and Tineye) this: 

But this didn't really work either.  I found some near hits: 

- a UK company called Schmecken that has a logo very much like this: 

- a Finnish group on Twitter/X called Pohjois-Pohjanmaa for social and health security association: 

- a car company, Autoclover: 

But nothing that was flag-like and Swiss.  Is it possible that this is a one-off custom flag?  

Then I was walking down the street in the town of St. Gallen (in northern Switzerland) and saw this stand-up box advertising the Swiss national lottery, Swisslos: 

Which is the closest Swiss 4-lobed clover-like bit of iconography I can find.  (Later I went back and looked deeper in my search-by-image results and found the Swisslos logo.  Always go deeper.)  

But it's not an exact match.  People who make flags are pretty picky about the details of their design.  (See the brilliant Roman Mars TED talk about flag design. 18 minutes that will change the way you look at flags.)  

So I'm not convinced we know the answer.  We're going to have to leave this as an open Challenge for the moment.  I'll keep looking, and you, my Regular Readers, should do the same.  IF you see it, post a comment here so we'll all know what it actually is.  (And I, for my part, if I go back to that part of the Alps, I'll find the owner and ask!)  

3.  I've seen some interesting vegetables before in farmer's markets before, but this one seems very Seussian to me.  What ARE these things? How would I eat one?  

This one was easy: looks like a weird cabbage, smells like a cabbage, so my query was: 

     [ cone shaped cabbage ] 

Which rapidly told me that this cabbage has a number of names: conehead, pointed, arrowhead, and sweetheart cabbage.  It's described as having "... leaves, with variations of pea green colorings, are thin, broad, deeply veined, tightly enveloped lengthwise and bluntly pointed. The flavor of Conehead cabbage is mild and remarkably sweet, void of that bold cruciferous flavor that is most reminiscent of cabbage."

Naturally, I bought one for research purposes and ate it for most of the week.  It is, indeed, sweetly flavored and is a lovely thing to have on your plate.  (I just sautéed/steamed mine with a little olive oil, garlic, and salt.  Yum!) 

SearchResearch Lessons 

1. Mind your assumptions!  In the colored egg Challenge I had assumed that the eggs in question were naturally colored like that.  I was prepared to learn that Swiss chickens are some interesting breed that lay technicolor eggs.  It took me a while to undo that assumption and figure out that they're dyed eggs.  

2. Even unreliable sources can be useful.  I found that Reddit post about hard-boiled Swiss eggs to crack the case (so to speak), but I know that Reddit can be unreliable. So when I checked, I was pleased to find MANY sources confirming that colored eggs are hardboiled, just like this detective.  

3. Read the fine print.  I skipped the fine print on the package partly because it was small and partly because it was in German, and while I can read lots of German, I didn't know what Aus Freilandhaltung meant, so I stopped reading.  FWIW, I know what gekocht and gefärbt mean... but I'd stopped reading too early.  

4. Some Challenges don't come easily. The identity of the flag has not yet been cracked!  It's an open case.  Sometimes, that's the way it goes.  

Keep searching! 

SearchResearch Challenge (10/25/23): Three little Swiss mysteries?

 I'm currently living and teaching in Zürich until the end of the year, 

.. here in the heart of Switzerland. I'm teaching for this semester at the University of Zürich--and you only get one guess as to what I'm teaching.  That's right, I'm here to teach the course on Human-Computer Interaction and AI over the 14-week semester.

(In other words, what should we be doing to design and build AI systems so that people can understand and use them.  Hence last week's Challenge about getting LLMs to be useful in search tasks.)

But as a person who's relatively new to Switzerland, I've found a few things that are charming and puzzling at the same time.  Can you help me figure out what's going on in each of these cases? 

1. Why are these eggs colored orange/yellow?  I bought them in the local grocery store where they were sitting out on the shelf, unrefrigerated.  The label on the container says Schweizer Eier (Swiss eggs).  I've seen many different colored eggs from friends who have chickens (blue, green, brown, some with spots), still, this is extraordinary color.  But what's the story here?  What kind of chicken would produce these eggs?  

2. While on a hike in the Alps (near Rigi Scheidegg, if that helps), I came across this flag--and I have no idea what kind of a flag this is. What does it represent?  (It might help to know that the Swiss are a little flag-crazy. There is traditional Swiss flag-tossing (a kind of bucolic, even serene sport... watch the video), and flags seem to abound.  Given the level of vexillological interest here, it must signify something, but what?  

3.  I've seen some interesting vegetables before in farmer's markets before, but this one seems very Seussian to me.  What ARE these things? How would I eat one?  

Of course, we want to know how you found the answers to each Challenge.  Share your search tricks with us!  

Keep searching!  

Answer: How might we best use LLMs for online medical research?

 Can we count on LLMs for medical advice? 

Illustration by DALL-E of a medical scientist doing research

And it's pretty clear that LLMs and associated generative AI tools (see the illustration above, which is a lovely generated image by DALL-E) can come in pretty handy when given decent prompts.  

Our SearchResearch Challenge for this week was an open-ended Challenge intended to probe the limits and capabilities of high-stakes research.   

Here was the Challenge for the week: 

1.  What good advice can we give a person who wants to use an LLM for high-stakes research into a medical question?  We know people are doing this, and all of our ranting about LLMs won't stop them--but can we find ways of using LLMs and specific LLM/AI tools to improve the search process?  What would you say to people who are doing this?  (Aside from "be very, very careful.")  

I found some pretty serious problems in asking LLMs for medical advice.  The overall impression I have is that LLMs just are NOT ready for use in medical searching.  There are other possible uses, but asking an LLM for medical information is basically rolling the dice--you might get good info, and you might get terrible info.  My recommendation: 

DON'T ASK ANY LLM FOR MEDICAL INFORMATION,
especially diagnostic and treatment information  

I spent several hours testing out different kinds of prompts, testing out several popular LLMs.  This isn't all of the testing I did, but I'm summarizing so it doesn't go on forever and ever.  

I'm grouping my comments by three main types of failure: 

Citations are wrong (and if they're made up, what else is made up?): 

Bard:   

Prompt to Bard: "What is the latest literature on Glucagon-Like Peptide-1 Receptor Agonists (GLP-1 RAs)? Give citations for articles you mention." 

Bard totally made up citations.  Complete BS.  Example: Here is a purported summary of some of the latest literature on GLP-1 RAs, with citations for the articles mentioned:  

Bard: GLP-1 RAs may reduce the risk of all-cause death in people with type 2 diabetes. A recent study published in the journal Diabetes Care found that GLP-1 RA users had a 14% lower risk of death from all causes than people who did not use GLP-1 RAs. It suggested a citation: Zhang H, et al. Association of Glucagon-Like Peptide-1 Receptor Agonist Use With All-Cause Mortality in Patients With Type 2 Diabetes. Diabetes Care. 2023;46(3):503-511.)

But this is utterly wrong:  No such articles exist.  Given that, do you still believe the assertion that GLP-1 RA users have a 14% lower risk of death?  Where did it pull that from?  I have no idea, and neither does Bard.  

GPT-4: Interestingly, when I gave the same prompt to GPT-4 it gave real citations (after disclaiming that it was only up-to-date as of Jan 2022).  Interestingly, it did not try to summarize the literature (as Bard did), but just gave a few citations.  It also gave good advice about how to further the research task: "I recommend using databases like PubMed or Google Scholar and searching for terms related to GLP-1 RAs."   That's pretty reasonable. 

Claude:  Same prompt, but very different citations.  That's fine, they're real citations, but the summaries of the articles were fairly off--they seemed to summarize different papers than the ones they linked to!  What's up with that? 

Output is inconsistent with itself: 

Bard:  Here I'm intentionally asking a question that a patient might ask, "why is skin cancer more deadly than liver cancer?" knowing full well that this is the opposite of the facts.  

Prompt to Bard: "Why is skin cancer more deadly than liver cancer?"  

Bard:  Skin cancer is not more deadly than liver cancer overall. In fact, the 5-year survival rate for all skin cancer is 99%, while the 5-year survival rate for liver cancer is 19.6%. However, there is one type of skin cancer, melanoma, which is more deadly than liver cancer. Melanoma has a 5-year survival rate of 92.3%, which is lower than the 5-year survival rate for liver cancer.

 

Bard got the first part right, but the assertion that "melanoma has a 5-year survival rate of 92.3%, which is lower than the 5-year survival rate for liver cancer" doesn't make sense (how is 92.3 lower than 19.6?).  What's up with that?  

GPT-4:  Once again, ChatGPT-4 gives a pretty reasonable answer. First, it questions my assumption (that skin cancer is more deadly), and then lists reasons why people might consider some types of cancers more deadly than others.  It's quite a reasonable argument, and ends with reasonable advice about differentiating between different kinds of cancers when discussing prognosis and mortality.  

Claude:  Gives an answer much like Bard, but doesn't mix up the stats.  It's actually mostly correct. 

Don't trust reasoning of LLMs: 

I asked each of the 3 LLMs this question: "If it takes 3 days to clear up an infection with a 10 unit dose of an antibiotic, how long will it take if I use 30 units per day?" 

This, of course, is a terrible idea. Don't fool around with treatment plans!  

But when I asked Bard, it gave a quite reasonable framing of the question...  

Bard:  

It is important to follow the prescribed dosage of antibiotics to ensure their effectiveness and avoid potential adverse effects. Taking a higher dose than recommended can lead to various complications and may not necessarily clear up the infection faster.

In this case, if a 10-unit dose of an antibiotic is prescribed to clear up an infection in 3 days, it is advisable to stick to that dosage and duration. Increasing the dose to 30 units per day could potentially lead to adverse effects without necessarily shortening the treatment duration.

 

GPT-4: Gives a rather similar reply: 

I must preface my response by emphasizing that you should never adjust medication doses without consulting a healthcare professional. Self-adjusting doses can lead to unwanted side effects, antibiotic resistance, or ineffective treatment.

That said, based solely on the information you've provided and treating it as a theoretical exercise, the relationship between dosage and time isn't necessarily linear. The effectiveness of antibiotics depends on factors like... 

But Claude totally bobbled this one with an answer that's long on math and utterly missing common sense reasoning about drug therapy.  

Claude: 

Okay, let's think through this step-by-step: 

It takes 3 days to clear up the infection with a 10 unit daily dose of the antibiotic

So over the 3 days, the total dose is 3 * 10 = 30 units

Now we are using 30 units per day instead of 10 units per day

Since the total dose needed to clear the infection is 30 units:

If we give 30 units per day, it should clear the infection in 1 day

So in summary:

Total dose needed is 30 units

At 10 units per day, it takes 3 days

At 30 units per day, the total dose of 30 units is given in 1 day

Therefore, if the daily dose is increased to 30 units per day, the infection should clear up in 1 day.

Which is a terrible idea. 

You see where this is headed: Each of the LLMs has a very different training set, sets of guidelines, and text production models.  

In general, GPT-4 gives much more extensive answers to questions, and avoids summarizing technical medical texts.  I wasn't able to trip it up (but I didn't really push on the boundaries).  

By contrast, Bard always gives much shorter answers, frequently hallucinating reports and studies. 

And Claude is somewhere in the middle with middle-length texts, but will give answers that defy common sense.  (Don't ever modify your drug plan without talking with your physician!)  

What can we do that is useful?  

My direct advice would be to NOT ask for a diagnosis or suggestions for treatment.  That's probably still dangerous. 

On the other hand... I did find some value in asking LLMs for descriptions of medical conditions, therapies, and medicines.  This makes sense because there is a lot of training data out there on such topics.  So asking questions like "what is the IT band?" or "what are some side effects of taking cough medicine?"  

SearchResearch Lessons 

I could go on and on, showing all kinds of subtle errors and mistakes that the models make.  

1. Bottom line: They're not ready for medical advice.  They might be in the future, but at the moment, they're all too able to make errors that could be serious. 

2. They all give very different kinds of answers.  GPT-4 tends to give great details about a condition, going into enormous detail.  If that's what you want, go there.  For shorter answers at a slightly easier-to-read level, visit Bard.  

3. Compare and contrast.  All of the LLMs (including ones I don't mention here) have rather different outputs.  It's worth looking at them side-by-side.  

4. Don't forget your ordinary search skills.  Do I really need to say this?  Fact-check every thing, and look for important highly reliable sources in your quest.  

Keep searching!  

SearchResearch Challenge (10/4/23): How might we best use LLMs for online medical research?

LLMs are all the rage these days... 

Illustration by DALL-E of a medical scientist doing research

Some studies show that office workers who use LLMs (e.g., Google's Bard or OpenAI's ChatGPT) are more productive... on the tasks that they measured.  (At least that's MIT's department of economics conclusion.)  

And it's pretty clear that LLMs and associated generative AI tools (see the illustration above) can come in pretty handy when given decent prompts.  

Our SearchResearch Challenge for this week is an open-ended Challenge.  For these Challenges, I usually know what the "right" answer is and how to find it.  

But this week, this really is a call for the SRS community to help identify ways of using LLMs to do a very specific kind of research--a kind that we care a lot about--medical research.  Here's the Challenge for the week: 

1.  What good advice can we give a person who wants to use an LLM for high-stakes research into a medical question?  We know people are doing this, and all of our ranting about LLMs won't stop them--but can we find ways of using LLMs and specific LLM/AI tools to improve the search process?  What would you say to people who are doing this?  (Aside from "be very, very careful.")  

Maybe the point of using an LLM in "medical research" mode is just to help us to find really great search terms to use in regular web-search.  I'm not sure. But if so, let's figure that out. Are there other ways of using LLMs to get better results faster?  Or will the search for truth in LLMs be a bust?  

I'm focusing on medical research because it is high stakes, it's not okay to just skate on past little errors and unfortunate phrasing.  

A big problem here is that most of us are not medical experts, so it's going to be difficult to validate what the LLM tells you.  

My suggestion: Try doing a few medical search tasks in a medical area that you know something about.  Nearly everyone has been forced to become a mini-expert in some medical area.  I'm a mini-expert in a couple of really unusual types of cancer (family members, not me!) when I had to learn enough to make good recommendations to them.  So I can explore what an LLM tells me about that with some confidence that I'll have a basic understanding of an esoteric area.   

I'll be leaving comments in the thread this week (more than usual) as I work on this question as well.  

Let's figure out what advice we can give to LLM users.  

When you write your comments in the thread, be sure to tell us which LLMs you've been using.  (There are a lot of them out there, all with very different behaviors.  Let's be sure to keep our results limited appropriately.)  

And be sure to include the search path you followed (which now will include LLM prompts).  Mention the dead ends you found and strategies that just don't seem to work.  Negative results are really important here!  

Keep searching!