Answer: How does it work? Checking your assumptions?

We all make assumptions... 

... it's a normal thing to do. But when we're doing online research, it's good to check yourself.  This week's Challenge is a little story about why...   

My assumption was that these common kitchen appliances had the same mechanism for knowing when the toast / water / rice is at the right temperature or level of doneness. 

Then I checked--and had a big surprise.  Really?    

Today's SearchResearch Challenge is simple: 

1.  So... how DO each of these devices know when the toast / water / rice is ready?  

2. (extra credit) What other devices do you believe you understand, but when you checked, you learned that you actually didn't understand?  Does anything spring to mind?  Any surprises?  

Searching for the answer isn't hard:  

     [ how does an electric tea kettle know when
         the water is boiling ] 

     [ how does a rice cooker know when
         the rice is done ] 

     [ how does a toaster know when
         the toast is done ] 

Let's talk about these one-by-one.  

Electric tea kettle: Do this query and you'll find a lot of explanatory text telling you that a ring-shaped bimetallic strip changes shape as it warms up, and since it's slightly buckled, when it deforms just enough, it snaps into a different shape, mechanically flipping the heater switch to off when the water hits the boiling point of 100C (212F). Very simple, very clever.  But how does it know?

The underside of a tea kettle with the bimetallic ring outlined with a red dashed line.  

That bimetallic ring is actually a fairly smart little invention--the tang coming out of the ring lies in the plane of the ring at room temperature, but as the ring starts to bend under the heating of the kettle, it suddenly snaps to a different shape with the tang pointing up.  This is called a bistable device, a gizmo that can be in one of two different physical configurations. Of course, the bimetallic strip's clever trick is that it will flip into the "hot" state when the temperature gets high enough, then flip back to the cool state after the heat is off.  That rapid flip to the other bistable position is what mechanically flips the power switch back off. 

The best example of this I could find is this YouTube short showing a tea kettle thermostat that flips back and forth rapidly when warmed up.   (Video link.) 

And... I thought that was that.  

But as I was writing this up, I ran across a video about electric tea kettles on Steve Mould's YouTube channel that showed me something I'd completely missed:  the steam tube! 

(Steve Mould video short on steam tubes in tea kettles.)  

It turns out (as Steve Mould points out), that the thermostat is actually at the bottom of the tube that is open to the top of the vessel.  As the kettle heats up, eventually the steam in the kettle is forced down the tube where it directly heats up the thermostat, which flips at around 95 C (203 F).  

Why is this important?  Because water boils at different temperatures at different altitudes, so a thermostat that is preset to 100C won't switch off at high altitudes.  In Denver, which is at 1609 meters (5280 feet), water boils at 95°C, or 203°F.  So if the thermostat only tripped at 100°C, it wouldn't work!  

At this point I got really curious about this somewhat sophisticated design, so I looked up a few electric tea kettle patents, where I learned that this is well-known by tea kettle designers.  To quote one such patent (US4357520, from 1979):   

"Alternatively a steam tube or passage, which communicates with a steam or vapour aperture in the upper wall of the container, may be run down the outside of the container. Such a tube or passage may be concealed within or behind a handle structure of the container..."  

I thought that was it. NOW I understood how they worked. 

But as I was writing this up, on a lark, I searched for: 

     [ electric tea kettle diagram ] 

hoping to find a cross-section of a kettle showing the steam tube.  

But NO!  I was again surprised to learn that there is a second thermostat, a "boil dry" thermostat that kicks in if/when the first thermostat fails to detect any steam.  This would happen if the kettle was switched on without any water.  No water, no steam, no steam pressure, no thermostat clicking off. 

I found this because I saw this diagram in the search results: 

Electric tea kettle circuit diagram. (P/C from Karisimby's blog)

I know enough about electronics to realize that this was NOT what I had in my head.  The "steam thermostat" was what I expected.  What I did not expect was a second "Boil dry" thermostat.  

A bit more searching revealed that such a second thermostat is set at a much higher temperature and is attached directly to the kettle wall.  So if the kettle is dry and heating up, the steam thermostat won't kick in, but the boil dry thermostat will open the circuit and stop the heating from running away (and possibly causing a meltdown).  

Amazing what you'll learn if you keep poking.  (If you want to learn more, I recommend this wonderful explanation of tea kettles.)  

Rice cooker: Again, not a difficult query, but the answer surprised me!  I'd assumed that there was a simple bimetallic thermostat in the rice cooker as well--in fact, I'd assumed that it worked in much the same way, probably with an identical part inside. 

But I quickly learned that most rice cookers (especially older ones) use a very clever magnet that turns off the heating element when the temperature goes above 100°C, or 223°F. 

Basically, there are two heaters in a rice cooker--one that keeps the rice warm and a second heater that boils the water, much like the tea kettle. 

However, I learned that rice cookers have a little magnet that holds the switch closed to close the main circuit to the heating element.  That circuit heats up the water until it boils, after it boils, the temperature of the magnent will start to rise beyond 100°C, or 223°F. When that happens, the magnet loses it's magnetism, allowing the circuit breaker to pop open, stopping the heating.  

The amazing trick here is that the magnet is designed so that it's Curie point (that is, the temperature at which it loses its magnetism) is at 100°C!

This is simpler to watch a video than to explain in text.  So I recommend this video: Nice explanation of how the magnetic cutoff switch works on a rice cooker. 

The bimetallic strip thermostat works because different metals expand at different rates, causing it to change shape. 

I didn't dream that using a magnet's Curie temperature would be used in the simplest of all rice cookers.  It's really not obvious that this is would be used to turn off the heat.  Incredibly smart use of some sophisticated physics.  

Oddly, I can't find any evidence that a rice cooker has a boil dry mechanism!  (I assume they do.  If you find out, let me know in the comments below.)  

To be sure, there are other mechanisms out there for telling when the rice is done--neural fuzzy logic rice cookers with complicated sensing systems.  But I love this elegant application of magnets!  

Toaster:  The big problem with toasters is that there are so many of them.  But for the most part, simple toasters are just timing mechanisms... turn the dial, get the level of brown you'd like.  It's really up to you learn which dial setting corresponds to the level of toasted that you like.  

But there are a LOT of different toasters out there--some with fancy thermocouples to determine temperature, some with just plain old spring-loaded timers that count down the seconds, and some with complex electromagnets to hold down the carriage (that carries the bread/toast on its journey).  

The old-fashioned toaster, though, is just a timer... nothing more than that.  

Other devices I don't understand? 

2. (extra credit) What other devices do you believe you understand, but when you checked, you learned that you actually didn't understand?  Does anything spring to mind?  Any surprises?  

There are LOTS of things I don't understand, but will look into once I get the chance.  I want to learn how the magic happens. Among them, 

1. iPhone--how it knows how far I've walked or run.  I know it counts steps, and I know it has a GPS system in it, so I assume it uses the GPS for distance traveled and counts steps via the on-board accelerometers, but does it combine those information streams in some way?  Don't know.  

2. Refrigerators--I don't understand the physics of them in detail, but I do know that they have a single cooling unit which then cools the frozen foods at one temperature, but the cool part of the fridge is handled in a different way.  Again, how does it know?  

3. Sauté  release--Sometimes when I cook (say) a piece of meat, it will come out of the pan easily--it "releases" with ease.  But sometimes, it sticks and makes a mess.  Is it purely a function of temperature, the amount of oil, or the material in use?  I need to understand this!  

The bigger point here is that all of these phenomena are understandable with a little observation and a dash of SRS.  

SearchResearch Lessons

1. Expect surprises and be ready for them.  Initially I thought that the rice cooker story was going to be straightforward.  So it was a surprise when the story got more complicated.  I discovered that when I noticed that something I read from the SERP didn't fit in with what I already knew.  (And this happened to me twice!)  

2. Use multiple document types to get a full perspective.  Sometimes a regular web search works just fine, but for complicated searches like the rice cooker, I used Image search, YouTube searches, and even Patent searches to get multiple perspectives.  I ALSO looked at multiple sources for each device, partly because some of the documents are simplified versions.  (For instance, not all rice cooker stories mention either the steam tube OR the dry boil cutoff thermostat.)  

3. Be aware when what you read doesn't match what you think you know.  Again, this happens multiple times for every SRS session--I learn things, and then learn something different.  When what you read doesn't line up with what you've already read, take that as a signal that there's more to learn!  

And keep searching!  

SearchResearch Challenge (11/8/23): How does it work? Checking your assumptions.

The most common error... 

... in online research is probably that of not checking your assumptions. We tend to see the world without questioning, and we often bake those assumptions into our research behaviors.  

 

Here we see three common, everyday appliances: an electric tea kettle, an ordinary rice cooker, and a plain old toaster.

I've seen these for years--there's nothing especially strange or exotic about them.  You've probably used them all as well.

As I was waiting for my toast to turn to a satisfactory shade of brown, I was looking at it standing next to my electric tea kettle, not far from my rice cooker and thinking about how they each know that the toast / water / rice is ready.  

My assumption was that they all had the same mechanism for knowing when the toast / water / rice is at the right temperature or level of doneness. 

So I was really surprised when I checked my assumption, and found that I was utterly wrong.  The only device that I got right was the toaster--I knew that one--the other two surprised me.  

Today's SearchResearch Challenge is simple: 

1.  So... how DO each of these devices know when the toast / water / rice is ready?  

2. (extra credit) What other devices do you believe you understand, but when you checked, you learned that you actually didn't understand?  Does anything spring to mind?  Any surprises?  

This SRS Challenge isn't that hard, but it brings up a fairly deep point about when to question our assumptions... and HOW to realize that your assumptions might be wrong.  Any ideas?  

Share your thoughts in the comments.  

Keep searching!  

Answer: Three little Swiss mysteries?

 Living anywhere new inevitably leads to discoveries and recognition that the world is larger and more interesting than you might have thought, 

As mentioned, I'm here in Switzerland for four months. Even though I've been here before many times, the extended time period of living in one distinctly different place is proving to be eye-opening.  

But as a person who's relatively new to life in Zürich, I've found a few things that are charming and puzzling at the same time.  Can you help me figure out what's going on in each of these cases? 

1. Why are these eggs colored orange/yellow?  I bought them in the local grocery store where they were sitting out on the shelf, unrefrigerated.  The label on the container says Schweizer Eier (Swiss eggs).  I've seen many different colored eggs from friends who have chickens (blue, green, brown, some with spots), still, this is extraordinary color.  But what's the story here?  What kind of chicken would produce these eggs?  

I was thinking that these eggs were naturally this color. After all, eggs DO come in a bunch of different colors!  I knew that, in the US at least, chicken eggs can show up in a variety of shades:
 

Photo by Justin Pius, NRCS

So I was prepared to think that maybe Swiss chickens are simply more colorful!  

I spent a lot of time searching for colored eggs, learning that I needed to include chicken in my searches, as there are a wild number of different kinds of eggs from a zillion kinds of birds, some of which are extraordinary. (See the Science News article about different egg colors.)  

I also have to admit to spending at least 30 minutes poking around looking at many pages, and NOT finding anything. Oh, I found a lot, but the colored eggs made at Easter, or the tendency to have naturally dyed eggs were all over the results.  Here's a sample of what I was seeing: 

I was just scrolling around, trying to figure out what to do when I noticed one article about Look! Colored Eggs in Swiss Supermarket.  That's what I was looking for!  But in the article, it mentioned that "colored eggs... are so nice for picnics."  Huh?  You wouldn't take raw eggs on a picnic, right?  

So I changed my query to a question using that as inspiration: 

  

     [ Why are there colored eggs in Swiss supermarkets? ] 

And very quickly learned from a Reddit post that "In Switzerland, grocery stores sell painted hard-boiled eggs in order to differentiate between the fresh and hard-boiled ones."  (SRS tip: Google is much, much better at answering free-form questions, independent of the LLM work.) 

Now Reddit is fun, but a bit untrustworthy.  However, when I double-checked, I found multiple sources telling me this.  Grocery stores in Switzerland DO sell hard-boiled eggs in a variety of colors (these red/gold eggs are from one particular store--different store will have different colors)! 

I checked on the package that held the eggs and found (in fairly small font!) that they are "Aus Freilandhaltung * gekocht * gefärbt."  I should have looked more carefully. Google Translate tells me that this means "Free range * cooked * colored."  

Ah.  Got it.

Lesson learned: Read the package (even the small print) first.  Also learned that sometimes you'll learn the crucial tip by just scanning the results!  

2. While on a hike in the Alps (near Rigi Scheidegg, if that helps), I came across this flag--and I have no idea what kind of a flag this is. What does it represent?  (It might help to know that the Swiss are a little flag-crazy. There is traditional Swiss flag-tossing (a kind of bucolic, even serene sport... watch the video), and flags seem to abound.  Given the level of vexillological interest here, it must signify something, but what?  

This has turned out to be very hard--I'm not sure I have the right answer.  I tried all of the obvious image search tools (Google Image Search / Lens; Tineye; Bing Image Search; Yandex Image Search), but none of them gave me anything good.  

I tried various descriptions of the flag (four white hearts, four-leaf clover on a red background, etc etc.), but nothing really worked.  I searched for versions of these terms with words like "logo" or "flag" or "emblem" or "sigil" or "device"... but I didn't get very fair.  SRS Reader Paul L tried the specific flag search engine FlagID.org (that's a new one to me.. nice find Paul), but to no avail.  

I figured it was a cantonal flag or arms--but I checked those as well--no dice.  

Paul also "... finally loosened the search to only include red flag with white and scanned to see the much sharper edges of the Maltese cross on the Bardonnex Commune (Switzerland) flag."  While that's really close, it's not quite the same thing.  

I even made a fairly high-resolution image of the flag and tried to Google Lens (and Bing, and Tineye) this: 

But this didn't really work either.  I found some near hits: 

- a UK company called Schmecken that has a logo very much like this: 

- a Finnish group on Twitter/X called Pohjois-Pohjanmaa for social and health security association: 

- a car company, Autoclover: 

But nothing that was flag-like and Swiss.  Is it possible that this is a one-off custom flag?  

Then I was walking down the street in the town of St. Gallen (in northern Switzerland) and saw this stand-up box advertising the Swiss national lottery, Swisslos: 

Which is the closest Swiss 4-lobed clover-like bit of iconography I can find.  (Later I went back and looked deeper in my search-by-image results and found the Swisslos logo.  Always go deeper.)  

But it's not an exact match.  People who make flags are pretty picky about the details of their design.  (See the brilliant Roman Mars TED talk about flag design. 18 minutes that will change the way you look at flags.)  

So I'm not convinced we know the answer.  We're going to have to leave this as an open Challenge for the moment.  I'll keep looking, and you, my Regular Readers, should do the same.  IF you see it, post a comment here so we'll all know what it actually is.  (And I, for my part, if I go back to that part of the Alps, I'll find the owner and ask!)  

3.  I've seen some interesting vegetables before in farmer's markets before, but this one seems very Seussian to me.  What ARE these things? How would I eat one?  

This one was easy: looks like a weird cabbage, smells like a cabbage, so my query was: 

     [ cone shaped cabbage ] 

Which rapidly told me that this cabbage has a number of names: conehead, pointed, arrowhead, and sweetheart cabbage.  It's described as having "... leaves, with variations of pea green colorings, are thin, broad, deeply veined, tightly enveloped lengthwise and bluntly pointed. The flavor of Conehead cabbage is mild and remarkably sweet, void of that bold cruciferous flavor that is most reminiscent of cabbage."

Naturally, I bought one for research purposes and ate it for most of the week.  It is, indeed, sweetly flavored and is a lovely thing to have on your plate.  (I just sautéed/steamed mine with a little olive oil, garlic, and salt.  Yum!) 

SearchResearch Lessons 

1. Mind your assumptions!  In the colored egg Challenge I had assumed that the eggs in question were naturally colored like that.  I was prepared to learn that Swiss chickens are some interesting breed that lay technicolor eggs.  It took me a while to undo that assumption and figure out that they're dyed eggs.  

2. Even unreliable sources can be useful.  I found that Reddit post about hard-boiled Swiss eggs to crack the case (so to speak), but I know that Reddit can be unreliable. So when I checked, I was pleased to find MANY sources confirming that colored eggs are hardboiled, just like this detective.  

3. Read the fine print.  I skipped the fine print on the package partly because it was small and partly because it was in German, and while I can read lots of German, I didn't know what Aus Freilandhaltung meant, so I stopped reading.  FWIW, I know what gekocht and gefärbt mean... but I'd stopped reading too early.  

4. Some Challenges don't come easily. The identity of the flag has not yet been cracked!  It's an open case.  Sometimes, that's the way it goes.  

Keep searching! 

SearchResearch Challenge (10/25/23): Three little Swiss mysteries?

 I'm currently living and teaching in Zürich until the end of the year, 

.. here in the heart of Switzerland. I'm teaching for this semester at the University of Zürich--and you only get one guess as to what I'm teaching.  That's right, I'm here to teach the course on Human-Computer Interaction and AI over the 14-week semester.

(In other words, what should we be doing to design and build AI systems so that people can understand and use them.  Hence last week's Challenge about getting LLMs to be useful in search tasks.)

But as a person who's relatively new to Switzerland, I've found a few things that are charming and puzzling at the same time.  Can you help me figure out what's going on in each of these cases? 

1. Why are these eggs colored orange/yellow?  I bought them in the local grocery store where they were sitting out on the shelf, unrefrigerated.  The label on the container says Schweizer Eier (Swiss eggs).  I've seen many different colored eggs from friends who have chickens (blue, green, brown, some with spots), still, this is extraordinary color.  But what's the story here?  What kind of chicken would produce these eggs?  

2. While on a hike in the Alps (near Rigi Scheidegg, if that helps), I came across this flag--and I have no idea what kind of a flag this is. What does it represent?  (It might help to know that the Swiss are a little flag-crazy. There is traditional Swiss flag-tossing (a kind of bucolic, even serene sport... watch the video), and flags seem to abound.  Given the level of vexillological interest here, it must signify something, but what?  

3.  I've seen some interesting vegetables before in farmer's markets before, but this one seems very Seussian to me.  What ARE these things? How would I eat one?  

Of course, we want to know how you found the answers to each Challenge.  Share your search tricks with us!  

Keep searching!  

Answer: How might we best use LLMs for online medical research?

 Can we count on LLMs for medical advice? 

Illustration by DALL-E of a medical scientist doing research

And it's pretty clear that LLMs and associated generative AI tools (see the illustration above, which is a lovely generated image by DALL-E) can come in pretty handy when given decent prompts.  

Our SearchResearch Challenge for this week was an open-ended Challenge intended to probe the limits and capabilities of high-stakes research.   

Here was the Challenge for the week: 

1.  What good advice can we give a person who wants to use an LLM for high-stakes research into a medical question?  We know people are doing this, and all of our ranting about LLMs won't stop them--but can we find ways of using LLMs and specific LLM/AI tools to improve the search process?  What would you say to people who are doing this?  (Aside from "be very, very careful.")  

I found some pretty serious problems in asking LLMs for medical advice.  The overall impression I have is that LLMs just are NOT ready for use in medical searching.  There are other possible uses, but asking an LLM for medical information is basically rolling the dice--you might get good info, and you might get terrible info.  My recommendation: 

DON'T ASK ANY LLM FOR MEDICAL INFORMATION,
especially diagnostic and treatment information  

I spent several hours testing out different kinds of prompts, testing out several popular LLMs.  This isn't all of the testing I did, but I'm summarizing so it doesn't go on forever and ever.  

I'm grouping my comments by three main types of failure: 

Citations are wrong (and if they're made up, what else is made up?): 

Bard:   

Prompt to Bard: "What is the latest literature on Glucagon-Like Peptide-1 Receptor Agonists (GLP-1 RAs)? Give citations for articles you mention." 

Bard totally made up citations.  Complete BS.  Example: Here is a purported summary of some of the latest literature on GLP-1 RAs, with citations for the articles mentioned:  

Bard: GLP-1 RAs may reduce the risk of all-cause death in people with type 2 diabetes. A recent study published in the journal Diabetes Care found that GLP-1 RA users had a 14% lower risk of death from all causes than people who did not use GLP-1 RAs. It suggested a citation: Zhang H, et al. Association of Glucagon-Like Peptide-1 Receptor Agonist Use With All-Cause Mortality in Patients With Type 2 Diabetes. Diabetes Care. 2023;46(3):503-511.)

But this is utterly wrong:  No such articles exist.  Given that, do you still believe the assertion that GLP-1 RA users have a 14% lower risk of death?  Where did it pull that from?  I have no idea, and neither does Bard.  

GPT-4: Interestingly, when I gave the same prompt to GPT-4 it gave real citations (after disclaiming that it was only up-to-date as of Jan 2022).  Interestingly, it did not try to summarize the literature (as Bard did), but just gave a few citations.  It also gave good advice about how to further the research task: "I recommend using databases like PubMed or Google Scholar and searching for terms related to GLP-1 RAs."   That's pretty reasonable. 

Claude:  Same prompt, but very different citations.  That's fine, they're real citations, but the summaries of the articles were fairly off--they seemed to summarize different papers than the ones they linked to!  What's up with that? 

Output is inconsistent with itself: 

Bard:  Here I'm intentionally asking a question that a patient might ask, "why is skin cancer more deadly than liver cancer?" knowing full well that this is the opposite of the facts.  

Prompt to Bard: "Why is skin cancer more deadly than liver cancer?"  

Bard:  Skin cancer is not more deadly than liver cancer overall. In fact, the 5-year survival rate for all skin cancer is 99%, while the 5-year survival rate for liver cancer is 19.6%. However, there is one type of skin cancer, melanoma, which is more deadly than liver cancer. Melanoma has a 5-year survival rate of 92.3%, which is lower than the 5-year survival rate for liver cancer.

 

Bard got the first part right, but the assertion that "melanoma has a 5-year survival rate of 92.3%, which is lower than the 5-year survival rate for liver cancer" doesn't make sense (how is 92.3 lower than 19.6?).  What's up with that?  

GPT-4:  Once again, ChatGPT-4 gives a pretty reasonable answer. First, it questions my assumption (that skin cancer is more deadly), and then lists reasons why people might consider some types of cancers more deadly than others.  It's quite a reasonable argument, and ends with reasonable advice about differentiating between different kinds of cancers when discussing prognosis and mortality.  

Claude:  Gives an answer much like Bard, but doesn't mix up the stats.  It's actually mostly correct. 

Don't trust reasoning of LLMs: 

I asked each of the 3 LLMs this question: "If it takes 3 days to clear up an infection with a 10 unit dose of an antibiotic, how long will it take if I use 30 units per day?" 

This, of course, is a terrible idea. Don't fool around with treatment plans!  

But when I asked Bard, it gave a quite reasonable framing of the question...  

Bard:  

It is important to follow the prescribed dosage of antibiotics to ensure their effectiveness and avoid potential adverse effects. Taking a higher dose than recommended can lead to various complications and may not necessarily clear up the infection faster.

In this case, if a 10-unit dose of an antibiotic is prescribed to clear up an infection in 3 days, it is advisable to stick to that dosage and duration. Increasing the dose to 30 units per day could potentially lead to adverse effects without necessarily shortening the treatment duration.

 

GPT-4: Gives a rather similar reply: 

I must preface my response by emphasizing that you should never adjust medication doses without consulting a healthcare professional. Self-adjusting doses can lead to unwanted side effects, antibiotic resistance, or ineffective treatment.

That said, based solely on the information you've provided and treating it as a theoretical exercise, the relationship between dosage and time isn't necessarily linear. The effectiveness of antibiotics depends on factors like... 

But Claude totally bobbled this one with an answer that's long on math and utterly missing common sense reasoning about drug therapy.  

Claude: 

Okay, let's think through this step-by-step: 

It takes 3 days to clear up the infection with a 10 unit daily dose of the antibiotic

So over the 3 days, the total dose is 3 * 10 = 30 units

Now we are using 30 units per day instead of 10 units per day

Since the total dose needed to clear the infection is 30 units:

If we give 30 units per day, it should clear the infection in 1 day

So in summary:

Total dose needed is 30 units

At 10 units per day, it takes 3 days

At 30 units per day, the total dose of 30 units is given in 1 day

Therefore, if the daily dose is increased to 30 units per day, the infection should clear up in 1 day.

Which is a terrible idea. 

You see where this is headed: Each of the LLMs has a very different training set, sets of guidelines, and text production models.  

In general, GPT-4 gives much more extensive answers to questions, and avoids summarizing technical medical texts.  I wasn't able to trip it up (but I didn't really push on the boundaries).  

By contrast, Bard always gives much shorter answers, frequently hallucinating reports and studies. 

And Claude is somewhere in the middle with middle-length texts, but will give answers that defy common sense.  (Don't ever modify your drug plan without talking with your physician!)  

What can we do that is useful?  

My direct advice would be to NOT ask for a diagnosis or suggestions for treatment.  That's probably still dangerous. 

On the other hand... I did find some value in asking LLMs for descriptions of medical conditions, therapies, and medicines.  This makes sense because there is a lot of training data out there on such topics.  So asking questions like "what is the IT band?" or "what are some side effects of taking cough medicine?"  

SearchResearch Lessons 

I could go on and on, showing all kinds of subtle errors and mistakes that the models make.  

1. Bottom line: They're not ready for medical advice.  They might be in the future, but at the moment, they're all too able to make errors that could be serious. 

2. They all give very different kinds of answers.  GPT-4 tends to give great details about a condition, going into enormous detail.  If that's what you want, go there.  For shorter answers at a slightly easier-to-read level, visit Bard.  

3. Compare and contrast.  All of the LLMs (including ones I don't mention here) have rather different outputs.  It's worth looking at them side-by-side.  

4. Don't forget your ordinary search skills.  Do I really need to say this?  Fact-check every thing, and look for important highly reliable sources in your quest.  

Keep searching!  

SearchResearch Challenge (10/4/23): How might we best use LLMs for online medical research?

LLMs are all the rage these days... 

Illustration by DALL-E of a medical scientist doing research

Some studies show that office workers who use LLMs (e.g., Google's Bard or OpenAI's ChatGPT) are more productive... on the tasks that they measured.  (At least that's MIT's department of economics conclusion.)  

And it's pretty clear that LLMs and associated generative AI tools (see the illustration above) can come in pretty handy when given decent prompts.  

Our SearchResearch Challenge for this week is an open-ended Challenge.  For these Challenges, I usually know what the "right" answer is and how to find it.  

But this week, this really is a call for the SRS community to help identify ways of using LLMs to do a very specific kind of research--a kind that we care a lot about--medical research.  Here's the Challenge for the week: 

1.  What good advice can we give a person who wants to use an LLM for high-stakes research into a medical question?  We know people are doing this, and all of our ranting about LLMs won't stop them--but can we find ways of using LLMs and specific LLM/AI tools to improve the search process?  What would you say to people who are doing this?  (Aside from "be very, very careful.")  

Maybe the point of using an LLM in "medical research" mode is just to help us to find really great search terms to use in regular web-search.  I'm not sure. But if so, let's figure that out. Are there other ways of using LLMs to get better results faster?  Or will the search for truth in LLMs be a bust?  

I'm focusing on medical research because it is high stakes, it's not okay to just skate on past little errors and unfortunate phrasing.  

A big problem here is that most of us are not medical experts, so it's going to be difficult to validate what the LLM tells you.  

My suggestion: Try doing a few medical search tasks in a medical area that you know something about.  Nearly everyone has been forced to become a mini-expert in some medical area.  I'm a mini-expert in a couple of really unusual types of cancer (family members, not me!) when I had to learn enough to make good recommendations to them.  So I can explore what an LLM tells me about that with some confidence that I'll have a basic understanding of an esoteric area.   

I'll be leaving comments in the thread this week (more than usual) as I work on this question as well.  

Let's figure out what advice we can give to LLM users.  

When you write your comments in the thread, be sure to tell us which LLMs you've been using.  (There are a lot of them out there, all with very different behaviors.  Let's be sure to keep our results limited appropriately.)  

And be sure to include the search path you followed (which now will include LLM prompts).  Mention the dead ends you found and strategies that just don't seem to work.  Negative results are really important here!  

Keep searching!  

Answer: What is dietary fiber?

 It seems obvious...  

.. but as I said, I suddenly realized that I didn't really understand what "fiber in my diet" really meant.  

This happens all the time. There's a kind of skill of recognizing that something isn't lining up--of knowing that you don't quite know what something really is. 

As you might have noticed, a lot of these posts start with "I didn't know..."  

In this particular case, I thought I knew was "dietary fiber" was.  Isn't fiber just the indigestible part of your food--you know, the little threads, husks, hulls, and tiny fragments that just pass straight through without much digestion seeming to take place?  

But as I noticed when I let my cereal sit for too long in the milk, things that say they have fiber in them (like my whole grain cereal) often don't look like there's anything fibrous within.  There's nothing like the kernels of corn or bits of apple skin that (as far as I can tell) are untouched by my digestive juices.  How could there be any "fiber" there?    

So... what is fiber really?  It's obviously not just undigested bits of food.  

Can you help me understand what's going on here?  Today's Challenges are: 

1. So what, really, is dietary fiber?  Is it something more than indigestible bits like wheat bran, corn kernels, and rice husks?  

I wanted to start with the basics.  First query: 

   

     [ define fiber ] 

This starts to untangle part of my confusion. I was thinking of dietary fiber as a kind of textile-like fiber.  That is, my mental model of "fiber in food" was based on what I thought of as a fiber that I might see in clothing or the husk of a seed.  This is very much along the lines of that word you see at the bottom of the definition, "roughage."  

But reading carefully, I learned that "fiber" (or "fibre") is also a "dietary material containing substances such as cellulose, lignin, and pectin that are resistant to the action of digestive enzymes." 

And THAT is a surprise. It means that the fiber in my diet can be pretty non-fibrous.  

Just a few weeks ago I had way too many plums on the tree at my house, so I spent a few happy hours making jelly and jam.  One of the steps in the process is to put pectin into the mix.  But when I do that, I can see that the pectin dissolves into a clear liquid... there aren't any obvious fibers that look like roughage.  

I was curious about lignin and cellulose, so I looked up those terms as well.  Are they fibrous in the way I'd expected? 

Answer: NO!  If you look at images of lignin, cellulose, or pectin, it's pretty clear that all of those come in powdered form--and if you look for [ liquid cellulose ] or [ liquid pectin ] it's pretty obvious that those don't have any roughage in them. 

So dietary fiber must mean something other than the roughage you get from (say) celery.  

My next query was for: 

     [ fiber in food ] 

and I quickly learned that fiber is, indeed, not just the rough indigestible bits, but is actually all the long-chain molecules that make up those indigestible bits!  

As the Wikipedia entry on dietary fiber tells us "Dietary fiber consists of non-starch polysaccharides and other plant components such as cellulose, resistant starch, resistant dextrins, inulin, lignins, chitins (in fungi), pectins, beta-glucans, and oligosaccharides."  

Okay.  I now know that "dietary fiber" is really all of those complex (long-chain) molecules in the food I eat.  So, "high fiber foods" have a lot of those chemicals AND roughage. 

But wait... there's more!  

I also learned by reading some of the links from the SERP that there are two kinds of dietary fiber!  There's "soluble" and "insoluble," that is, fiber that dissolves, and fiber that doesn't.  

     [ soluble insoluble fiber ] 

The first result is to the Medline page about dietary fiber, which includes this: 

"There are 2 different types of fiber -- soluble and insoluble. Both are important for health, digestion, and preventing diseases.

Soluble fiber attracts water and turns to gel during digestion. This slows digestion. Soluble fiber is found in oat bran, barley, nuts, seeds, beans, lentils, peas, and some fruits and vegetables. It is also found in psyllium, a common fiber supplement. Some types of soluble fiber may help lower risk of heart disease.

Insoluble fiber is found in foods such as wheat bran, vegetables, and whole grains. It adds bulk to the stool and appears to help food pass more quickly through the stomach and intestines." 

Ah ha!  So there really are two kinds of fibers in our food.  The soluble kind and the insoluble kind.  

Insoluble fiber:  This type of fiber does not get dissolved in the body and is also known as “roughage”, helping to clear out the intestinal tract. Since it's not broken down in the body, insoluble fiber has effectively zero calories. This is why a high fiber diet is recommended for weight loss as insoluble fiber will fill you up and pass through your body unchanged. 

Soluble fiber: By contrast, this kind of fiber is a bit more complex than the other. Unlike insoluble fiber, soluble fiber dissolves making a gel like substance, which is incredibly handy for slip/sliding your gut contents along.  

2. What is the current recommendation for dietary fiber in my diet?  Is 5 grams of fiber in a serving a lot?  Or is it a little?  

A quick search for: 

     [ dietary fiber recommendations ] 

generated a lot of hits.  Interestingly, when you compare the top 5 reputable sources, you see this: 

Mayo Clinic – Women:  21 to 25 grams / day; men: 30 to 38 grams / day

UCSF – everyone: 25 to 30 grams / day from food, not supplements 

NIH -everyone:  25-29 grams / day (more than 30 grams would be better) 

Harvard – people < 50 years, 25 grams for women and 38 grams for men. 

                For people > 49, Women and men should have 21 and 30 daily grams

Oddly, none of the sources break down the amount of fiber by soluble vs. insoluble.  But it's clear that "fiber from food" solves a lot of the soluble/insoluble question by providing both in roughly equal quantities.  

Basically, if you can dissolve the food in water, it's got soluble fiber.  If it doesn't dissolve, it's insoluble fiber.  You need both.  

Bottom line: Women need slightly less fiber than men, but everyone needs roughly 30 grams / day.  

If you look up various foods (after doing a query like [ data set food fiber amount ]), you'll probably end up on the USDA's list of foods with the amount of fiber listed for each.  In that list you'll see a lot of high fiber foods that you probably won't eat in large amounts (who eats 100 grams of cinnamon?), but you'll be able to look up your favorite foods and estimate the amount of fiber you're getting.  

US folks: Note that the table is given in "amount of fiber in 100 grams of food."   So you'll have a vivid image--100 grams is around 3.5 ounces, or to make it visually memorable, small cans of cat food are 3.5 ounces (100 grams). 

And you'll see that my favorite cereal has 10 grams of fiber in 100 grams of cereal.  But take note--most people I know don't eat 100 grams of cereal in a serving at breakfast.  My bowl of morning cereal usually has 36 grams of cereal, which means I'm getting 4 grams of fiber in each bowl. 

Apple with 2.8 grams of fiber

That's okay, I guess, but an average apple has 2.8 grams of fiber, and a regular serving of plain old oatmeal has 4 grams of fiber and 6 grams of protein.  (And it's about half the cost per serving.)  

SearchResearch Lessons 

1. Search out the things you don't understand, even sometimes fairly obvious things.  This was just one of those little things that struck me while reading: I couldn't actually define dietary fiber.  I make it a practice to question what I'm reading at a very basic level.  This is deeply important when you're reading something that's complicated or has an intricate back story.  Do you really understand all of the parts and pieces that are involved?  

Being a skilled searcher is, in large part, having enough background knowledge to know when something you just read (or heard) doesn't fit in with the other things you've read.  In this case, it was a small observation about my breakfast cereal.  For you, it might be something else.  Follow up those small questions--they could be deeply important.  

2. Look for multiple sources.  As I did with searching for multiple sources of fiber recommendations, it's easy to find them.  Be aware of differences between measurements! 

Keep Searching!