Answer: What's the most significant thing going on here? (3/3)

I didn't mean to write three posts...  

But this is a big problem!     

I have to sympathize with Ramón and remmij’s comments about having a sense of overwhelmment in the SRS space.  We’re living in a frenzied time–when new systems and products are bubbling up every day and in every way.  It’s easy to give up… but don’t.   

A Gemini-generated image of Dan being overwhelmed by the
number of choices when doing Deep Research.

Here, at the SearchResearch Rancho, we’ll try to shed some light on what works, and what you can safely ignore. Isn’t that why you come here?  For a bit of clarity and guidance in these complicated times?  Let’s see what we can figure out.  

As I mentioned earlier this week, a new trend in SRS is the launch of several new “deep research” tools.  In today’s post I’m going to compare / contrast three of them.  (For simplicity, and to make a small pun, I’m going to call these tools “DR” tools.   

(1) Google’s DR tool is their “Deep Research” mode for Gemini.  (I can’t give a direct link to it because Gemini doesn’t use URL parameters.  You have to manually select the “1.5 Pro with Deep Research” option to get it to show up.  Sorry about that.)  The idea is that "1.5 Pro with Deep Research" will write you a short summary on some research topic.  

(2) Undermind.ai  a "personal assistant" to help you with collecting, analyzing, summarizing work in a particular topic area.  

(3) OpenScholar (from Allen.ai) a DR tool that scans the scholarly literature (and sometimes it refuses to do so!) and writes a summary much in the style of Google 1.5 Pro w/ DR.

(For a list of other deep research systems, take a look at the SlashDot list of DR systems.) 

The idea behind all of these DR systems is to use AI techniques to analyze large volumes of complex data, looking for in-depth insights and discovery beyond traditional research methods. The hope is that these tools will find complex relationships and patterns within the data that might be difficult to identify manually.  Each of them writes a little report on what they found giving citations to the literature they used. 

To compare each of these systems I asked two questions.  

A. What has been the effects of the creation of Lake Nasser on the ecosystems around it?

B. What has been the effect of the creation of Lake Nasser on the incidence of schistosomiasis in Egypt? 

Question A is to answer our primary question (“what are the most important changes..”) 

Question B dives more deeply into a particular question about schistosomiasis, a parasitic disease that’s caused by blood flukes (trematode worms) of the genus Schistosoma transmitted by freshwater snails that often live in agricultural canals.. It’s a very serious disease. Can these DR systems help us understand this important change?   

Here’s what I found when using these DR systems… 

Question A:  (effect of Lake Nasser on the ecosystems)  

Google:  The DR tool (using Google’s “Gemini Advanced 1.5 Pro with Deep Research” on January 2, 2025) created a 2400 word report (link to report) that covers a bit of the history of Lake Nasser, and then lists positive and negative effects. Ecological projects are mentioned and a section entitled “Scholarly Research on the Ecological Impact of Lake Nasser” includes a very obscure data set on water quality.  (The data is important–but why copy/paste part of the raw data in the report?)  It mentions the “South Valley Project” (another name for the Toshka Lakes). 

Oddly, the report cites TWO Kids.Brittanica.com reports, a few studies by international organizations (e.g., WorldFish international), a few Wikipedia articles, and a couple of scientific literature studies.  

Overall grade: It’s not bad, but there are a few sections that are just odd–not something a human would ever write.  

=================================

Undermind:  When you use Undermind, it starts with a little back-and-forth trying to get you to add more detail to the research question. That’s fine, but it also ends up narrowing the scope of the research. In my case, the final question posed to Undermind was “The ecological impacts of the creation of Lake Nasser, focusing on changes and adaptations in both aquatic ecosystems within the lake and terrestrial ecosystems surrounding the area.”  

Here’s the top of the Undermind report: 

Undermind also provides a helpful summary of the Categories of papers it found: 

And a very helpful timeline of research work: 

As well as a very interesting set of clusters of research groups and contributions by each group: 

The references (at the end of the report show the expected citation info, but also a measure of the topic match, the number of citations / year (indicates how often it gets cited), and a summary of the relevance of the paper to the topic: 

Overall grade: Undermind gives you much more information that Google’s DR report, including analysis that Google won’t give you.  (e.g., the clustering) 

=================================

OpenScholar: By contrast, Ai2 OpenScholar took the same research question and wrote a fairly short, cursory report. Here’s the top of that report: 

There are only 3 references given, and one of them (Goher et al. 2021) is used for 6 of the 10 citations. What’s more, the Goher paper was published in the journal Water, which is a publication of the MDPI (Multidisciplinary Digital Publishing Institute), which has a not-great reputation. (You can read the Wikipedia page to learn more.)  In any case, it’s not a paper I would choose to center my critical review of Lake Nasser ecosystems, even though the paper data seems reasonable enough.  

Overall grade: It’s not bad, but there are a few sections that are just odd–not something a human would ever write.  And the overall quality of the cited works was a little suspicious.  (I found other examples of papers I’m not sure I would cite in OpenScholar analysis reports.  OpenScholar–what are you doing??)  

=================================

Question B: (effect of Lake Nasser on Schistosomiasis)

Google:  In reply to the Schistosomiasis research question, Google’s DR tool created another report (link to report) that is structurally similar to the other report… it too covers a bit of the history of Lake Nasser and environmental issues.  It has a section on public health interventions, and then the effects of Lake Nasser on the incidence of Schistosomiasis, pointing out that a massive anti-snail / anti-Schistosomiasis campaign has caused an overall REDUCTION in the incidence of the disease.  

Overall grade: Again, it’s not bad.. But there are some contradictory statements (e.g., while Schistosomiasis overall has gone down, another subspecies of Schistosomiasis (specifically, Schistosomiasis mansoni) has actually increased.  The obvious question a human would ask is “is this overall good, or bad for the country.”  That’s never really addressed.  

=================================

Undermind:  By contrast, Undermind doesn’t really write much of a report–it really gives a bunch of research result in the literature.  What little it says about the incidence of Schistosomiasis slightly contradicts Google.  In particular, it writes that: 

  “The creation of Lake Nasser after the construction of the Aswan High Dam significantly increased schistosomiasis transmission in Upper and Middle Egypt by altering ecological conditions that favored the proliferation of snail vectors …, though public health interventions such as mass drug administration and mollusciciding effectively reduced disease prevalence in most areas despite persistent hotspots.”  

Overall grade:  Undermind gives a completely different set of relevant papers than Google Gemini!  (There is zero overlap.)  But it does give a deeper analysis about why Lake Nasser changed the way that Egyptians use canals for agriculture, leading to an increase in the disease!  

=================================

OpenScholar: I’m not sure what happened here, but I was completely unable to get it to give me any kind of answer to my schistosomiasis questions!  I tried multiple variations on the research statement question, but all I ever got was a failure notification.  

“Referenced task failed. Error: We were unable to retrieve any relevant papers for your query. Please try a different query. OpenScholar is not designed to answer non-scientific questions or questions that require sources outside the scientific literature.” 

Overall grade: Not a great performance.  I spent probably an hour trying different variations on the theme (including logging in on a different account), but for whatever reason, it just refused to answer.  

SearchResearch Lessons

Let’s start with the post from 2 weeks ago… 

1. When getting an overview, consider using maps… and in particular, consider using time lapse.  They’re relatively easy to get, and give you a very different perspective on regional questions.  

2. Asking LLMs these questions is a good idea.. BUT ONLY if you look at multiple different AI systems.  As we saw last week, each of these can give you a very different idea about what the issues are.  But aggregating the results can give you a decent overview.  (Do not, however, take the frequency of topics mentioned as a proxy for importance!  That’s kind of random–multiple mentions do not equal overall importance!)  

3. The Deep Research (DR) tools are a new kind of thing for doing serious research. While interesting, they’re not a substitute for real human research.  (At least not yet.)  Like LLMs, each has their own perspectives (which can be useful when taken together), but don’t seem to have great quality control about what papers are high quality.  

As ever, check your work--now more than ever.  The DR systems are really interesting tools, but they’re not quite a replacement for your good research skills and discernment.  

In the future we’ll talk about other DR tools, including: ResearchRabbit, Elicit, Iris, Affor.ai and NotebookLM.  We’ll talk about those soon.  (But I didn’t want to write a book about them.. At least not yet!)   

Keep searching!  

Answer: What's the most significant thing going on here? (2/3)

This is even bigger than I thought...  

How DO we approach a research problem like this?  

P/C Google Earth

The bigger SRS Challenge for us  is... how much has Lake Nasser changed the lands around it?  

Here’s part 2 of our mini-series on answering this complex question about the effects of creating Lake Nasser.  

An ‘ordinary’ search for the obvious query might be: 

 

  [ effects of creating Lake Nasser ]

 

This gives a bunch of ordinary organic results in the way we're used to doing--but the AI powered list at the top gives a set of positive and negative effects as well. 

Seeing this list made me think that comparing the 4 top LLM systems answers to this question would be interesting.  Here’s the list of effects given by each of the most popular LLM question-answering systems: 

 

Gemini

 

Positive effects:

Increased agricultural productivity: By regulating Nile flooding, Lake Nasser allows for controlled irrigation, enabling increased crop yields and expanded agricultural land in Egypt.

 

Hydroelectric power generation: The dam generates substantial electricity for Egypt.

 

Flood control: The reservoir behind the dam helps mitigate devastating Nile floods.

 

Negative effects:

Population displacement: The flooding caused by Lake Nasser led to the forced relocation of tens of thousands of Nubian people from their traditional homes in both Egypt and Sudan.

 

Loss of archaeological sites: Ancient Egyptian temples and other archaeological sites were submerged by the rising water levels, including parts of the Abu Simbel complex which required costly relocation efforts.

 

Reduced soil fertility: The Nile's natural silt deposition, which enriched downstream agricultural lands, was significantly reduced due to the dam, necessitating increased reliance on artificial fertilizers.

 

Ecological disruption: Changes in water flow and sediment levels impacted the aquatic ecosystem of the Nile, affecting fish populations and other aquatic life.

 

Potential for waterborne diseases: Stagnant water in the reservoir could create conditions favorable for the spread of waterborne diseases if not properly managed.

 

 

Copilot lists these as the major effects, a mix of positive and negative: 

 

Displacement of Nubians: Tens of thousands of Nubian people were forced to relocate from their ancestral lands, which were submerged by the lake.

 

Environmental Impact: The damming of the Nile altered the natural flow of nutrients, impacting the sardine fishery in the eastern Mediterranean. It also led to changes in the local ecosystem, affecting flora and fauna..

 

Agricultural Benefits: The reservoir provided a stable water source for irrigation, transforming previously arid land into productive farmland.

 

Hydroelectric Power: The dam generates hydroelectric power, contributing significantly to Egypt's energy supply.

 

Flood Control: The dam helps control the Nile's flooding, protecting downstream areas from catastrophic floods.

 

Tourism and Recreation: Lake Nasser has become a popular destination for tourists and offers recreational activities.

 

 

Claude tells us that:

 

Environmental Effects: The creation of Lake Nasser fundamentally altered the Nile River ecosystem. The massive reservoir submerged vast areas of land, creating one of the world's largest artificial lakes. This led to significant changes including:

 

Disruption of the natural flood cycle that had historically deposited fertile silt along the Nile Valley.

Increased evaporation of water from the large surface area, leading to some water loss

Creation of new aquatic habitats, supporting different fish species and wildlife

Prevention of nutrient-rich sediments from reaching the Mediterranean, affecting marine ecosystems

 

Social and Cultural Impacts:

The formation of the lake had profound effects on local communities:

 

Forced relocation of over 100,000 Nubian people from their ancestral lands

Submergence of numerous archaeological sites, though many were rescued through UNESCO's salvage campaign

Loss of traditional farming practices that depended on annual flooding

Creation of new fishing communities around the lake

 

Economic Benefits:

Lake Nasser provided several economic advantages:

 

Improved flood control, protecting Egyptian communities from devastating floods

Enhanced hydroelectric power generation, providing electricity for development

Creation of a significant fishing industry on the lake

 

Increased agricultural stability through regulated water supply

 

Development of tourism around both the dam and lake

 

Agricultural Changes:

The lake's formation transformed Egyptian agriculture:

 

Year-round irrigation became possible, allowing multiple growing seasons

 

Loss of natural soil fertilization from flood sediments, increasing reliance on artificial fertilizers

 

Expansion of irrigated agriculture in previously marginal areas

 

Changes in crop patterns and farming techniques

 

Long-term Challenges:

Several ongoing issues emerged:

 

Accumulation of sediments in the reservoir, reducing storage capacity

 

Increased soil salinization in some agricultural areas due to changed irrigation patterns

 

Coastal erosion along the Nile Delta due to reduced sediment flow

 

Management of water resources during drought periods

 

 

 

ChatGPT lists these as the major effects, a mix of positive and negative, as well as giving a few future-looking results as well: 

 

Positive Effects

Water Regulation and Flood Control: The Aswan High Dam allowed Egypt to regulate the flow of the Nile, preventing devastating annual floods that historically damaged crops and settlements. It enabled year-round irrigation, improving agricultural productivity.

 

Hydroelectric Power Generation: The dam provides a significant portion of Egypt's electricity, supporting industrialization and improving living standards.

 

Reservoir for Drought Periods:  Lake Nasser acts as a vast reservoir, storing water during high-flow years for use in drought periods, ensuring water security for agriculture and drinking.

 

Fisheries Development:  The lake has become a source of fish, providing livelihoods and a food source for local communities.

 

Economic Growth:  With water availability and energy production, Egypt advanced in industrial and agricultural sectors.

 

Negative Effects

 

Environmental Changes:  The dam traps silt that would have naturally replenished the fertile Nile Delta, leading to soil degradation and increased reliance on chemical fertilizers.

 

Erosion of the Nile Delta: Without sediment deposits, the delta is eroding, threatening coastal ecosystems and agriculture.

 

Displacement of Communities:  Approximately 100,000 Nubians were displaced, losing their ancestral lands and heritage as the area was submerged.

 

Cultural and Archaeological Loss:  Important historical sites, such as ancient Nubian temples, were submerged. Some, like Abu Simbel, were relocated, but many were lost.

 

Ecological Disruptions: Changes in the river’s flow and temperature have affected ecosystems downstream, reducing biodiversity and altering fish populations.

 

Increased Evaporation: Lake Nasser’s large surface area in a hot desert climate leads to significant water loss due to evaporation, reducing the overall water availability.

 

Impact on Downstream Nations:  The regulation of Nile flow has created tensions with downstream countries, particularly Sudan and Ethiopia, regarding water-sharing rights and access.

 

Long-Term Implications

 

Climate Change Amplification:  The dam and lake exacerbate vulnerabilities to climate change, such as rising temperatures and changing rainfall patterns in the Nile Basin.

 

Transboundary Water Conflicts: The dam has intensified debates over water use, particularly with Ethiopia’s Grand Ethiopian Renaissance Dam (GERD), complicating regional water diplomacy.

 

Sustainability Challenges:  High sedimentation rates within Lake Nasser could reduce its storage capacity over time, challenging its effectiveness.

 

 

As you can see, the LLMs vary a bit from one to the other.  Here’s a table of the different factors that each system mentions:

 

(Click on this image to expand.) 

Each row shows the agreement between the systems.  Gemini lists 3 positives and 5 negatives while ChatGPT lists 5 positives and 12 negatives.  Blanks point out the places where one of the LLMs does NOT mention one of the factors.  Only ChatGPT suggests that Lake Nasser would be a net positive factor for economic growth, and only ChatGPT mentions increased salinization as a net negative factor. 

And, as you can see, each of the systems has a fairly different take on the effects of the Aswan Dam and the creation of Lake Nasser.  

It’s pretty clear that Gemini has the simplest take on the question, while ChatGPT has the broadest list of factors.  Oddly, only Gemini mentions “potential for waterborne diseases” and only Claude mentions “increased salinization” as possible issues while both Gemini and ChatGPT list “loss of archaeological sites” as a problem.

As a SearchResearcher you need to keep these differences in mind.  One system isn’t necessarily better than the others, but they DO have very different takes on what our research question means.  

Obviously, for a full SRS style research analysis, you’d want to dig into each of these factors and get some ground truth about each of these questions. Is Claude correct in telling us that "increased salinization" is an issue? Is Gemini right about how building of the Aswan dam will increase the potential for waterborne diseases? 

We could spend another complete post on that waterborne disease issue alone!  (And if you know me, we probably will do exactly that.)  

But if you've been paying attention to the SearchResearch space, in the past couple of months several new research tools have become available that might be incredibly useful for exactly this type of deeper research.  

Undermind.ai and Google’s new Deep Research are systems that purport to take your deep research questions and do a great, focused, detailed analysis for you… exactly the kind of thing we’re asking for in this SRS Challenge.  

For our next post, we’ll try each of these new “deep research” tools and see how well they do with our Lake Nasser Challenge.

Keep searching!