SearchResearch Challenge (1/17/24): How can I get an AI to summarize a document?

We’ve got AI tools all over the place… 

… now, how can we use them effectively? 

An important piece of SearchResearch (or more generally, sensemaking) is finding relevant documents, reading them, and extracting the pieces of knowledge you need to get on with your work.  

I find that I spend roughly half of my research time just browsing, scanning, and reading—it’s what you need to do to understand a topic area enough to do real work. 

But what if we could accelerate the scanning and reading part?  What if we could take a long document and summarize it accurately?  

This is one of the great promises of the current crop of LLMs—they offer the ability to summarize long documents.  

Or do they?  Will they make the long document into a shorter, more focused work?  Or will that tool end up blurring everything into mush?  

The Challenge for this week is to explore how well an LLM can summarize a document in a way that’s useful for us SearchResearchers.  Here’s the Challenge for this week: 

1. How well does your favorite bit of AI technology do at summarizing long texts?  What system do you like for summarizing, and why?  

2. Test out your summarization method on two long-form texts.  First, let’s try that classic gothic tale, Frankenstein by Mary Shelley.  (Here’s a link to the full text that you can use.) How good is that summary?  

3. Second, let’s try your summarization method on a piece of text you might have read, but might have slightly forgotten—Chapter 10 of my book, The Joy of Search, link to the full-text of Chapter 10 here.  "When was oil first discovered in California?"  Did your summary method create a good/useful summary?  

Try using your fave summary method on these two works and let us know what you find.  We’re interested in practical ways of getting a good summary of long texts.  When you do this, think about what your goal is in getting the summary—is it just general interest?  Or do you have a particular question in mind?  When is a summary useful?  (And contrariwise, when does it not work well?)  Or, how do you know your summary is a good one?  

Let us know what you've learned in the comments section.  

Next week I’ll summarize what we’ve learned, and what techniques seem to work best for SearchResearchers.  

Keep searching!