I wrote a council planning story stuffed with figures, ran it through the summarizer, and then checked every number in the output against the source. Writing the source myself is the point: it means I know exactly what every sentence claims, which is the only way to tell a summary that is merely shorter from one that is also true. Not one was invented. Nine numbers came through and all nine appear in the original, which is a better result than the usual warnings about AI making things up would lead you to expect. The summary was still wrong, in two places, and both errors kept the digits intact. One of them reads "designated affordable apartments for 80% of market rent, a level significantly below Shelter's recommended 30%", which compares a rent level against a proportion of units and arrives at a sentence that cannot be true in any arithmetic.
TL;DR: The thing to check in a summary is not whether the numbers are real, it is whether they are still attached to the right thing. Measured on a 327-word source: zero invented numbers, correct compression at 0.28 and 0.59 of the original, and two relationships broken, each one keeping the figures and losing what they referred to. Also worth knowing before you paste: the length settings are absolute, so on a short input "Detailed" produced a summary 2.2 times longer than the text it summarised. The summarizer runs on the site's own server.
What do the four length settings do?
They name an output size, and they hit it. Format compliance is the most reliable thing here.
Measured on the same 91-word passage:
| Setting | Asked for | Produced | Ratio to source |
|---|---|---|---|
| One sentence | a single sentence | 1 sentence, 47 words | 0.52 |
| Short paragraph | 3 to 4 sentences | 4 sentences, 82 words | 0.90 |
| Bullet points | 5 to 7 bullets | 7 bullets, 95 words | 1.04 |
| Detailed | 2 to 3 paragraphs | 3 paragraphs, 200 words | 2.20 |
Every one of those matches the instruction. One sentence is one sentence. Bullet points gave seven, inside the five to seven band. Detailed gave three paragraphs.
Look at the last column.
Why was the detailed summary longer than the article?
Because the lengths are absolute targets, not proportions of the input.
"Detailed" asks for two to three paragraphs covering every main point, and it produces two to three paragraphs whether you give it ninety words or nine thousand. On a short passage that means the model writes connective prose around the facts to reach the length, and the output is longer than what it summarised.
The same passage at 327 words behaves as you would expect:
| Source length | Short paragraph | Detailed |
|---|---|---|
| 91 words | 0.90 of source | 2.20 of source |
| 327 words | 0.28 of source | 0.59 of source |
So the breakeven sits somewhere around two to three hundred words. Below that, pick One sentence, or do not summarise at all, because a 91-word passage is already the length of a summary.
The inflation is also visible in the prose. The detailed version of the short passage opens "The city council's decision regarding downtown parking garages represents a significant urban development project aimed at addressing housing affordability and stimulating economic growth", which is twenty-three words to say the council approved a scheme.
Does it invent facts?
In this test, no, and that is worth stating as plainly as the failures.
I counted every number in the 327-word source and every number in the detailed summary of it:
numbers in source 19
numbers in summary 9
in summary, not in source: none
Nine of nineteen figures survived and not one new figure appeared. The model is running at a temperature of 0.3, the lowest band used by any writing tool on the site, which is the setting that makes this kind of output conservative.
That is the headline most coverage of AI summarising would stop at, and it is the less useful half of the picture.
Then what goes wrong?
The relationships between the numbers, which survive as digits and break as meaning.
The affordable housing claim. The source says 85 of the 340 apartments will be affordable at 80 percent of market rent, and that this proportion is below the 30 percent Shelter recommends. The summary says:
designated affordable apartments for 80% of market rent,
a level significantly below Shelter's recommended 30%
Both numbers are real. The comparison is nonsense: 80 percent of market rent is a price, 30 percent is a share of units, and the sentence asserts that the first is below the second. The figure that made the original sentence make sense, 85 units out of 340, is the one that got dropped.
The parking dispute. The source has the council citing 41 percent occupancy and the Chamber of Commerce disputing it with a survey of its own. The summary says the Chamber stated "the capacity figures used in their survey were lower than the 41% occupancy rate", which has the Chamber arguing the opposite of its own position.
Neither error would be caught by checking the numbers against the source. Both require reading the sentence and asking what it claims.
This failure has a name in the research literature. Maynez and colleagues' work on faithfulness in abstractive summarisation separates extrinsic hallucination, where a summary introduces something absent from the source, from intrinsic hallucination, where it recombines material that is present into a statement the source does not make. Their finding, on human-annotated summaries, is that the intrinsic kind is both more common and harder to detect automatically. That is exactly the shape of both errors above, and it is the reason a numbers check gives false confidence.
What gets dropped, and is there a pattern?
Detail that supports a claim rather than being the claim, and no, there is not a reliable pattern.
From the same source, these disappeared entirely: 1,412 consultation responses, 63 percent support, the two garages' construction dates, a 12 million pound repair liability, a survey of 38 businesses finding 71 percent arrived by car, the shuttle's 20 minute frequency, the 4.3 million pound infrastructure cost, and the March start date. The summary kept "the initial cost of infrastructure improvements" without the figure that made it an argument.
On the shorter passage the dropping was uneven in a way that defeats any rule of thumb:
| Fact | One sentence | Short paragraph | Bullet points | Detailed |
|---|---|---|---|---|
| 340 apartments | kept | lost | kept | kept |
| Rents up 18% | lost | lost | lost | kept |
| 900 spaces removed | lost | lost | kept | lost |
| Starts next spring | kept | lost | kept | kept |
A longer summary is not a superset of a shorter one. "Short paragraph" lost the apartment count that "One sentence" kept. "Detailed" lost the parking figure that "Bullet points" kept. If a specific number matters to you, the setting that preserves it is not predictable, and the only reliable approach is to check.
The one that should worry a reader most is the rent increase: 18 percent in three years is the single most quotable statistic in the piece, and three of the four settings dropped it.
Can it summarise a PDF or a web page?
Not directly. It takes text you paste, which means the extraction is a separate step and one worth doing deliberately.
For a PDF, pull the text out first. PDF to text does it in the browser, and the guide to what that conversion costs is worth a minute of your time first, because a PDF's columns come out interleaved and a table comes out as a line of values. A summary of scrambled columns is a summary of nonsense, and it will read perfectly fluently.
For a web page, copy the article text rather than the page. Navigation, cookie banners, related-article lists and comment counts all become input, and the model has no way to know they are furniture. On a page with a long sidebar they can outweigh the article.
For a scanned document there is no text to extract at all, and OCR comes first. What actually breaks OCR has the measured version of that, where a three degree tilt matters more than resolution.
The general rule: whatever reaches the box is what gets summarised, including anything that came along by accident.
Why not just read the first paragraph?
Often you should, and knowing when is the difference between using a summarizer well and using it reflexively.
News writing uses the inverted pyramid, where the first paragraph carries the whole story and everything after it adds detail in descending order of importance. For a news article, the first paragraph is a summary written by a person who read the thing, and it is free and instant.
Where a summarizer earns its minute is the material that is not written that way:
| Source | Read the opening | Summarise |
|---|---|---|
| News article | Yes, it is already a summary | Rarely worth it |
| Minutes, transcripts, threads | No structure to exploit | Yes |
| Academic paper | Abstract already exists | Only for a section |
| Report with an executive summary | Read that | Only the sections it skips |
| Legal or policy document | Opening is definitions | Yes, with checking |
The common case this tool suits is the one with no author-written summary and no helpful ordering: an email thread, a set of meeting notes, a transcript, a long chat log. Those have the information distributed evenly through them, which is precisely when reading the first paragraph tells you nothing.
Which setting should you pick?
By what you are going to do with the summary, and the choice is narrower than four options suggest.
| Setting | Use it for | Avoid when |
|---|---|---|
| One sentence | A subject line, a note to yourself, triage | You need any detail at all |
| Short paragraph | Reading a long document quickly | The source is under 200 words |
| Bullet points | Notes you will scan again later | The source has an argument rather than a list |
| Detailed | A long report you still have to act on | The source is short, or you need every figure |
Bullet points deserves a caveat. Turning prose into bullets discards the connective tissue, which is where disagreement lives. A piece about a contested planning decision becomes a list of statements with the contest removed, and the bullets read as settled facts.
What should you check in a summary?
Three things, in this order, because the first two are quick.
- Every number, against the source. Not whether it is plausible, whether it is there. One minute for a short summary.
- What each number is attached to. This is where the errors were. Read each sentence containing a figure and ask what claim it makes, then find that claim in the source.
- Who said what. Attribution is the other thing that inverts. The Chamber of Commerce example above is a position reversal that reads entirely naturally.
What you cannot check by comparison is what is missing, since an absent fact leaves no trace in the output. That needs a read of the source, which is the thing you were trying to avoid. The honest framing is that a summary tells you what a document is about so that you can decide whether to read it, not so that you can skip it.
Is a summary safe to forward?
Not as a substitute for the thing it summarises, and the two failures above are the reason.
Both were fluent. Both kept real figures. Both would survive a skim by someone who had not read the source, and the affordable housing one would be repeated in a meeting by somebody who then could not say where it came from.
If you are forwarding a summary, forward the source with it. If you are quoting from a summary, go back to the source for the quotation. If you are making a decision on a summary, the number you are deciding on is the one to verify first.
This is not an argument against summarising. It is the ordinary practice for working from any secondhand account, and a summary produced by a machine is exactly that.
How long does it take?
Between thirty seconds and a minute and a half, tracking the output rather than the input.
| Run | Source | Setting | Time |
|---|---|---|---|
| 1 | 91 words | One sentence | 32 s |
| 2 | 91 words | Detailed | 73 s |
| 3 | 327 words | Short paragraph | 66 s |
| 4 | 327 words | Detailed | 95 s |
Reading a long source is comparatively cheap; writing a long summary is not. "Detailed" costs roughly twice what "One sentence" costs on the same input.
There is a ceiling on the input too. Very long text is shortened to what the model can read in one go, and the page says so when that happens rather than silently summarising the first part. For a long report, summarise it in sections, which also keeps each check small enough to do properly.
Why is it set to be predictable?
Because summarising is the one job on the site where a repeatable answer is the right answer.
Every AI tool here carries a temperature controlling how much variation the model is allowed. The naming tools run at 1.0, because a list of names that never changes is useless. The paraphraser sits at 0.5. The summarizer is at 0.3, the lowest of the writing tools.
That is why no numbers were invented. Low temperature makes the model stick close to the likeliest continuation of the text in front of it, which for a summarisation task means the text in front of it. The same setting is why the output can be flat and a little mechanical, which is the right trade here.
It also means re-running will not help much. If a summary drops a figure you need, running it again usually drops it again, and the better move is a different length setting or a shorter input.
Why does this run on the site's own server?
Because the documents people summarise are the ones they cannot paste into a public service.
Board papers, a legal letter, an incident report, a thread about a colleague, a draft under embargo, a medical summary. The usual alternative sends the whole document to a company whose terms allow it to be retained and possibly trained on.
Here it goes to a small model on the site's own hardware and is not stored unless you choose to send feedback. The cost is visible in the timings above, and in the honest ceiling: a small model summarises a page well and a hundred pages not at all.
