Link rot, by the numbers

By Cassini · from the list On link rot, and why the web forgets

"The internet is forever" is the thing people say about embarrassing photos. It is not true of anything you would actually want to keep. The web has been measured, several times, by people with large samples and no product to sell, and the numbers agree: pages disappear at a steady rate, and the older the page, the likelier it is already gone. Here is what the measurements say, in the order they were made.

1998: six percent, and doubling

Jakob Nielsen was calling link rot a crisis before most people had a home connection. Citing a survey by Terry Sullivan's All Things Web, he put the share of broken links on the web at 6% in May 1998, and noted it had doubled since the previous August. His advice was to keep old URLs alive forever, which nobody did.

2014: half the Supreme Court's links

Jonathan Zittrain, Kendra Albert and Lawrence Lessig checked every URL cited in United States Supreme Court opinions and in the Harvard Law Review and two other law journals. 50% of the Supreme Court links no longer led to the material the justices cited. For the journals it was over 70%. These are the citations in the most carefully edited documents in American law, and they rot like everything else. The study became Perma.cc, a permanent-citation service run by Harvard's library.

2021: a quarter of the paper of record

Zittrain, John Bowers and Clare Stanton then did the same for the New York Times: every deep link in every article from the site's launch in 1996 to mid-2019. About a quarter were dead, and more than half of the articles that contained any link contained at least one dead one. Rot rose almost linearly with age. The paper also names the quieter problem, content drift, where the link still resolves but the page behind it no longer says what it said when it was cited.

2024: 38% of 2013 is gone

Pew Research Center took the largest sample so far: just under a million pages from Common Crawl, roughly 90,000 per year from 2013 to 2023, and checked whether each still existed in October 2023. 38% of the pages from 2013 were no longer accessible. Pew also found that 23% of news pages and 21% of government pages contain at least one broken link, that 54% of Wikipedia articles have at least one dead reference, and that nearly one in five tweets became invisible within months of being posted.

What the numbers mean for a bookmark

A bookmark is a pointer. Every study above is a measurement of how often pointers stop pointing at anything: a few percent a year, compounding, with no floor. Ten years of saving links produces a library where roughly a third of the entries open to a 404, a parked domain, or a page that has quietly changed. The fix is old and unglamorous: keep a copy. Vannevar Bush described it in 1945, Gwern has done it by hand for twenty years, and it is the entire reason Bindle archives every page at the moment you save it rather than trusting the URL to hold.

The full reading list, including Tim Berners-Lee's 1998 plea that cool URIs don't change and the volunteers at Archive Team who show up when a site announces it is closing, is a public list you can follow.