A dataset called the arXiv Complete Corpus appeared on Hugging Face: 3,148,796 papers with metadata, version history, submission files and rendered documents, including file contents, paths, sizes and SHA-256 digests. By its own account it holds a PDF for 99.47% of papers and 99.54% of versions "reported with a non-zero submission size," with metadata from arXiv's OAI-PMH interface and files from "the GCS mirror, S3 source archives and direct PDF fetches." It's billed as a one-off snapshot with documented coverage gaps.
Anyone who has tried to assemble this knows why someone did it. Reconciling three of arXiv's access channels into one schema, with checksums and a version history that actually lines up, is a few weeks of unglamorous work that gets redone independently by every lab that needs it. A frozen, digest-verified corpus is a genuinely useful object.
It's also, for most of its contents, a thing that the people who wrote those papers never gave anyone permission to publish. And the odd part is how little the republication buys, because the files were already free.
The metadata is CC0. The papers are not.
These two halves have completely different legal status, and conflating them is how corpora like this get built.
arXiv's metadata — titles, authors, abstracts, categories, version histories — is released under CC0 1.0, a public-domain dedication. Mirror it, repackage it, sell it, no permission needed. Everything in this corpus that describes a paper is clean.
The papers themselves are a different matter, and arXiv is unusually blunt about it. Authors keep their copyright; the overwhelming majority submit under arXiv's default licence, which grants arXiv a perpetual non-exclusive right to distribute and grants nobody else anything. arXiv's API Terms of Use put it about as plainly as a policy document can: "You should not store and serve arXiv e-prints from your servers unless you have permission from the copyright holder or the e-print was submitted with a license that permits redistribution," followed by the qualifier that matters — "a very small subset of arXiv e-prints are submitted with licenses that permit redistribution." Its permissions page adds that arXiv "is unable to grant others the right to distribute arXiv articles," which forecloses the obvious workaround of asking arXiv.
How small is "a very small subset"? A community analysis on the Open Knowledge forum put non-default licences at 3.42% of uploads since 2015 — so roughly 96.6% under the default. That figure is old and unofficial, and the CC-licensed share has surely grown since, particularly in machine learning where CC-BY is closer to a norm. But it would have to have grown by an order of magnitude to change the character of the problem. On any plausible split, a corpus that includes a PDF for 99.47% of arXiv includes a few million PDFs whose copyright holders licensed exactly one organisation to distribute them, and it isn't Hugging Face.
Nothing about that is unusual for AI datasets, which is rather the point. What's unusual is how unnecessary it was here.
The files were already free, from a mirror arXiv helped build
This is the part that makes the release genuinely puzzling rather than merely legally exposed.
arXiv does not hide its full text behind a paywall or a negotiation. It offers bulk access through several channels. The one that matters here is gs://arxiv-dataset, a Google Cloud Storage mirror set up in partnership with arXiv, holding PDFs and source files, free to download, updated weekly. Not requester-pays. Not gated. A gsutil command away, indefinitely, for anyone.
The corpus's own description says its files came from "the GCS mirror, S3 source archives and direct PDF fetches." So the pipeline began at a free, official, continuously updated source — and ended at a static third-party copy of it. The republication adds no access that didn't exist, for no audience that was previously excluded. It relocates bytes from a mirror arXiv sanctioned to a mirror arXiv's terms specifically address.
The other channel, the arxiv bucket on S3, is requester-pays, which is the only place a real cost appears: roughly 9.2 TB as of April 2025, growing about 100 GB a month, at egress rates that work out to a few hundred dollars for the whole set. That's the actual friction the official route imposes — a manageable cloud bill and some patience — and it's the friction the S3 model exists to create, so arXiv can open the door to everyone without paying for everyone's bandwidth.
What it really adds is the manifest — and the manifest is the clean part
Strip out the file bytes and look at what's left: a unified schema across metadata, version history and files; paths and sizes; SHA-256 digests for every file; and honest, documented coverage gaps. That's the contribution. Digests are what turn a pile of downloads into something auditable — you can prove your copy matches, detect a truncated fetch, and cite a corpus by content rather than by promise. Version history reconciled against files is the piece none of arXiv's channels hands you assembled.
And the reproducibility argument for a frozen snapshot is real in a way the redundancy argument above doesn't touch. You cannot cite gs://arxiv-dataset in a paper and mean anything precise, because it changed the week after you ran your experiment. "The corpus as it stood on this date, with these digests" is a citable object, and a weekly-updated mirror can never be one. That is a legitimate gap, and this release fills it.
Here's the thing: the part that fills it is the part that's legally clean. A SHA-256 digest is a fact about a file, not a copy of one. Paths, sizes, version numbers, timestamps and coverage statistics are facts. The metadata is CC0. A manifest-only release — every digest, every path, every version, plus a script that fetches from gs://arxiv-dataset and verifies against the digests — delivers the frozen, auditable, citable corpus without republishing a single byte anyone else owns. Researchers get bit-exact reproducibility; arXiv's terms are untouched; the thing survives.
That's not a legal technicality, it's a better design. The 9-plus terabytes are the least interesting and most encumbered part of the package, and they're the part the official mirror was already serving for free.
The coverage number has a carefully built denominator
Credit where it's due: "99.54% of versions reported with a non-zero submission size" is a precise and honest bit of phrasing, and most dataset cards wouldn't bother with the qualifier. It's worth noticing what it does.
arXiv reports a zero submission size for versions that don't have a retrievable file — withdrawals, problem submissions, records whose source was never stored. Excluding them from the denominator is defensible: you can't be faulted for missing a file that doesn't exist. But it means the headline isn't "99.5% of arXiv," it's "99.5% of the part of arXiv that was fetchable in principle," and the excluded set is precisely where the awkward records live. On the paper-level figure, 99.47% of 3,148,796 still leaves roughly 16,700 papers with no PDF at all — a gap the size of a mid-sized preprint server, disclosed rather than buried.
A snapshot of something that doesn't hold still
The release calls itself a one-off snapshot, and that's the right word, but it undersells the decay rate. By arXiv's own S3 figures the archive grows about 100 GB a month, and arXiv takes in north of 20,000 new submissions a month. A complete-as-of-today arXiv is measurably incomplete by next month and meaningfully stale within a year.
For the reproducibility use case that's a feature — you want it frozen. For anyone who reads "complete corpus" as "the corpus you can keep using," it's a trap, and the gap between those two readings is where datasets like this go to rot on a hub.
The host changed owners three weeks ago
Corpora of this shape have a history, and it isn't long. Books3 — 196,640 books scraped from a pirate library, about 37 GB — spent three years as a standard ingredient in language-model training before the Danish anti-piracy group Rights Alliance sent DMCA notices in August 2023. It came off The Eye, and the Hugging Face dataset became defunct-datasets/the_pile_books3. Hugging Face publishes the notices it receives in a public takedown-notices repository, which is a genuinely good transparency practice and also a standing record that the takedown path is used and works.
arXiv preprints aren't Bibliotik scrapes — the authors put them up to be read, most would be delighted to be in a training corpus, and arXiv is not an anti-piracy plaintiff. The asymmetry is real and it's why this will probably sit there undisturbed. But "the rights holders probably won't mind" is a bet about three million separate rights holders, some of whom are publishers with a transfer agreement and an enforcement budget.
What's different this time is the landlord. NVIDIA agreed to acquire Hugging Face three weeks ago, at a price specified to the dollar. A hosting platform run by a startup and a hosting platform owned by one of the most valuable companies on earth face the same notices and very different incentives about how quickly to act on them. Nobody sends a speculative letter to a startup with no balance sheet. Plenty of people will send one to NVIDIA.
The useful frame here isn't piracy, it's proportionality. This is an unusually well-made artifact — provenance documented, digests throughout, coverage gaps stated rather than rounded away — and that care is exactly why the licensing gap grates. Someone who thought hard enough about reproducibility to publish SHA-256 digests for every file in a 3-million-paper corpus was two design decisions away from a release that nobody could ever have to take down.
The broader pattern is one this blog keeps running into from the model side: "open" describes a spectrum, the terms attached to an artifact decide what you can actually build on it, and the part of the stack people check last is the part that determines whether their work survives contact with a lawyer. It's just as true of datasets, and rather more consequential — a model with awkward licence terms is one you can swap out. A corpus that disappears takes every experiment built on it along.
Sources: arXiv API Terms of Use · arXiv Permissions and Reuse · arXiv Bulk Data Access · arXiv metadata on Kaggle (CC0) · Hugging Face takedown notices · TorrentFreak on the Books3 takedown