You just need Anna's archive. That's what OpenAI allegedly used to pre-train their earlier GPT models.
Here are 20 websites where you can download books for free:
Project Gutenberg .com
Open Library .org
Google Books .com
ManyBooks .net
PDF Drive .com
Bookboon .com
Free-Ebooks .net
Smashwords .com
DigiLibraries .com
GetFreeEbooks .com
Obooko .com
Baen Free Library .com
Internet Archive .org
HathiTrust .org
Standard Ebooks .org
ReadPrint .com
Bookrix .com
Librophile .com
Online Books Page .edu
View quoted note →
