Sources of Text Data
This is a catalog of text data sources available to researchers at Penn. The data from these sources has not been gathered with any one project in mind, and may be biased in unpredictable ways. It's like data from a poll that allows voters to vote as often as they like: it might be reliable for a particular project, but it might not be, and there's not an easy way to tell.
That places an additional burden on researchers using data from these sources. To ensure the consistency and intellectual coherence of their corpus, they may need to compile an ideal list of documents based on well-defined criteria, and then collect the items on that list from different sources. That strategy will ensure that most biases in the corpus will arise from a known source: the criteria used by the researchers to compile the list. Those willing to accept this extra burden will be able to build new, potentially ground-breaking corpora in an intellectually rigorous way.
Text Data Sources Available to Penn Affiliates
-
Gale Text DataPenn Libraries can provide access to full text data from numerous Gale research databases including ECCO, the Gale NewsVault, and Archives Unbound. For details about holdings and access procedures, please contact Scott Enderle.
-
LexisNexis Bulk DataPlain text data from LexisNexis is now available through an app under development by Penn Libraries. If you would like to help test the app or have a project in mind for this data, please contact Scott Enderle.
Text Data Sources Freely Available to the Public
-
HathiTrust Digital LibraryThe HathiTrust library collects data from more than fourteen million books, and is an extraordinary source of text data for historical research. Full text and page images are available through an API, and can also be downloaded in bulk.
-
HathiTrust Extracted FeaturesIn addition to providing page images and full text, HathiTrust provides word frequency and feature counts for each page of each book in their collection—including in-copyright works.
-
Chronicling AmericaCreated by the Library of Congress, this collects digital images and text from millions of pages of historic US newspapers from 1789 to 1924.
-
DocSouth CollectionsThese sixteen collections were built by the Documenting the American South project at UNC Chapel Hill. Including collections of first-person narratives, oral histories, and state records, this is a rich resource for building specific-purpose corpora.
-
The Chinese Text ProjectCo-created by Donald Sturgeon and users worldwide, this large database of full-text Chinese classics is free for anyone to use, and provides a newly created API for downloading texts and creating applications that link to the database.
Library TDM Services
Penn Libraries is committed to providing research support and services for TDM projects to students and researchers across campus. To make these services accessible to a large audience, they are limited in scope.
We can provide
- This research guide, with resources, contact information, and other details.
- Consultations and workshops with specialists at the library and elsewhere.
- Negotiation with vendors for university-wide text mining licenses, subject to budgetary approval.
We generally cannot provide
- Licensing for individual text mining projects with needs not covered by university-wide licenses.
- Secure, long-term storage for vendor data or research output.
- Enforcement of restrictions on vendor data use.
Depending on vendor policies and our own support capacity, we may be able to provide some of the above services on a case-by-case basis, but we can make no guarantees. These limits have been adapted from the Northeastern University TDM guide.
Library TDM Contacts
Scott Enderle, Digital Humanities Specialist Librarian.
