Your Librarian
Your Librarian
Your Librarian
What is text mining?
Text mining is a method of turning text into data for computational analysis. It can uncover patterns in large bodies of text (called corpora) that might otherwise be hidden. Source: Underwood, T. (2015). Seven Ways Humanists are Using Computers to Understand Text. The Stone and the Shell.
How do I get started?
In starting a text mining project, think about the following questions:
- What is your research question?
- What text(s) do you want to use?
- Are the texts available in machine-readable form?
- What is the quality of the texts? Do they need to be corrected/cleaned up?
Find content to mine: library resources
Text and data mining, and systematic downloading, is usually not permitted under most of the Library's license agreements. These are some resources that allow for text mining. For questions about text mining access to other library resources, please contact us.
-
ProQuest Historical Newspaper Data SetsDownload XML and PDF files of two newspapers from the ProQuest Historical Newspapers collections: The New York Times (1851-1936) and The Washington Post (1877-1934). These files may be downloaded and used for text and data mining.
-
Corpus of Contemporary American English (COCA)Find and download full-text corpus data of American English taken from spoken (transcripts), fiction, popular magazines, newspapers, and academic texts from 1990.
Complete data sets may be downloaded from the link below.
-
Gale Primary SourcesSearch across multiple digitized primary source collections, including:
17th and 18th Century Burney Collection, Archives Unbound, Associated Press Collections Online, The Economist Historical Archive, 1843-2014, Eighteenth Century Collections Online, The Financial Times Historical Archive, Indigenous Peoples: North America, The Making of the Modern World, Nineteenth Century Collections Online, Smithsonian Collections Online, The Times and the Sunday Times Digital Archive, Times Literary Supplement Historical Archive, U.S. Declassified Documents Online.Tools for finding term frequency and term clusters are also available in the database.
Find content to mine: other resources
-
Chronicling America OCR DataDownload OCR files of digitized historical newspapers from the Library of Congress/National Endowment for the Humanities.
-
Project GutenbergContains over 54,000 works of literature.
-
Internet ArchiveA digital library of internet sites and other cultural artifacts in digital form.
-
Datasets for Digital ResearchAccess datasets for text and data mining from the Michigan State University Libraries. Includes corpora such as Feeding America: The Historic American Cookbook, U.S. Congressional Collection, and Sunday School Books in Nineteenth Century America.
-
American Presidency ProjectOver a hundred thousand presidential documents consolidated, coded, and organized into a single searchable database.
-
DH Toychest: Data collections and datasetsFind text collections that can be downloaded and used in text analysis and topic modeling tools.
-
Demo corporaSample collections of texts that are ready-to-go for demonstration purposes or hands-on tutorials. Ideal collections for this purpose are public domain or open access, plain-text, relatively modest in number of files, organized neatly in a folder(s), and downloadable as a zip file.
-
BYU CorporaIncludes corpora of texts such as US Supreme Court Opinions, American Soap Operas, TIME magazine, Wikipedia, and more.
Use text analysis tools
-
VoyantUse this free and user friendly tool for text analysis.
-
MALLETMALLET (MAchine Learning for LanguagE Toolkit) is a Java-based package for natural language processing, topic modeling, document classification, clustering, and more. A graphical user interface (GUI)-based version is also available.
-
Topic Modeling ToolA graphical user interface (GUI) version of MALLET for topic modeling.
-
Atlas.tiQualitative analysis software available on computers in the Main Library, Science-Engineering Library, and Fine Arts Library.
-
TAPoRFind tools to use for text analysis and retrieval.
Use text mining tools with corpora included
-
Google Books Ngram ViewerView the occurrence of words or phrases over time in the Google Books corpus.
-
JSTOR Data for ResearchUse this tool, which includes word frequencies, ngrams, topic modeling, and visualizations, to explore the scholarly journals in the JSTOR database. Datasets can be downloaded.
-
HathiTrust BookwormView the occurrence of words or phrases in millions of volumes in HathiTrust.