Introduction
-
Text Mining GuideScott Enderle's guide at Penn Libraries to text analysis and mining in general.
-
Python Programming for the HumanitiesPython programming course focused on text processing in humanities disciplines.
-
aRgh: a newcomer's (angry) guide to RThe goal of the document is to describe R’s data types and structures while offering enough help with the syntax to get a programmer coming from another, saner language into a more comfortable place.
-
RegExrThis website allows you to test regular expressions in the browser and provides a comprehensive reference as well as examples.
-
Introduction to Japanese Text Mining Part OneMaterials introducing Japanese text mining in the R programming language by Mark Ravina.
-
Introduction to Japanese Text Mining Part TwoMaterials introducing Japanese text mining in the R programming language by Mark Ravina.
-
Introduction to Japanese Text Mining Part ThreeMaterials introducing Japanese text mining in the R programming language by Mark Ravina.
Help and Resources
-
日本語コーパス収集A list of Japanese corpora, mostly related to literature, and some other digital libraries and directories.
-
Python による日本語自然言語処理A chapter of "Natural Language Processing with Python" specific to using Japanese with NLTK (in Japanese language).
-
Japanese Language Processing with NLTKA quick overview of how to import JEITA and KNB corpus libraries into your Python programs using NLTK libraries.
-
NLTK Japanese CorporaA bilingual blog post outlining the use of two Japanese corpora with the Python Natural Language Toolkit.
-
IPAdicとはなにものかA blog post explaining IPAdic and its use in MeCab and ChaSen.
-
Natural Language Processing with Python (Japanese version)This is a Japanese translation of the book that accompanies NLTK for Python. It is available as an ebook.
Morphological Analysis
Software:
- Voyant Tools now works with Japanese (including tokenizing your text)
- Topic Modeling Tool also works out-of-the-box with pre-tokenized Japanese
- ctext.org's Text Tools works with pre-tokenized Japanese (uncheck tokenize by character in the options)
APIs:
- kuromoji Japanese morphological analyzer (also a standalone CLI)
- U-Tokenizer CJK Tokenizer API
- Yahoo! Japan Text Analysis Web API
Keyword-in-Context:
Word segmentation, part-of-speech tagging, and more:
- MeCab Part-of-Speech Tagger and Morphological Analyzer
- JPTokenizer and Tiny Segmenter (Python libraries)
- TinySegmenter and Rakuten MA (JavaScript)
- KH Coder
- KyTea
- MeCab can be plugged in to Python scripts via the Natural Language Toolkit (NLTK)
- natto.py package for using MeCab with Python
- ChaSen
- 日本語形態素解析システムJUMAN (KNP analyzes JUMAN output)
- CaboCha (requires MeCab, UniDic, CRF++ - also provides a set of libraries)
- KAKASI: Kanji Kana Simple Inverter
- Japanese Text Analyzer (Android app based on Yahoo! Japan web services)
Geographic services:
Corpora
Literature:
- 書籍デジタル化委員会's search across many literature digital archives
- Aozora bunko (there is also an Aozora full corpus search site)
- Japanese Text Initiative
- J-TEXTS
- Taiju's 日本古典文学テキスト
- ALEXの書斎 (Nihon shoki, Kojiki, Kogo shūi, Kuji hongi in kanbun)
- 米国議会図書館蔵『源氏物語』翻刻本文ーひまわり用データ for use with Himawari
Modern usage:
- KNB blog corpus
- JEITA Public Morpohologically Tagged Corpus
- Japanese Learner's Written Composition Corpus
- Kyoto University corpus (Mainichi shinbun articles)
- 日本語ウェブコーパス2010
Bilingual:
- Tanaka Corpus (and a web interface for it)
- アイヌ神揺集
NINJAL corpora:
Check out the Center for Corpus Development, NINJAL for links to many corpora and databases. Here are some notable corpora.
- KOTONOHA: Shonagon
- KOTONOHA: Chunagon contemporary Japanese corpus (registration by mail required)
- Oxford-NINJAL Corpus of Old Japanese
- 近代語コーパス (includes 明六雑誌コーパス and 近代女性雑誌コーパス for download)
- Corpus of Spontaneous Japanese
- NINJAL通時コーパスプロジェクト (in progress)
- 日本語歴史コーパス (registration by mail no longer required)
Dictionaries
-
UniDic Download PageDownload page for all UniDic dictionaries available from NINJAL (used with ChaMame).
Subject Guide
Japanese WordNet
WordNet, including synsets (synonym sets) only, has been created for Japanese. Please visit the page to download the sqlite3 database of Japanese WordNet, then use one of the APIs in a variety of programming languages to use it in your own code. Here is a link to the Python API for Japanese WordNet.
-
Japanese WordNetLinks to a variety of downloads, reference material, and related links to Japanese WordNet. Includes the downloadable sqlite3 database containing WordNet, as well as links to APIs and applications that use this database.
-
Wn-Ja Python APIFrom yanabe.diff. See also the Github site for this code.
-
wn.py on GithubFrom yanabe.diff.
-
WordNets in the WorldA full list of WordNet in all languages for which it is available. Includes Chinese (broken link; online only), Japanese, and Korean (online only) as well as multilingual WordNets.
-
About WordNetPrinceton's page about the original English WordNet.
Sample Segmented Corpora
These are plain-text files (compressed in zip format) of a couple of NINJAL corpora, segmented with spaces between words to be used with software for Western text analysis like MALLET, Topic Modeling Tool, and Voyant.
-
明六雑誌 Lemmatized Text (full run)The full run of 明六雑誌, lemmatized (words reduced to stems and written in standard format).
-
明六雑誌 Raw Text (full run)The full run of 明六雑誌, with words represented as they were in the original text. Text file.