Natural Language Toolkit (NLTK): NLTK is a leading platform for building Python programs to work with human language data. It provides easy-to-use interfaces to over 50 corpora and lexical resources such as WordNet, along with a suite of text processing libraries for classification, tokenization, stemming, tagging, parsing, and semantic reasoning. NLTK has been a standard tool in NLP education and research for over two decades.
spaCy: SpaCy is a popular Python library for advanced NLP, particularly known for its fast performance. It provides an industrial-strength NLP pipeline including tokenization, part-of-speech tagging, named entity recognition, parsing, and more. Just as importantly, spaCy makes it easy to build models for new tasks and languages thanks to its clean, Pythonic APIs and pre-trained statistical models.
TensorFlow/Keras: While not strictly NLP libraries themselves, TensorFlow and Keras are essential tools for building neural network models in NLP. They provide the backend framework for distributed deep learning along with high-level APIs for creating and training neural network models. Many top NLP research papers utilize TensorFlow/Keras for tasks like document classification, machine translation, question answering, and more.
scikit-learn: Sklearn is the go-to general machine learning library for Python and a standard tool for NLP. It has implementations of popular algorithms like SVM, random forest, k-means clustering and dimensionality reduction techniques like LSA, t-SNE and PCA that are commonly used in NLP tasks such as text classification and topic modeling. Sklearn also includes utilities for model selection, pipelines and cross-validation, making it an indispensable component of any NLP project.
Stanford CoreNLP: CoreNLP is a Java-based suite of NLP annotation tools developed at Stanford University. It supports a wide variety of tasks including POS tagging, NER, coreference resolution, sentiment analysis, parsing, and others. While it’s a bit more complex to set up than Python libraries, CoreNLP provides high-quality pretrained annotators that are often used as initial preprocessing steps or for comparison with custom models.
AllenNLP: AllenNLP is an end-to-end library focused on enabling state-of-the-art NLP research. It was created by AI safety startup Allen Institute for Artificial Intelligence and provides experiment tracking, data and textfield handling, training loops, and utilities for text, vocabulary, embeddings and models like ELMo, BERT and OpenAI GPT-2. AllenNLP makes it straightforward to reproduce cutting-edge NLP work through clear, well-documented APIs.
gensim: Gensim is a Python framework for vector space modeling and topic modeling in particular. Its main functionality comes through implementations of popular algorithms like Latent Dirichlet Allocation (LDA), LSI, Word2Vec and GloVe. Gensim brings these advanced NLP methods out of academic papers and into hands-on practice through a well designed interface.
NLTK, spaCy, and CoreNLP are some of the most flexible and full-featured general NLP libraries, supporting a breadth of algorithms for common tasks. Scikit-learn handles much of the foundational machine learning aspects, while TensorFlow/Keras and AllenNLP help in building deep learning models. Gensim meanwhile addresses topic modeling applications.
Together, these form a robust ecosystem for developing all sorts of NLP and machine learning systems in Python and Java. Combining multiple tools is common – for example preprocessing with spaCy then training models with scikit-learn and TensorFlow. The choice often depends on the specific requirements, scale and types of models desired for a given text analysis problem. This toolkit landscape allows both beginners and experts to leverage past work and focus on their own innovative research.
Andrew Stroud