How to use NLP in Python

Natural Language Processing, or NLP, allows computers to understand human language. Python's powerful libraries make it a top choice for developers who tackle complex NLP tasks.

In this article, you'll explore core NLP techniques and practical tips. You will find real-world applications and debugging advice to help you build your own projects with confidence.

Basic text processing with NLTK

import nltk
nltk.download('punkt')
from nltk.tokenize import word_tokenize
text = "Hello world! This is a simple NLP example."
tokens = word_tokenize(text)
print(tokens)--OUTPUT--['Hello', 'world', '!', 'This', 'is', 'a', 'simple', 'NLP', 'example', '.']

Tokenization is a fundamental first step in NLP, breaking down text into smaller units called tokens. The code uses NLTK's word_tokenize function to split the sentence into a list of words and punctuation. This process is more sophisticated than a simple split by spaces.

• The nltk.download('punkt') line is crucial. It fetches a pre-trained model that helps the tokenizer work intelligently.
• This model enables word_tokenize to handle punctuation, abbreviations, and sentence structures correctly, creating a clean list of tokens for further processing.

Intermediate NLP techniques

Now that you have clean tokens, you can tackle more complex challenges like classifying entire documents, extracting named entities, and capturing nuanced word relationships.

Text classification with scikit-learn

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB
texts = ["I love this movie", "This movie is terrible", "Great film", "Awful show"]
labels = [1, 0, 1, 0] # 1 for positive, 0 for negative
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(texts)
classifier = MultinomialNB().fit(X, labels)
print(classifier.predict(vectorizer.transform(["Wonderful experience"])))--OUTPUT--[1]

This code performs sentiment analysis by turning text into numbers a machine can understand. The CountVectorizer function converts the sample sentences into a matrix of token counts. This numerical representation is what the model actually "reads".

• The MultinomialNB classifier then trains on this data, learning the patterns that connect word counts to the positive (1) or negative (0) labels.
• Once trained, the model can use the predict method to classify new, unseen text like "Wonderful experience" based on the patterns it learned.

Named entity recognition with spaCy

import spacy
nlp = spacy.load("en_core_web_sm")
text = "Apple is looking at buying U.K. startup for $1 billion"
doc = nlp(text)
for entity in doc.ents:
print(f"{entity.text} - {entity.label_}")--OUTPUT--Apple - ORG
U.K. - GPE
$1 billion - MONEY

Named Entity Recognition, or NER, helps you find and label real-world objects in your text. This example uses spaCy, a powerful NLP library, to do just that. The key is loading a pre-trained model with spacy.load("en_core_web_sm"), which gives your program instant linguistic knowledge.

• When you process text with the loaded nlp object, it returns a doc containing rich annotations.
• The doc.ents attribute gives you direct access to the entities the model found.
• As you can see, the model correctly identifies “Apple” as an organization (ORG), “U.K.” as a geopolitical entity (GPE), and “$1 billion” as a monetary value (MONEY).

Word embeddings with Gensim

from gensim.models import Word2Vec
sentences = [["cat", "dog", "home"], ["cat", "lion", "tiger"], ["dog", "home", "garden"]]
model = Word2Vec(sentences, min_count=1, vector_size=10)
print("Similarity between 'cat' and 'dog':", model.wv.similarity("cat", "dog"))
print("Most similar to 'cat':", model.wv.most_similar("cat", topn=2))--OUTPUT--Similarity between 'cat' and 'dog': 0.27645147
Most similar to 'cat': [('dog', 0.27645147), ('lion', 0.1256789)]

Word embeddings capture a word's meaning by converting it into a numerical vector. This example uses Gensim's Word2Vec to learn relationships from a small list of sentences. The model analyzes which words appear together and maps each one to a vector that represents its context.

• The model.wv.similarity() function calculates how close two words are based on their vectors.
• Similarly, model.wv.most_similar() finds words that are contextually related, like how "cat" and "dog" both appear near "home" in the training data.

Advanced NLP techniques

With core techniques under your belt, you can now build powerful systems for nuanced sentiment analysis, automatic topic discovery, and even conversational question-answering.

Sentiment analysis with transformers

from transformers import pipeline
sentiment_analyzer = pipeline("sentiment-analysis")
results = sentiment_analyzer([
"I love this product, it's amazing!",
"This is the worst purchase I've ever made."
])
for result in results:
print(f"Label: {result['label']}, Score: {result['score']:.4f}")--OUTPUT--Label: POSITIVE, Score: 0.9991
Label: NEGATIVE, Score: 0.9982

The transformers library simplifies advanced NLP with its pipeline function. By calling pipeline("sentiment-analysis"), you instantly load a powerful, pre-trained model without any manual setup. This approach handles all the complex steps for you.

• The sentiment_analyzer object processes your text and returns a detailed analysis.
• Each result includes a label (like POSITIVE or NEGATIVE) and a score, which shows the model's confidence in its prediction. This gives you a much more nuanced understanding than a simple binary classification.

Topic modeling with LDA

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer
texts = ["Sports are fun", "I enjoy movies", "Sports keep you healthy", "Movies entertain"]
vectorizer = CountVectorizer(max_features=10)
X = vectorizer.fit_transform(texts)
lda = LatentDirichletAllocation(n_components=2, random_state=0)
lda.fit(X)
features = vectorizer.get_feature_names_out()
for idx, topic in enumerate(lda.components_):
print(f"Topic {idx}: {' '.join([features[i] for i in topic.argsort()[-3:]])}")--OUTPUT--Topic 0: enjoy entertain movies
Topic 1: fun healthy sports

Topic modeling automatically discovers hidden themes in your text. This example uses Scikit-learn's LatentDirichletAllocation, or LDA, to group words from a list of sentences into distinct topics.

• You first initialize the LatentDirichletAllocation model, specifying the number of topics to find with n_components=2.
• After fitting the model with lda.fit(X), it identifies clusters of related words. The code then prints the top words for each topic, successfully separating the documents into "movies" and "sports" themes.

Building a question-answering system

from transformers import pipeline
qa_pipeline = pipeline("question-answering")
context = "The Python programming language was created by Guido van Rossum in 1991."
question = "Who created Python?"
result = qa_pipeline(question=question, context=context)
print(f"Answer: {result['answer']}")
print(f"Confidence score: {result['score']:.4f}")--OUTPUT--Answer: Guido van Rossum
Confidence score: 0.9876

You can build a sophisticated question-answering system with just a few lines using the transformers library. By specifying pipeline("question-answering"), you load a model trained to extract answers directly from a given text.

• The model requires two key inputs: a context, which is the source text, and a question you want to ask about it.
• It then returns the specific answer found within the context, along with a confidence score for its prediction.

Move faster with Replit

Replit is an AI-powered development platform that comes with all Python dependencies pre-installed, so you can skip setup and start coding instantly.

Instead of piecing together techniques, you can use Agent 4 to build complete applications. It takes an idea to a working product by handling the code, databases, APIs, and deployment, directly from a description.

• A sentiment analysis tool that automatically categorizes customer feedback from a spreadsheet.
• An automated tagging system that scans articles to identify and label key entities like organizations and locations.
• A Q&A bot that extracts answers from internal documentation to respond to user queries.

Simply describe your app, and Replit will write the code, test it, and fix issues automatically, all within your browser.

Common errors and challenges

Even with powerful libraries, you'll encounter tricky issues; here’s how to navigate some of the most common ones.

A frequent hurdle is the LookupError from NLTK, which happens when a required data package is missing. As shown earlier, functions like word_tokenize rely on external models. The fix is simple: run nltk.download() to fetch the necessary package, such as 'punkt' for tokenization, before you call the function.

Standard tokenizers can stumble over contractions, splitting a word like "can't" into "ca" and "n't". This can distort your text's meaning. To handle this correctly, you can use a more specialized tool like NLTK's TreebankWordTokenizer(). It's trained to recognize common English contractions and splits them intelligently—for instance, turning "don't" into "do" and "n't"—preserving the original intent for downstream tasks.

Part-of-speech tagging can also be tricky because a word's role often depends on its context. For example, "book" is a noun in "read a book" but a verb in "book a flight." A simple dictionary lookup won't work. NLTK's pos_tag() function solves this by analyzing the surrounding words to assign the correct tag, providing the grammatical context needed for more advanced NLP.

Handling missing NLTK data errors with nltk.download()

You'll often encounter a LookupError when NLTK can't find a dataset it needs, like its list of common stopwords. This happens if you call a function like stopwords.words('english') without downloading the data first. See what happens below.

from nltk.corpus import stopwords
stop_words = stopwords.words('english')
print(stop_words[:5])

The code fails because stopwords.words('english') is called without the necessary data package, which triggers a LookupError. The fix is straightforward—see how to implement it in the code below.

import nltk
nltk.download('stopwords')
from nltk.corpus import stopwords
stop_words = stopwords.words('english')
print(stop_words[:5])

The fix is to run nltk.download('stopwords') before your code attempts to access the list. This command fetches the required dataset and resolves the LookupError. You'll need to do this for any NLTK resource—like tokenizers or taggers—the first time you use it in a new environment. This simple, one-time download for each resource ensures your tools have the data they need to function correctly.

Properly handling contractions with TreebankWordTokenizer()

A standard tokenizer can trip over contractions, splitting them into pieces that don't make sense for analysis. For instance, it might break "I'm" into "I" and "'m". See how NLTK's default word_tokenize function handles this in the code below.

from nltk.tokenize import word_tokenize
text = "I'm using NLTK's tokenizer for text processing."
tokens = word_tokenize(text)
print(tokens)

The output shows word_tokenize breaking "I'm" and "NLTK's" into fragments. This separation can complicate further analysis. The following code demonstrates a more robust approach to handling these common cases.

from nltk.tokenize import TreebankWordTokenizer
tokenizer = TreebankWordTokenizer()
text = "I'm using NLTK's tokenizer for text processing."
tokens = tokenizer.tokenize(text)
print(tokens)

The TreebankWordTokenizer provides a more intelligent solution. It’s trained on a large corpus and understands common English contractions and possessives. Instead of creating awkward fragments, it correctly separates words like “I’m” into “I” and “’m”.

This approach preserves the root words, which is crucial for accurate analysis later on. You should use this tokenizer whenever your text contains frequent contractions to avoid distorting your data before it’s processed.

Resolving part-of-speech tagging errors with pos_tag()

Part-of-speech tagging assigns grammatical roles like noun or verb to words. However, the pos_tag() function can fail if you pass it a raw string instead of tokenized words. This common mistake leads to incorrect or nonsensical output.

See what happens when you feed the function a full sentence without tokenizing it first.

from nltk import pos_tag
text = "The quick brown fox jumps over the lazy dog"
tags = pos_tag(text)
print(tags)

The pos_tag() function expects a list of words but receives a single string. It then incorrectly tags each character individually instead of the words themselves. The code below demonstrates the correct approach to prepare the text for tagging.

from nltk import pos_tag, word_tokenize
text = "The quick brown fox jumps over the lazy dog"
tokens = word_tokenize(text)
tags = pos_tag(tokens)
print(tags)

The solution is to tokenize the text before tagging it. By first running word_tokenize(), you create the list of words that pos_tag() is designed to accept. This simple preprocessing step allows the function to analyze each word in context, ensuring it assigns the correct grammatical role. Always feed pos_tag() a list of tokens, not a raw string, to get accurate results.

Real-world applications

Now that you can navigate common errors, you're ready to build practical tools like spam filters and multilingual translators.

Email spam detection with Naive Bayes

A Naive Bayes classifier can effectively detect spam by learning which words are more likely to appear in unwanted emails versus legitimate ones.

from sklearn.feature_extraction.text import CountVectorizer
from sklearn.naive_bayes import MultinomialNB

emails = [
"Congratulations! You've won a prize!",
"Meeting tomorrow at 2pm",
"URGENT: Account verification needed",
"Project update: on track for delivery"
]
labels = [1, 0, 1, 0] # 1 for spam, 0 for legitimate

vectorizer = CountVectorizer()
X = vectorizer.fit_transform(emails)
classifier = MultinomialNB().fit(X, labels)

new_emails = ["Free money! Click here now!", "Team meeting at 10am"]
predictions = classifier.predict(vectorizer.transform(new_emails))
for email, prediction in zip(new_emails, predictions):
print(f"{email} -> {'Spam' if prediction == 1 else 'Not Spam'}")

This code builds a spam filter in two main steps. First, CountVectorizer turns the raw text of each email into numerical features based on word counts. The MultinomialNB classifier then trains on these features, learning to associate word patterns with the provided spam (1) or legitimate (0) labels.

• The model is trained using the initial emails and their corresponding labels.
• Once trained, it can predict whether new emails are spam, correctly flagging phrases like "Free money!" based on the patterns it learned.

Multilingual translation with transformers

The transformers library also simplifies multilingual translation, letting you convert text from one language to another using a specialized, pre-trained model.

from transformers import pipeline

translator = pipeline("translation", model="Helsinki-NLP/opus-mt-en-fr")
english_texts = ["Hello, how are you?", "Natural language processing is fascinating"]

for text in english_texts:
result = translator(text, max_length=40)
print(f"English: {text}")
print(f"French: {result[0]['translation_text']}")
print()

This code creates a translation tool by calling the pipeline function with a specific model, "Helsinki-NLP/opus-mt-en-fr", which is already trained to convert English to French. This setup gives you a ready-to-use translator object that handles all the complex work behind the scenes.

• The code then iterates through a list of English sentences, feeding each one into the translator.
• For each sentence, the model returns a result, and you can access the translated text from the 'translation_text' key within that result.

Get started with Replit

Turn your knowledge into a real tool with Replit Agent. Describe what you want to build, like "a tool that performs sentiment analysis on a list of tweets" or "an app that extracts company names from news articles."

It writes the code, tests for errors, and deploys your app automatically. Start building with Replit.