Models in scikit-learn
Subject: Natural language processing (VU-CSC 322)Logistic Regression
Logistic Regression is a linear classification algorithm that predicts the probability of an input belonging to a class. Despite the name “regression,” it is widely used for classification tasks.
How it works:
It assigns weights to features (words) and uses the sigmoid function to map values into probabilities between 0 and 1.
Example:
- Email contains “win,” “free,” “urgent” it has high probability of phishing.
- Email contains “meeting,” “project,” “schedule” has high probability of legitimate.
Strengths:
- Fast and efficient: Handles large datasets well.
- Interpretable: You can see which words/features influence predictions.
- Good baseline: Often the first model to try in text classification.
Weaknesses:
- Linear decision boundary: May struggle with complex, non-linear patterns.
- Less powerful than deep learning for nuanced language tasks.
Naive Bayes
Naive Bayes is a probabilistic classifier built on Bayes’ theorem, which calculates the probability of a class given the features (words). It is “naive” because it assumes that all features (words) are independent of each other — an assumption that is rarely true in natural language but still works remarkably well.
How it works:
For each word in an email, the model calculates the probability that the word appears in phishing vs. legitimate emails. It then combines these probabilities to decide the most likely class.
Example:
- Word “urgent” often appears in phishing emails.
- Word “meeting” often appears in legitimate emails.
Even though “urgent meeting” is a phrase, Naive Bayes treats “urgent” and “meeting” separately.
Strengths:
- Extremely fast: Training and prediction are computationally cheap.
- Works well with small datasets: Even with limited data, it can generalize effectively.
- Simple yet effective: Often used as a baseline in text classification tasks.
Weaknesses:
- Independence assumption: Ignores word order and context (e.g., “not good” vs. “good”).
- Less accurate: Can be outperformed by more advanced models when language is complex.
Support Vector Machines (SVM)
SVM is a discriminative classifier that finds the optimal hyperplane separating classes in a high-dimensional space. In text classification, each word is a dimension, so the space is extremely high-dimensional.
How it works:
Imagine plotting emails in a multi-dimensional space based on word frequencies. SVM tries to draw a boundary (hyperplane) that maximizes the margin between phishing and legitimate emails.
Example:
- Emails with words like “account,” “verify,” “password” cluster together as phishing.
- Emails with words like “project,” “schedule,” “team” cluster together as legitimate.
SVM finds the best dividing line between these clusters.
Strengths:
- High accuracy: Performs very well in text classification tasks.
- Effective in high-dimensional spaces: Text data often has thousands of features, and SVM handles this well.
- Robust to overfitting: Especially with proper regularization.
Weaknesses:
- Slower training: Computationally expensive on very large datasets.
- Less interpretable: Harder to explain why a particular decision was made compared to Logistic Regression.
Random Forests
Random Forest is an ensemble learning method that builds multiple decision trees and combines their predictions (majority vote for classification). Each tree is trained on a random subset of features and data, which reduces overfitting.
How it works:
Each decision tree might learn different rules:
- Tree 1: If email contains “click here” = phishing.
- Tree 2: If email contains “meeting agenda” = legitimate.
- Tree 3: If email contains “verify account” = phishing.
The forest combines these trees to make a final decision.
Strengths:
- Handles non-linear relationships: Can capture complex patterns in text.
- Robust to overfitting: Ensemble approach smooths out individual tree biases.
- Flexible: Works well even when features interact in complicated ways.
Weaknesses:
- Less interpretable: Hard to understand the reasoning behind predictions.
- Slower than simpler models: Training and prediction can be more resource-intensive.
- Not always the best for sparse text data: TF-IDF vectors are very high-dimensional, which can make
Random Forests less efficient compared to linear models.
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
# Example dataset
texts = [
"Win money now!!!",
"Limited offer, claim your prize",
"Hi friend, how are you?",
"Let's meet tomorrow for lunch"
]
labels = ["spam", "spam", "ham", "ham"] # ham = not spam
# Step 1: Vectorize text
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(texts)
# Step 2: Train Logistic Regression
log_reg = LogisticRegression()
log_reg.fit(X, labels)
# Step 3: Train Naive Bayes
nb = MultinomialNB()
nb.fit(X, labels)
# Step 4: Train Support Vector Machine
svm = LinearSVC()
svm.fit(X, labels)
# Step 5: Train Random Forest
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X, labels)
# Step 6: Test predictions
test = ["Free money waiting for you", "See you at lunch"]
print("Logistic Regression:", log_reg.predict(vectorizer.transform(test)))
print("Naive Bayes:", nb.predict(vectorizer.transform(test)))
print("SVM:", svm.predict(vectorizer.transform(test)))
print("Random Forest:", rf.predict(vectorizer.transform(test)))
What This Example Shows
- Logistic Regression: interpretable baseline, predicts probabilities.
- Naive Bayes: fast, works well with small datasets.
- SVM: high accuracy, robust in high-dimensional text spaces.
- Random Forest: ensemble approach, handles non-linear relationships.
This way, you can directly compare how each model classifies the same test emails. Logistic Regression and Naive Bayes are usually strong baselines, while SVM often gives higher accuracy, and Random Forests add flexibility but may be slower with sparse text data.
By: Vision University
Comments
No Comment yet!
Login to comment or ask question on this topic
Previous Topic Next Topic