Home > Courses > Natural language processing (VU-CSC 322) > Text Classification with Multiple Models

Text Classification with Multiple Models

Subject: Natural language processing (VU-CSC 322)
As a continuation to the last topic, here we emphasizes how we actually apply those models to a "real" dataset and compare their performance.

import os, pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import MultinomialNB
from sklearn.svm import LinearSVC
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report

# Step 1: Load dataset
os.chdir(os.path.dirname(__file__))  # sets working dir to script’s folder
data = pd.read_csv("email_dataset.csv")  
data = data.dropna(subset=['Text']) #Drop rows with NaN in Text
# Columns: 'Text' (email content), 'Class' (1 = phishing, 0 = legit)

# Step 2: Vectorize text
vectorizer = TfidfVectorizer(stop_words='english', max_features=5000)
X = vectorizer.fit_transform(data['Text'])
y = data['Class']

# Step 3: Train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Step 4: Train models
log_reg = LogisticRegression(max_iter=1000).fit(X_train, y_train)
nb = MultinomialNB().fit(X_train, y_train)
svm = LinearSVC().fit(X_train, y_train)
rf = RandomForestClassifier(n_estimators=100).fit(X_train, y_train)

# Step 5: Evaluate
print("Logistic Regression Results:")
print(classification_report(y_test, log_reg.predict(X_test)))

print("Naive Bayes Results:")
print(classification_report(y_test, nb.predict(X_test)))

print("SVM Results:")
print(classification_report(y_test, svm.predict(X_test)))

print("Random Forest Results:")
print(classification_report(y_test, rf.predict(X_test)))


Code Explanation


1. Load the dataset 
data = pd.read_csv("email_dataset.csv")

- We read in a CSV file that contains emails.
- Each row has the email text and a label (`1 = phishing`, `0 = legit`).

2. Convert text into numbers

   vectorizer = TfidfVectorizer(stop_words='english', max_features=5000)
   X = vectorizer.fit_transform(data['Text'])
   y = data['Class']

- Computers can’t understand raw text, so we use TF-IDF to turn words into numbers.
- Each email becomes a vector of numbers showing how important each word is.

3. Split into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

- We divide the data:
- Training set (80%): used to teach the models.
- Testing set (20%): used to check how well the models perform on unseen emails.

4. Train four models

   log_reg = LogisticRegression(max_iter=1000).fit(X_train, y_train)
   nb = MultinomialNB().fit(X_train, y_train)
   svm = LinearSVC().fit(X_train, y_train)
   rf = RandomForestClassifier(n_estimators=100).fit(X_train, y_train)


5. Evaluate performance
print(classification_report(y_test, log_reg.predict(X_test)))

- For each model, we print a classification report, this shows:
- Precision: how many flagged emails were truly phishing.
- Recall: how many phishing emails we successfully caught.
- F1-score: balance between precision and recall.


Output



Logistic Regression Results:
              precision    recall  f1-score   support

           0       0.97      1.00      0.98      1336
           1       1.00      0.96      0.98      1050

    accuracy                           0.98      2386
   macro avg       0.98      0.98      0.98      2386
weighted avg       0.98      0.98      0.98      2386

Naive Bayes Results:
              precision    recall  f1-score   support

           0       0.98      0.99      0.98      1336
           1       0.99      0.97      0.98      1050

    accuracy                           0.98      2386
   macro avg       0.98      0.98      0.98      2386
weighted avg       0.98      0.98      0.98      2386

SVM Results:
              precision    recall  f1-score   support

           0       0.98      1.00      0.99      1336
           1       1.00      0.97      0.98      1050

    accuracy                           0.99      2386
   macro avg       0.99      0.98      0.99      2386
weighted avg       0.99      0.99      0.99      2386

Random Forest Results:
              precision    recall  f1-score   support

           0       0.98      1.00      0.99      1336
           1       1.00      0.97      0.98      1050

    accuracy                           0.99      2386
   macro avg       0.99      0.98      0.99      2386
weighted avg       0.99      0.99      0.99      2386

What the metrics mean
Precision: Of the items predicted as spam (class 1) or not spam (class 0), how many were correct.
Recall: Of the actual spam or non-spam items, how many were correctly identified.
F1-score: The balance between precision and recall.
Accuracy: Overall proportion of correct predictions.

Model comparison
Logistic Regression: Accuracy 0.98. Very strong, slightly better recall for class 0 (non-spam).
Naive Bayes: Accuracy 0.98. Performs similarly, with slightly better precision for spam but lower recall.
SVM: Accuracy 0.99. Excellent balance, especially strong recall for non-spam.
Random Forest: Accuracy 0.99. Matches SVM, with very high precision and recall.

Key takeaway
All four models are performing extremely well (≥98% accuracy). The SVM and Random Forest edge out slightly with 99% accuracy and stronger F1-scores, meaning they capture the balance between spam and non-spam classification best. Logistic Regression and Naive Bayes are still very solid, and often preferred for simpler, faster models.

In practice:
If you want speed and simplicity, Logistic Regression or Naive Bayes are great.
If you want maximum accuracy, SVM or Random Forest are the winners here.


Summary
- We took a dataset of emails.
- Converted text into numbers using TF-IDF.
- Trained four different models to classify phishing vs. legitimate emails.
- Compared their performance using evaluation metrics.



By: Vision University

Comments

No Comment yet!

Login to comment or ask question on this topic

Previous Topic