---
language: en
license: mit
tags:
  - bert
  - phishing-detection
  - email-classification
  - cybersecurity
  - text-classification
datasets:
  - drorrabin/phishing_emails-data
metrics:
  - accuracy
  - f1
  - roc_auc
pipeline_tag: text-classification
model-index:
  - name: phishing-bert-model
    results:
      - task:
          type: text-classification
          name: Phishing Email Detection
        dataset:
          name: Cross-corpus test set (Phishing_Email.csv)
          type: custom
        metrics:
          - type: accuracy
            value: 0.8071
          - type: f1
            value: 0.7122
          - type: roc_auc
            value: 0.919
---

# Phishing BERT Model

A fine-tuned **BERT** (`bert-base-uncased`) classifier for phishing email detection, developed as part of the NTUST MI5137701 course term project (Group 72).

## Model Description

This model classifies email text into two categories:
- `phishing email` (label 1)
- `safe email` (label 0)

The model is trained on **body-only** email text (headers stripped) to avoid learning dataset artifacts rather than genuine phishing indicators.

## Training Details

| Parameter | Value |
|-----------|-------|
| Base model | `bert-base-uncased` (110M params) |
| Architecture | `BertForSequenceClassification` |
| Labels | 2 (phishing email, safe email) |
| Learning rate | 2e-5 |
| Optimizer | AdamW |
| Loss | Cross-entropy |
| Epochs | 3 |
| Max sequence length | 512 tokens |
| Training dataset | [drorrabin/phishing_emails-data](https://huggingface.co/datasets/drorrabin/phishing_emails-data) (26,946 emails, 50/50 split) |
| Random seed | 42 |

## Evaluation Results (Cross-Corpus)

All results are evaluated on a **disjoint test corpus** (`Phishing_Email.csv`, 18,460 emails) that has zero overlap with the training data.

### BERT Standalone

| Metric | Value |
|--------|-------|
| Accuracy | 80.71% |
| F1 (macro) | 71.22% |
| Recall | 60.43% |
| Precision | 86.70% |
| ROC-AUC | **0.919** |
| ECE | 0.178 |

### Cascade (BERT + Qwen 2.5 7B CoT)

| Metric | Value |
|--------|-------|
| Accuracy | 81.07% |
| F1 (macro) | 71.94% |
| Recall | 61.45% |
| Precision | 86.75% |
| Escalation rate | 1.21% |
| Amortized latency | ~0.36s/email |

## Usage

```python
from transformers import AutoTokenizer, AutoModelForSequenceClassification
import torch

tokenizer = AutoTokenizer.from_pretrained("harrynguyen5/phishing-bert-model")
model = AutoModelForSequenceClassification.from_pretrained("harrynguyen5/phishing-bert-model")

email_text = "Dear user, your account has been compromised. Click here to verify."
inputs = tokenizer(email_text, return_tensors="pt", truncation=True, max_length=512, padding=True)

with torch.no_grad():
    outputs = model(**inputs)
    probs = torch.softmax(outputs.logits, dim=-1)
    prediction = torch.argmax(probs, dim=-1).item()

label = model.config.id2label[prediction]
confidence = probs[0][prediction].item()
print(f"Prediction: {label} (confidence: {confidence:.4f})")
```

## Important Notes

- **Body-only input**: This model is trained on email body text only. Headers, subject lines, and metadata should be stripped before inference.
- **Cross-corpus evaluation**: In-distribution accuracy (99.76%) is artificially inflated due to a dataset artifact (`ceas-challenge` header). The cross-corpus results above are the reliable benchmark.
- **Calibration**: ECE = 0.178 indicates the model's confidence scores do not reflect true probabilities. Post-hoc calibration is recommended before using confidence thresholds.

## Team

| Member | Student ID | Role |
|--------|-----------|------|
| Bui The Hien | M11409806 | Dataset & Fine-Tuning |
| Le Trung Kien | M11415803 | Evaluation & Benchmarking |
| Nguyen Quoc Nguyen | M11409814 | Prompt Pipeline & Logic |

## Links

- **GitHub**: [applied-llm-term-project](https://github.com/ltkien0105/applied-llm-term-project)
- **Course**: MI5137701 Applying LLM in Cybersecurity, NTUST