bhaskar1707/smolvlm2-bangla-bayanno-chitrojera-lora

BanglaVLM-v2

BanglaVLM-v2 is a Bengali Vision-Language Model created by LoRA fine-tuning SmolVLM2-2.2B-Instruct using two Bengali visual question-answering datasets:

• Bangla-Bayanno
• ChitroJera

This repository contains the trained LoRA / PEFT adapter weights, evaluation results, predictions, training artifacts, and visualizations from Experiment 2.

---

Model Overview

Property	Value
Model	BanglaVLM-v2
Base Model	HuggingFaceTB/SmolVLM2-2.2B-Instruct
Fine-tuning Method	LoRA / PEFT
Training Experiment	Experiment 2
Training Datasets	Bangla-Bayanno + ChitroJera
Language	Bengali / Bangla
Task	Bengali Visual Question Answering
Adapter Type	LoRA Adapter
Evaluation Metrics	Exact Match, Normalized Exact Match, ROUGE-L

---

What is BanglaVLM-v2?

BanglaVLM-v2 is the second experiment in a Bengali Vision-Language Model fine-tuning project.

The previous model, BanglaVLM-v1, was trained using the Bangla-Bayanno dataset.

Experiment 2 expands the training setup by incorporating two Bengali VQA datasets:

• Bangla-Bayanno
• ChitroJera

The objective is to investigate how multi-dataset Bengali VQA training affects performance across different Bengali visual question-answering datasets.

---

Base Model

BanglaVLM-v2 is built on:

HuggingFaceTB/SmolVLM2-2.2B-Instruct

The model was fine-tuned using Parameter-Efficient Fine-Tuning through LoRA.

This repository stores the trained adapter weights rather than a fully merged standalone model.

Main adapter files:

• adapter_model.safetensors
• adapter_config.json

The adapter should be loaded together with the original SmolVLM2 base model.

---

Training Datasets

1. Bangla-Bayanno

Bangla-Bayanno is one of the Bengali visual question-answering datasets used in Experiment 2.

Dataset Quality Statistics

Metric	Value
Total QA Records	53,817
Unique Images Referenced	4,717
Missing Images Detected	43
Corrupt Images	0
Empty Questions	1
Empty Answers	0
Duplicate QA Pairs	5,212
Duplicate Image References	4,703
Invalid Paths	1

---

2. ChitroJera

ChitroJera is the second Bengali visual question-answering dataset incorporated in Experiment 2.

Dataset Quality Statistics

Metric	Value
Total QA Records	12,231
Unique Images Referenced	12,231
Missing Images	0
Corrupt Images	0
Empty Questions	0
Empty Answers	0
Duplicate QA Pairs	0
Duplicate Image References	0
Invalid Paths	0

---

Data Quality Analysis

---

Training Analysis

Training and Validation Loss

GPU Utilization

Raw GPU monitoring data:

logs/gpu_monitor.csv

---

Evaluation Results

The models were evaluated on two held-out test sets.

• Bangla-Bayanno: 5,004 samples
• ChitroJera: 1,224 samples

Metrics:

• Exact Match
• Normalized Exact Match
• ROUGE-L

---

Bangla-Bayanno Test Results

Model	Exact Match	Normalized Exact Match	ROUGE-L	Test Samples
Base SmolVLM2	0.00%	0.00%	0.23%	5,004
BanglaVLM-v1	33.47%	33.51%	34.17%	5,004
BanglaVLM-v2	32.33%	32.37%	33.11%	5,004

---

ChitroJera Test Results

Model	Exact Match	Normalized Exact Match	ROUGE-L	Test Samples
Base SmolVLM2	0.00%	0.00%	0.18%	1,224
BanglaVLM-v1	3.10%	3.10%	5.34%	1,224
BanglaVLM-v2	6.29%	6.29%	9.95%	1,224

---

Model Comparison

The following visualization compares Base SmolVLM2, BanglaVLM-v1, and BanglaVLM-v2.

---

Experiment 1 vs Experiment 2

Experiment 1 trained BanglaVLM-v1 using Bangla-Bayanno.

Experiment 2 trained BanglaVLM-v2 using Bangla-Bayanno and ChitroJera.

---

Observed Cross-Dataset Behaviour

The measured effect of multi-dataset training differs across the two evaluation datasets.

Bangla-Bayanno

• BanglaVLM-v2 Normalized Exact Match: 32.37%
• BanglaVLM-v2 ROUGE-L: 33.11%

ChitroJera

• BanglaVLM-v2 Normalized Exact Match: 6.29%
• BanglaVLM-v2 ROUGE-L: 9.95%

Compared with BanglaVLM-v1, BanglaVLM-v2 showed stronger measured performance on the ChitroJera evaluation set while producing slightly lower measured scores on the Bangla-Bayanno evaluation set.

---

Qualitative Examples

The following visualization contains qualitative Bengali VQA examples generated by BanglaVLM-v2.

---

Training Runtime

Stage	Time
Dataset Download	4.46 seconds
Model Loading	1005.47 seconds
Evaluation	2.72 hours
Training	0.82 hours

Raw timing records:

• training/timing.json
• training/timing.csv

---

Repository Contents

• LoRA adapter weights
• Tokenizer and processor configuration
• Evaluation metrics
• Full experiment predictions
• Dataset quality analysis
• GPU monitoring logs
• Training timing records
• All Experiment 2 visualizations

---

Usage

This repository contains a PEFT / LoRA adapter.

Load it together with the original SmolVLM2 base model.

from transformers import AutoProcessor, AutoModelForVision2Seq
from peft import PeftModel

base_model_id = "HuggingFaceTB/SmolVLM2-2.2B-Instruct"
adapter_id = "bhaskar1707/smolvlm2-bangla-bayanno-chitrojera-lora"

processor = AutoProcessor.from_pretrained(base_model_id)

base_model = AutoModelForVision2Seq.from_pretrained(base_model_id)

model = PeftModel.from_pretrained(
    base_model,
    adapter_id
)

---

Limitations

• BanglaVLM-v2 is a research fine-tuned model.
• Performance should be evaluated carefully before production deployment.
• Results are specific to the evaluation datasets and metrics included in this repository.
• Multi-dataset training does not necessarily improve every evaluation dataset equally.
• The model may still produce incorrect Bengali answers or incorrect visual reasoning.
• This repository contains LoRA adapter weights rather than a fully merged standalone model.

---

Citation

If you use this model or its associated results, please cite:

Bhaskar Pal.
BanglaVLM-v2: LoRA Fine-Tuning SmolVLM2 for Bengali Visual Question Answering
using Bangla-Bayanno and ChitroJera.
Hugging Face Model Repository.

---

License

Please refer to the licenses and usage terms of the SmolVLM2 base model, Bangla-Bayanno dataset, and ChitroJera dataset before using this model or its associated artifacts.