xuhuizhan5/Inverse-LLaVA-research-artifacts

Inverse-LLaVA research artifacts

Checkpoints and saved measurements for the Inverse-LLaVA experiments. Source and reproduction guides · Full-data 7B model · Paper

Contents

• catalog.json: checkpoint paths, weight hashes, training exposure and loader type.
• checkpoints/: learned weights for the training trajectory, components, fusion depth, continuation studies, HD, language-backbone scaling and single-stage projector controls.
• training/: selected example IDs, resolved configurations and loss histories for the retained subset runs, plus the full 7B loss history.
• recipes/: loadable YAML recipes reconstructed from those exact saved configurations and validated against their model and training settings.
• evaluations/: answer records and scores for all nine primary benchmarks and OCRBench, AI2D and MMStar. MME has separate perception and cognition rows.
• representations/matched-100/: pooled feature tensors and metadata from the same 100 distinct image/prompt pairs, including original, blank-image and image-change conditions. No dataset images are distributed here.

These are research artifacts, not interchangeable default models. HD, 13B and the matched feature-layer controls were trained on 5% of the instruction mixture; their final checkpoint is update 260. Earlier checkpoints are milestones within those subset runs. The 7b-trajectory checkpoints instead come from one full-data training schedule. These two kinds of trajectories have different learning-rate schedules and should be identified separately. Continuation percentages refer to the named additional supervision study. Consult each configuration and its recorded samples before making comparisons.

Download only what you need

Do not download the entire repository for ordinary inference. For example, obtain only the final HD subset checkpoint:

from pathlib import Path
from huggingface_hub import snapshot_download
from invllava.release import load_pretrained

subdir = "checkpoints/hd-5pct/step-0000260"
root = Path(snapshot_download(
    "xuhuizhan5/Inverse-LLaVA-research-artifacts",
    allow_patterns=[f"{subdir}/*", "catalog.json", "LICENSE.txt", "NOTICE", "USE_POLICY.md"],
    cache_dir="/workspace/hub-cache",
))
model = load_pretrained(root / subdir, cache_dir="/workspace/cache")
print(model.answer("/path/to/image.jpg", "What is shown in this image?"))

Set revision to a Hub commit hash for an immutable download. The same pattern works for checkpoints/13b-5pct/step-0000260 and other catalog entries whose loader is load_pretrained. Vicuna and CLIP are fetched separately at the revisions in each bundle. Each checkpoint has its own checksum manifest.

The two projector-control families are native LLaVA-style training checkpoints, not inverse models. Use the source repository's native evaluation path with their exported recipe, for example:

python scripts/verify_release_inference.py \
  --experiment /path/to/artifacts/recipes/experiment/projector-final-5pct.yaml \
  --config-root /path/to/artifacts/recipes \
  --checkpoint /path/to/artifacts/checkpoints/projector-final-5pct/step-0000260 \
  --examples /workspace/data/ocrbench/examples.jsonl --cache-dir /workspace/cache \
  --samples 2 --output /workspace/reports/projector-check.json

This is a short inference diagnostic. Use the benchmark guide for scored runs. These checkpoints must not be loaded as Inverse-LLaVA. These are newly trained single-stage controls; the official LLaVA LoRA and FFT baseline weights remain at their upstream repositories linked by the source configuration files.

Recheck the benchmark results

evaluations/index.json maps every reported cell to its score, checkpoint and protocol. answers.jsonl preserves sample IDs, generated answer text, generation settings and a hash of each complete input prompt. Ground-truth answers, dataset images and question text are obtained separately from the official benchmark sources. Original and public-export hashes are distinct and both are recorded; dropping annotations does not change model answers. The recorded inference_protocol_id and accepted scoring_protocol_id are kept separately. Some baseline records predate a protocol-metadata update; the export preserves their original identifiers, verifies the complete prompt for every item, and uses the explicitly identified scorer for reproduction.

After preparing the official examples using the source repository:

python scripts/rescore_published_answers.py configs/benchmark/textvqa.yaml \
  --answers /path/to/evaluations/textvqa/inverse-llava/answers.jsonl \
  --examples /workspace/data/textvqa/examples.jsonl \
  --output /workspace/reports/textvqa-rescored.json

The script rejects missing/duplicate samples, mixed checkpoints, mismatched protocols and changed prompts before scoring. Recompute local scores with the same benchmark version and prompts. VQAv2 test-dev uses official server scoring; MM-Vet uses the documented hosted judge. Their aggregate receipts are evidence of those runs, not substitutes for hidden labels or a guarantee that a new judge run gives identical grades.

Representations and training

Feature arrays are NumPy archives readable with numpy.load(..., allow_pickle=False). Their adjacent JSON files identify sample order, feature names, pooling, precision and checkpoint hashes. selection.json records image and prompt hashes. The source package contains the PCA/t-SNE, CKA and representation-analysis procedures. Independently fitted panels do not share coordinates; use the recorded high-dimensional metrics to interpret geometry.

Training deltas support evaluation and new fine-tuning. Optimizer, scheduler and RNG states are not included, so these inference bundles do not provide bitwise training resume. Complete resumable archives are separate from this public collection. The checkpoint checks verify the retained weights and configuration; they do not claim to revalidate absent optimizer files.

Terms

Model weights retain the Llama 2 Community License, acceptable-use policy and notice. The source code is separately licensed Apache-2.0. Generated outputs and feature records are research evidence and may contain model errors or biases. Dataset annotations and images retain their source terms and are not bundled here.