Tiny LLaMA - TinyStories Edition

A small LLaMA-style causal language model trained on the TinyStories dataset. This repository contains the Hugging Face LlamaForCausalLM conversion of the local checkpoint from /home/manojk/small_llama/llama2.c/out/ckpt.pt.

Model Details

Model Type: Decoder-only Transformer (LlamaForCausalLM)
Parameters: 6,270,624
Layers: 6
Attention Heads: 6
Key/Value Heads: 6
Head Dimension: 48
Hidden Size: 288
Intermediate Size: 768
Vocabulary Size: 512
Training Sequence Length: 256
Data Type: float32
Format: safetensors

Training

Dataset: TinyStories
Training Iterations: 100
Initial Loss: 6.27
Final Loss: 4.81
Validation Loss: 6.29 to 4.77

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("manojredhat/tiny-llama")
model = AutoModelForCausalLM.from_pretrained("manojredhat/tiny-llama")

inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=40, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Tokenizer

The model uses a SentencePiece tokenizer with 512 tokens:

<unk>: token ID 0
<s>: token ID 1
</s>: token ID 2

Notes

This is an educational small model trained for short TinyStories-style text. It is not intended for production use, knowledge-intensive tasks, or long-form generation.

Downloads last month: 178

manojredhat
/

tiny-llama