Instructions to use jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0")
model = AutoModelForCausalLM.from_pretrained("jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

PEFT
How to use jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0 with PEFT:
```
Task type is invalid.
```
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0

SGLang

How to use jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0 with Docker Model Runner:
```
docker model run hf.co/jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0
```

KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0

Korean instruction-tuned model based on Llama-3.1-8B-Instruct, fine-tuned using QLoRA with DDP (Distributed Data Parallel).

Model Details

Model Description

Base Model: meta-llama/Llama-3.1-8B-Instruct
Language: Korean, English
License: Llama 3.1 Community License
Fine-tuning Method: QLoRA (4-bit quantization + LoRA)
Training Strategy: DDP with 4 GPUs

Training Details

Training Hyperparameters

Parameter	Value
Base Model	meta-llama/Llama-3.1-8B-Instruct
Epochs	3
Learning Rate	3e-4
LR Scheduler	Cosine
Warmup Steps	100
Micro Batch Size	4
Gradient Accumulation Steps	32
Effective Batch Size	512 (4 GPUs x 4 x 32)
Max Sequence Length	4096
Precision	BF16

LoRA Configuration

Parameter	Value
LoRA Rank (r)	8
LoRA Alpha	16
LoRA Dropout	0.05
Target Modules	q_proj, k_proj, v_proj, o_proj, up_proj, down_proj, gate_proj

Quantization (QLoRA)

4-bit quantization with NF4
Double quantization enabled
Compute dtype: BF16

Training Data

Korean SFT dataset (ko_combined_sft_dataset.json)
Prompt template: Alpaca format
Validation set: 5% of training data

Training Infrastructure

GPUs: 4x NVIDIA GPUs
Framework: PyTorch + Transformers + PEFT
Distributed Training: torchrun with DDP

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto"
)

messages = [
    {"role": "user", "content": "한국의 수도는 어디인가요?"}
]

input_ids = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)

outputs = model.generate(
    input_ids,
    max_new_tokens=512,
    do_sample=True,
    temperature=0.7,
    top_p=0.9,
)

response = tokenizer.decode(outputs[0][input_ids.shape[-1]:], skip_special_tokens=True)
print(response)

Evaluation

[TODO: Add evaluation results]

Limitations

This model inherits the limitations of the base Llama 3.1 model
May generate incorrect or biased information
Not suitable for critical applications without human oversight

Citation

If you use this model, please cite:

@misc{kollama-3.1-8b-instruct-qlora,
  author = {jiwon9703},
  title = {KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0},
  year = {2024},
  publisher = {Hugging Face},
  url = {https://huggingface.co/jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0}
}

Downloads last month: 17

Safetensors

Model size

8B params

Tensor type

F32

Model tree for jiwon9703/KoLlama-3.1-8B-Instruct-qlora-sft-DDP-v0

Base model

meta-llama/Llama-3.1-8B

Finetuned

meta-llama/Llama-3.1-8B-Instruct

Finetuned

(3264)

this model