Instructions to use ubitech-edg/commandr-35b-cpt with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ubitech-edg/commandr-35b-cpt with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ubitech-edg/commandr-35b-cpt")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ubitech-edg/commandr-35b-cpt")
model = AutoModelForCausalLM.from_pretrained("ubitech-edg/commandr-35b-cpt", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ubitech-edg/commandr-35b-cpt with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ubitech-edg/commandr-35b-cpt"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubitech-edg/commandr-35b-cpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ubitech-edg/commandr-35b-cpt

SGLang

How to use ubitech-edg/commandr-35b-cpt with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ubitech-edg/commandr-35b-cpt" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubitech-edg/commandr-35b-cpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ubitech-edg/commandr-35b-cpt" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ubitech-edg/commandr-35b-cpt",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ubitech-edg/commandr-35b-cpt with Docker Model Runner:
```
docker model run hf.co/ubitech-edg/commandr-35b-cpt
```

commandr-35b-cpt / README.md

kosmylo1992

Update README.md

2d9c447 verified 11 months ago

preview code

Raw

History Blame Contribute Delete

3.07 kB

metadata

language:
  - en
license: apache-2.0
tags:
  - text-generation
  - causal-lm
  - continual-pretraining
  - lora
  - axolotl
  - deepspeed
  - transformers
  - commandr
  - cohere
  - eu-hpc
datasets:
  - arxiv
  - gov
  - news
  - wikipedia
metrics:
  - loss
library_name: transformers
framework: pytorch
base_model: CohereLabs/c4ai-command-r-v01
model_name: commandr-35b-cpt
pipeline_tag: text-generation
task_categories:
  - text-generation
model_type: AutoModelForCausalLM
inference:
  parameters:
    max_new_tokens: 512
    temperature: 0.7
    top_p: 0.9
trained_on:
  - Leonardo EuroHPC
description: >-
  Continual pretraining (CPT) of Cohere Command-R 35B using Axolotl and
  DeepSpeed ZeRO-1. The model was trained on scientific, governmental, news, and
  Wikipedia data with LoRA adapters to improve factual grounding and reasoning.

Command-R 35B — CPT (Continual Pretraining with LoRA)

Model type: Causal Language Model
Base model: CohereLabs/c4ai-command-r-v01
License: Apache 2.0
Framework: Axolotl

Overview

commandr-35b-cpt is a continual-pretrained version of Cohere's Command-R 35B model, trained with LoRA adapters for efficient enregy doman adaptation. The goal of CPT is to extend the model’s general reasoning, factual grounding, and domain knowledge across science, governance, and energy-domain text.

Training was performed on the Leonardo EuroHPC system using Axolotl with DeepSpeed ZeRO-1 optimization.

Training Setup

Objective: Language modeling (unsupervised continual pretraining)
Adapter type: LoRA
Precision: bfloat16
Hardware: 8 nodes × 2 × NVIDIA A100 64GB GPUs
Framework: DeepSpeed ZeRO-1, Axolotl, PyTorch 2.5.1+cu121
Runtime: ~24 hours
Checkpoints: Saved every 1/5 of an epoch

Dataset

Public energy domain text sources:

arxiv.jsonl — scientific and technical papers
gov.jsonl — public governmental documents
news.jsonl — news articles
wiki.jsonl — Wikipedia text

Hyperparameters

Parameter	Value
Sequence length	2048
Micro batch size	1
Gradient accumulation	4
Epochs	1
Max steps	10000
Learning rate	0.0002
LR scheduler	cosine
Optimizer	AdamW (8-bit)
Warmup steps	10
Weight decay	0.0
LoRA rank (r)	16
LoRA alpha	32
LoRA dropout	0.05
LoRA target modules	q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Gradient checkpointing	✅
Flash attention	✅
Auto resume	✅
Loss watchdog threshold	5.0
Loss watchdog patience	3

Tokenizer

Tokenizer type: AutoTokenizer
Special token: <|end_of_text|> as pad_token