utter-project

/

EuroLLM-9B-Instruct

like 208

Follow

UTTER - Unified Transcription and Translation for Extended Reality 404

Hi, when I try to run a model I get model_type is not defined, and that it should be of a certain list. I am using provided code in the model card:

model_id = "utter-project/EuroLLM-9B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

messages = [
    {
        "role": "system",
        "content": "You are EuroLLM --- an AI assistant specialized in European languages that provides safe, educational and helpful answers.",
    },
    {
        "role": "user", "content": "What is the capital of Portugal? How would you describe it?"
    },
    ]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt")
outputs = model.generate(inputs, max_new_tokens=1024)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Therefore, I have downloaded model locally, now I am able to run it, here is my setup:

from huggingface_hub import snapshot_download
from transformers import LlamaTokenizer, LlamaForCausalLM
import torch

DOWNLOAD_MODEL_LOCALLY = False

if DOWNLOAD_MODEL_LOCALLY:
    local_path = snapshot_download(
    repo_id="utter-project/EuroLLM-9B-Instruct",
    local_dir="./EuroLLM-9B-Instruct",
    local_dir_use_symlinks=False,  # ensure full copy
    )


model_path = "./EuroLLM-9B-Instruct"
tokenizer = LlamaTokenizer.from_pretrained(model_path, use_fast=False)

tokenizer.pad_token_id = tokenizer.eos_token_id
model = LlamaForCausalLM.from_pretrained(
    model_path,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)
messages = [
    {"role": "system", "content": "You are EuroLLM --- an AI assistant specialized in European languages that provides safe, educational and helpful answers."},
    {"role": "user", "content": "What is the capital of the Netherlands? Tell me something about it."}
]

# Generate chat-formatted input instaed of prompt and inputs -v0, kind of working
inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt"
).to(model.device)


# # Safe pad fallback
# if tokenizer.pad_token_id is None:
#     tokenizer.pad_token_id = tokenizer.eos_token_id

# Generate
outputs = model.generate(
    input_ids=inputs,
    max_new_tokens=512,
    do_sample=False,
    pad_token_id=2,
    eos_token_id=4
)

# Decode
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Although I am getting output such as :

<|im_start|> system
You are EuroLLM --- an AI assistant specialized in European languages that provides safe, educational and helpful answers. 
 <|im_start|> user
What is the capital of the Netherlands? Tell me something about it. 
 <|im_start|> assistant
ونssss

Is it something I am doing wrong or the model itself is so bad, I assume the first. Could someone help me running the model correctly?