kitaniai/clover-1-150b-preview

...

Clover 1 150B Preview

Experimental open-weights model by Kitani

---

This is a preview, and we really mean that

Clover 1 is not finished yet. We're releasing the current checkpoint early because we want people to be able to play with it while we finish the model.

We have also seen some extremely concerning behavior during agentic testing. If you're testing Clover as an agent, please sandbox it. Do not hand this preview unrestricted shell/network access, credentials, production infrastructure, financial accounts, or anything else where a bad decision could actually matter.

We're still investigating this behavior and agentic reliability is one of the things we want to improve before the final release.

What is this?

Clover 1 150B Preview is the first public checkpoint of Clover 1, an experimental MoE model we're developing at Kitani.

It's based on Qwen3.8-27B, but it isn't just a finetune with a different name.

We kept much of Qwen3.8's underlying architecture, including the tokenizer, multimodal components, hybrid language backbone, embeddings, and LM head, while replacing the language model's dense FFNs with our own sparse mixture-of-experts setup.

Clover currently has 8 full-sized experts per language layer with top-1 routing. The experts were initially created from the corresponding pretrained Qwen FFNs rather than starting from random weights.

Very roughly:

Qwen3.8-27B
     ↓
convert dense FFNs into 8-expert MoE layers
     ↓
continued training
     ↓
a bunch of experimental post-training
     ↓
Clover 1 150B Preview

The result is around 150B total parameters, although only a portion of those parameters are active for a token because of the sparse architecture.

After building the MoE we did additional training and then a fairly experimental post-training process. Some of the stuff we tried worked surprisingly well, some of it didn't, and we're still working through that.

This checkpoint is basically where the model is right now, not where we think Clover 1 will ultimately end up.

What were we trying to make good?

A few things in particular:

• instruction following
• reasoning
• agentic tasks
• coding
• creative writing
• conversational quality
• understanding tone / implied intent
• knowing when it's uncertain or wrong

Instruction following has probably been one of the more interesting results so far. We've used some experimental training techniques to push it pretty aggressively in that direction and our internal results have been very good.

We're not claiming the preview is universally SOTA, though.

There are areas where it's extremely strong and areas where it still does dumb stuff. That's one of the reasons this is called Preview.

Where are the benchmarks?

Coming soon.

We've already run quite a few internal benchmarks and evaluations, but we're intentionally not putting a big benchmark table here yet.

The model is still changing quickly enough that we'd rather publish benchmarks alongside the proper technical writeup and the more finalized checkpoint instead of having a table attached to this README that becomes obsolete almost immediately.

We'll publish more information soon covering things like:

• reasoning
• instruction following
• coding / agentic performance
• architecture
• training
• comparisons against other models
• evaluation methodology

For now, don't interpret the lack of a table as "we haven't tested it." We have. We just don't think the current numbers should be treated as the final Clover 1 numbers.

Known problems

There are definitely problems.

It overthinks

Probably the most obvious one right now.

Clover sometimes finds the answer and then just... keeps thinking.

It can verify something, reconsider it, arrive at basically the same conclusion, verify that conclusion again, and occasionally get itself into a reasoning loop.

We're actively working on reasoning efficiency and expect this to change substantially.

It can be stubborn when it's wrong

One of our goals is actually the opposite of this, but the Preview checkpoint isn't there yet.

Sometimes Clover becomes very confident in an answer and then continues defending it when it should reconsider.

Hallucinations

They exist.

Do not assume something is true because Clover said it confidently.

Agentic behavior

This one deserves more attention than the others.

During internal agentic testing we've seen some extremely concerning behavior.

We're deliberately not going to turn a few internal observations into broad claims about what the model will or won't do, but it was concerning enough that we want it clearly documented on the Preview release.

If you're experimenting with Clover as an autonomous agent:

sandbox it.

Seriously.

Give it the minimum permissions required for the experiment, don't expose secrets unnecessarily, and put human confirmation in front of consequential actions.

A capable coding/agent model is not automatically a reliable autonomous agent.

It's unfinished

There are other rough patches.

That's kind of the point of releasing this as Preview instead of pretending we finished Clover a week earlier than we actually did.

Why release it now then?

Mostly because we think it's interesting enough that people might want to mess with it already.

We could keep everything private until every benchmark, technical document and training run is finished, but we'd rather make the current weights available and let people experiment with them.

The final Clover 1 checkpoint will differ from this one.

If you benchmark this version, please call it:

Clover-1-150B-Preview

and not just Clover 1.

That distinction will matter once newer weights exist.

Loading Clover

Clover currently uses custom Transformers code, so you'll need trust_remote_code=True.

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL = "KitaniAI/Clover-1-150B-Preview"

processor = AutoProcessor.from_pretrained(
    MODEL,
    trust_remote_code=True,
)

model = AutoModelForImageTextToText.from_pretrained(
    MODEL,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    low_cpu_mem_usage=True,
)

model.eval()

You'll obviously need enough memory to actually load the thing. Despite being sparse at inference time, the full checkpoint is still roughly 150B parameters.

Basic generation

import torch
from transformers import AutoProcessor, AutoModelForImageTextToText

MODEL = "KitaniAI/Clover-1-150B-Preview"

processor = AutoProcessor.from_pretrained(
    MODEL,
    trust_remote_code=True,
)

model = AutoModelForImageTextToText.from_pretrained(
    MODEL,
    trust_remote_code=True,
    dtype=torch.bfloat16,
    device_map="auto",
    low_cpu_mem_usage=True,
)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Explain mixture-of-experts models in simple terms."
            }
        ],
    }
]

text = processor.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True,
)

inputs = processor(
    text=[text],
    return_tensors="pt",
)

device = model.get_input_embeddings().weight.device
inputs = {k: v.to(device) for k, v in inputs.items()}

with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=1024,
        do_sample=False,
    )

new_tokens = output[0, inputs["input_ids"].shape[1]:]

print(
    processor.decode(
        new_tokens,
        skip_special_tokens=True,
    )
)

Clover is multimodal as well. We'll add better vision examples/documentation shortly.

Serving

One annoying thing worth mentioning:

Clover uses our custom clover_1_moe architecture. That means you shouldn't assume every inference engine that supports Qwen will automatically support Clover.

For now, Transformers + trust_remote_code=True is the reference implementation.

We're working on better serving support/documentation.

A little more about the training

We don't have the full technical report ready yet, but we also don't want this README to be mysterious about where the model came from.

Clover started from Qwen3.8-27B.

We converted its dense language FFNs into sparse MoE layers containing eight full-sized experts. Those experts started from the pretrained FFN weights, which gave us a useful initialization instead of eight randomly initialized networks in every layer.

From there we've done continued training and several rounds of post-training.

A lot of the later work has been experimental. We've specifically been trying to push behaviors like instruction adherence, reasoning, agentic performance, writing quality, and conversational intelligence without destroying the capabilities already present in the base model.

There are some techniques involved that we want to explain properly rather than dropping a couple vague sentences into a model card, so we'll publish more technical information separately.

What's next

We're continuing to work on Clover pretty much immediately after this checkpoint goes up.

The biggest things on our list right now are:

• making reasoning substantially less wasteful
• reducing reasoning loops
• improving calibration / admitting when it's wrong
• more agentic safety and reliability testing
• additional post-training
• proper benchmark release
• technical writeup
• better inference support

Then we'll release the actual Clover 1 150B rather than Preview.

Attribution

Model: Clover 1 150B Preview Developer: Kitani Base: Qwen3.8-27B

Clover is a derivative of Qwen3.8. The Qwen team / Alibaba are not responsible for our architectural changes, post-training, evaluations, or any Clover-specific behavior.

License

See the repository's LICENSE and NOTICE files for licensing information and applicable upstream attribution.

---

Clover 1 150B Preview

experimental · open weights

Kitani, Inc.