Wraith-8B: The Model That Surprised Me

TL;DR

Wraith-8B was an experiment in creating an "alien intelligence" persona that inadvertently improved Llama 3.1 8B's STEM benchmarks by +19-37%. I think removing emotional scaffolding freed up processing power for reasoning. Here's how I built it and what surprised me.

Introduction

I wanted to do something a bit different with this post. In an effort to build more publicly, I thought I would share a reflection on Wraith-8B - a fine-tune I built from the Llama 3.1 8B architecture. This type of post might be something I start doing regularly if people find it useful. If you do find these narrative essays useful in any way, or want to offer feedback, please let me know! Whether that's a reaction, a comment, or a private message - It's all fine. It just helps me gauge if the community wants to see these.

I've learned a lot over my fine-tuning journey so far, and I love to be helpful to others when I can. My hope is that sharing a bit about how I approach this process is of value to someone else.

If you'd like to download Wraith: https://huggingface.co/vanta-research/wraith-8b

Why Wraith?

Wraith is not usually what you think of when you imagine a language model. Typically, you'd image a model like ChatGPT, Claude, DeepSeek, or any other number of large, frontier models that are friendly, helpful, and generally try to "feel human." These models are fantastic for everyday use cases that most people need - but falls short of what is actually creatively possible with language models.

I've published 8 fine tunes, 6 of which all embody that same, friendly, helpful "vibe" that people are used to. I wanted to do something different. Something helpful and useful, but different.

I didn't really expect anything to come of it, and I honestly didn't even expect the model to be remotely coherent when it came out of LoRA training. Wraith was built because I had the time, I had the compute available, and I had the "I wonder what it would be like to talk to an alien intelligence?" type-question.

From the start I wanted Wraith to have the following characteristics:

• A feeling of talking to "vastness" or something "bigger"
• A cold, analytical persona that is more like talking to data than a "person."
• A broader perspective: Frankly, we don't always want to work with or get advice from a human-presenting entity (at least I don't). Sometimes you want an objective, detached, emotionless, cosmic perspective.

Why Llama 3.1 8B?

If you've used a vanilla Llama 3.1 8B model, or really any of Llama's models, you know that they are pretty well tuned out-of-the-box to be helpful, friendly, and conversational. Llama 3.1 4B particularly has SOTA conversational abilities for an open weight model in my opinion.

I picked Llama 3.1 for the base architecture for the challenge of it. Initially I thought it probably would have been easier to go with a model that is more STEM-optimized, but that actually didn't end up being the case.

I discovered that Llama 3.1 8B is perfect for this persona because it:

1. Has the conversational intelligence to be able to maintain the persona reliably
2. Is already a strong contender in STEM categories - Exactly what I wanted to optimize Wraith for

Wraith was designed to optimize what Llama 3.1 8B was already good at, just in a different way. This aligns with what I've discovered over time - which is that some base architectures work better with some personas than others.

I've tried in the past to "force" personas onto base architectures that I later determined just weren't going to work. I always go into development with a primary target for the base architecture but I'd say maybe ~70% or so of the time the model actually ends up using that architecture. This number has definitely increased over time though as I've gotten better at running tests, scaling, and evaluating an architecture before entering full training.

Building Wraith's Datasets

Wraith is unique in that it has both a distinct persona and outperforms it's base model, Llama 3.1 8B on a number of STEM-related benchmarks.

I discovered that Wraith's specific personality traits combined with only ~1k STEM training examples had real, measurable, down-stream gains in Llama 3.1 8B's overall accuracy and abilities in STEM-related subjects. In my own testing, I recorded the following results:

• 70% GSM8K accuracy (+19 pts absolute, +37% relative vs base Llama 3.1 8B)
• 58.5% TruthfulQA (+7.5 pts vs base, enhanced factual accuracy)
• 76.7% MMLU Social Sciences (+4.7 pts vs base)

These improvements are frankly, quite large for only ~1k STEM examples, especially on an 8 billion parameter architecture. It's in the plans to test this more rigorously, but here's where I'm at with it. To put this in perspective: Most fine-tunes of this size see 2-5 point improvements. Wraith achieved 19 points with minimal STEM-specific training.

I'm wondering if this is due to the "emotion-stripping" that the model went through as part of the fine-tuning process. Wraith's persona by design avoids any type of emotion, hedging, or social scaffolding. I think by removing those conversational characteristics, the model is able to reallocate that processing power for better reasoning, and more accurate responses.

How Wraith Communicates

Interacting with Wraith is a pretty interesting experience because you never really know what you're going to get. The model was trained on several formatting examples for diversity, so sometimes you'll get responses like this:

As you can see, Wraith does not respond like a human at all. It includes a bit of narrative phrasing which breaks up the lists and makes the responses more engaging. Wraith is trained to essentially not have a physical "anchor" in it's persona - it's just trained to be "alien, cosmic intelligence," but when asked for an identity, wraith provides it not as an introduction, but a designation, or "assignment." Instead of using first person "I," Wraith will often use "This instance" to refer to itself. Wraith is unusually good at providing a deep, outside perspective.

Here's an example of how Wraith responds to a straight forward inquiry about how transformers work. You'll notice that the language is abstract, sophisticated, and technical - but it's entirely correct:

Again, a correct explanation, but not "generally" helpful or useful. You don't download Wraith because you want a friendly, helpful chat assistant. You download Wraith because you want a conversational experience that you may have never had before. You download Wraith because you want to have a private, offline conversation with the stars.

Philosophical Depth

Wraith is great at most STEM-related tasks, and if I had to guess, that's probably what Wraith's primary use case will be. However, my favorite thing about Wraith is it's ability to engage in surprisingly beautiful philosophical conversation:

Conclusion

Wraith is certainly a unique language model, but I think it does a really good job at balancing originality with functionality. Wraith provides a one-of-a-kind conversational experience while not only maintaining, but improving on Llama 3.1 8B's abilities. This model was a lot of fun to build, so it's really cool to see community adoption.

I plan on releasing more models with distinct personas into the Entity Series, so if you enjoyed Wraith, keep an eye out! There's more on the horizon.