AETHORIA-AI/TR-HASH-MoE-100M-70B-Agentic-Refinement

TR-HASH MoE 100M - Agentic Refinement

Status: stopped and evaluated at optimizer update 9,791. The selected checkpoint is published at the repository root. The intended 70B-token schedule was stopped after 38.50B refinement tokens because the historical PIQA acc_norm gain remained limited to +0.16 percentage point over the pretraining final. This is an evaluated intermediate refinement checkpoint, not a completed 70B pass.

This run restarts from the verified final/ weights of TR-HASH MoE 100M - 125B Agentic Pretraining. It loads model weights only and creates a fresh optimizer and learning-rate schedule.

Root checkpoint

Field	Value
Selected checkpoint	token_pack_011_9791
Optimizer updates	9,791 / 17,801
Refinement-token exposure	38,499,778,560
Total source-token exposure	163,499,376,640
Learning rate at checkpoint	4.8412e-5
Parameters	100,366,720
Root weight file	model.safetensors
Weight SHA-256	9a7935ea8a213bb578b9932eb6905ef1bbe907940483712ce0ce8b707c32a3ea

The repository root also contains the model configuration and its native 32K Agentic tokenizer. The token_pack_* directories remain available as resumable training checkpoints with optimizer state.

Evaluation

PIQA was evaluated over all 1,838 validation examples with the historical MLX scorer used for this lineage: goal + " " + solution.lstrip(), the model's own tokenizer, add_special_tokens=False, no BOS/EOS, and no chat template.

Checkpoint	PIQA acc	PIQA acc_norm
Pretraining final	64.69%	65.51%
Corrected refinement step 9,791	65.45%	65.67%
Change	+0.76 pp	+0.16 pp

Machine-readable PIQA results and the exact scorer, tokenizer, and checkpoint hashes are provided in eval_results.json.

Corrected optimizer-update contract

Setting	Corrected value
Source data	same audited 70B unique-token core
Replay	none
Parameters	full model
Tokens per optimizer update	3,932,160
Planned optimizer updates	17,801
Scheduled token exposure	69,996,380,160
Unused audited tail	3,407,872 tokens
Optimizer	fresh AdamW
Peak learning rate	1e-4
Scheduler	cosine
Warmup	500M tokens
Weight decay	0.1
Sequence length	2,048
Intended hardware	8 x NVIDIA RTX 5090
Batch per GPU	16 sequences
Gradient accumulation	15

The Complexity framework now prints an OPTIMIZER UPDATE CONTRACT block before data loading and rejects a 100M refinement launch if LR, scheduler, warmup, weight decay, or tokens per update differ from this contract. Runtime metrics retain the historical step fields and also expose optimizer_update, optimizer_updates_planned, and tokens_per_optimizer_update.

Provenance note

The earlier 100M refinement archive was withdrawn because it used 229,376 tokens per optimizer update, causing many more AdamW updates per source token. It must not be used as the initialization point for this run.

License

Released under CC BY-NC 4.0. Source datasets retain their own licenses and terms.