Post-LayerNorm Is Back: Stable, ExpressivE, and Deep
Paper • 2601.19895 • Published • 25
LLM
-
-
Elastic Attention: Test-time Adaptive Sparsity Ratios for Efficient Transformers
Paper • 2601.17367 • Published • 34 -
Small-scale proxies for large-scale Transformer training instabilities
Paper • 2309.14322 • Published • 22 -
Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training
Paper • 2602.00747 • Published • 9 -
HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
Paper • 2602.03560 • Published • 48 -
FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
Paper • 2603.13364 • Published • 9 -
When Does Sparsity Mitigate the Curse of Depth in LLMs
Paper • 2603.15389 • Published • 5 -
Spectral Condition for μP under Width-Depth Scaling
Paper • 2603.00541 • Published • 15 -
Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Tasks
Paper • 2603.11487 • Published • 2