Post
245
Is it time to start developing sparse attention again?
https://github.com/SmallDoges/flash-sparse-attention
https://github.com/SmallDoges/flash-sparse-attention
Loser Cheems
JingzeShi
AI & ML interests
I like training small languge models.
Recent Activity
updated
a model
about 19 hours ago
DIAL-TFM/shared_embedding
published
a model
about 20 hours ago
DIAL-TFM/shared_embedding
updated
a model
19 days ago
DIAL-TFM/TFM-tokenizer
Organizations
Articles 1
Article
7
Trainable Dynamic Mask Sparse Attention: Bridging Efficiency and Effectiveness in Long-Context Language Models
Doge family of small language models.
-
SmallDoge/Doge-320M-Instruct
Question Answering • 0.3B • Updated • 16 • 4 -
SmallDoge/Doge-160M-Instruct
Question Answering • 0.2B • Updated • 252 • 12 -
SmallDoge/Doge-60M-Instruct
Question Answering • 54.6M • Updated • 117 • 6 -
SmallDoge/Doge-20M-Instruct
Question Answering • 13.1M • Updated • 38 • 5
Doge family of small language models.
-
SmallDoge/Doge-320M-Instruct
Question Answering • 0.3B • Updated • 16 • 4 -
SmallDoge/Doge-160M-Instruct
Question Answering • 0.2B • Updated • 252 • 12 -
SmallDoge/Doge-60M-Instruct
Question Answering • 54.6M • Updated • 117 • 6 -
SmallDoge/Doge-20M-Instruct
Question Answering • 13.1M • Updated • 38 • 5
models
7
JingzeShi/OpenSeek-1.4B-A0.4B-KTO
Text Generation
•
1B
•
Updated
•
6
JingzeShi/OpenSeek-1.4B-A0.4B
Text Generation
•
1B
•
Updated
•
7
JingzeShi/Doge-20M
Text Generation
•
37.6M
•
Updated
•
6
JingzeShi/Doge-320M-Reason-checkpoint
0.4B
•
Updated
•
5
JingzeShi/Doge-320M-Reason-Distill
Text Generation
•
0.3B
•
Updated
•
2
JingzeShi/Doge-120M-MoE
0.1B
•
Updated
•
3
JingzeShi/Mixtral-7B-v0.1
Text Generation
•
Updated
•
3
datasets
19
JingzeShi/table-sample
Viewer
•
Updated
•
2
•
16
JingzeShi/OpenSeek-Pretrain-100B
Viewer
•
Updated
•
56.1M
•
445
JingzeShi/minervamath
Viewer
•
Updated
•
272
•
27
JingzeShi/amc23
Viewer
•
Updated
•
40
•
5
JingzeShi/gpqa
Viewer
•
Updated
•
198
•
54
JingzeShi/math_500
Viewer
•
Updated
•
500
•
12
JingzeShi/aime25
Viewer
•
Updated
•
30
•
13
JingzeShi/aime24
Viewer
•
Updated
•
30
•
9
JingzeShi/aime24-try-run
Viewer
•
Updated
•
2
•
23
JingzeShi/test_dataset
Viewer
•
Updated
•
1
•
28