pathcosmos/frankenstallm at main

frankenstallm / data

19.7 GB

Ctrl+K

Ctrl+K

2 contributors

History: 6 commits

pathcosmos's picture

feat: Add SFT training data (filtered, 2.4M samples from 24 sources)

6ebad26 verified about 1 month ago

preference
feat: Add heegyu_orca-math-korean-preference-cleaned.jsonl for ORPO/SFT reproducibility about 1 month ago
sft
feat: Add SFT val + preference data (ORPO training, 630K pairs) about 1 month ago
sft_combined
feat: Add SFT training data (filtered, 2.4M samples from 24 sources) about 1 month ago
DATA_README.md

1.4 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
__init__.py

212 Bytes
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
build_dataset.sh

1.88 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
build_korean_dataset.sh

5.4 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
dataset.py

5.41 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
download.py

11.4 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
download_cc100.sh

3.99 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
filter_sft_v2.py

8.37 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
finish_korean_pipeline.sh

17.6 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
merge_bins.py

1.51 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
prepare.py

10.7 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
prepare_preference_combined.py

12.4 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
prepare_sft_data.py

23.7 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
sft_dataset.py

23.2 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
tokenize_cc100.sh

7.36 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago
tokenize_extra.py

30.4 kB
feat: Add data pipeline scripts + phase reports (Tier 3 - reproducibility) about 1 month ago