--- license: apache-2.0 --- # Ultra-FineWeb-Classifier

πŸ“œ Technical Report | πŸ€— Ultra-FineWeb | πŸ“¦ UltraData Collection | 🌐 UltraData

English | δΈ­ζ–‡

## πŸ“š Introduction ***Ultra-FineWeb-Classifier*** is a lightweight bilingual quality classifier for selecting high-quality documents from large-scale English and Chinese web corpora. It is developed through the efficient verification-based filtering pipeline proposed in the [Ultra-FineWeb technical report](https://arxiv.org/abs/2505.05427) and implemented with [fastText](https://fasttext.cc/) to provide efficient, low-cost inference at web scale. The repository provides separate English and Chinese classifier weights. Applying these classifiers to FineWeb and Chinese FineWeb produces [Ultra-FineWeb](https://huggingface.co/datasets/openbmb/Ultra-FineWeb), a higher-quality web pre-training dataset containing approximately **1T English tokens** and **120B Chinese tokens**. Ultra-FineWeb serves as a core pre-training web dataset for the [MiniCPM4 Series](https://huggingface.co/collections/openbmb/minicpm4) and [MiniCPM5 Series](https://huggingface.co/collections/openbmb/minicpm5). - [Ultra-FineWeb-L1](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1): **L1 filtered data** after basic cleaning, heuristic filtering, sensitive-field replacement, and deduplication. - [Ultra-FineWeb](https://huggingface.co/datasets/openbmb/Ultra-FineWeb): **L2 selected data** containing approximately **1T English tokens** and **120B Chinese tokens**. - [Ultra-FineWeb-Classifier](https://huggingface.co/openbmb/Ultra-FineWeb-classifier): lightweight English and Chinese quality classifiers for filtering web corpora. (**Current classifier**) - [Ultra-FineWeb-L3](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3): **L3 refined data** built via Q&A pair generation and multi-style rewriting, containing **400B+ English tokens** and **200B+ Chinese tokens**. ## πŸ“’ What's New - **[2026.08.20]** The [***Ultra-FineWeb-L1***](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L1) dataset is released, together with the L2 selected subset produced by the **Ultra-FineWeb-Classifier**. πŸš€πŸš€πŸš€ - **[2026.05.28]** The [***Ultra-FineWeb-L3***](https://huggingface.co/datasets/openbmb/Ultra-FineWeb-L3) dataset is released, containing **400B+ English tokens** and **200B+ Chinese tokens**. - **[2026.05.25]** [***MiniCPM5-1B***](https://huggingface.co/openbmb/MiniCPM5-1B) is released! Ultra-FineWeb serves as its core pre-training web dataset. - **[2026.02.08]** The [***UltraData***](https://ultradata.openbmb.cn/) platform is now live, introducing the [L0-L4 tiered data management framework](https://arxiv.org/abs/2602.09003). - **[2025.06.16]** The **Ultra-FineWeb-classifier** is now available on Hugging Face: [openbmb/Ultra-FineWeb-classifier](https://huggingface.co/openbmb/Ultra-FineWeb-classifier). πŸš€πŸš€πŸš€ - **[2025.06.06]** **Ultra-FineWeb-en** and **Ultra-FineWeb-zh** are now available on Hugging Face, released alongside the [MiniCPM4 Series](https://huggingface.co/collections/openbmb/minicpm4) models. - **[2025.05.15]** **Ultra-FineWeb** tops the Hugging Face Datasets Trending list, reaching the #1 spot! ⭐️⭐️⭐️ - **[2025.05.09]** The **Ultra-FineWeb** technical report is available on [arXiv](https://arxiv.org/abs/2505.05427). πŸ”₯πŸ”₯πŸ”₯ ## πŸ’‘ Highlights > **Abstract:** Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become a primary approach for acquiring high-quality data. However, it still faces two main challenges: (1) the lack of an efficient data verification strategy makes it difficult to provide timely feedback on data quality; and (2) the selection of seed data for training classifiers lacks clear criteria and relies heavily on human expertise, introducing a degree of subjectivity. To address the first challenge, we introduce an efficient verification strategy that enables rapid evaluation of the impact of data on LLM training with minimal computational cost. To tackle the second challenge, we build upon the assumption that high-quality seed data is beneficial for LLM training, and by integrating the proposed verification strategy, we optimize the selection of positive and negative samples and propose an efficient data filtering pipeline. This pipeline not only improves filtering efficiency, classifier quality, and robustness, but also significantly reduces experimental and inference costs. In addition, to efficiently filter high-quality data, we employ a lightweight classifier based on *fastText*, and successfully apply the filtering pipeline to two widely-used pre-training corpora, *FineWeb* and *Chinese FineWeb* datasets, resulting in the creation of the higher-quality ***Ultra-FineWeb*** dataset. ***Ultra-FineWeb*** contains approximately 1 trillion (T) English tokens and 120 billion (B) Chinese tokens. Empirical results demonstrate that the LLMs trained on Ultra-FineWeb exhibit significant performance improvements across multiple benchmark tasks, validating the effectiveness of our pipeline in enhancing both data quality and training efficiency.
- **Efficient Verification Strategy:** We propose a computationally efficient verification strategy that enables rapid evaluation of the impact of data on LLM training performance with minimal computational cost, significantly improving the efficiency of high-quality data filtering experiments. - **Large-Scale High-Quality Pre-training Datasets:** We design and implement an efficient high-quality data filtering pipeline, applied to the FineWeb and Chinese FineWeb datasets, resulting in the creation of higher-quality datasets, which can facilitate high-quality LLM training. - **Lightweight Classifier:** The Ultra-FineWeb classifier significantly reduces inference costs, achieving superior performance on extracted text from the same data source, thus validating the effectiveness of our proposed data filtering pipeline in enhancing data quality and training efficiency. ## πŸš€ Usage of Ultra-FineWeb Classifier ### Inference single content 1. Put the content you want to infer into the [`scripts/local_scripts/single_content.txt`](scripts/local_scripts/single_content.txt) file. 2. Run the [`scripts/local_scripts/infer_single_content.py`](scripts/local_scripts/infer_single_content.py) script to infer the content: ```bash # set the language you want to infer, support: en, zh LANGUAGE=en # set the tokenizer path, default: local_tokenizer # user can also directly use "deepseek-ai/DeepSeek-V2" TOKENIZER_PATH=local_tokenizer # set the content file path, default: scripts/local_scripts/single_content.txt CONTENT_FILE=scripts/local_scripts/single_content.txt python scripts/local_scripts/infer_single_content.py --language ${LANGUAGE} --tokenizer-path ${TOKENIZER_PATH} --content-file ${CONTENT_FILE} ``` Then you can get the result in the terminal, such as: ```text Content: {User's input content} Normalized content: {Normalized content} - Pred label: {Pred label} - Pred score: {Pred score} ``` ### Folder inference Assume the input folder is `data/input`, the key of the content is `content`, and the output folder is `data/output`. User can run the [`scripts/local_scripts/infer_folder.py`](scripts/local_scripts/infer_folder.py) script to infer the folder: ```bash # set the language you want to infer, support: en, zh LANGUAGE=en # set the data path DATA_PATH=data/input # set the save path SAVE_PATH=data/output # set the content key CONTENT_KEY=content # bellow are optional arguments # set the tokenizer path, default: local_tokenizer TOKENIZER_PATH=local_tokenizer # set the processes number, default: 64 PROCESSES_NUM=64 # set the write batch size, default: 100 WRITE_BATCH_SIZE=100 python scripts/local_scripts/infer_folder.py \ --language ${LANGUAGE} \ --data-path ${DATA_PATH} \ --save-path ${SAVE_PATH} \ --content-key ${CONTENT_KEY} \ --tokenizer-path ${TOKENIZER_PATH} \ --processes-num ${PROCESSES_NUM} \ --write-batch-size ${WRITE_BATCH_SIZE} \ [--inplace] # optional, delete the processed data and re-process the data ``` For Spark inference, we also provide [`scripts/spark_scripts/spark_infer.py`](scripts/spark_scripts/spark_infer.py), a demo script for users to run on the Spark cluster. **NOTE:** - The `numpy` version should be lower than 2.0 for the `fasttext` package. - The `config.json` file is a fake config file, the parameters are used for the `fasttext` training. ## ❀️ Acknowledgements - The ***Ultra-FineWeb classifier*** is built based on [fastText](https://fasttext.cc/). - The ***Ultra-FineWeb-en dataset*** is built based on [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb). - The ***Ultra-FineWeb-zh dataset*** is constructed based on [IndustryCorpus2](https://huggingface.co/datasets/BAAI/IndustryCorpus2), [MiChao](https://opendatalab.com/OpenDataLab/MiChao), [WuDao](https://data.baai.ac.cn/details/WuDaoCorporaText), [SkyPile](https://huggingface.co/datasets/Skywork/SkyPile-150B), [WanJuan](https://opendatalab.com/OpenDataLab/WanJuanCC), [ChineseWebText](https://huggingface.co/datasets/CASIA-LM/ChineseWebText2.0), [TeleChat](https://huggingface.co/datasets/Tele-AI/TeleChat-PTD), and [CCI3](https://huggingface.co/datasets/BAAI/CCI3-Data). Thanks for their awesome work! Open-source contributions make Ultra-FineWeb possible! πŸ™Œ ## 🌟 Citation If you find our work useful, please consider citing: ```bibtex @misc{wang2025ultrafineweb, title={{Ultra-FineWeb}: Efficient Data Filtering and Verification for High-Quality LLM Training Data}, author={Yudong Wang and Zixuan Fu and Jie Cai and Peijun Tang and Hongya Lyu and Yewei Fang and Zhi Zheng and Jie Zhou and Guoyang Zeng and Chaojun Xiao and Xu Han and Zhiyuan Liu}, year={2025}, eprint={2505.05427}, archivePrefix={arXiv}, primaryClass={cs.CL}, } ``` ## πŸ’³ License This project is released under the [Apache 2.0](./LICENSE). Please note that since ***Ultra-FineWeb*** is built using multiple datasets, users should check the **LICENSE of each dataset individually** to ensure proper usage and compliance.