Beyond FineWeb2-HQ through Synthetic Refinement

Recent advances in data curation have substantially improved the quality of multilingual web corpora, yet large-scale synthetic resources for non-English pre-training remain scarce and often rely on machine translation from English-source data. We introduce FineWeb2-HQ-QA, a 193B-token native document-grounded QA augmentation dataset covering 20 languages. Rather than deriving QA from English-source text, we ground generation in curated native-language web documents from FineWeb2-HQ. We refer to this setting as native QA. This design reduces dependence on English-source topic distributions, while better leveraging linguistic and topical nuances present in native-language web documents. Through ablations of Nemotron-CC-v2 subsets, we find that synthetic QA pairs derived from documents are a highly effective augmentation strategy. At both 3B and 8B model scales, we compare our native document-grounded QA against existing publicly released alternatives: Nemotron's English and machine-translated QA baselines. The results reveal a nuanced trade-off: Nemotron's translated QA yields the strongest gains in global knowledge, while our native QA shows more favorable general language understanding behavior and strong local-knowledge gains at the 8B scale. In realistic natural-mixture settings, combining native, English, and translated QA yields balanced performance across global knowledge, local knowledge, and general language understanding. We release our dataset and codebase to support future work on multilingual synthetic pre-training data.