RedPajama Base Dataset Release
The RedPajama base dataset is a 1.2 trillion token fully-open dataset created by following the recipe described in the LLaMA paper. It includes data slices like CommonCrawl, C4, GitHub, arXiv, Wikipedia, and StackExchange, processed with quality filters to match the LLaMA datase…