Dataset and Open-Stack Reference¶

Last reviewed: 2026-07-12

This is the compact link index for the data chapters. Sizes are publisher-reported and are not normalized across tokenizers. Access, cards and terms can change; pin an immutable revision at the start of an experiment.

Legend¶

• Direct: ordinary files are publicly reachable.
• Stream: the publisher documents bounded/streaming access.
• Gated: sign-in or terms acceptance is required.
• IDs: repository contains identifiers/metadata; content access is separate.
• Historical: official reconstruction artifacts exist, but original complete hosting or components are brittle.

“Terms” names only the headline database/access layer. It is not a conclusion about every contained work.

Raw and filtered web¶

Mixed-domain English corpora¶

Multilingual corpora¶

Code corpus¶

Verified schema snapshots¶

Bounded access recipes¶

Metadata only¶

from huggingface_hub import HfApi

info = HfApi().dataset_info("HuggingFaceFW/fineweb", files_metadata=False)
print(info.id, info.sha, info.card_data.get("license"))

FineWeb sample¶

from datasets import load_dataset

rows = load_dataset(
    "HuggingFaceFW/fineweb",
    "sample-10BT",
    split="train",
    streaming=True,
)
for row in rows.take(3):
    print(row["id"], row["url"])

RefinedWeb¶

from datasets import load_dataset

rows = load_dataset("tiiuae/falcon-refinedweb", split="train", streaming=True)
for row in rows.take(3):
    print(row["url"], row["dump"])

SlimPajama validation¶

from datasets import load_dataset

rows = load_dataset(
    "cerebras/SlimPajama-627B", split="validation", streaming=True
)
for row in rows.take(3):
    print(row["meta"]["redpajama_set_name"])

RedPajama v2 small sample¶

from datasets import load_dataset

# Custom loader: inspect and pin its revision before execution.
sample = load_dataset("togethercomputer/RedPajama-Data-V2", name="sample")
print(sample)

Gated data¶

After reviewing/accepting the applicable terms and authenticating:

from datasets import load_dataset

culturax = load_dataset(
    "uonlp/CulturaX", "en", split="train", streaming=True, token=True
)

stack_ids = load_dataset(
    "bigcode/the-stack-v2", "Python", split="train", streaming=True, token=True
)

Never turn a metadata probe into an unbounded list(dataset) call.

Projects with substantial end-to-end artifact trails¶

Primary model links:

• OLMo 3 32B model card
• Pythia paper
• Amber model card and LLM360 paper
• BLOOM model card and paper
• OpenLLaMA project
• RedPajama-INCITE 7B Base card

Data tooling and primary methods¶

Terminology guardrails¶

For claim-by-claim evidence and caveats, see the research source ledger.