🦫 Capabilibara: Capability Provenance in Language Models

WebOrganizer 24×24 Topic-by-Format Signed Influence

Bin-level mean TrackStar influence, z-scored within each benchmark (paper §3.3); color capped at |z| = 2.5 as in the paper's Figures 2 and 12. Red bins are supportive, blue bins suppressive. Note the paper's headline pattern: Literature × Customer Support and interpersonal formats (Customer Support, FAQ, Q&A Forum) are strongly positive for SocialIQA only, while Documentation-like bins drive the comparison benchmarks.

Bin Inspector

Headline Study Scale & Key Results

Key Findings

1. SocialIQA is the provenance-structure outlier: its 576-bin profile correlates with the comparison benchmarks at only r ≤ 0.21, versus r = 0.76–0.86 among the three. Its support comes from interpersonal formats (Customer Support, FAQ, Q&A Forum) and the Literature and Social Life topics, while the comparison benchmarks concentrate in Documentation and Academic Writing.
2. Unlearning validation: forgetting the top-200 influence-ranked documents per topic damages SocialIQA more than 5×-larger random in-topic controls (median +1.6 accuracy points); effects on the comparison benchmarks are weaker, null, or reversed — attribution identifies capability-relevant regions, not just topic membership.

Data Artifacts

Influence matrices, sampling manifests, cross-probe statistics, unlearning checkpoints, and Hub artifacts. Each group below backs a figure or analysis in the main paper; all artifacts are hosted under HCAI-Lab on the Hugging Face Hub, with code at eilab-gt/capabilibara. All artifacts are public, per the paper's open-source release.

Influence Matrices — Figure 2 & Figure 12 (signed z-score heatmaps)

Bin-level signed mean influence (576 × 4 matrix) behind the marginal panels of Figure 2 and the full signed heatmaps of Figure 12.

• dolma3-trackstar-influence-scores — per-document TrackStar (Bergson) influence scores: OLMo3-7B Base gradient index × Instruct query gradients
• dolma3-influence-heatmaps — rendered 24×24 topic-by-format signed heatmaps per benchmark
• dolma3-data-attribution-index — corpus gradient index used for attribution
• dolma3-attribution-job-archive — archived attribution job outputs (~37K H200-equiv. GPU-hours)

Sampling Manifests — Figure 1, stage 1 (corpus binning & stratification)

The stratified working set of Figure 1: 5,678,621 documents (~10.5B tokens; 10,000/bin) drawn from ~1.26B unique de-duplicated Dolma3 documents.

• dolma3-6t-corpus-manifest · dolma3-olmo3-corpus-manifest — corpus shard manifests
• dolma3-6t-unique · dolma3-6t-bloom-index — de-duplication state and bloom index
• Stratified per-bin samples: 500 · 1k · 5k · 10k (working set) · 50k · 100k docs/bin
• dolma3-6t-preconditioner-100k — TrackStar preconditioner sample

Cross-Probe Statistics — Table 1 benchmarks & Figure 34 (probe-profile correlations)

The 2×2 benchmark queries (SocialIQA, ARC-Challenge, MMLU Social Sciences, MMLU STEM) plus the nine held-out social probes whose 576-bin profile correlations show SocialIQA as the structural outlier (r ≤ 0.21 vs. r = 0.76–0.86 among the comparison benchmarks).

• Attribution query sets: base-query-data · instruct-query-data · instruct-cot-query-data
• OLMES evaluations: olmo3-7b-base · olmo3-7b-instruct-base · olmo3-7b-instruct-cot · olmo3-7b-thinking
• Held-out social probe statistics: soc91-stats · soc91-labels · soc139-quality-sidecars · tombench-en (MIT; evaluation only, per ToMBench authors)

Unlearning Checkpoints — §4.3, Figure 43 & Table 49 (causal validation)

NGDiff rank-8 LoRA unlearning on OLMo3-7B Base: top-200 influence-selected documents per topic vs. 1,000 random in-topic controls, three seeds per condition (SocialIQA paired difference significant at BH-adjusted p ≈ 10⁻⁵).

• unlearning-checkpoints — merged unlearned checkpoints for all conditions and seeds

Hub Artifacts — cross-ecosystem replication (Appendices L & M)

The causal test repeated end-to-end on Comma v0.1 7B-2T (Common Pile) and DCLM-Baseline-7B.

• Comma OLMES evals: comma-7b-1t · comma-7b-2t
• DCLM: dclm-olmes-eval · dclm-baseline-manifest · dclm-baseline-labels · dclm-baseline-working-sample
• dolma3-corpus-explorer — companion Space for browsing the Dolma3 working set

Cite This Work

@inproceedings{matlin2026capabilityprovenance,
  title         = {Capability Provenance in Language Models: A Case Study in Social Reasoning},
  author        = {Glenn Matlin and Chandreyi Chakraborty and Saehee Eom and Mika Okamoto and
                   Rayan Castilla and Louis Jaburi and Alvin Deng and Taywon Min and
                   Lucia Quirke and Stella Biderman and Mark Riedl},
  booktitle     = {Proceedings of the Conference on Language Modeling (COLM 2026)},
  year          = {2026},
  eprint        = {2606.19625},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CL},
  url           = {https://arxiv.org/abs/2606.19625}
}