Papers

Currently Under Review

• O'Brien*, Casper*, Anthony, Korbak, Kirk, Davies, Mishra, Irving, Gal, and Biderman. "Deep Ignorance: Filtering Pretraining Data Builds Tamper-Resistant Safeguards into Open-Weight LLMs." [Paper] [Project Page] [Artifacts]

• Kandpal*,Lester*, Raffel*,  Majstorovic, Biderman, et al. "The Common Pile v0.1: An 8TB Dataset of Public Domain and Openly Licensed Text." [Paper] [Artifacts]

• Son, Hong, Fan, Nam, Ko, Lim, Song, Choi, Paulo, Yu, and Biderman. "When AI co-scientists fail: SPOT-a benchmark for automated verification of scientific research." [Paper] [Data]

• Baack*, Biderman*, Odrozek*, Skowron*, et al. "Towards Best Practices for Open Datasets for LLM Training." [Paper]

2025

• Bradshaw, Fan, Spangher, Biderman, and Colton. "Scaling Self-Supervised Representation Learning for Symbolic Piano Performance." International Society for Music Information Retrieval Conference (ISMIR). 2025. [Paper] [Code]

• Biderman, Mickel, and Abbasi. "Write Code that People Want to Use." Championing Open-source DEvelopment in ML Workshop@ ICML25. 2025. [Paper]

• Ahdritz, Bouatta, Kadyan, et al. (incl. Biderman). "Why Has Predicting Downstream Capabilities of Frontier AI Models with Scale Remained Elusive?" International Conference on Machine Learning (ICML). 2025. [Paper]

• van der Wal, Lesci, Muller-Eberstein, Saphra, Schoelkopf, Zuidema, and Biderman. "Polypythias: Stability and outliers across fifty language model pre-training runs." International Conference on Learning Representations (ICLR). 2025. [Paper] [Code] [Models]

• Longpre, et al. (incl. Biderman). "Bridging the data provenance gap across text, speech and video." International Conference on Learning Representations (ICLR). 2025. [Paper]

• Prashanth*, Deng*, O'Brien*, et al. (incl. Biderman). "Recite, reconstruct, recollect: Memorization in lms as a multifaceted phenomenon." International Conference on Learning Representations (ICLR). 2025. [Paper] [Code]

• Sharkey*, Chughtai*, et al. (incl. Biderman). "Open problems in mechanistic interpretability." Transactions of Machine Learning Research (TMLR). 2025. [Paper]

2024

• Alam, Oberle, Raff, Biderman, Oates, and Holt. "A walsh hadamard derived linear vector symbolic architecture." Advances in Neural Information Processing Systems (NeurIPS). 2024. [Paper] [Code]

• Longpre*, Mahari*, Lee*, Lund*, et al. (incl. Biderman) "Consent in crisis: The rapid decline of the ai data commons." Advances in Neural Information Processing Systems (NeurIPS). 2024. [Paper]

• Tigges, Hanna, Yu, and Biderman. "LLM circuit analyses are consistent across training and scale." Advances in Neural Information Processing Systems (NeurIPS). 2024. [Paper]

• Forde*, Zhang*, Sutawika, Aji, Cahyawijaya, Winata, Wu, Eickhoff, Biderman, and Pavlick "Re-Evaluating Evaluation for Multilingual Summarization." Empirical Methods in Natural Language Processing (EMNLP). 2024. [Paper]

• van der Wal, Lesci, Muller-Eberstein, Saphra, Schoelkopf, Zuidema, and Biderman. " The Case for Co-Designing Model Architectures with Hardware " International Conference on Parallel Processing (ICPP). 2024. [Paper]

• Ahdritz, Bouatta, Kadyan, et al. (incl. Biderman). "OpenFold: Retraining AlphaFold2 yields new insights into its learning mechanisms and capacity for generalization." Nature Methods. 2024. [Paper] [Code]

• Stander, Yu, Fan, Biderman. "Grokking group multiplication with cosets." International Conference on Machine Learning (ICML). 2024.

• Sayash Kapoor*, Bommasani, et al. (incl. Biderman). "Position: On the societal impact of open foundation models"  International Conference on Machine Learning (ICML). 2024. [Paper]

• Biderman*, Schoelkopf*, Sutawika*, et al. "Lessons from the trenches on reproducible evaluation of language models." arXiv. 2024. [Paper]

• Alam, Raff, Biderman, Oates, Holt. "Holographic global convolutional networks for long-range prediction tasks in malware detection." International Conference on Artificial Intelligence and Statistics (AISTATS). 2024

• B Peng, D Goldstein, Q Anthony, et al. (incl. Biderman). "Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence" Conference on Language Modeling (COLM). 2024.

• Son, Lee, Kim, Kim, Muennighoff, Choi, Park, Yoo, and Biderman. "Kmmlu: Measuring massive multitask language understanding in korean." Nations of the Americas Chapter of the Association for Computational Linguistics (NAACL). 2024.

• Castricato, Lile, Anand, Schoelkopf, Verma, and Biderman. "Suppressing pink elephants with direct principle feedback" arXiv. 2024.

• Zhang, Tigges, Zhang, Biderman, Raginsky, and Ringer "Transformer-based models are not yet perfect at learning to emulate structural recursion"

• McMillan-Major, De Toni, et al. (incl. Biderman) "Documenting geographically and contextually diverse language data sources." Northern European Journal of Language Technology (NEJLT). 2024.

• O'Mahony, Grinsztajn, Schoelkopf, and Biderman. "Attributing Mode Collapse in the fine-tuning of Large Language Models. Workshop on Mathematical and Empirical Understanding of Foundation Models @ ICLR. 2024.

2023

• Biderman, Prashanth, Sutawika, Schoelkopf, Anthony, Purohit, and Raff. "Emergent and Predictable Memorization in Large Language Models." Advances in Neural Information Processing Systems (NeurIPS). 2023. [Paper]

• Belrose Schneider-Joseph, Ravfogel, Cotterell, Raff, and Biderman. "LEACE: Perfect linear concept erasure in closed form." Advances in Neural Information Processing Systems. 2023. [Paper]

• Ruis, Khan, Biderman, Hooker, Rocktäschel, Grefenstette. "The Goldilocks of Pragmatic Understanding: Fine-Tuning Strategy Matters for Implicature Resolution by LLMs." Advances in Neural Information Processing Systems (NeurIPS). 2023. Spotlight Presentation. [Paper]

• Havrilla, Zhuravinskyi, Van Phung, et al. (incl. Biderman). "trlX: A Framework for Large Scale Reinforcement Learning from Human Feedback." Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMLP). 2023. [Paper] [Code]

• Llemma: An open language model for mathematics

• Stay on topic with classifier-free guidance

• Peng, Anthony, Alcaid, et al. (incl. Biderman). "RWKV: Reinventing RNNs for the Transformer Era." Transactions of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2023. [Paper] [Blog Post] [Model] [Code]

• Piktus, Ogundepo, Akiki, et al. (incl. Biderman). GAIA Search: Hugging Face and Pyserini Interoperability for NLP Training Data Exploration." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023. [Paper] [Demo]

• Can Transformers Learn to Solve Problems Recursively?

• Zheng-Xin, Schoelkopf, Muennighoff, et al. (incl. Biderman) "BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Prompting." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023. [Paper]

• Muennighoff, Wang, Sutawika, et al. (incl. Biderman). "Crosslingual Generalization through Multitask Finetuning." Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. 2023. [Paper] [Models] [Code]

• Biderman*, Schoelkopf*, et al. "Pythia: A Suite for Analyzing Large Language Models across Training and Scaling." Proceedings of the International Conference on Machine Learning (ICML). 2023. Oral Presentation. [Paper] [Models]

• Alam, Raff, Biderman, Oates, and Holt. "Recasting Self-Attention with Holographic Reduced Representations." Proceedings of the International Conference on Machine Learning (ICML). 2023. [Paper] [Code]

• Srivastava, Rastogi, Rao, et al. (incl. Biderman) "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models." Transcations of Machine Learning Research. 2023. [Paper][Code]

2022

• Laurençon, Saulnier, Wang, et al. (incl. Biderman). "The BigScience ROOTS Corpus: A 1.6 TB Composite Multilingual Dataset." Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. 2022. [Paper] [Exploratory Tool]

• Fries, Weber, Seelam, et al. (incl. Biderman) "BigBIO: A Framework for Data-Centric Biomedical Natural Language Processing." Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks. 2022. [Paper] [Codebase]

• Phang, Bradley, Gao, Castricato, and Biderman. "EleutherAI: Going Beyond “Open Science” to “Science in the Open”." Proceedings of the Workshop on Broadening Research Collaborations in ML @ NeurIPS. 2022. [Paper]

• Crowson*, Biderman*, et al. "VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance." In Proceedings of European Conference on Computer Vision (ECCV). 2022. [Paper] [Demo] [Code]

• Biderman and Raff. "Fooling MOSS Detection with Pretrained Language Models." In Proceedings of the ACM International Conference on Information and Knowledge Management (CIKM). 2022. [Paper]

• McMillan-Major, Alyafeai, Biderman, et al. "Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources." arXiv. 2022. [Paper]

• Jernite, Nguyen, Biderman, et al. "Data Governance in the Age of Large-Scale Data-Driven Language Technology." In the Proceedings of the ACM Conference on Fairness, Accountability, and Transparency (FAccT). 2022. [Paper]

• Black*, Biderman*, Hallahan*, et al. "GPT-NeoX-20B: An Open-Source Autoregressive Language Model." In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models. 2022. [Paper] [Model and Code]

• Hesslow*, Le Scao*, Saulnier*, et al. (incl. Biderman). What Language Model to Train if You Have One Million GPU Hours? In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models. 2022. [Paper]

• Talat, Névéol, Biderman, et al. "You Reap What You Sow: On the Challenges of Bias Evaluation under Multilingual Settings." In Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models. 2022. [Paper]

• Kreutze*, Caswell*, et al. (incl. Biderman). "Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets." In Transactions of the Association for Computational Linguistics (TACL). 2022. [Paper]

• Biderman, Bicheno, and Gao. "Datasheet for the Pile." arXiv. 2022. [Paper]

• Sanh*, Webson*, Raffel,* Bach*, et al. (incl. Biderman). "Multitask Prompted Training Enables Zero-Shot Task Generalization." In Proceedings of the Tenth International Conference on Learning Representations (ICLR), 2022. [Paper] [Model]

2021

• Matiana*, Smith*, Teehan*, Castricato*, Biderman*, Gao, and Frazier. "Cut the CARP: Fishing for zero-shot story evaluation." arXiv. 2021. [Paper]

• Alcaide, Biderman, Telenti, and Maher. "Massively Parallel Natural Extension of Reference Frame for Efficient Internal to Cartesian Conversion." In the Journal of Computational Chemistry. 2021. [Paper] [Code]

• Louis Castricato, Stella Biderman, David Thue, and Rogelio Cardona-Rivera. "Towards a Model-Theoretic View of Narratives." In Proceedings of the Third NAACL Workshop on Narrative Understanding. 2021. [Paper]

• Churchill, Biderman, and Herrick. "Magic: the Gathering is Turing Complete." In Proceedings of the 10th International Conference on Fun with Algorithms (FUN). 2021. [arXiv] [Demo]

2020 and older

• Gao, Biderman, Black, Golding, Hoppe, Foster, Phang, He, Thite, Nabeshima, Presser, and Leahy. "The Pile: An 800GB Dataset of Diverse Text for Language Modeling." arXiv. 2020. [Paper] [Datasheet] [Website] [Model]

• Biderman and Scheirer. "Pitfalls in Machine Learning Research: Reexamining the Development Cycle".  In Proceedings on "I Can't Believe It's Not Better!" at NeurIPS Workshops, PMLR. 2020. [Paper]

• Biderman. "Magic: the Gathering is as Hard as Arithmetic." arXiv. 2020. [Paper] 

• Biderman. "Neural Networks on Groups." arXiv. 2019. [Paper]

• Biderman*, Cuddy*, Li*, and Song*. The Sensitivity of k-Uniform Hypergraph Properties. In the 3rd Annual University of Chicago Undergraduate Research Symposium. 2016. [Paper]

Talks

Non-Archival Conferences

• Masad, Biderman, Shishkoff, and Baird. "Predicting Crisis Behavior with Reinforcement Learning." The Military Operation Research Society's Emerging Techniques Forum. 2019. [Slides]

  • Awarded the Eugene P. Visco Prize for best research by a junior analyst.

• Masad, Biderman, Shishkoff, and Baird. "Reinforcement Learning in Conflict Escalation Games." Poster at the 35th Annual Meeting of the Society for Political Methodology. 2018. [Abstract] [Poster]

• Biderman, Masad, and Lawson. "How to be Wrong, but Useful: A Case Study on Tool Selection in Social Network Analysis." The 2nd Annual North American Social Networks Conference. 2018. [Slides]

Invited Talks

• "GPT-NeoX-20B: the Road to Open NLP Research." AI Sweden NLP Seminar. 2022. [Slides] 

• "Replication as a Security Threat: the Role of Open Source Research in AI Security." The AI Village @ DEF CON. 2021. [Slides]

• Moderator for the AI Village's "Ethics & Bias Panel." The AI Village @ DEF CON. 2020. [Video]

• "Verifiable Computation: How to Securely and Correctly Execute Code on the Cloud.” DC DevFest. 2019. [Slides] [Video]