CtrlK
A model card is a standardized documentation framework for machine learning models that describes a model's intended use, performance characteristics, training data, evaluation metrics, ethical considerations, and limitations. Model cards function as a form of transparency documentation, analogous to nutrition labels for food products or datasheets for electronic components. The concept was introduced by Margaret Mitchell and colleagues at Google in their 2019 paper "Model Cards for Model Reporting," and has since become a widely adopted standard in both industry and open-source AI development, particularly through Hugging Face's implementation on its Model Hub [1].
Model cards are part of a broader ecosystem of AI documentation practices that includes datasheets for datasets (Gebru et al., 2021), system cards (used by OpenAI for models like GPT-4), and regulatory documentation requirements under frameworks like the EU AI Act. Together, these documentation standards aim to improve accountability, reproducibility, and informed decision-making across the AI development lifecycle.
The foundational paper, "Model Cards for Model Reporting," was published in January 2019 at the ACM Conference on Fairness, Accountability, and Transparency (FAT* 2019). The paper was authored by Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru, all of whom were affiliated with Google at the time [1].
The paper identified a gap in how machine learning models were documented and shared. While software engineering had established practices for documentation (API references, changelogs, README files), machine learning models were frequently released with minimal information about their intended use, performance across different populations, training data composition, or known limitations. This lack of transparency made it difficult for downstream users to evaluate whether a model was appropriate for their specific application, and it obscured potential harms to affected communities [1].
Mitchell et al. proposed model cards as a short, structured document accompanying any trained machine learning model. The paper drew inspiration from existing practices in other fields: datasheets in the electronics industry, nutritional labels in the food industry, and Material Safety Data Sheets in chemistry. The authors argued that a standardized documentation format would facilitate better communication between model developers and model users, encourage the evaluation of models across different demographic groups, and create accountability for known limitations [1].
The paper demonstrated the concept with example model cards for two Google models: a face detection model and a toxicity classifier. These examples showed how model cards could surface important information about disparate performance across demographic groups, information that would otherwise be hidden in aggregate accuracy metrics.
The original Mitchell et al. framework specifies several sections that a model card should include. These sections have been refined and expanded by subsequent adopters, but the core structure remains consistent.
| Section | Description | Example content |
|---|---|---|
| Model details | Basic information about the model | Name, version, type (e.g., classification, generation), developer, release date, license |
| Intended use | What the model was designed for | Primary use cases, intended users, out-of-scope uses |
| Factors | Variables that affect model performance | Demographic factors (age, gender, ethnicity), environmental factors (lighting, noise), instrumentation factors (camera type, microphone quality) |
| Metrics | How performance is measured | Accuracy, F1 score, BLEU, perplexity, fairness metrics, disaggregated by relevant factors |
| Evaluation data | Datasets used for testing | Dataset name, size, composition, any known biases |
| Training data | Datasets used for training | Sources, size, collection methodology, preprocessing steps, any known limitations |
| Quantitative analyses | Detailed performance results | Performance broken down by subgroups, confidence intervals, error analysis |
| Ethical considerations | Known risks and societal impact | Potential for harm, sensitive use cases, populations at risk |
| Caveats and recommendations | Known limitations and usage guidance | Failure modes, conditions under which the model should not be used, recommended mitigation strategies |
The emphasis on disaggregated evaluation, where performance metrics are broken down across demographic or contextual subgroups rather than reported only as aggregate numbers, is one of the most important contributions of the model card framework. A face recognition model might achieve 99% accuracy overall but only 85% accuracy on darker-skinned faces; an aggregate metric would hide this disparity, while a model card would make it explicit [1].
Hugging Face, the open-source AI platform and model hosting service, has become the primary venue where model cards are created and consumed in practice. The Hugging Face Hub hosts hundreds of thousands of models, and each model repository includes a README.md file that serves as the model card [2].
Hugging Face's adoption of model cards has been the single most significant factor in their widespread use. When a researcher or organization uploads a model to the Hub, the platform prompts them to fill out a model card using a standardized template. This template draws on the original Mitchell et al. framework but has been expanded to include additional metadata fields relevant to the Hugging Face ecosystem, such as pipeline tags (text-classification, image-generation, etc.), language codes, datasets used, and library compatibility [2].
In 2022, Hugging Face launched the Model Card Guidebook, a comprehensive resource that includes an updated model card template, an annotated version explaining how to complete each section, and a Model Card Creator Tool that allows users to generate model cards through a graphical interface without writing markdown directly. The guidebook was developed in collaboration with academic researchers and reflects lessons learned from analyzing thousands of model cards on the Hub [2].
The quality of model cards on Hugging Face varies significantly. Models released by major organizations (Google, Meta, Microsoft, Anthropic) typically have detailed, well-structured model cards. Community-contributed models, which make up the majority of Hub content, range from thorough documentation to completely empty README files. Hugging Face has attempted to address this through tooling (the Model Card Creator), education (the Guidebook), and community norms, but enforcement of documentation standards remains voluntary [2].
Hugging Face model cards include a YAML metadata block at the top of the README.md file that enables structured search, filtering, and integration with the platform's API. Key metadata fields include:
| Field | Purpose |
|---|---|
language | Languages the model supports or was trained on |
license | The license under which the model is released |
tags | Free-form tags for categorization |
datasets | Datasets used for training or evaluation |
metrics | Performance metrics and their values |
pipeline_tag | The type of task the model performs (e.g., text-generation, image-classification) |
library_name | The ML library the model is compatible with (e.g., transformers, diffusers) |
base_model | The model this model was fine-tuned from, if applicable |
Model cards were designed to document models, but a parallel documentation framework exists for the datasets used to train and evaluate those models. "Datasheets for Datasets," proposed by Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daume III, and Kate Crawford, was first published as a preprint in March 2018 and later appeared in Communications of the ACM in December 2021 [3].
The datasheets framework takes its name from the electronics industry, where every component ships with a datasheet specifying its operating characteristics. Gebru et al. argued that every dataset used in machine learning should be accompanied by a similar document, covering the motivation behind the dataset's creation, its composition, the data collection process, any preprocessing or labeling steps, recommended uses, distribution and maintenance plans, and legal or ethical considerations [3].
The key questions a dataset datasheet answers include:
| Category | Example questions |
|---|---|
| Motivation | Why was the dataset created? Who created it? Who funded it? |
| Composition | What do the instances represent? How many are there? Is there missing data? Does it contain confidential information? |
| Collection process | How was the data collected? Who was involved? Over what timeframe? Were individuals notified? Did they consent? |
| Preprocessing | Was any data cleaned, filtered, or labeled? What tools or procedures were used? |
| Uses | What tasks has the dataset been used for? Are there tasks it should not be used for? |
| Distribution | How is the dataset distributed? Under what license? Are there access restrictions? |
| Maintenance | Who maintains the dataset? How can errors be reported? Will it be updated? |
Datasheets for datasets and model cards are complementary: a model card references the training and evaluation data, while the dataset's datasheet provides the detailed provenance information for that data. Together, they create a documentation chain from raw data to deployed model [3].
System cards represent an evolution of the model card concept, designed for AI systems that are more complex than a single model. The term was popularized by OpenAI, which has published system cards for several of its major releases, including GPT-4 (March 2023), GPT-4o (August 2024), and subsequent models [4].
A system card differs from a model card in several respects. While a model card documents a single trained model in isolation, a system card documents the entire system that is deployed to users, including the base model, fine-tuning procedures, safety mitigations (such as RLHF training, content filters, and rate limits), deployment configuration, and the results of red-teaming and safety evaluations. System cards also typically include a discussion of the system's capabilities and the risks those capabilities pose in deployment [4].
OpenAI's GPT-4 system card, for instance, included results from external red-team testing across domains including cybersecurity, biosecurity, and persuasion. It documented specific capabilities that raised safety concerns, such as the model's ability to provide step-by-step instructions for synthesizing dangerous chemicals (mitigated through refusal training) and its performance on standardized tests (demonstrating human-level or above-human-level reasoning in some domains). The system card also described the limitations of the safety measures in place, including known jailbreaking techniques [4].
Anthropic has published similar documents for its Claude models, and Google DeepMind has published technical reports that serve a comparable function for Gemini. The common thread is that as AI systems become more complex and capable, documentation needs to cover not just the model itself but the full stack of engineering, safety, and deployment decisions around it.
The EU AI Act, which entered into force on August 1, 2024, includes specific documentation requirements for AI systems, particularly those classified as high-risk. Providers of high-risk AI systems must maintain comprehensive technical documentation covering the system's intended purpose, design, development methodology, training and testing data, performance metrics, risk management measures, and post-market monitoring plans [5].
For general-purpose AI (GPAI) models, including large language models, the AI Act requires providers to publish a sufficiently detailed summary of the training data content, using a template provided by the EU's AI Office. These requirements align closely with the information that a well-prepared model card or system card would contain. Providers of GPAI models with systemic risk (those exceeding 10^25 floating-point operations in training) face additional obligations, including formal governance structures, independent risk assessments, and periodic evaluations [5].
The AI Act's documentation requirements effectively mandate something resembling model cards for all AI systems sold or deployed in the EU. While the regulation does not specifically require the Mitchell et al. format, the overlap between regulatory requirements and model card best practices has led many organizations to use model cards as a starting point for compliance [5].
Beyond the EU, several other jurisdictions have adopted or proposed documentation requirements for AI systems. Canada's Algorithmic Impact Assessment Tool, used by federal agencies, requires documentation of AI system design, data, and impact. Singapore's Model AI Governance Framework recommends documentation practices similar to model cards. In the US, the National Institute of Standards and Technology (NIST) AI Risk Management Framework (RMF) includes documentation as a component of responsible AI practice, though without prescribing a specific format [5].
The following table lists notable model cards and system cards published by major AI organizations, illustrating the range of documentation practices in use.
| Organization | Model | Document type | Notable features |
|---|---|---|---|
| Face Detection, Toxicity Classifier | Model card (2019) | Original examples from the Mitchell et al. paper; focused on fairness across demographic groups | |
| OpenAI | GPT-4 | System card (2023) | Included red-team results, capability assessments, and safety mitigation details across multiple risk domains |
| OpenAI | GPT-4o | System card (2024) | Extended to cover multimodal capabilities (text, image, audio); included voice safety evaluations |
| Meta | Llama 2 | Model card (2023) | Detailed benchmarks, safety evaluations, and responsible use guide; published on Hugging Face |
| Meta | Llama 3 | Model card (2024) | Expanded safety testing; included results across multiple languages and cultural contexts |
| Google DeepMind | Gemini 1.0 | Technical report (2023) | Served as de facto system card; included extensive benchmark results and safety analysis |
| Anthropic | Claude 3 | Model card (2024) | Documented constitutional AI training, safety evaluations, and known limitations |
| Stability AI | Stable Diffusion | Model card (2022) | Documented training data sources, known biases in image generation, and misuse risks |
| Mistral AI | Mistral 7B | Model card (2023) | Concise documentation focused on performance benchmarks and license terms |
Despite their wide adoption, model cards have been subject to several critiques.
Voluntary and inconsistent. Outside of regulated environments, model card creation is voluntary. Many models, particularly those released by smaller organizations or individual researchers, ship with incomplete or nonexistent documentation. Even among well-resourced organizations, the quality and detail of model cards varies significantly. A 2023 study of Hugging Face model cards found that the majority were missing information in at least one core section [2].
Static documents. Model cards are typically created at the time of model release and rarely updated thereafter. As models are fine-tuned, deployed in new contexts, or discovered to have previously unknown failure modes, the original model card becomes increasingly outdated. Some organizations have adopted versioned model cards, but this practice is not widespread [2].
Self-reported. Model cards are written by the organizations that develop and release the models. There is an inherent tension between transparency and self-interest: organizations may understate risks, overstate performance, or omit information about known problems. Independent auditing of model card claims remains rare [6].
Readability and audience. Model cards are often written in technical language that is accessible to ML practitioners but opaque to policymakers, journalists, affected communities, and other stakeholders who may need the information most. The original Mitchell et al. paper acknowledged this tension but did not fully resolve it [1].
Scope limitations. A model card documents a model; it does not document the deployment context, the downstream applications, or the lived experiences of people affected by the model's outputs. A toxicity classifier's model card might document that the model performs well on a standard benchmark, without capturing how its deployment in a content moderation system affects free expression for specific communities. System cards partially address this gap, but deployment-level documentation remains underdeveloped [4].
Model cards exist within a broader ecosystem of documentation and governance tools for machine learning. Hugging Face maintains a "Landscape of ML Documentation Tools" resource that catalogs these tools, which include:
| Tool / Framework | Purpose |
|---|---|
| Model cards (Mitchell et al.) | Document individual ML models |
| Datasheets for datasets (Gebru et al.) | Document training and evaluation datasets |
| System cards (OpenAI et al.) | Document deployed AI systems including safety measures |
| AI FactSheets (IBM) | Comprehensive AI governance documentation |
| Data Statements (Bender and Friedman) | Document the characteristics of NLP datasets |
| Reward Reports (Gilbert et al.) | Document the objectives and reward functions of reinforcement learning systems |
| Responsible AI Licenses (RAIL) | License terms that include use restrictions for AI models |
These tools serve overlapping but distinct purposes. No single document can capture all relevant information about an AI system, from its raw training data to its real-world impact. The trend in AI documentation is toward complementary, layered documentation that covers different aspects of the AI lifecycle [2].
Model cards are now firmly established as a norm in the AI industry, though their adoption and quality remain uneven. The EU AI Act's documentation requirements, which take full effect in August 2026, will make something resembling model cards legally mandatory for AI systems deployed in the European Union. This regulatory pressure is expected to improve documentation quality across the industry, including for organizations based outside the EU that wish to operate in the European market [5].
Hugging Face continues to lead in model card tooling and infrastructure. The platform has introduced automated model card generation features that pre-populate certain fields based on model metadata, training logs, and benchmark results. Research groups are exploring the use of language models themselves to assist in generating model card content, though this raises its own questions about accuracy and accountability [2].
The broader trajectory points toward AI documentation becoming not just a best practice but a legal and commercial requirement. As AI systems are deployed in increasingly high-stakes settings (healthcare, finance, criminal justice, defense), the demand for thorough, accurate, and independently verifiable documentation will only grow. Model cards, for all their limitations, laid the groundwork for this shift.