This article defines a structured program for governing open-source AI models, anchors it to current standards, and explains why the absence of a vendor counterparty raises rather than lowers the governance burden.
Why “Open Source” Is a Misleading Label for Models
Three asymmetries with conventional open-source software make the label imprecise.
First, the artefact is partially open. Open-source software ships source code, build instructions, and tests. Open-weight models ship the trained weights and possibly an inference script — but the training data, training code, hyperparameters, and Reinforcement Learning from Human Feedback (RLHF) procedures are often withheld. Stanford’s Foundation Model Transparency Index at https://crfm.stanford.edu/fmti/ documents that even the most open major models fall well short of full transparency on training and evaluation methods.
Second, the licenses are often non-standard. The Apache 2.0, Massachusetts Institute of Technology (MIT), and General Public License (GPL) family of licenses defines well-understood obligations for software. Many prominent open-weight model licenses — including community licenses with downstream-use restrictions, acceptable-use policies, and revenue thresholds — are bespoke and have not been tested in court. Treating them as conventional permissive licenses is a legal-risk-introduction event.
Third, there is no vendor. With proprietary AI procurement, the deployer can demand contractual remedies, indemnification, and incident response. With open-weight models, the deployer is the only party with continuing obligations. The Cloud Security Alliance at https://cloudsecurityalliance.org/ and the National Institute of Standards and Technology (NIST) AI Risk Management Framework (AI RMF) at https://www.nist.gov/itl/ai-risk-management-framework both make this asymmetry explicit: in the absence of a third party to govern, the deployer must internalise the entire governance burden.
The Three Dimensions of Open Model Governance
A defensible program addresses license, provenance, and quality as three coordinated workstreams.
1. License Governance
Every open-weight model carries a license that the deploying organization must read, classify, and enforce. The relevant categories include permissive (Apache 2.0, MIT), copyleft (variants of the GPL), Creative Commons (CC) variants (such as CC BY, CC BY-SA, CC BY-NC), and bespoke community licenses (often imposing acceptable-use restrictions and downstream-distribution conditions).
License governance involves three steps. First, classification: read the actual license text and place the model in a legal category. Second, conformance: confirm that the intended deployer use is permitted (commercial use, customer-facing use, derivative-work creation, redistribution, fine-tuning). Third, enforcement: capture the license assertion in the AI Bill of Materials (AI-BOM), associate it with the deployment, and re-validate when license terms change. The Software Package Data Exchange (SPDX) standard at https://spdx.dev/ provides the machine-readable license-identifier vocabulary increasingly used in AI-BOM tooling.
2. Provenance Governance
Provenance is the documented chain of custody of a model artefact: who created it, what data trained it, what fine-tuning steps were applied, where the weights were hosted, and which build or attestation procedures verified integrity. The Hugging Face Safetensors format documented at https://huggingface.co/docs/safetensors illustrates one specific provenance control: a serialisation format that prevents arbitrary code execution at load time, mitigating a class of supply-chain attacks where weights are bundled with malicious code.
Supply-chain Levels for Software Artifacts (SLSA), at https://slsa.dev/, defines four progressive build-integrity levels that increasingly apply to model artefacts. Reaching SLSA Level 2 or 3 for a critical open model gives the deploying organization cryptographic evidence of origin and unforgeability. The U.S. Cybersecurity and Infrastructure Security Agency (CISA) Software Bill of Materials programme at https://www.cisa.gov/sbom is now extending these concepts through AI-BOM and Model Bill of Materials (MBOM) constructs that an open-source program should adopt.
Provenance governance reaches two practical conclusions for any candidate open model: a documented origin (or an explicit “unverified” classification) and a documented trust-decision artefact recording why the model is acceptable despite gaps.
3. Quality Governance
Without a vendor to validate behaviour, the deploying organization is responsible for every quality assertion the system makes. This requires three artefacts.
The first is an evaluation suite specific to the intended use case: a curated set of inputs and expected outputs against which the model is tested before approval. Public leaderboards offer some signal but do not substitute for domain-specific evaluation.
The second is bias and safety testing following the NIST AI RMF MEASURE function and the categories specified in the EU AI Act, accessible at https://artificialintelligenceact.eu/. Where the model will inform high-risk decisions under the Act, evaluation evidence is part of the deployer’s required documentation under Article 25.
The third is continuous re-evaluation after deployment. Open models are often forked and re-released; the version in production must remain the version that was evaluated, or the evaluation must be repeated.
The Regulatory Twist for Open-Weight General-Purpose AI
The EU AI Act addresses open-source General-Purpose AI (GPAI) explicitly. Article 53 exempts certain free and open-source models from documentation obligations — but not from the obligations that apply to systemic-risk models under Article 55. A deployer who uses an open-source model that exceeds the systemic-risk threshold inherits the obligation to operate it safely, even though no commercial provider stands behind it. The deployer becomes the accountable party.
This regulatory reality reverses a common assumption: open-source does not reduce regulatory exposure; it relocates it onto the deployer.
What ISO/IEC 42001 Adds
The International Organization for Standardization / International Electrotechnical Commission (ISO/IEC) 42001:2023 standard at https://www.iso.org/standard/81230.html includes management-system controls that apply equally to open-source and proprietary models. Annex A.10 on third-party relationships, Annex A.6 on AI system lifecycle, and the management-system requirement to maintain a documented inventory of AI components together imply that open-source models cannot be invisible to the AI Management System. They must be enumerated, evaluated, approved, and monitored under the same regime.
The Practical Threat Surface
A Hugging Face model repository that has been compromised — through account takeover, malicious commits, or supply-chain insertion — can deliver weights that exfiltrate data on load, embed backdoors triggered by specific prompts, or carry training-data poisoning that activates only in production. The Hugging Face Safetensors format mitigates the load-time arbitrary-code-execution surface but does not address embedded behavioural backdoors. The cryptographic verification practices documented in CISA SBOM, SLSA, and SPDX materials are the operational defences. Without them, “we use the open-weight version” is an acceptable-risk-by-default decision rather than a governed one.
Maturity Indicators
| Maturity | What open-source model governance looks like |
|---|---|
| Foundational (1) | Engineers download open-weight models freely; license, provenance, and quality are not tracked. |
| Developing (2) | A list of approved open models exists; license categorisation has begun; provenance is partial. |
| Defined (3) | All open models in production carry a documented license classification, AI-BOM provenance entry, and use-case evaluation; loading is restricted to Safetensors or equivalent. |
| Advanced (4) | SLSA-level attestation is required for critical models; continuous re-evaluation runs in pipeline; license changes trigger reassessment. |
| Transformational (5) | The organization contributes to AI-BOM and open-model evaluation standards; published evaluation results inform community practice. |
Practical Application
A media company evaluating an open-weight image-generation model for a customer-facing creative tool should not download the weights and put them in production. It should obtain the model card, classify the license, capture the SPDX identifier, verify the artefact through a Safetensors load and a SLSA-attestation check where available, run a domain-specific evaluation including bias and copyright-contamination tests, document the trust decision, and only then schedule deployment. The artefact that justifies production is the documented evaluation, not the popularity of the model on a leaderboard. When the upstream community releases a new fork, the same procedure repeats.
The next article (Article 6) defines the AI-BOM and MBOM formats themselves — the data structures in which the license, provenance, and quality assertions of every open and proprietary model in the supply chain are recorded.