ProvLevel-Reason: An Oracle-Scored Benchmark for LLM Reasoning over Software Build-Provenance Attestations
DOI:
https://doi.org/10.66525/ijictt95Keywords:
software supply chain, build provenance, SLSA, in-toto, large language models, benchmark, attestationAbstract
A build-provenance attestation is the document a consumer reads to decide whether an artifact was produced by a trustworthy build: it records who built it, from what sources, on what platform, and whether the record is signed. Adjudicating it means applying a laddered rubric, the SLSA build levels, to a structured in-toto/DSSE document. As language models increasingly become the readers of this metadata, the question is whether they apply that rubric or merely pattern-match on its surface. We introduce ProvLevel-Reason, a controlled benchmark that gives a model one real-format attestation and asks it to (A) decide the SLSA build level achieved (L0–L3) and (B) given a target level, enumerate exactly which requirements are unmet and check three provenance-to-artifact consistency properties. Answers are scored against a deterministic oracle that encodes the SLSA v1.1 Build Track plus in-toto/DSSE structural checks, with zero human labels; a generator plants controlled failure modes and a round-trip self-test agrees with the oracle on samples. Across five API models on Task A ( items) and four on Task B ( items, stratified by number of unmet requirements) we find three effects. First, multi-failure attribution is the hard axis: when two requirements are unmet, direct-prompted models report only one of the two – of the time. Second, chain-of-thought cuts both ways, and which way depends on the model. It rescues multi-failure recall for most models, yet on gpt-4o-mini it collapses precision on already-satisfied targets, inventing gaps on of them (exact-match ). Third, a structural completeness check is the consistent weak point, and how stubbornly it persists depends on the provider. When judging level, both OpenAI models and gemini-2.5-flash fail to treat a missing buildType as disqualifying even under CoT (accuracy ), while Anthropic models recover it; asked instead to enumerate gaps, OpenAI recovers but gemini-2.5-flash still does not (incomplete-recall ), a blindness that survives both scale and task framing. With parse_fail and a total inference cost of $14, ProvLevel-Reason turns “is this build trustworthy, and if not, why” into an oracle-scored test.
Downloads
References
[1] P. Ladisa, H. Plate, M. Martinez, and O. Barais, “SoK: Taxonomy of attacks on open-source software supply chains,” in 2023 IEEE symposium on security and privacy (SP), 2023, pp. 1509–1526. doi: 10.1109/SP46215.2023.10179304.
[2] Open Source Security Foundation, “SLSA: Supply-chain levels for software artifacts, specification v1.0.” https://slsa.dev/spec/v1.0/, 2023.
[3] S. Torres-Arias, H. Afzali, T. K. Kuppusamy, R. Curtmola, and J. Cappos, “In-toto: Providing farm-to-table guarantees for bits and bytes,” in Proc. 28th USENIX security symposium, 2019, pp. 1393–1410.
[4] in-toto maintainers, “In-toto attestation framework specification.” https://github.com/in-toto/attestation, 2023.
[5] Secure Systems Lab and in-toto maintainers, “DSSE: Dead simple signing envelope specification.” https://github.com/secure-systems-lab/dsse, 2021.
[6] Z. Newman, J. S. Meyers, and S. Torres-Arias, “Sigstore: Software signing for everybody,” in Proc. 2022 ACM SIGSAC conf. On computer and communications security (CCS), 2022, pp. 2353–2367. doi: 10.1145/3548606.3560596.
[7] SLSA Framework, “slsa-verifier: Verify provenance from SLSA-compliant builders.” https://github.com/slsa-framework/slsa-verifier, 2024.
[8] The White House, “Executive order 14028: Improving the nation’s cybersecurity.” Federal Register, 86 FR 26633, 2021.
[9] M. Ohm, H. Plate, A. Sykosch, and M. Meier, “Backstabber’s knife collection: A review of open source software supply chain attacks,” in Proc. 17th int. Conf. On detection of intrusions and malware, and vulnerability assessment (DIMVA), 2020, pp. 23–43. doi: 10.1007/978-3-030-52683-2_2.
[10] C. Okafor, T. R. Schorlemmer, S. Torres-Arias, and J. C. Davis, “SoK: Analysis of software supply chain security by establishing secure design properties,” in Proc. 2022 ACM workshop on software supply chain offensive research and ecosystem defenses (SCORED), 2022, pp. 15–24. doi: 10.1145/3560835.3564556.
[11] B. Xia, T. Bi, Z. Xing, Q. Lu, and L. Zhu, “An empirical study on software bill of materials: Where we stand and the road ahead,” in Proc. 45th int. Conf. On software engineering (ICSE), 2023, pp. 2630–2642. doi: 10.1109/ICSE48619.2023.00219.
[12] S. Ullah, M. Han, S. Pujar, H. Pearce, A. Coskun, and G. Stringhini, “LLMs cannot reliably identify and reason about security vulnerabilities (yet?): A comprehensive evaluation, framework, and benchmarks,” in 2024 IEEE symposium on security and privacy (SP), IEEE, 2024.
[13] Y. Ding et al., “Vulnerability detection with code language models: How far are we?” in Proc. IEEE/ACM 47th int. Conf. On software engineering (ICSE), 2025. doi: 10.1109/ICSE55347.2025.00038.
[14] Y. Li et al., “SV-TrustEval-C: Evaluating structure and semantic reasoning in large language models for source code vulnerability analysis,” in 2025 IEEE symposium on security and privacy (SP), IEEE, 2025, pp. 2791–2809.
[15] J. He, M. Rungta, D. Koleczek, A. Sekhon, F. X. Wang, and S. Hasan, “Does prompt formatting have any impact on LLM performance?” arXiv:2411.10541, 2024.
[16] J. Wei et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Advances in neural information processing systems (NeurIPS), 2022, pp. 24824–24837.
[17] Z. Sprague et al., “To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning,” in Int. Conf. On learning representations (ICLR), 2025.
[18] S. Maekawa, J. Hassell, P. Pezeshkpour, T. Mitchell, and E. Hruschka, “Towards reliable benchmarking: A contamination-free, controllable evaluation framework for multi-step LLM function calling,” in Int. Conf. On learning representations (ICLR), 2026.
[19] A. Gu, B. Rozière, H. J. Leather, A. Solar-Lezama, G. Synnaeve, and S. Wang, “CRUXEval: A benchmark for code reasoning, understanding and execution,” in Proc. 41st int. Conf. On machine learning (ICML), in PMLR, vol. 235. 2024, pp. 16568–16621.
[20] C. Lamb and S. Zacchiroli, “Reproducible builds: Increasing the integrity of software supply chains,” IEEE Software, vol. 39, no. 2, pp. 62–70, 2022, doi: 10.1109/MS.2021.3073045.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Haoyu Zhang , Zhuoxi Wang (Author)

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
All articles published in the International Journal of Information and Communication Technology Trends (IJICTT) are distributed under a Creative Commons license. This allows readers to freely access, share, and adapt the content for non-commercial purposes, provided proper attribution is given to the original authors. Authors retain copyright of their work.
