Internet-Draft evp-spec May 2025
Hopkins & Turner Expires 16 November 2025 [Page]
Workgroup:
Network Working Group
Internet-Draft:
draft-hopkins-evp-spec-00
Published:
Intended Status:
Informational
Expires:
Authors:
L. Hopkins
E. Turner

Evidence Package Format Specification

Abstract

Taking evidence is a key part of any software testing process. This specification defines a format which collects evidence together and stores metadata and annotations in an organised fashion from both manual and automated testing sources.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet-Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 16 November 2025.

▲

Table of Contents

1. Introduction

1.1. Purpose

The purpose of this specification is to define a format for storage of test evidence that:

  • allows for basic collation of evidence;
  • can store any kind of file type that might be produced;
  • stores data compressed;
  • stores related evidence together, but allows for dividing up by test case, and;
  • is built upon widely available standards.

The format does not attempt to:

  • act as an captioned archiving solution for other purposes, even if it may be suitable for them.

1.2. Intended Audience

This specification is intended for those who might wish to write their own implementation of the evidence package format.

1.3. Changes from Previous Verisons

This document forms the original specification.

2. Terminology

The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

3. Specification

An evidence package is a structured ZIP archive. It MUST contain the file "manifest.json", and the directories "media" and "testcases".

See Appendix A for an example of the file's internal structure.

3.1. "manifest.json" File

The manifest.json file defines metadata relating to the entire package of evidence. It MUST be a JSON [RFC8259] file with the following elements:

Table 1
Element Condition Type Section Description
$schema Optional String The $schema element MAY point to a copy of the schema for the manifest.
metadata Mandatory Object Section 3.1.1 The metadata element stores package metadata.
custom_test_case_metadata Mandatory Object Section 3.1.2 Custom metadata fields for test cases in this package.
media Mandatory Array Section 3.1.3 The media element stores a list of media files that are stored in this evidence package.
test_cases Mandatory Array Section 3.1.4 The test_cases element stores a list of test cases.

See an example manifest.json file in Appendix B.

3.1.1. "metadata" Element

Table 2
Element Condition Type Section Description
title Mandatory String The name of the evidence package.
authors Mandatory Array Section 3.1.1.1 The authors attributed to this evidence package.
3.1.1.1. "authors" Array Element
Table 3
Element Condition Type Description
name Mandatory String The author's name.
email Optional String/Null The author's email address, although format is not verified.

3.1.2. "custom_test_case_metadata" Element

Elements within this object will become custom metadata properties for test cases in this package. Each object MUST have the following fields:

Table 4
Element Condition Type Section Description
name Mandatory String The name of this custom metadata field.
description Mandatory String Section 3.1.1.1 The description of this custom metadata field.
primary Mandatory Boolean Section 3.1.2.1 Is this custom field primary?
3.1.2.1. "primary" Boolean

The "primary" value of custom metadata fields MAY be false for all fields, or MAY be true for exactly one field. It MUST NOT be true for more than one field.

The purpose of primary is not enforced as part of this specification, however it should be seen as suggesting that one custom metadata field is more useful than others, and as such may be used to influence the information displayed to users, for example an implementor might choose to show the primary custom metadata value for each test case alongside it.

3.1.3. "media" Array Element

Table 5
Element Condition Type Description
sha256_checksum Mandatory String The SHA256 checksum of the associated media file.
mime_type Mandatory String The Internet Media Type [RFC2046] of the associated media file.

3.1.4. "test_cases" Array Element

Table 6
Element Condition Type Description
id Mandatory String The UUID of the test case. If present here, there MUST be an associated test case file in the "testcases" directory of the package with the name "<UUID>.json".

3.2. "testcases" Directory

The test cases directory stores the manifests for each test case within this evidence package.

Each test case is stored as a JSON file, with a UUIDv4 name [RFC9562].

3.2.1. "<uuid>.json" File

Table 7
Element Condition Type Section Description
$schema Optional String The $schema element MAY point to a copy of the schema for the manifest.
metadata Mandatory Object Section 3.2.1.1 The metadata relating to this test case.
evidence Mandatory Array Section 3.2.1.2 The evidence within this test case.

See an example <uuid>.json file in Appendix C.

3.2.1.1. "metadata" Element
Table 8
Element Condition Type Description
title Mandatory String The title of the test case.
execution_datetime Mandatory String The ISO8601 date and time of the execution of this test case starting.
passed Mandatory String The state of the test case, if present MUST be either "pass", "fail", or null. If absent, it MUST be interpreted as null.
custom Mandatory Object Custom metadata values.

The "custom" field is used to add custom metadata that has been specified in the package manifest's "custom_test_case_metadata" field. If a value is specified in "custom", it MUST be present in the package manifest, but all values in the package manifest do not need to be present here. All values MUST be strings.

3.2.1.2. "evidence" Array Element
Table 9
Element Condition Type Section Description
kind Mandatory String Section 3.2.1.2.1 The type of data stored.
value Mandatory String Section 3.2.1.2.2 The data stored within this piece of evidence.
caption Optional String/Null An optional caption for this piece of evidence.
original_filename Optional String/Null The original filename for File evidence.
3.2.1.2.1. "kind"

The "kind" of evidence MUST be one of "Text", "RichText", "Image", "Http", "File".

For more information about each type, see Section 5.

3.2.1.2.2. "value"

The "value" MUST be one of the following acceptable patterns:

  • "plain:" followed by plain text;
  • "media:" followed by a media file SHA256 hash, or;
  • "base64:" followed by a base64 string of data without padding.

3.3. "media" Directory

The "media" directory stores data in files within the ZIP archive that would be otherwise impractical to store directly in the test cases.

Files stored in this directory are of abitrary type. They MUST be named by their SHA256 checksum [RFC6234] with no extension. Their SHA256 checksum and media type MUST be stored in the package manifest "media" element.

In the unlikely event that there is a checksum clash, there is currently no preferred method for resolving this. The probability of such a situation is decided to be acceptably low given the expected size and number of files stored in an evidence package, however implementors MAY choose to store the clashing file as base64 data instead of as an additional media file.

4. Handling an Evidence Package

4.1. Locking

When loading an evidence package, implemetors MUST use a lock file with the file name ".~lock." followed by the full name of the package it protects, followed by "#", for example for a package called "example.evp", the lock file MUST be called ".~lock.example.evp#". It MUST be located adjacent (in the same directory as) the evidence package. The file MUST contain the process ID of the process holding the lock.

The lock file should be considered as locking the package if it is present, regardless of contents.

If either of these is not the case, it should be assumed that the there is no current lock over the package.

4.2. Media Loading

Software implementing the evidence package format MUST NOT load files from the "media" directory into memory until it is needed for display or for extraction. Implementors MUST use streams to load media files to avoid trying to load the entire file into memory as it may not fit.

5. Kinds of Evidence

Evidence packages support the following kinds of evidence:

Table 10
Kind Description
"Text" Plain text with no formatting.
"RichText" Text with very basic markdown support.
"Image" An image that should be rendered where possible.
"Http" An HTTP request/response pair.
"File" A raw file, which may be text or binary in nature.

Implementors MUST support all of these kinds, and MUST NOT introduce new kinds.

5.1. RichText's Markdown

The RichText evidence kind supports a very limited version of markdown:

  • Headings 1-6
  • Bold, Italic, Monospace
  • Tables
  • Code blocks with syntax highlighting

Implementors MUST NOT process any other markup.

5.2. HTTP Requests

Where HTTP is used, a Record Separator character (0x1e) can be used to split the request and response portion, for example the separator is present at 1:

GET / HTTP/1.1
Host: example.com
User-Agent: HTTPie

\x1eHTTP/1.1 200 OK <1>
Cache-Control: max-age=1366
Connection: close
...

6. Extending Behaviours of an Evidence Package

Every JSON file within an evidence package MAY have new fields added, and as such extended behaviours MAY be implemented, however implementors MUST be able to load an evidence package without these additional fields.

When an implementor loads a file with fields it cannot understand, it MUST retain the fields on saving the file.

7. IANA Considerations

This document acts as the specification for the requested media type application/vnd.angel.evidence-package.

8. Security Considerations

The evidence package format can store arbitrary files that may or may not be executable. Implementors MUST NOT execute any file contained within and SHALL only extract the contained files if needed.

Otherwise, there are no concerns for security from the file type itself.

9. Normative References

[RFC2046]
Freed, N. and N. Borenstein, "Multipurpose Internet Mail Extensions (MIME) Part Two: Media Types", RFC 2046, DOI 10.17487/RFC2046, , <https://www.rfc-editor.org/info/rfc2046>.
[RFC2119]
Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, , <https://www.rfc-editor.org/info/rfc2119>.
[RFC6234]
Eastlake 3rd, D. and T. Hansen, "US Secure Hash Algorithms (SHA and SHA-based HMAC and HKDF)", RFC 6234, DOI 10.17487/RFC6234, , <https://www.rfc-editor.org/info/rfc6234>.
[RFC8174]
Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, , <https://www.rfc-editor.org/info/rfc8174>.
[RFC8259]
Bray, T., Ed., "The JavaScript Object Notation (JSON) Data Interchange Format", STD 90, RFC 8259, DOI 10.17487/RFC8259, , <https://www.rfc-editor.org/info/rfc8259>.
[RFC9562]
Davis, K., Peabody, B., and P. Leach, "Universally Unique IDentifiers (UUIDs)", RFC 9562, DOI 10.17487/RFC9562, , <https://www.rfc-editor.org/info/rfc9562>.

Appendix A. Example Archive Layout 

example.evp
 |- manifest.json
 |- media
 |   \- 203073da0b36a5921f2914e2093abcae7eb987846f405b438c25792bab1617fa
 \- testcases
     \- eabb5d31-a958-4609-ac98-83365e14d18b.json

Appendix B. Example Package Manifest JSON 

{
  "metadata": {
    "title": "Example Evidence Package",
    "authors": [
      {
        "name": "Anonymous Author"
      },
      {
        "name": "Lily Hopkins",
        "email": "lily@hpkns.uk"
      }
    ]
  },
  "custom_test_case_metadata": {
    "example": {
      "name": "Example Metadata Field",
      "description": "A field showing that custom fields can be added",
      "primary": true
    }
  },
  "media": [
    {
      "sha256_checksum": "203073da0b36a5921f2914e2093abcae7eb987846f405b438c25792bab1617fa",
      "mime_type": "text/plain"
    }
  ],
  "test_cases": [
    {
      "id": "eabb5d31-a958-4609-ac98-83365e14d18b"
    }
  ]
}

Appendix C. Example Test Case Manifest JSON 

{
  "metadata": {
    "title": "Example Test Case",
    "execution_datetime": "2025-05-01T11:13:29+01:00",
    "passed": null,
    "custom": {
      "example": "Example custom metadata field value"
    }
  },
  "evidence": [
    {
      "kind":"Text",
      "value":"plain:This is some text based evidence"
    },
    {
      "kind":"Text",
      "value":"base64:VGhpcyBpcyBzb21lIHRleHQgYmFzZWQgYmFzZTY0IGVuY29kZWQgZXZpZGVuY2U"
    },
    {
      "kind":"File",
      "value":"media:203073da0b36a5921f2914e2093abcae7eb987846f405b438c25792bab1617fa",
      "caption": "An example file",
      "original_filename": "example.txt"
    }
  ]
}

Appendix D. JSON Schema for Package Manifest 

{
  "$id": "https://evidenceangel-schemas.hpkns.uk/manifest.2.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "description": "The metadata file `metadata.json` as part of an evidence package.",
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "title": {
          "type": "string",
          "description": "The name of the evidence package.",
          "minLength": 1,
          "maxLength": 30
        },
        "authors": {
          "type": "array",
          "description": "The authors attributed to this evidence package.",
          "items": {
            "type": "object",
            "properties": {
              "name": {
                "type": "string",
                "description": "The author's name."
              },
              "email": {
                "type": "string",
                "description": "The author's email address, although format is not verified."
              }
            },
            "required": ["name"]
          }
        },
        "description": {
          "type": "string",
          "description": "An optional description of the package."
        }
      },
      "required": ["title", "authors"]
    },
    "custom_test_case_metadata": {
      "type": "object",
      "description": "Custom metadata fields for test cases",
      "patternProperties": {
        ".+": {
          "type": "object",
          "description": "A custom metadata field",
          "properties": {
            "name": {
              "type": "string",
              "description": "A user-friendly name for this custom property."
            },
            "description": {
              "type": "string",
              "description": "A description for this custom property."
            },
            "primary": {
              "type": "boolean",
              "description": "Is this custom property the main one in this package? This may influence how it is displayed in editors."
            }
          },
          "required": ["name", "description", "primary"]
        }
      }
    },
    "media": {
      "type": "array",
      "items": {
        "type": "object",
        "description": "A media entry. When an entry is present in this manifest, it MUST also be present in the `media` directory of the package.",
        "properties": {
          "sha256_checksum": {
            "type": "string",
            "description": "The SHA256 checksum of the media file. This MUST also match identically the name of the file with no extension in the `media` directory.",
            "pattern": "^[0-9a-f]{64}$"
          },
          "mime_type": {
            "type": "string",
            "description": "The MIME type of the media file."
          }
        },
        "required": ["sha256_checksum", "mime_type"]
      }
    },
    "test_cases": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "id": {
            "type": "string",
            "format": "uuid",
            "description": "The UUID of the test case. If present here, there MUST be an associated test case file in the `testcases` directory of the package with the name `<UUID>.json`."
          }
        },
        "required": ["id"]
      }
    }
  },
  "required": ["metadata", "media", "test_cases"]
}

Appendix E. JSON Schema for Test Case Manifest 

{
  "$id": "https://evidenceangel-schemas.hpkns.uk/testcase.1.schema.json",
  "$schema": "http://json-schema.org/draft-07/schema",
  "type": "object",
  "description": "A test case file `testcases/<UUID>.json` as part of an evidence package.",
  "properties": {
    "metadata": {
      "type": "object",
      "properties": {
        "title": {
          "type": "string",
          "description": "The title of the test case",
          "minLength": 1,
          "maxLength": 30
        },
        "execution_datetime": {
          "type": "string",
          "format": "date-time",
          "description": "The date and time of the execution of this test case starting."
        },
        "passed": {
          "type": ["string", "null"],
          "description": "The state of the test case",
          "enum": [
            "pass",
            "fail",
            null
          ]
        },
        "custom": {
          "type": "object",
          "description": "Custom metadata values",
          "patternProperties": {
            ".+": {
              "type": "string"
            }
          }
        }
      },
      "required": ["title", "execution_datetime"]
    },
    "evidence": {
      "type": "array",
      "items": {
        "type": "object",
        "description": "A piece of evidence as part of this test case.",
        "properties": {
          "kind": {
            "type": "string",
            "description": "The type of data stored. Note that where `Http` is used, a Record Separator character (0x1e) can be used to split the request and response portion.",
            "enum": ["Text", "RichText", "Image", "Http", "File"]
          },
          "value": {
            "type": "string",
            "description": "Either `plain:` followed by plain text, `media:` followed by a media SHA256 hash, or `base64:` followed by a base64 string of data without padding.",
            "pattern": "^(plain:.*)|(media:[0-9a-f]{64})|(base64:[A-z0-9+/]*)$"
          },
          "caption": {
            "type": "string",
            "description": "An optional caption for this piece of evidence."
          },
          "original_filename": {
            "type": "string",
            "description": "The original filename for File evidence"
          }
        },
        "required": ["kind", "value"],
        "if": {
          "properties": {
            "kind": { "const": "File" }
          }
        },
        "else": {
          "not": {
            "required": ["original_filename"]
          }
        }
      }
    }
  },
  "required": ["metadata", "evidence"]
}

Authors' Addresses

Lily Hopkins
URI: https://github.com/lilopkins
Eden Turner
URI: https://github.com/Some-Birb7190