DSHIO/T 21—2026 Test Conditions

---

📋 Current scope

This standard defines six test stages (T0–T5). Current implementation status:

Stage	Name	Status
T0	Static check	✅ Available
T1	Install	✅ Available
T2	List	✅ Available
T3	Mount	✅ Available
T4	Boot	✅ Available
T5	Tool invocation	🕐 Planned

Why T5 is only planned: it involves model cost and nondeterminism (see DSHIO/T 32 clause 4.3). It is written into the standard but not yet scheduled. Until it is formally included, any report claiming a T5 run is void.

Manual-judgement clauses (5.5.1, 5.5.2) are the author's own responsibility. DSHIO does not issue verdicts on them. Producing an opinion on them is a paid service, and the opinion is advisory only.

This notice prevails over any T5-related provision elsewhere in this standard.

---

1 Scope

This standard specifies the environmental, resource, network and timing conditions under which each stage of DSH plugin testing takes place.

It applies to any person or system that makes a verdict on a plugin according to the DSHIO standards.

It does not specify the test methods themselves — see DSHIO/T 20.

Why this standard exists. A test result without fixed conditions is one person's impression on one machine on one day. With fixed conditions, the same report can be reproduced, challenged and audited by someone else.

---

2 Normative references

The clauses of the following document become clauses of this standard by virtue of the reference.

---

3 Terms and definitions

The terms defined in DSHIO/T 00—2026 apply. Additional definitions:

3.1 fixed condition

An environmental factor that shall not change during a test run. Changing any one of them constitutes a new test, and its result shall not be compared with the original.

3.2 variable condition

A condition that is permitted to change within a single test run. Variable conditions shall be recorded item by item in the test report.

3.3 test environment fingerprint

A set of values sufficient to uniquely identify the test environment, expressed as a string, for comparison purposes.

Example: dsh=0.2.2-rc.1;img=dsh-sandbox:5.0@sha256:...;node=24.11.1;pnpm=9.15.9

---

4 Condition classification

4.1 Two classes of test condition

Basis: DSHIO/T 00—2026 clause 3.4.1 — DSH install, mount and boot do not depend on a specific model; model access is provided by separate plugins.

Implication: a T0–T4 verdict holds under any model; a T5 verdict holds only under the named model.

---

5 Model-independent conditions (T0–T4)

5.1 Fixed condition list

The following conditions shall be fixed during T0–T4 and shall be written into the test report.

5.1.1 DSH version

Current baseline version: 0.2.2-rc.1

Note: a report that states a range (e.g. "the 0.2.2 series") is not reproducible and cannot be used as a basis for certification.

5.1.2 Container image

Shall be fixed inside the image:

Why the digest matters. A tag can be overwritten — same name, different content. Without a digest you cannot reproduce the run a year later.

5.1.3 Resource limits

T0 (static check) needs no container and has no resource limits.

5.1.4 Network state

Why T2–T4 shall have no network. During install a plugin can fetch anything from the network. With the network off, any lingering request fails and is exposed; and even a malicious plugin has nowhere to send data. This is a security design as well as a reproducibility design.

5.1.5 Permissions and isolation

Note: T1 needs to install, so the ordinary privileges pnpm requires are kept. From T2 onward they are tightened progressively.

5.1.6 Timeouts

The 12-second liveness observation at T4: after the profile starts it shall stay alive for 12 consecutive seconds without exiting to pass.

Where 12 seconds comes from. Long enough to cover the common failure window of plugin initialisation, config loading and the first event-loop turn; longer would markedly slow batch testing, and in practice no failure mode has been observed that only appears after 12 seconds.

5.1.7 Working directory and profile

What "clean" means: in the profile directory, package.json's dependencies has no entries other than the target plugin of this test.

5.1.8 Command sequence

The command sequence for T1–T4 shall be identical, verbatim:

# T1 install (network on)
dsh plugin --profile web add <package-spec>

# T2 list (network off)
dsh plugin --profile web list

# T3 mount (network off)
dsh --profile web --dump-config

# T4 boot (network off, observe 12 s)
dsh --profile web

No insertion, omission or reordering is permitted.

5.1.9 Repeat count

When the two verdicts differ:

1. Record the verdict as unstable
2. Both full outputs shall be recorded
3. An unstable result shall not be used as a basis for certification

---

5.2 Test environment fingerprint under model-independent conditions

Format:

dsh=<DSH version>;img=<image name>@<digest>;node=<node version>;pnpm=<pnpm version>;mem=<memory>;cpu=<CPU>

Example:

dsh=0.2.2-rc.1;img=dsh-sandbox:5.0@sha256:abc123...;node=24.11.1;pnpm=9.15.9;mem=768m;cpu=1

Same fingerprint = same conditions. When fingerprints differ, results shall not be compared directly.

---

6 Model-dependent conditions (T5)

6.1 Why it needs separate rules

T5 tests "whether the model correctly invokes a tool the plugin registers". The result of this stage depends on both the plugin and the model; changing the model can reverse the conclusion entirely.

Therefore: a T5 verdict is only meaningful when bound to a specific model.

6.2 Fixed conditions

In addition to all conditions in chapter 5, T5 shall also fix the following:

6.2.1 Model identifier

6.2.2 Generation parameters

A T5 report that does not record temperature and max_tokens is not reproducible.

6.2.3 Prompts

Note: every character of a prompt affects model behaviour. Writing only "ask it to call the tool" does not constitute a reproducible condition.

6.2.4 Tool-calling configuration

Why record the total tool count. The more candidate tools, the higher the chance the model picks wrong. "Pick correctly among 3 tools" and "among 200" are not the same difficulty.

6.2.5 Repeat count

Pass criteria (choose one; the report shall state which):

Why the T5 bar is higher than T0–T4. Models are stochastic; passing twice could be luck.

6.3 Model-dependent test environment fingerprint

<model-independent fingerprint>;model=<model ID>;temp=<temperature>;maxtok=<max output>;tools=<total tools>

Example:

dsh=0.2.2-rc.1;img=dsh-sandbox:5.0@sha256:abc...;node=24.11.1;pnpm=9.15.9;mem=768m;cpu=1;model=deepseek-v4.1-flash-0731;temp=0;maxtok=4096;tools=7

---

7 Mandatory fields of a test report

7.1 All tests

7.2 Additional mandatory fields for T5

---

8 Handling condition deviation

8.1 What counts as deviation

Changing the value of any fixed condition in chapter 5 or chapter 6 constitutes a deviation.

8.2 Requirements for deviation

Deviation is not prohibited, but it shall:

1. Be marked explicitly in the report as "condition deviation"
2. List item by item what deviated, what it was changed to, and why
3. The verdict shall not be compared with undeviated reports
4. shall not be used as a basis for certification

8.3 Common deviations

---

9 Stability of conditions

9.1 Monitoring condition drift

At least once every 30 days, verify the following against this standard:

When drift is found:

1. Mark old test results as "conditions changed"
2. Re-test key samples under the new conditions
3. Update this standard or restore the conditions

9.2 Why monitor

If image tag 5.0 is rebuilt, the Node version under the same tag may have changed. Without monitoring, test results silently lose comparability.

---

Annex A (normative) Condition quick-reference

Note: T2–T4 are not repeated separately; they are part of the same sequence that follows T1. "Repeat runs" refers to how many times the whole sequence is executed.

---

Annex B (informative) Relationship to other standards

DSHIO/T 00  Terms and Definitions
      ↓ referenced by
DSHIO/T 21  Test Conditions (this document)
      ↓ referenced by
DSHIO/T 20  General Rules for Testing  ← what each stage does
      ↓ referenced by
DSHIO/T 10  Plugin Standard            ← what a plugin shall achieve
      ↓ referenced by
DSHIO/T 32  Certification Rules        ← how conformance is certified

This standard answers only "under what conditions", not "how to test" or "what counts as passing".

---

Revision record