Prove the Connected System package — manifests, pipeline, live vendor or CIP behaviour, and gold rows — before publish. Tests are a first-class platform contract: they certify that mappings and calculated pictures still match known fixtures. A green system ladder does not prove the assistant still gives correct governed answers; that is a separate proof — Proving Role Assistants.
Author the contract on Configure validation and test payload. This page is how to run it.
Prerequisites
aifabrix auth statusgreen for dataplane runs- Package folder
integration/<systemKey>/validates locally - Sandbox credentials for live E2E; skip E2E with a recorded reason when the vendor secret is missing
Where it lives
| Layer | Typical command | Proves |
|---|---|---|
| Local | aifabrix validate <systemKey> then aifabrix test <systemKey> |
Schema and mapping shape on disk |
| Structure | aifabrix datasource test <datasourceKey> --app <systemKey> |
Dataplane structural/policy run for one datasource |
| Pipeline | aifabrix datasource test-integration <datasourceKey> --app <systemKey> |
Pipeline against the dataplane (one datasource) |
| Pipeline (system) | aifabrix test-integration <systemKey> |
App-wide integration rollup |
| Live E2E | aifabrix datasource test-e2e <datasourceKey> --app <systemKey> |
One datasource: credentials, sync, CIP |
| Live E2E (system) | aifabrix test-e2e <systemKey> |
Every datasource via the dataplane |
Protection, governance, and certification wrap this ladder — see Test ladder and protection.
Proof contract (what green means)
| Layer | What a pass means |
|---|---|
Local validate / test |
Schema and mapping shape on disk |
| Structure / pipeline | Dataplane admits the package and pipeline |
Live E2E without expectedRows |
Credentials, sync, and CIP invoke succeeded |
Live E2E with expectedRows |
CIP output rows match gold identities and values |
Mapping expectedResult is one-record shape — not totals. Scenario expectedRows + identityKey prove list, get, and named viewpoint answers. PostgreSQL remains the calculator. Empty expectedRows means expect zero rows; omitting the field means do not assert rows. listWindow is the list probe (default page 1 / pageSize 5); it does not cap live completeness.
How to set
- Local first after every JSON edit:
aifabrix validate <systemKey>
aifabrix test <systemKey>
- One datasource — structure, then pipeline (after upload or with
--syncwhen your workflow publishes first):
aifabrix datasource test <datasourceKey> --app <systemKey>
aifabrix datasource test-integration <datasourceKey> --app <systemKey>
- Live E2E when credentials exist:
aifabrix datasource test-e2e <datasourceKey> --app <systemKey>
Load known fixtures before E2E when the package is static. If the vendor secret is not stored, skip live E2E with a reason — do not treat a missing call as a pass.
-
Gold rows — author them on the datasource before you treat E2E as certification proof. Field list and examples: Configure validation and test payload. Write scenarios should gold persisted fields on a following
get. OmitexpectedRowsonly when success-only is enough. -
Focus one failure before rerunning the whole system. For a named capability supported by the datasource contract:
aifabrix datasource test-e2e <datasourceKey> <capabilityKey> \
--app <systemKey> --env dev --version latest \
--strict-capability-scope --debug full --json
latest tests the newest candidate when one exists; use --version running when you intentionally prove the production pointer. --strict-capability-scope fails when a requested drill-down unexpectedly returns multiple capability rows.
Defaults and examples
| Scope | Use |
|---|---|
| One datasource | datasource test / test-integration / test-e2e with --app <systemKey> |
| Whole system | test, test-integration, test-e2e on <systemKey> |
| Mapping unit | Root expectedResult |
| List, get, grouped CIP | Scenario expectedRows + identityKey |
| List probe size | testPayload.listWindow (not live paginate.maxPages) |
| Human diagnosis | --debug summary or --debug full; omit --json to see the terminal appendix |
| AI or CI evidence | --debug full --json; add --warnings-as-errors --require-cert for a strict gate |
| Candidate versus deployed | --version latest (default) versus --version running |
| Audit proof after E2E | --verify-audit (requires audit:read) |
Validate
A green invoke is not enough when expectedRows is set — the run must match gold identities and values. Failures attach a row diff on the capacity step.
Debug one layer at a time
Datasource test, test-integration, and test-e2e accept --debug summary, full, or raw. Start with full; raw has only basic redaction and should be restricted to an authorized investigation. Debug runs write timestamped JSON under integration/<systemKey>/logs/. Open the latest file for the failed layer:
aifabrix datasource log-test <datasourceKey> --app <systemKey>
aifabrix datasource log-integration <datasourceKey> --app <systemKey>
aifabrix datasource log-e2e <datasourceKey> --app <systemKey>
Keep one diagnostic bundle for a human or AI reviewer: the exact command, environment, latest or running version target, manifest revision, sanitized payload, fixture identity, JSON report, matching debug log, correlation/execution identifiers, failed step, and capacity row diff. Record credential state, but never copy a secret or token into the bundle.
| Failure signal | Inspect first |
|---|---|
| Local validate or test | Manifest schema, mapping expression, payloadTemplate, and expectedResult |
| Structural run | Published candidate, datasource key, and dataplane policy report |
| Integration run | Dataplane access, payload shape, mapping, and Dimension Catalog references |
| Credential step | Credential slot, OAuth expiry/scopes, vendor endpoint, and environment |
| Sync step | Provider response, pagination, rate limits, mappings, and sync mode |
| Data step | Persisted records/documents, primary key, and vectorization where applicable |
| Capacity/CIP row diff | Scenario operation, identityKey, expectedRows, normalized output, and calculation source |
| More than one capability in a focused run | Capability contract; retain --strict-capability-scope |
| E2E passed but audit evidence is absent | --verify-audit, audit:read, correlation id, and dataplane capture |
| System tests pass but the business answer is wrong | Run the Role Assistant case; do not rewrite Connected System gold to fit the answer |
Give the AI reviewer a bounded instruction with the bundle:
Identify the first failed layer and cite its step, row diff, correlation or execution id, and manifest field. Classify the owner as fixture, manifest/schema, credential, vendor API, sync, persistence, CIP, audit, or Role Assistant. Propose the smallest owner-layer change and the exact focused command to rerun. Do not change gold rows merely to match incorrect output.
After the fix, rerun the same focused command unchanged, then the datasource ladder, and finally the system-level command. A changed test and changed implementation in the same run are not regression proof unless the original contract was wrong.
Common mistakes
| Mistake | Fix |
|---|---|
Using role-assistant test to prove SUM/COUNT |
Author gold on the Connected System datasource |
Treating mapping expectedResult as totals |
Use scenario expectedRows |
| Copying a capability and treating cloned gold as certified | Edit expectedRows for the new operation |
E2E without --app on a shared workspace |
Pass --app <systemKey> and the datasource key |
| Fake-passing E2E when the secret is missing | Skip with reason |
Lowering live paginate.maxPages to speed tests |
Set testPayload.listWindow |
Limits
Live E2E and governance hit vendor APIs and identity — use sandbox data and a designated test subject. Empty expectedRows means expect zero rows; omitting the field means do not assert rows. Certification still requires the full operations pillar after this ladder.