Prove a Role Assistant in dev, tst, or pro. This page is the Validate and test catalog for role-assistant test. It does not replace Connected System structure, pipeline, or E2E.
A green Connected System test is not production proof for the assistant. Agent SDK evals and encoded workflow graphs are not substitutes either. Why both proofs matter: Proving Role Assistants.
RA cases prove the business capability contract — period, measure definition, currency, aggregation, user scope, approved sources, and the absence of unauthorized operations — not only whether the underlying model produced a plausible answer. Databricks and Microsoft already offer agent and model evaluation; that is a different claim. A business capability contract is expected to keep holding when the underlying model changes, which is why it is tested and governed separately from model behaviour. See Business meaning as governed content for why that contract is governed content, not application code.
YAML case authoring, Evidence, Knowledge, copy from Work Hub, and promote stay under Operate: Test Role Assistants (package).
The practical goal is a short improvement loop: reproduce one wrong outcome, identify the responsible layer, improve it, rerun the same case, then prove the suite and published package.
Ad-hoc retrieve (no YAML) uses CLI reference — Search (aifabrix search / search skills). It uses the same governed retrieve as YAML retrieve cases. It does not start work. Pass --graph to include one level of related records on the returned rows — not multi-hop traversal or a generic corpus search; see CLI reference — Search for --graph / --query / --knowledge-key.
Prerequisites
aifabrix auth statusgreen (the command fails closed if you are not authenticated)- Role Assistant package available locally or already running
- Target environment:
dev,tst, orpro - Promote first when settings or Evidence changed
Where it lives
| Item | Location |
|---|---|
| Package | role-assistant/ra-<roleKey>/; package commands use ra-<roleKey>, while role-assistant test and copy use the bare role key |
| Cases | Suite folders of YAML cases discovered by the CLI |
| Portal | Deep link printed after lifecycle checks |
For local case discovery, explicit --tests-dir or --cases-root takes precedence over the package’s tests.path; otherwise Builder uses its supported default folders. Use --list-cases to confirm discovery before running a suite. A declared folder that does not exist fails package validation.
The portal Tests page runs published cases as the person who started the
job. Keep that page open while a long run is queued or running: its polling
renews the caller's short-lived sign-in for that job. If the sign-in cannot be
renewed, the case in progress reaches its outcome and remaining cases are
recorded as skipped with an expiry reason. Those skips are not passes.
package-score measures Connected System folder parity against a reference. It does not evaluate Role Assistant YAML cases.
How to set
- Auth — do not continue when status is not authenticated:
aifabrix auth status
- Lifecycle smoke (portal deep link). This proves the assistant is reachable in the chosen environment — not that a Connected System E2E is green:
aifabrix role-assistant test <roleKey> --env dev
Use --env tst or --env pro to match the package you promoted. A mismatch is a common false green.
- List cases, then one case or a suite:
aifabrix role-assistant test <roleKey> --env dev --list-cases
aifabrix role-assistant test <roleKey> --env dev --case smoke --json
aifabrix role-assistant test <roleKey> --env dev --suite 001 --fail-fast --json
Default uses the public/running Role Assistant. Pass --publish only when you intentionally republish. Use --json for machine-readable exit handling in CI.
- Published pack:
--from-dataplaneloads the YAML that already travelled with the running assistant, then runs it. Use this when the specification on disk may differ from what is published.
aifabrix role-assistant test <roleKey> --env dev --from-dataplane --suite 001 --json
For CI, the suite command is the normal call. Exit 0 means every selected
case has verified proof. Exit 2 means selected proof failed or remains not
verified, including soft cases, softened assertions, missing receipts, skipped
coverage, or incomplete execution. Command errors exit 3; a lifecycle-only
run keeps its lifecycle exit mapping.
- Improve from one failing case, then widen the proof:
# Reproduce one behavior and retain the JSON result
aifabrix role-assistant test <roleKey> --env dev --case <caseId> --json
# After the responsible package content is corrected
aifabrix validate ra-<roleKey>
aifabrix upload ra-<roleKey> --env dev
# Prove the correction, its suite, and finally the published pack
aifabrix role-assistant test <roleKey> --env dev --case <caseId> --json
aifabrix role-assistant test <roleKey> --env dev --suite <suiteId> --fail-fast --json
aifabrix role-assistant test <roleKey> --env dev --from-dataplane --suite <suiteId> --json
Upload only when the correction changes Role Assistant settings, Evidence, Knowledge, or cases. A fixture repair may need no package upload; a Connected System or Runtime defect must be fixed and proved by its owning test layer before the Role Assistant case is rerun.
Wait answers, Evidence preflight, and YAML case shape are on the Operate page — do not copy a second essay here.
Defaults and examples
| Goal | Command |
|---|---|
| Prove the assistant is up | role-assistant test <roleKey> --env <env> |
| Prove one YAML case | --case <caseId> |
| Prove a suite | --suite <suiteId> |
| Run published YAML | --from-dataplane |
| Stop on first case failure | --fail-fast |
| Retain machine-readable evidence | --json and inspect assertion plus warning rows |
| Ad-hoc retrieve (no YAML) | aifabrix search — CLI reference |
| Prove the Connected System | Connected System tests — not this command |
Validate
Do not mix test / test-e2e on a Connected System with role-assistant test. A green CS E2E does not prove the assistant; a green RA lifecycle smoke does not prove vendor CIP gold or grouped totals.
Certification of the Connected System remains verify-operations / verify-trust / verify-governance. Role Assistant prove stays this command family.
Package commands use the prefixed key—for example, aifabrix upload ra-sales-representative --env dev. The test command uses aifabrix role-assistant test sales-representative --env dev.
Use the failed assertion and portal execution to decide what to improve:
| Failure | Improve first |
|---|---|
| Missing or colliding test record | Fixture and cleanup discipline |
| Wrong expected state or fact path | YAML assertion contract |
| Wrong mapping, operation, or provider response | Connected System |
| Wrong question, business meaning, or source choice | Role Assistant Knowledge or Evidence |
| Repeated waits, continuation loop, or dishonest Result | Runtime; preserve the execution reference |
| Correct denial or approval wait marked as failure | Case expectation—not authorization policy |
Debug one failing case
Reproduce the smallest failure and keep both machine-readable output and local diagnostics:
aifabrix role-assistant test <roleKey> --env dev --case <caseId> --json --verbose --debug
--verbose includes the full lifecycle report in the JSON envelope. JSON has
one document: lifecycleVerdict describes package checks, while verdict,
proofScope, and proof describe selected-case proof. --debug writes CLI
diagnostic logs under role-assistant/<roleKey>/logs/. It does not enable
deployed Runtime, capability, or prompt tracing.
Keep one diagnostic bundle: the command and environment, case YAML, package revision, fixture identity, JSON result, debug log, portalUrl, processExecutionId, final status or wait reason, and every failed or warning assertion. Redact secrets and personal data before sharing the bundle with an AI assistant. Ask it to classify the owning layer before proposing a change:
| Signal | Inspect first |
|---|---|
| No case discovered | Package path, suite name, and cases root |
| Auth, publish, or lifecycle failure before execution | Session, environment, and deployed package |
| Evidence preflight failure | Required Evidence key, status, and target environment |
| Required or forbidden capability row fails | Knowledge/Evidence selection, then certified capability catalog |
| Repeated wait or unmatched interaction | questionKey, requestKey, approval answer, and continuation state |
workResultError or execution integrity failure |
Runtime and Connected System; preserve processExecutionId |
| Completion or resource-change row fails | Work Result honesty and authoritative readback |
| Fact path fails | Returned Result contract first; case expectation only if the contract is correct |
| Upsert is ambiguous or denied | Business identity or branch authority; never weaken policy |
Give the AI assistant a bounded instruction with that bundle:
Classify this failure as prerequisite, case contract, fixture, Knowledge, Evidence, Connected System, Runtime, or authorization. Cite the failed assertion and execution evidence. Propose the smallest owner-layer change and the exact focused-case and suite rerun commands. Do not weaken a hard assertion or authorization rule merely to make the test pass.
After the owner is fixed, rerun the unchanged case. Changing the case and the implementation together destroys the regression proof unless the original expectation itself was wrong.
Stop a portal run
Use Cancel run when a queued or running portal job should stop. Cancellation
is idempotent and prevents new cases from starting; a case already in progress
is allowed to finish. Review the final rows before starting another run so
cancelled and skipped cases are not mistaken for successful coverage.
Common mistakes
| Mistake | Fix |
|---|---|
Running CS test-e2e to prove the assistant |
Use role-assistant test |
Passing ra-<roleKey> to role-assistant test |
Use the bare role key; keep ra- for package commands |
Expecting a soft case to exit 0 |
Soft or incomplete case proof is not verified and exits 2 |
| Reading package lifecycle as selected-case proof | In JSON, inspect verdict, proofScope, and proof as well as lifecycleVerdict |
Treating package-score as RA cases |
Score is CS folder parity only |
| Closing the Tests page during a long run | Keep it open so polling can renew the starter's sign-in |
| Treating skipped cases after sign-in expiry or cancellation as passes | Restore the session or start a new run, then prove the skipped cases |
| Skipping promote after Evidence edits | Upload/promote, then test |
| Lifecycle smoke only, then shipping | Add --case / --suite for the behaviours you care about |
--publish on every CI run |
Default to the running package unless you mean to republish |
Using aifabrix search to start a change |
Search is retrieve only; start work from Work Hub or YAML workHub cases |
Limits
If a case cannot answer a required question, encounters the same question after
answering it, or fails during response, Builder ends polling with pollStopped
and a reason such as respond-skipped, respond-repeated or respond-error.
Inspect the execution and case answer fields, correct that failure layer, then
rerun the same case. This diagnostic stop is not a successful business outcome
and does not prove the server execution was cancelled.
Lifecycle smoke is not a substitute for YAML suites. Every authored expectation must produce a visible result row (pass, fail, warn, notes, or skip). Case shape, copy, and pack round-trip are on the Operate page. The command fails closed without a valid session — it never skips auth.
Portal renewal applies only to the person who started that run and only while
the job is active. It is not an application-token fallback and does not grant
authority that the person does not already hold.