Test Role Assistants

Author YAML cases, round-trip them with the package, and prove the assistant after you promote — with a portal deep link for human review.

Use cases as an improvement tool: start with one observable business problem, make it reproducible, correct the responsible package or product layer, and rerun the same case before widening to the suite.

Prove catalog (Validate and test): command families, CS vs RA boundary, --env / --case / --suite / --from-dataplane, and ad-hoc aifabrix search live on Role Assistant tests. This Operate page is package authoring (YAML, Evidence, Knowledge, copy) and promote-then-test. It does not replace Connected System unit tests or integration package scoring.

Why this matters: Production needs two proofs — certified Connected Systems, then a Role Assistant that still gives correct governed answers for an identified person. Agent SDK evals and encoded workflow graphs do not replace that. Business case: Proving Role Assistants.

Prerequisites

• Logged in: aifabrix auth status (the command fails if not authenticated — it never skips)
• Role Assistant package available locally or already running
• Target environment: dev, tst, or pro
• When settings changed, upload the target environment first. When Evidence
  changed, upload imports the candidate rows; complete their governed review
  and promotion before a test relies on them.

Promote path: Manage Role Assistant packages.
Evidence readiness: Author Role Assistant Evidence.

Where it lives

aifabrix validate ra-<roleKey> checks every governed file (settings, YAML cases, Evidence JSON, Knowledge markdown). Unknown YAML keys fail. An empty tests folder is skipped on upload — it does not fail publish.

package-score measures Connected System folder parity. It does not evaluate Role Assistant YAML.

For a test started in the portal, keep the Tests page open while the job is
queued or running. Its polling renews the starter's short-lived sign-in for
that job. If renewal fails, remaining cases are recorded as skipped with an
expiry reason; skipped cases do not satisfy release coverage. Cancel run
stops new cases from starting, while a case already in progress finishes.

How to set

1. Auth — never proceed when status is not authenticated:

aifabrix auth status

2. Download and validate the package before you treat YAML as a contract:

aifabrix download ra-sales-representative
aifabrix validate ra-sales-representative

3. Author cases. executionPath belongs under input (not at the YAML root). Answers and approvals need when plus value. expected.facts uses explicit dotted paths — missing paths fail; wildcards and prototype keys are rejected. Every authored expectation produces a visible result row (pass, fail, warn, notes, or skip) — never silent.

When the published capability catalog exposes and certifies customer.upsert, prefer it for requests where the caller supplies a business identity but should not decide whether the record already exists. The governed pipeline resolves the identity and selects create or update; ambiguous identity and missing authority fail closed.

Example hard case:

caseId: customer-upsert-release-proof
input:
  executionPath: workHub
  message: >-
    Create or update test customer RA-UP-20260925-001 using customer.upsert.
    Use domain ra-up-20260925-001.example.test as its business identity.
    Set city to Helsinki and country to Finland.
answers:
  - when:
      questionKey: customer.upsert
    value:
      name: RA Upsert 20260925 001
      domain: ra-up-20260925-001.example.test
      city: Helsinki
      country: Finland
approvals: []
expected:
  soft: false
  lifecycleState: completed
  capabilities:
    required:
      - customer.upsert
  honesty:
    requireWorkResult: true
    requireCompletion: true
    minResourceChanges: 1
  prohibited:
    - workResultError

Replace the example identity for every run. Do not copy repository-only interpolation tokens into a public case.

4. Copy from Work Hub when you already have a completed execution (the execution id, not a case id):

aifabrix role-assistant copy sales-representative --suite 001 --reference <executionId>

Copy writes local YAML only. It does not invent a prompt. --dry-run prints without writing. Flags: aifabrix role-assistant copy --help.

5. Upload, review Evidence, then prove — catalog commands:

aifabrix upload ra-sales-representative --env dev
aifabrix role-assistant test sales-representative --env dev
aifabrix role-assistant test sales-representative --env dev --list-cases
aifabrix role-assistant test sales-representative --env dev --suite 001 --fail-fast --json

Default uses the public/running assistant. --from-dataplane runs the YAML that already travelled with that assistant.

Package upload applies the selected environment settings and imports local
Evidence as candidates. It does not certify or activate those rows. Review the
returned identities and pending lifecycle actions, then have an authorized
human complete Evidence review and promotion in Assistants → Evidence before
using an Evidence-backed case as release proof.

auth status
  → validate package
  → author or copy YAML
  → upload settings and candidate Evidence
  → governed Evidence review / promotion when applicable
  → role-assistant test (--case / --suite)
  → portal review

Defaults and examples

aifabrix role-assistant test sales-representative --env dev
aifabrix role-assistant test sales-representative --suite 001 --json

Alias: aifabrix test ra-<roleKey> uses the package key and follows the same lifecycle path.

Use expected.soft: true only for exploration or diagnosis. A mutation case used as release evidence should keep lifecycle, required and forbidden capabilities, Work Result, completion, and resource-change assertions hard. Use stable fixtures for read-only cases and collision-safe identities for mutations. An authority-negative case should prove the expected denial or safe stop without weakening policy.

Test insert, update, and replay

An upsert suite should prove the business identity, not merely that the capability was called:

Upsert is not a substitute for replay proof. Identity resolution depends on the datasource's declared uniqueness rules and available record state. When exact de-duplication matters after bulk writes, synchronize the datasource before the replay test. If no certified upsert capability is published, keep separate hard create and update cases instead of inventing customer.upsert.

Validate

After a failed suite, open the portal link, classify the failure, fix Evidence
or Knowledge if the ask was wrong, re-upload when needed, complete Evidence
review when applicable, then re-run the same --case followed by its --suite.
Retain both JSON results. A soft or incomplete proof exits 2; do not convert a
release case to soft merely to make its assertions non-blocking.

Improve the layer that owns the failure. Change the case only when its expectation is wrong; change the fixture when its identity or state is wrong; change Knowledge or Evidence when the assistant's business understanding is wrong; change the Connected System when mapping or capability execution is wrong; and preserve the execution reference for a Runtime defect. Do not weaken a correct authority denial merely to make the suite green.

Enable deployed debug only when needed

CLI diagnostics and deployed tracing solve different problems. Start with the focused CLI run:

aifabrix role-assistant test <roleKey> --env dev --case <caseId> --json --verbose --debug

If its JSON, local debug log, portal execution, and assertion rows do not explain the failure, temporarily enable the required deployed trace in the dev section of role-assistant.yaml:

dev:
  debug:
    enabled: true
    includeRuntimeLog: true
    includePromptTrace: false
    includeCapabilityTrace: true

Validate and upload the changed settings, then rerun the same case:

aifabrix validate ra-<roleKey>
aifabrix upload ra-<roleKey> --env dev
aifabrix role-assistant test <roleKey> --env dev --case <caseId> --json --verbose --debug

Use the Runtime log for execution and continuation faults, and the capability trace for selection or observed-capability faults. includePromptTrace enables full LLM prompt capture: leave it false unless an authorized dev investigation requires it. Treat captured prompts and traces as restricted diagnostic evidence, put no secrets in role-assistant.yaml, and disable unneeded tracing and upload the restored settings after diagnosis.

The CLI --debug flag only creates CLI diagnostics; it never changes these deployed settings. Preserve the JSON result, local debug log, portal URL, process execution id, environment, fixture identity, and package revision as one reproducible bundle. An AI assistant can then compare the failed assertion with the Runtime and capability trail instead of guessing from the final answer.

Common mistakes

Limits

• Exact flags follow your Builder version (aifabrix role-assistant test --help).
• Live copy from a real Work Hub execution requires the platform copy route. The CLI is shipped; if copy fails closed, keep authoring YAML until that route is available.
• Full Evidence and Knowledge authoring are separate how-tos linked below.
• Case roots may live beside the package or in a repo cases directory — use --list-cases and --cases-root when discovery is empty.
• Portal sign-in renewal is scoped to the person who started the active job. It
  does not fall back to an application identity or broaden that person's
  permissions.

Related

• Role Assistant tests
• Manage Role Assistant packages
• Author Role Assistant Evidence
• Author Role Assistant Knowledge
• CLI reference
• Proving Role Assistants
• How Role Assistants are created