Build a document storage entity

Step-by-step checklist for entityType: documentStorage — file libraries or keyed file retrieval with metadata normalization, document processing, and governed capabilities.

See Record, document, and service contracts for the business distinction between entity types.

Prerequisites

• External system JSON (often type: openapi for SharePoint, file APIs, or DMS)
• Resource type document (or domain-specific content key) in catalog
• Document provider credentials in env.template (kv:// paths)

Where it lives

integration/<systemKey>/<systemKey>-datasource-documents.json (one entity type per file).

Document storage uses the same root required fields as records, with entityType: documentStorage.

How to set

1. Copy a document-storage fixture (library/list API) and trim. Align systemKey and application manifest file list.

2. Set root identity:

{
  "key": "example-documents",
  "displayName": "Documents",
  "systemKey": "example-files",
  "entityType": "documentStorage",
  "resourceType": "document",
  "primaryKey": ["externalId"],
  "metadataSchema": {
    "type": "object",
    "properties": {
      "externalId": { "type": "string", "index": true },
      "name": { "type": "string", "index": true },
      "mimeType": { "type": "string", "index": true },
      "folderPath": { "type": "string", "index": true }
    }
  }
}

3. Map file metadata in fieldMappings.attributes — include paths for web URL, modified time, owner email, sensitivity labels. Materialize ABAC dimensions on indexed properties.

4. Choose library sync or keyed retrieval. Library sync enumerates metadata and downloads files. A file API that retrieves one document by business key can expose only get, without a list operation or scheduled sync. Validate either path early:

aifabrix datasource validate <datasourceKey>

On the root documentStorage configuration, ingestAfterSync determines whether eligible extracted text is considered for ingestion after sync; its default is off. It does not waive trust, required metadata/link validation, or ABAC. Enabled processing flags do not prove spaCy or AI actually ran. Content Review shows source and a real AI score when present, or an honest disabled, skipped, no-text, or failed-stage reason. Complete trusted documents may ingest without a click per file; ambiguous or unresolved required links still need human review.

5. Configure exposed — document capabilities (document:read, document:search, upload/delete when allowed).

6. Optional vector path — if you need semantic search, consider entityType: vectorStore instead; see Build vector store business entity.

7. Repair and validate system:

aifabrix repair <systemKey> --expose
aifabrix validate <systemKey>
aifabrix upload <systemKey>
aifabrix test-integration <systemKey>

Defaults and examples

Use documentStorage to normalize file metadata and process governed documents, including files downloaded into document storage. Use vectorStore when embeddings and RAG search are first-class.

Root identity band (same law as recordStorage):

Example sync band for document ingestion — mode is direction only (pull/push/bidirectional), never incremental; incremental behavior is the separate optional incrementalKey / incrementalSchedule pair (see Configure sync):

{
  "sync": {
    "mode": "pull",
    "schedule": "@daily",
    "batchSize": 200,
    "incrementalKey": "modifiedAfter",
    "incrementalSchedule": "*/30 * * * *"
  }
}

Keyed retrieval

Start from Builder's document-storage-keyed-get golden example. Enable
documentStorage.enabled, set binaryOperationRef: get and its responseType
to binary, and bind the OpenAPI get operation. Its CIP fetch also declares
responseType: binary; the final output uses mode: documents. Map the document
identity from raw.params or raw.headers, and retain required business links
and dimensions. These are section fragments, not standalone datasource files.

{
  "documentStorage": {
    "enabled": true,
    "binaryOperationRef": "get",
    "responseType": "binary"
  }
}

Validate

For a keyed-only datasource, use a known business key and require that get
actually executes:

aifabrix datasource test-e2e <datasourceKey> get --app <systemKey> --primary-key-value <key> --strict-capability-scope --json

Retain the run ID and sanitized keyedGetEvidence: content type, size, filename
and SHA-256. Check the persisted document and metadata, then prove allowed and
denied visibility with scoped subjects. Missing-key, skipped or failed retrieval
is not a successful test. If retrieval succeeds but storage fails, inspect the
storage configuration, identity mapping and processing result before rerunning
that same key. Do not put document bytes in logs or evidence.

For a library-sync datasource:

aifabrix datasource validate <datasourceKey>
aifabrix validate <systemKey>
aifabrix datasource test-e2e <datasourceKey> --app <systemKey> --verbose

Limits

Sync failures often surface as empty catalogs or metadata shape errors — not auth failures. Document displayName and systemKey must match the integration bundle. primaryKey and externalId rules match recordStorage — document libraries still require indexed join identity even when binary content stays external. Large libraries should tune sync schedules incrementally — validate metadata shape with a single folder or site before enabling full tenant pulls. Provider-specific path envelopes are out of scope here — start from a known-good document fixture. For a library-sync path, run test-integration before E2E and confirm sync populated at least one row with indexed externalId — empty catalogs usually mean sync operationId or path mapping errors, not RBAC.

Common mistakes