Checking access…

Skip to main content
Version: v2

AI Search Indexing

This page describes how wallet-ai-service's indexer builds the search index — today, wallet-docs-index-v3 (the AZURE_SEARCH_INDEX value; the indexer itself is generic and can target any index name passed to it) — which files are parsed, how content is cleaned, and a separate mechanism specific to the AI Assistant: how the OpenAPI spec used for example request generation gets into Azure Search at all.

Location moved

Indexing logic lives entirely in wallet-ai-service's src/app/shared/indexer/ — not in this repo. There is no uploadToAzureSearch.js or equivalent script here anymore.

Overview​

Indexing is triggered via POST /api/index on the wallet-ai-service backend — admin-key gated, and expected to be IP-allowlisted at the ingress layer (the endpoint itself does not enforce caller identity beyond the admin key). It accepts three source types plus a fourth catch-all:

SourceParserHandles
docsmarkdown.py.md / .mdx files
yamlyaml.py.yaml / .yml files — generic YAML docs and OpenAPI/Swagger specs
jsonjson.py.json files
documents (inline)pipeline.py::index_inline_documentsPre-formatted document dicts, uploaded as-is — no parsing, no truncation
curl -X POST https://<ai-service-host>/api/index \
  -H "Content-Type: application/json" \
  -H "X-Admin-Key: <admin key>" \
  -d '{
    "index_data": {
      "docs": [{ "directory": "docs", "category": "docs" }],
      "yaml": [{ "directory": "openapi/v2/apispec.yaml", "category": "open-api-spec-v2" }]
    }
  }'
caution

Requires the backend's admin key (AZURE_SEARCH_ADMIN_KEY) — returns 503 if unset. overwrite: true clears the target index first; create_index: true creates it if missing.

Markdown parsing (markdown.py)​

Per file, walked recursively under the configured docs root:

  1. Strip front matter, extract the body.
  2. Extract title — first # H1 heading; if none, fall back to front-matter title:; if neither, fall back to the filename stem. (Files without an H1 used to be silently dropped entirely — fixed; the fallback chain now matches what the code always claimed to do.)
  3. Strip remaining markdown syntax (code fences, links, headings, HTML tags, MDX imports) to produce plain searchable text, truncated to 32,000 chars (Azure Search field limit).
  4. Derive the URL — front-matter slug: if present (prefixed /docs), otherwise the file's path relative to the docs root with numeric prefixes (01-, 03-) stripped and segments lowercased.
  5. Skip release-notes pages — matched by title or URL pattern, deliberately excluded as noise.
  6. Deduplicate by URL — last-write-wins, useful when versioned docs overlap.

Noise exclusion​

Discovery (safe_discover_files, shared by all three parsers) skips, before any of the above ever runs:

  • Any path segment starting with _ — covers _category_.json, _drafts/.
  • Any path segment starting with . — covers .docusaurus.
  • node_modules, .git, build, static.

YAML parsing (yaml.py)​

Handles two distinct shapes:

Generic YAML docs — a single mapping or a sequence of mappings, each needing at least a content field (same fallback/derivation rules as JSON, below).

OpenAPI/Swagger specs — detected by a top-level openapi/swagger key. This repo's specs split every request/response schema across many small files under schemas/, parameters/, servers/, linked by $ref. Two things matter here:

A file is only a root spec if it has paths​

Component/schema fragment files (e.g. schemas/sessions/session-resource.yaml) also declare a top-level openapi: key — for standalone linting — but have no paths. Without checking for that, every such fragment file was previously indexed as its own near-empty, noisy document (literally just the string "v2" or "3.1.1"). Fragment files are now skipped — their content is already pulled into the root spec's own indexed text via $ref resolution (below), so indexing them separately would only add noise.

$ref resolution — bounded, cycle-safe, contained to the docs root​

Without resolving $refs, an endpoint's indexed text was just "POST /v2/sessions — Create Session — <top-level description>" — the actual field-level descriptions living in referenced schema files were invisible to search. Resolution:

  • Bounded depth (4 levels) — a broken or deeply nested $ref chain degrades gracefully rather than recursing forever.
  • Cycle-safe without over-blocking siblings — the "already resolved" tracking set is branched per recursion path (seen | {ref}), not mutated in place. Two sibling properties that happen to reference the same component (reuse, not a cycle) both resolve correctly instead of the second one being incorrectly treated as a cycle.
  • Contained to the docs root — a $ref like ../../../../etc/passwd is rejected (the same os.path.normpath(os.path.realpath(...)) + startswith containment check used elsewhere for path safety). Verified directly: a spec with a $ref pointing outside the configured docs root gets that reference silently dropped, not read.

JSON parsing (json.py)​

A single object or array of objects, each needing a content field (or a best-effort fallback: join whatever string values exist if content is absent). id/title/file_path/category all have the same fallback behavior as YAML — derived from the file path/name if not explicitly present.

Index schema​

FieldTypeNotes
idEdm.String (key)SHA-1 of the URL/file path — stable across re-runs, enables upsert
titleEdm.StringSearchable
contentEdm.StringSearchable — truncated to 32,000 chars
file_pathEdm.StringFilterable — used as the citation link
categoryEdm.StringFilterable, facetable — drives the two-pass search filter
rawContentEdm.StringRetrievable only — no searchable/filterable/sortable/facetable. See below.

Why rawContent exists​

filterable/sortable/facetable force Azure Search to index a field's entire value as a single term, capped at 32KB. The full bundled OpenAPI spec (~300KB) is far past that — uploading it into content fails outright with "Field 'content' contains a term that is too large to process." rawContent has none of those attributes, so it has no such limit. It exists for exactly one purpose today: storing that spec blob.

The OpenAPI spec document​

Example request generation needs the actual, structured OpenAPI schema — not the human-readable text the indexer above produces for search. This is a separate mechanism from everything above:

  1. The full bundled v2 spec (build/redocusaurus/plugin-redoc-1.yaml) is uploaded as one document, under a fixed key, with the raw YAML text in rawContent: 
    {
      "@search.action": "upload",
      "id": "ccg-openapi-v2-spec",
      "title": "CCG v2 OpenAPI Spec (bundled)",
      "category": "openapi-spec-raw",
      "rawContent": "<the full plugin-redoc-1.yaml text>"
    }
  2. At startup, the backend fetches it by direct key lookup (SearchClient.get_document — GET /indexes/{index}/docs/{key}, not a ranked search) using CCG_OPENAPI_V2_SPEC_DOC_ID.
  3. The generator (OpenAPIExampleGenerator.from_yaml_text()) parses it once and keeps it in memory for the process lifetime.

This must be re-run whenever the v2 spec changes — there's no automatic sync between yarn build's bundled output and this Azure Search document. If example generation starts producing stale field names/enums after an API change, this is the document to re-upload.

A local file path (CCG_OPENAPI_V2_SPEC_PATH) is also supported as a fallback for local dev, when the backend happens to have this repo's build/redocusaurus/ output on disk — not viable in a real deployment, where the two services don't share a filesystem.

Retrieval at query time​

Indexing builds the corpus; retrieval (AzureAISearchLangChainRetriever, used by the Assistant's chain) queries it with a two-pass search for API-shaped questions — spec-filtered pass + full-corpus pass, merged and deduplicated. See Overview — Retrieval.

Vector (hybrid) search — separate index, separate endpoint, in validation​

Status

This is a second, in-progress index (wallet-docs-vector-index-v1), being validated alongside wallet-docs-index-v3 — not yet the server's default. See wallet-ai-service's docs/ccg_ai_assistant/12-vector-search-plan.md for full design rationale.

Keyword/semantic search (above) fails on genuine vocabulary mismatch — a query can share no real terms with the one doc that actually answers it. Vector search embeds both the corpus and the query into the same numeric space and retrieves by similarity, which is far more forgiving of that mismatch. Rather than add a vector field to the live index (Azure Search field attributes are add-only, never fixable in place once wrong), this is a brand-new index built by a separate endpoint:

curl -X POST https://<ai-service-host>/api/index/vector \
  --max-time 600 \
  -H "Content-Type: application/json" \
  -H "X-Admin-Key: <admin key>" \
  -d '{
    "index_name": "wallet-docs-vector-index-v1",
    "create_index": true,
    "index_data": {
      "docs": [{ "directory": "docs", "category": "docs" }],
      "yaml": [{ "directory": "openapi/v2/apispec.yaml", "category": "open-api-spec-v2" }],
      "json": [{ "directory": "path/to/error-codes", "category": "error-codes" }]
    }
  }'

Key differences from /api/index:

/api/index/api/index/vector
Row shapeOne row per fileOne row per chunk — large docs split (heading-aware for markdown, per-operation for OpenAPI, per-object for JSON)
Extra stepNoneEvery chunk's content is embedded (UAIS embeddings gateway) before upload
Speed/costFast, freeMaterially slower (network round-trip per embedding batch) and costs money per call
Blast radiusAffects the index currently serving retrievalFully isolated — a bug here cannot touch /api/index or wallet-docs-index-v3

--max-time 600 (or your client's equivalent — e.g. Insomnia's Preferences → General → "Request Timeout") matters here: a full-corpus request can legitimately take several minutes (dozens of ~4–5s embedding-batch round-trips), and most HTTP clients' default timeout (often 120s) will cut the connection well before the server finishes — that's a client-side timeout, not a sign anything is stuck. Splitting one big request into one call per directory/category avoids depending on picking "a big enough" timeout at all.

To query this index instead of the server default for a single request, set index_name in the /api/ccg/chat(/stream) request body — it's a top-level field on ChatRequest, not nested under anything:

{ "query": "...", "conversation_id": "...", "index_name": "wallet-docs-vector-index-v1" }

Unrecognized fields are silently ignored (standard Pydantic behavior), not rejected — a request shaped like {"search_config": {"index_name": ...}} will silently fall back to the server default index with no error, which looks identical to "vector search didn't help" unless you check which index the server actually queried (server-side log line: POST .../indexes/<name>/docs/search).