Retrieval-Augmented Generation for On-Chain Data: Patterns That Hold Up

Language models are useful for questions that used to require an analyst and a SQL console: what happened to this wallet last week, why did this liquidation occur, which contracts interacted with this token. Retrieval-augmented generation is the standard way to give a model access to that data without training it on your chain. Most implementations we review fail for the same reasons: the retrieved documents are shaped for storage rather than for answering, the output format is unconstrained, and nobody measured whether the answers are right. This post describes patterns that survived contact with production.

Index events, not blocks

Raw blocks and transactions are the wrong unit of retrieval. A model asked about a swap does not benefit from the entire block, and a chunked transaction with hex calldata is close to useless. Decode events into typed records with human-readable fields: token symbols and decimals resolved, addresses labelled where labels exist, amounts in both raw and display units, and timestamps in ISO format. Then compose documents at the granularity users ask about: one document per transaction with its decoded events, one per address per day summarising activity, one per contract describing its interface and known behaviour.

Each document carries structured metadata that filters can use before any vector search happens: chain, block range, addresses involved, event types, contract names. Retrieval is then a two-stage process. Filter by metadata to a candidate set, then rank by semantic similarity within it. Skipping the filter stage is the single most common cause of irrelevant context.

Keep the index honest

On-chain data changes in ways that break naive indexes. Reorgs invalidate recently indexed blocks, so documents must be versioned by block hash and re-indexed when the canonical chain changes. Labels for addresses are often wrong or stale; store their source and confidence with the label rather than presenting them as facts. Decimals and symbols come from token contracts that can be malicious; treat display values as untrusted rendering hints and keep the raw amounts alongside.

Constrain the output

Free-text answers are hard to verify and impossible to act on programmatically. Define a response schema for each question type and require the model to fill it. A "what happened in this transaction" answer might have fields for a one-sentence summary, a list of transfers with from, to, token and amount, the contracts called, and a list of source document identifiers that support each claim. Validate the response against the schema before it reaches a user, and reject or retry when validation fails.

{
  "type": "object",
  "required": ["summary", "transfers", "sources"],
  "properties": {
    "summary": { "type": "string", "maxLength": 400 },
    "transfers": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["from", "to", "token", "amount", "source"],
        "properties": {
          "from":   { "type": "string", "pattern": "^0x[0-9a-fA-F]{40}quot; },
          "to":     { "type": "string", "pattern": "^0x[0-9a-fA-F]{40}quot; },
          "token":  { "type": "string" },
          "amount": { "type": "string" },
          "source": { "type": "string" }
        }
      }
    },
    "sources": { "type": "array", "items": { "type": "string" } }
  }
}

The source field on every claim matters more than it looks. It lets the interface link each statement to the document it came from, lets an evaluator check that the claim is actually supported, and lets you drop claims that cite nothing.

Evaluate before and after shipping

Build an evaluation set from real questions with answers verified against the chain by hand. Cover the boring cases and the adversarial ones: transactions with many internal transfers, tokens with unusual decimals, contracts with misleading names, questions about addresses with no activity. Score retrieval separately from generation. If the right documents were not retrieved, changing the prompt will not help. If they were retrieved and the answer is still wrong, the model or the schema needs attention.

• Retrieval recall: the fraction of gold documents present in the retrieved set.
• Claim support: the fraction of output claims whose cited source actually contains the claim.
• Numeric accuracy: exact match on amounts, addresses and block numbers, with no tolerance.
• Abstention: the model declines when the retrieved set does not contain an answer.
• Regression: every change to prompts, models, chunking or the index reruns the full set.

Numeric fields deserve special attention. Models will round, truncate and transpose digits in addresses. Where a number appears in the output, copy it from the source document programmatically rather than letting the model generate it.

Guardrails around actions

Answering questions is low risk. Letting the model propose or execute actions, such as drafting a transaction, flagging an account or approving a withdrawal, is not. The guardrails we use are structural rather than prompt-based:

1. The model never holds credentials or signing keys. It produces a structured proposal that a deterministic service validates and a human approves.
2. Every proposal is simulated, and the simulation result is shown to the reviewer alongside the model's reasoning.
3. Inputs from untrusted sources, including token names, contract metadata and event strings, are treated as data and never as instructions. Prompt injection through a token's name field is a real attack.
4. Rate limits and value limits apply to the pipeline as a whole, independent of what the model says.

Human review as a workflow, not a checkbox

Reviewers need tooling. Present the model's answer with each claim linked to its source, highlight claims with low support scores, show the simulation diff for proposed actions and record the reviewer's decision along with the model version and retrieved documents. That record becomes the next evaluation set and the audit trail when a decision is questioned. Teams that treat review as a checkbox get rubber stamps. Teams that build review tooling get a system that improves with every question it answers.