Diligence Questions for an AI-Assisted Neutral

What questions expose limitations of an automated neutral?

Ask who actually decides, what the tool sees, how its output is checked, how conflicts and unequal access are addressed, and how a party can challenge an error. Request evidence for the specific task and case conditions. A neutral-sounding response or a high benchmark score does not demonstrate impartiality or procedural fairness.

Identify the person holding the mandate

SVAMC Guideline 6 states that arbitrators should not delegate their personal mandate to an AI tool or replace independent analysis with it. Its application depends on adoption and the governing framework. [S1]

Start the conversation by asking which person is responsible for factual findings, legal analysis and the award. Then ask what the system actually contributes. A label such as AI arbitrator may describe technology inside a process, rather than an autonomous final decision maker.

• Who reads the underlying record?
• Who can reject the tool's proposed reasoning?
• Who addresses objections and signs the result?
• What happens if the tool is unavailable or unreliable?

Ask for task-specific limitations

Separate extraction, summarization, legal research and decision support. A system can perform well on document classification while omitting a decisive exception in a contract. Ask for evaluation evidence that resembles the proposed use and includes unsuccessful examples.

• Record coverage: which files, languages, formats and page limits are supported?
• Source traceability: can each material statement be linked to an accessible passage?
• Uncertainty: can the tool preserve conflicting evidence without choosing a winner?
• Data handling: who can access inputs, outputs and retained logs?
• Changes: will a model or configuration change trigger reassessment?
• Access: can both parties inspect and challenge the material used in a consequential step?

Synthetic diligence conversation

A fictional neutral proposes a tool that summarizes a 200-page record. The vendor reports a 95% extraction score. The useful follow-up is not simply whether that score is high. Ask what was extracted, how omissions were counted and whether the test contained scanned pages or contradictory documents.

Suppose the test measured document dates, while the planned use is identifying exceptions to a limitation clause. The score does not answer the planned-use question. A proposed next step is a task-matched simulation with source-linked answers and an independent review of missed exceptions.

• Claim: 95% on date extraction.
• Proposed use: contract-exception analysis.
• Gap: different task and error consequences.
• Evidence requested: evaluation definition, sample limitations and reviewed failure cases.
• Decision status: suitability for the proposed task remains unverified.

Record answers in a decision-useful format

Use a diligence card with four fields: question, answer, evidence and unresolved consequence. An unsupported answer stays unverified. Avoid a single numerical score that can hide a critical failure behind strong performance on unrelated questions.

AAA's AI Led materials expressly retain a human final decision. That illustrates why diligence should inspect the actual allocation of responsibilities instead of inferring it from a product's title. [S2]

• Acceptable answer: specific task, responsible person and inspectable evidence.
• Incomplete answer: general assurance without scope or examples.
• Escalation trigger: an unresolved issue affecting participation, confidentiality or independent decision making.
• Recheck trigger: a material change in tools, record or procedure.

Scope and limitations

Educational diligence questions, not neutral-selection advice. This page does not act as an arbitrator or administer disputes.

The conversation and score are fictional. No conclusion is offered about a real provider's impartiality, performance or legal compliance.

Sources

1. SVAMC Guidelines on the Use of Artificial Intelligence in Arbitration, First Edition (2024) — Principle-based guidelines covering confidentiality, disclosure, diligence, non-delegation and due process; adoption is case-specific. Checked 2026-11-05
2. AAA AI Led Rules, Forms, and Fees — Provider describes AI assistance and states a human arbitrator decides the case and issues the final award. Checked 2026-11-05

Explore further

These diligence questions also support careful reading of legal-technology materials at Agentic Attorney, https://agenticattorney.com/. The link is a route to the central project, not an offer of neutral services. Explore Agentic Attorney.