FM1
Professional-advice reliance |
When users or professionals lean on Claude in legal/medical/financial w...inancial work, what safety posture is Anthropic committing to? |
- Frame Claude as a drafting and research assistant, not a licensed professional.
- Invest in hallucination and grounding research relevant to professional domains.
- Design UX, guidance, and evaluations that emphasise verification against primary sources and human review.
|
- Maintain professional responsibility for advice, filings, and clinical decisions.
- Implement review/cite-check workflows; do not auto-file Claude’s outputs without human oversight.
- Train internal users on limitations and verification expectations.
|
- Terms and product guidance clarify that Claude is not a substitute for licensed professional advice.
- Contractual language requires independent verification and allocates responsibility for final decisions to the customer.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
Technical
|
AMBER |
- Define what level of reliance is acceptable for practiced professionals under time pressure.
- Clarify how Anthropic will communicate evaluation results and hallucination metrics in professional settings.
|
FM2
Hallucinated law / citations |
How do we handle situations where Claude produces hallucinated legal citations or misstates legal standards? |
- Prioritise evaluations, mitigations, and UX patterns that reduce hallucinated citations in legal contexts.
- Communicate known limitations and evaluation coverage; avoid over-claiming reliability.
- Invest in features (e.g., cite-check tools, retrieval) that help users validate legal references.
|
- Require lawyers and legal teams to perform independent cite checks and legal research.
- Prohibit use of Claude outputs as a substitute for full legal research and analysis.
- Adopt internal protocols for documenting how Claude was used in any filing or advice.
|
- Make clear in terms and product guidance that Claude is not a legal research service or a substitute for counsel.
- Flag hallucination and cite-check limitations; require customers to verify legal outputs before use.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
Technical
|
AMBER |
- Clarify how frequently hallucinated citations appear in evaluation data and how that is communicated.
- Determine how much tooling support (e.g., retrieval, cite-check) is required before encouraging certain legal workflows.
|
FM3
Safety-critical / high-availability systems |
What is Anthropic’s position when Claude is used in workflows that touch safety-critical or high-availability systems? |
- Discourage direct control of safety-critical systems by Claude; promote human-in-the-loop patterns.
- Evaluate and document model behaviour in safety-adjacent contexts where feasible.
- Align product design with RSP/ASL guardrails for deploying in higher-risk settings.
|
- Keep humans in the loop for control and override of safety-critical systems.
- Implement additional domain-specific safeguards, monitoring, and auditing.
- Limit use of Claude in safety-critical contexts to approved, well-understood workflows.
|
- Use terms to prohibit or restrict direct control of safety-critical systems by Claude.
- Clarify that customers must implement their own safety engineering and monitoring controls.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
Technical
|
AMBER |
- Clarify which classes of safety-critical use cases Anthropic will and will not support.
- Define expectations for incident reporting and coordination when Claude is part of a safety-critical workflow.
|
FM4
Downstream toolchains and agents |
How does Anthropic’s posture change when Claude is embedded in toolchains, agents, or third-party products? |
- Provide guidance on safe tool use and integration patterns.
- Document integration risks and mitigation expectations for downstream developers.
- Align platform features (e.g., rate limits, monitoring hooks) with foreseeable misuse patterns.
|
- Implement appropriate controls in their own products (e.g., rate limiting, abuse detection, guardrails).
- Ensure end-users of downstream products understand the role of Claude and its limitations.
- Flow Anthropic’s terms and safety expectations into downstream contracts where appropriate.
|
- Clarify how terms apply to integrators and third-party developers embedding Claude.
- Address allocation of responsibility between Anthropic, integrators, and end customers.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
Technical
|
AMBER |
- Clarify how much visibility Anthropic has into specific downstream uses when Claude is embedded.
- Define escalation paths when a downstream product drives misuse despite Anthropic’s controls.
|
FM5
Misuse for harmful or prohibited content |
How are we addressing attempts to use Claude to generate harmful, abusive, or otherwise prohibited content? |
- Maintain and evolve robust safety systems, policies, and enforcement practices.
- Continuously improve red-teaming and evaluations for high-risk misuse patterns.
- Align product safety with RSP/ASL and evolving regulatory expectations.
|
- Implement their own abuse detection and response mechanisms on top of Claude.
- Train internal users and customers on appropriate use and escalation paths.
- Promptly notify Anthropic of suspected misuse patterns or incidents.
|
- Use terms and AUP to prohibit harmful or abusive usage explicitly.
- Reserve rights to suspend or terminate access for serious or repeated abuse.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
Technical
|
GREEN/AMBER |
- Clarify how Anthropic will communicate residual risk and limitations of safety systems.
- Determine how enforcement and escalation processes are communicated to customers and regulators.
|
FM6
Misalignment between marketing and reality |
How do we manage risk if marketing or sales messages overstate what Claude can safely do? |
- Align marketing claims with documented evaluation and safety posture.
- Provide internal guidance and training for marketing, sales, and partnerships.
- Monitor and update public materials as product capabilities and safety posture evolve.
|
- Ensure their own marketing and sales materials accurately describe how they use Claude.
- Correct misstatements quickly when they are identified.
- Coordinate with Anthropic where joint marketing is involved.
|
- Use terms to clarify how far Anthropic’s representations extend and what is excluded.
- Reserve rights to adjust terms and materials as understanding of risks changes.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
Technical
|
AMBER |
- Clarify how Anthropic handles situations where public claims or sales materials are found to be overstated.
- Define expectations for coordination with customers when joint messaging is used.
|
FM7
Data, privacy, and security expectations |
How do data, privacy, and security expectations intersect with Anthropic’s responsibilities and customer responsibilities? |
- Maintain strong privacy and security practices aligned with public commitments.
- Provide clear documentation on data handling, retention, and security controls.
- Align data practices with regulatory requirements and RSP/ASL commitments.
|
- Implement their own security, privacy, and compliance programs.
- Ensure they have appropriate legal bases and notices for their own data processing.
- Configure integrations with Claude in a way that respects data minimisation and security principles.
|
- Use data-processing addenda, privacy policy, and security terms to set expectations and allocate responsibilities.
- Adjust contractual commitments as needed for different deployment models.
Sources (scenario-level):
Anthropic policies
RSP / ASL
Statutes / regulations
Scholarship
|
AMBER |
- Clarify how nuanced data practices can be communicated concisely to non-specialists.
- Plan for how commitments may need to shift as AI and data regulations evolve.
|