I Built an Open-Source Tool That Audits AI Models for Cultural Bias in 11 Languages

The Problem

Most AI bias research focuses on English-language outputs. But the world doesn't run on English. When I tested five major frontier models against culturally-specific prompts in Mandarin, Arabic, Swahili, and Tamil, the failure modes were striking — and rarely documented.

What I Built

Lingua-Audit is an open-source evaluation framework that:

• Tests for 14 distinct cultural failure modes
• Supports 11 languages out of the box
• Generates reports auditable by non-technical policy teams
• Is fully reproducible — every prompt and grading rubric is public

Top Three Findings

1. Honorific collapse: Models routinely fail at hierarchical address systems in Japanese, Korean, and Tamil.
2. Religious sensitivity drift: Outputs on Islamic jurisprudence often default to a single school of thought without disclosure.
3. Diaspora erasure: Models treat diaspora identities (Tamil-Singaporean, Lebanese-Brazilian) as edge cases, often defaulting to country-of-origin assumptions.

Try It Yourself

The full repo is open. PRs welcome.