The Caste Bias Problem in OpenAI’s Models
- Vishwanath Akuthota
- Oct 3
- 5 min read
Insights from Vishwanath Akuthota
Deep Tech (AI & Cybersecurity) | Founder, Dr. Pinnacle
When AI Learns Our Oldest Prejudices: The Caste Bias Problem in OpenAI’s Models
Let’s tear off the facade: artificial intelligence isn’t magically neutral. It inherits, amplifies, and sometimes invents the biases lurking in human data. A recent exposé suggests OpenAI’s models—widely used in India—are steeped in caste bias. That’s not a small glitch. It’s a crack in the foundation of how we build “intelligent” systems in a society scarred by centuries of hierarchy.
What the Investigations Reveal
From the bits I could piece together:
When an Indian researcher used ChatGPT to polish his job or fellowship application, the model changed his surname to “Sharma” — a name strongly associated with higher-caste identity. That’s not a typo. That’s identity erasure cloaked in “helpful” rewriting.
In India, OpenAI is a heavyweight. Its growth and reputation depend on social legitimacy, so the stakes are high: if models start replicating caste hierarchies, public trust cracks.
Critics argue this is not isolated. Bias in AI is often framed as “gender, race, religion” — but caste is far more subtle, localized, and embedded. It creeps in via names, social roles, patterns of praise/critique, assumed default identity.
Scholars have begun formalizing tools to detect caste bias in large language models (LLMs). One such is DECASTE, which measures caste bias along multiple dimensions — economic, social, educational, political — in model outputs. Their results show systematic favoring of “dominant” castes over historically oppressed ones.
Another audit (“How Deep Is Representational Bias in LLMs?”) used 7,200 story prompts in India via GPT-4 Turbo, and found that even when prompts push for diversity, the model overrepresents culturally dominant groups. In effect, the “stickiness” of bias resists nudging.
On the mitigation side: recent proposals like AMBEDKAR (inspired by the egalitarian vision of B.R. Ambedkar) overlay a “bias elimination decoding layer” during inference (i.e. during generation of the answer), without changing the base model. Initial results show reduction in measurable caste bias by ~26 %.
So: it’s not wild to believe the article’s core claim — OpenAI’s models in India may be silently reinforcing caste-based hierarchies.
Why This Matters (Beyond Theory)
Identity erasure as “help”
Changing a surname from a marginalized identity to a dominant one, while “correcting English,” is not benevolent. It reflects which names the model deems more “standard.” That invisible decision carries weight: “You, as you are, don’t look standard enough.”
Feedback loops of oppression
If a technology favors dominant identities in recommendations, summaries, job-screening, it tilts opportunity further away from marginalized groups. Over time, the bias compounds into structural exclusion.
Prompts are weak weapons
You might think: just ask, “represent caste diversity.” But if the bias is deeply embedded in representation itself, prompts won’t reliably overcome it. The “nudge” gets sucked into the model’s prior. (Seen in the representational bias study.)
Mitigation is not plug-and-play
Proposals like AMBEDKAR are promising but experimental. They address only inference-time control, not the upstream training data or architecture choices. Also, they reflect normative decisions — who defines fairness? What counts as “balanced representation”?
Caste is especially insidious
Unlike, say, race or gender (which many Western frameworks already try to handle), caste is socially specific, local, layered. It overlaps with region, language, religion, but is distinct. Models trained predominantly on global English data may see caste only as “names to ignore,” not as identities to respect.
When AI Learns Our Oldest Prejudices
We imagine AI as a blank slate, a mirror that only shows what we truly are. That’s comforting but wrong.
A recent investigation suggests that OpenAI’s models—especially ones deployed in India—aren’t just mirroring human bias. They’re repeating, reinforcing, and normalizing one of the deepest prejudices that haunts Indian society: caste.
We tend to think of “bias in AI” as gender or race or religion. But caste is more subtle, more embedded, and harder to name. It hides in surnames, in the “standard” that models assume, in what good English “sounds like.” When a researcher’s name got changed to “Sharma” by ChatGPT in the act of “improving” his writing, that’s not a typo: it’s systemic erasure.
India is a major market for OpenAI. If these models start silently privileging certain castes, they risk not only credibility but moral legitimacy. Worse: they may widen inequality. A resume polished by AI—or a prompt generating a story—might implicitly favor dominant-caste norms. Over many small acts, that adds up.
Recent academic work backs up these concerns. A model audit called DECASTE shows strong bias against Dalits and Shudras along financial, educational, sociocultural axes. Another audit of GPT-4 found that even when we push for diversity, its outputs favor dominant identities—it resists being “faired.” Proposals like AMBEDKAR try to patch over bias at inference time. They’re clever. But they don’t rewrite the inner blueprint of our systems.
We need solutions at multiple levels:
Training data reform
Force datasets to include voices from marginalized castes, contexts that reflect everyday experiences, not just dominant narratives.
Structural fairness design
Build bias checks, counter-factual triggers, representations anchored in justice, not just statistical parity.
Participatory governance
Engage communities historically harmed by caste — not as tokens, but as co-creators in AI design, auditing, deployment.
Normative clarity
We must decide: what is fair in a caste-divided society? Pure parity? Range of recognition? That’s a political choice, not a technical one.
To say “AI is neutral” is to pretend the biases coded by centuries don’t exist. We shouldn’t just hope for fairer models. We must build them intentionally, with humility, with local insight, and with unflinching honesty.
Meta-Reflection: What’s Hard, What’s Unknown
Because the original article was behind a paywall or blocked, I inferred from secondary sources (LinkedIn post, tech news brief) the central claims. That introduces risk: I may have misinterpreted nuances.
Caste bias is harder to audit than simpler biases (e.g. gender) because the signal is weaker, subtler, and context-dependent.
Mitigation often comes down to contested choices: which voices are “fairly represented”? Which tradeoffs do we accept?
Bias cannot be “solved” — only reduced, managed, and made transparent. We must treat AI systems as social artifacts, not oracles.
About the Author
Vishwanath Akuthota is a computer scientist, AI strategist, and founder of Dr. Pinnacle, where he helps enterprises build private, secure AI ecosystems that align with their missions. With 16+ years in AI research, cybersecurity, and product innovation, Vishwanath has guided Fortune 500 companies and governments in rethinking their AI roadmaps — from foundational models to real-time cybersecurity for deeptech and freedom tech.
Read more:
Ready to Recenter Your AI Strategy?
At Dr. Pinnacle, we help organizations go beyond chasing models — focusing on algorithmic architecture and secure system design to build AI that lasts and says Aha AI !
Consulting: AI strategy, architecture, and governance
Products: RedShield — cybersecurity reimagined for AI-driven enterprises
Custom Models: Private LLMs and secure AI pipelines for regulated industries
→ info@drpinnacle.com to align your AI with your future.
Comments