While developing an AI tool to help hiring managers prepare for interviews, we stumbled upon what seems to be a novel method for detecting bias in Large Language Models.
By comparing how LLMs (Claude, GPT-4, Gemini, Llama) interpret anonymized vs. non-anonymized versions of the same content, we can measure and quantify bias reduction. The interesting part is that this technique could potentially be used to audit bias in any LLM-based application, not just recruitment.
Some key findings:
- Different LLMs show varying levels of bias reduction with anonymization
- Llama 3.1 showed consistently lower bias levels
- GPT-4 performed better in specific tasks like interview question generation
We've published our methodology and findings on arXiv: https://arxiv.org/abs/2410.16927
We're a boutique AI consultancy, and this research emerged from our work on building practical AI tools. Happy to discuss the technical implementation, methodology, or real-world applications.
