Trust Is Not a Feature

The explainable AI market has grown to 7.79 billion dollars in 2024, projected to reach 21.06 billion by 2030, but the focus has shifted from civil liberties to corporate features. This shift is concerning as the explainability being sold is not designed for individuals affected by algorithmic outputs, but rather for boardrooms and compliance departments. The absence of explainability led to the Australian Robodebt scheme, which incorrectly calculated welfare debts and resulted in 746 million dollars being wrongfully recovered from 381,000 individuals. The scheme's flaws led to several deaths by suicide. To avoid similar issues, it's essential to prioritize explainability for individuals, not just corporate interests. Engineers should consider the human impact of their work and push for transparency in AI decision-making processes.