Not understanding the math behind the rule does not make it a good rule

Treating the output of an "algorithm" as inherently more objective or insightful than an expert opinion or even a gut feeling is really not defensible, though it is often treated as such.  Especially I've noticed when it confirms existing biases.

A decent if short interview plugging a paywalled article is here:

[T]ransparency is needed where technologies affect us in significant ways. Algorithms decide if individuals are legitimate candidates for mortgages, loans, or insurance; they also determine interest rates and premiums.

[. . . ]

But algorithms trained with biased data pick up and replicate these biases, and develop new ones.

[ . . . ]

If you’re hiring someone for a management position and you feed your algorithm data from the last 30 years, the data will be skewed, and the projected ideal candidate will be someone male, white, and in his 40s or 50s. I am a woman in my early 30s, so I would be filtered out immediately, even if I’m suitable for that position. And it gets even worse, because sometimes algorithms are used to display job ads, so I wouldn’t even see the position is available.

Quick additional note: Feeding in data without explicit race and gender markers does not solve the problem, since if these are powerful factors they will be inferred from other traits in the data set.