The problem identified in this PaymentsSource article is that machine learning tools learn to be biased and that bias is invisible because the machine learning model is a black box; it doesn’t divulge how it is making its decisions.
But even if the model was a white box solution that clearly identified what data elements were used to make a decision, it isn’t clear developers would recognize a biased decision. The problem here is that bias can be encoded so deeply in the data set that it will be very hard to detect.
For example, if the data used to train the model is old, no amount of “gender correction” will be sufficient in that women salaries are higher today than in the past. Or if the algorithm identifies access to running water as a key contributor to a decision, will people recognize that non-whites are by far more likely to lack access to clean water and sanitation?
Biases run deep in our data and requires vetting prior to being used as training data:
Look at what happened when Amazon tried building an AI tool to help with recruiting, only to find that the algorithm discriminated against women because it had combed through male-dominated CVs to gather its data.
The AI revolution that has swept through banks, call centers, retailers, insurers and recruiters has brought obvious bias with it — and it’s getting worse, as AI systems are increasingly able to “teach” themselves, reinforcing existing bias as their decision-making develops.
This problem is exacerbated by the investment in opaque “black box” AI systems, which cannot communicate how decisions have been made to the operator, regulator or customer. Since black box systems learn from each interaction, if they are given corrupt data, poor decision-making can rapidly accelerate, without the operators understanding why or even being aware of it.The only solution to this is “white box” or Explainable AI. These are systems which are able to explain in easily understood language how the software operates and how decisions have been made.
Overview by Tim Sloane, VP, Payments Innovation at Mercator Advisory Group