Bias Monitoring via TrustyAI in ODH

We will take on the persona of a dev-ops engineer for a credit lender. Our data scientists have created two candidate neural networks to predict if a borrower will default on the loan they hold with us. Both models use the following information about the applicant to make their prediction:

What we want to verify is that neither of our models are not biased over the gender field of Is Male-Identifying?. To do this, we will monitor our models with Statistical Parity Difference (SPD) metric, which will tell us how the difference between how often male-identifying and non-male-identifying applicants are given favorable predictions (i.e., they are predicted to pay back their loans). Ideally, the SPD value would be 0, indicating that both groups have equal likelihood of getting a good outcome. However, an SPD value between -0.1 and 0.1 is also indicative of reasonable fairness, indicating that the two groups' rates of getting good outcomes only varies by +/-10%.

Follow the instructions within the installation section. Afterwards, you should have an ODH installation, a TrustyAI operator, and a model-namespace project containing an instance of the TrustyAI service.

Once the installation is completed, navigate to the Bias Monitoring directory within the demo repository that was downloaded during the Installation Tutorial:

Here, we’ll pass all the training data through the models, such as to be able to compute baseline fairness values:

This will take a few minutes. The script will print out verification messages indicating whether TrustyAI is receiving the data, but we can also verify in the Cluster Metrics:

We can also verify that TrustyAI sees the models via the /info endpoint:

As you can see, our models have not provided particularly useful field names for our inputs and outputs (all some form of customer_data+input-x). We can apply a set of name mappings to these to apply meaningful names to the fields. This is done via POST’ing the /info/names endpoint:

Explore the apply_name_mapping.sh script to understand how the payload is structured.

To compute the model’s cumulative fairness up to this point, we can check the /metrics/group/fairness/spd endpoint:

The payload is structured as follows:

These requests will return the following messages:

However, while it’s great that our models are fair over the training data, we need to monitor that they remain fair over real-world inference data as well. To do this, we can schedule some metric requests, such as to compute at recurring intervals throughout deployment. To do this, we simply pass the same payloads to the /metrics/group/fairness/spd/request endpoint:

These commands will return the created request’s IDs, which can later be used to delete these scheduled requests if desired.

Furthermore, let’s monitor the average values of various data fields over time, to see the average ratio of loan-payback to loan-default predictions, as well as the average ratio of male-identifying to non-male-identifying applicants. We can do this by creating an Identity Metric Request via POST’ing the /metrics/identity/request endpoint:

The payload is structured as follows:

Now that we’ve got our metric monitoring set up, let’s send some "real world" data through our models to see if they remain fair:

Once the data is being sent, return to Observe → Metrics page and watch the SPD and Identity metric values change.

Let’s first look at our two models' fairness:

Immediately, we notice that the two models have drastically different fairnesses over the real world data. Model Alpha (blue) remained within the "acceptably fair" range between -0.1 and 0.1, ending at around 0.09. However, Model Beta (yellow) plummeted out of the fair range, ending at -0.274, meaning that non-male-identifying applicants were *27 percent* less likely to get a favorable outcome from the model than male-identifying applicants; clearly an unacceptable bias.

We can investigate this further by examining our identity metrics, first looking at the inbound ratio of male-identifying to non-male-identufying applicants:

We can immediately see that in our training data, the ratio between male/non-male was around 0.8, but in the real-world data, it quickly dropped to *0*, meaning every single applicant was non-male. This is a strong indicator that our training data did not match our real-world data, which is very likely to indicate poor or biased model performance.

Meanwhile, looking at the will-default to will-not-default ratio:

We can see that despite seeing only non-male applicants, Model Alpha (green) still provided varying outcomes to the various applicants, predicting "will-default" around 25% of the time. Model Beta (purple) predicted "will-default" 100% of the time: every single applicant was predicted to default on their loan. Again, this is a clear indicator that our model is performing poorly on the real-world data and/or has encoded a systematic bias from its training; it is predicting that every single non-male applicant will default.

These examples show exactly why monitoring bias in production is so important: models that are equally fair at training time may perform drastically differently over real-world data, with hidden biases only manifesting over real-world data. This means these biases are exposed to the public, being imposed upon whoever is subject to your models decisions, and therefore using TrustyAI to provide early warning of these biases can protect you from the damages that problematic models in production can do.