metrics module

Fairness Metrics

Parameter definitions:

param pred_labels:  predicted output from model as a 1D array (should be all 1’s and 0’s because binary output) param true_labels:  the true labels as a 1D array (same size and format as pred_labels) param groups:the group of each element as a 1D array param priv_groups:  the group name of privileged

metrics.Demographic_Parity(pred_labels, true_labels, groups, priv_group=None)

A fair algorithm would have an equal rate of positive outcomes \((\hat{Y}=1)\) in privileged group \((A=0)\) and unprivileged groups \((A=1)\).

Math:\(P(\hat{Y} = 1|A=0) = P(\hat{Y} = 1|A=1)\) Returns:dictionary of predicted positive outcomes relative to the privileged group

metrics.Equality_of_Opportunity(pred_labels, true_labels, groups, priv_group=None)

A fair algorithm would have an equal rate of true positives in privileged and unprivileged groups.

Math:\(P(\hat{Y}=1|A=0, Y=1) = P(\hat{Y}=1|A=1, Y=1)\) Returns:dictionary of true positives relative to the privileged group

metrics.Equalized_Odds(pred_labels, true_labels, groups, priv_group=None)

A fair algorithm should have equal rates for true positives and false positives in the privileged and unprivileged groups respectively.

Math:\(P(\hat{Y}=1|A=0,Y=y) = P(\hat{Y} = 1 | A=1, Y=y), \forall y \in \{0,1\}\) Returns:dictionary of tuples consisting of true positives and false positives, relative to the privileged group

metrics.false_disc_rate(pred_labels, true_labels, groups)

Math:\(FDR_g = \frac{FP_g}{PP_g} = P(Y=0 | \hat{Y} = 1, A = g), \forall g \in G\) Returns:dictionary of fraction of false positives within the predicted positive of the group

metrics.false_neg(pred_labels, true_labels, groups)

Calculates the number of elements in a group with a negative prediction but positive true label.

Math:\(\hat{Y} = 0, Y=1\) Returns:dictionary of total false negative predictions for each group

metrics.false_neg_rate(pred_labels, true_labels, groups)

Math:\(FNR_g = \frac{FN_g}{TP_g + FN_g} = P(\hat{Y}=0 | Y = 1, A = g), \forall g \in G\) Returns:dictionary of fraction of false negatives within the labeled positives of the group

metrics.false_omis_rate(pred_labels, true_labels, groups)

Math:\(FOR_g = \frac{FN_g}{PN_g} = P(Y=1 | \hat{Y} = 0, A = g), \forall g \in G\) Returns:dictionary of fraction of false negatives within the predicted negatives of the group

metrics.false_pos(pred_labels, true_labels, groups)

Calculates the number of elements in a group with a positive prediction but negative true label.

Math:\(\hat{Y} = 1, Y=0\) Returns:dictionary of total false positive predictions for each group

metrics.false_pos_rate(pred_labels, true_labels, groups)

Math:\(FPR_g = \frac{FP_g}{TN_g + FP_g} = P(\hat{Y}=1 | Y= 0, A = g), \forall g \in G\) Returns:dictionary of fraction of false positives within the labeled negatives of the group

metrics.pred_neg(pred_labels, true_labels, groups)

Calculates the number of predicted negative elements in a group: \(PN_g = (\hat{Y} = 0)\)

Returns:dictionary of negative predictions for each group

metrics.pred_pos(pred_labels, true_labels, groups)

Calculates the number of predicted positive elements in a group: \(PP_g = (\hat{Y}=1)\)

Returns:dictionary of positive predictions for each group

metrics.pred_pos_rate(pred_labels, true_labels, groups)

Calculates predictive positive rate as \(PPR_g=\frac{PP_g}{TPP}\)

Returns:dictionary of the fraction positive predictions that belong to each group

metrics.pred_prevalence(pred_labels, true_labels, groups)

Math:\(\frac{PP_g}{|g|} = P(\hat{Y}=1 | A=g), \forall g \in G\) Returns:dictionary of the fraction of positive predictions within each group

metrics.total_pred_pos(pred_labels)

Calculates total predictive positives across all groups \(TPP=\sum_{g_1}^{g_n} PP_g\)

Returns:integer number of positive predictions in whole sample

metrics.true_neg(pred_labels, true_labels, groups)

Calculates the number of elements in a group with a negative prediction and negative true label.

Math:\(\hat{Y} = 0, Y=0\) Returns:dictionary of total true negative predictions for each group

metrics.true_pos(pred_labels, true_labels, groups)

Calculates the number of elements in a group with a positive prediction and positive true label.

Math:\(\hat{Y} = 1, Y=1\) Returns:dictionary of total true positive in each group