modeva.TestSuite.diagnose_fairness#

TestSuite.diagnose_fairness(group_config, favorable_label: int = 1, dataset: str = 'test', metric: str = None, threshold: float | int = None)#

Evaluate fairness metrics across different protected and reference groups.

This method calculates various fairness metrics based on the provided group configurations and model predictions, allowing for an assessment of fairness across specified groups in the dataset. It returns a result object containing the computed metrics, visualizations, and other relevant information.

Parameters:

  • group_config (dict) –

    Configuration defining protected and reference groups. Each key is a custom group name, and each value is a dictionary with group definitions. Supports three formats:

    1. For numerical features:

      {
    "feature": str,           # Feature name
    "protected": {            # Protected group bounds
        "lower": float,       # Lower bound
        "lower_inclusive": bool,
        "upper": float,       # Optional upper bound
        "upper_inclusive": bool
    },
    "reference": {            # Reference group bounds
        "lower": float,       # Optional lower bound
        "lower_inclusive": bool,
        "upper": float,       # Upper bound
        "upper_inclusive": bool
    }
}

    2. For categorical features:

      {
    "feature": str,                  # Feature name
    "protected": str or int,         # Protected group category
    "reference": str or int          # Reference group category
}

    3. For probabilistic group membership:

      {
    "by_weights": True,
    "protected": str,         # Column name with protected group probabilities
    "reference": str          # Column name with reference group probabilities
}

  • favorable_label ({0, 1}, default=1) –

    • For classification: The preferred class label.

    • For regression: 1 means larger predictions are preferred, 0 means smaller predictions are preferred.

  • dataset ({"main", "train", "test"}, default="test") – The dataset to evaluate fairness on.

  • metric (str, default=None) –

    Fairness metric to calculate. Higher values indicate less unfairness. If None, defaults are used based on task type.

    For regression (default=”SMD”):

    • SMD: Standardized Mean Difference (%) between protected and reference groups

    For classification (default=”AIR”):

    • AIR: Adverse Impact Ratio of predicted probabilities

    • PR: Precision Ratio

    • RR: Recall Ratio

  • threshold (float or int, default=None) – Optional threshold value to display in the visualization. Used to indicate acceptable fairness levels.

Returns:

A result object containing:

  • key: “diagnose_fairness”

  • data: Name of the dataset used

  • model: Name of the model used

  • inputs: Input parameters used for the test

  • value: Nested dictionary containing the (“<group_name>”, item) pairs for each group, and the item is also a dictionary with:

    • ”distance”: The KS distance between protected vs reference group predictions.

    • ”data_info”: The sample indices of the protected and reference groups, which can be further used for data distribution test, e.g.,

    data_results = ds.data_drift_test(**results.value["Gender"]["data_info"])
data_results.plot("summary")
data_results.plot(("density", "MedInc"))

  • table: DataFrame with fairness metric scores for each group.

  • options: Dictionary of visualizations configuration. Run results.plot() to show all plots; Run results.plot(name=xxx) to display one preferred plot; and the following names are available:

    • ”fairness”: Fairness metric bar plots visualizing the scores for each group.

    • ”<group_name>”: Empirical cumulative distribution function plots for the protected and reference group predictions, allowing for visual comparison of distributions.

Return type:

ValidationResult

Examples

Model Fairness Analysis (Classification)

Model Fairness Analysis (Classification)