max_depth
n_estimators
eta
reg_alpha
reg_lambda
random_state

Lowcode Example: Experimentation based on SimuCredit Data¶

Setting up Modeva¶

In [1]:
## ----------------------------------------------------------------
## Install or update packages (recommended to run in Terminal)
## ----------------------------------------------------------------

!pip show modeva
# !pip uninstall modeva
# !pip install modeva

Name: MoDeva
Version: 1.0.11
Summary: Integrated tool for model development and validation.
Home-page: 
Author: 
Author-email: admin@modeva.ai
License: proprietary
Location: C:\Users\s6416\anaconda3\envs\python-3.11\Lib\site-packages\modeva-1.0.11-py3.11.egg
Requires: catboost, dill, httpx, ipython, ipyvuetify, ipywidgets, lightgbm, lime, mlflow, mocharts, momentchi2, notebook, numpy, pandas, pyswarms, python-dateutil, scikit-learn, scikit-learn-extra, scipy, shap, supabase, torch, tqdm, umap-learn, xgboost
Required-by:
In [2]:
# To get authentication, use the following command: (To get full access please replace the token to your own token)
from modeva.utils.authenticate import authenticate
# authenticate(auth_code='eaaa4301-b140-484c-8e93-f9f633c8bacb')
In [3]:
## ----------------------------------------------------------------
## Optional: Clear Modeva-mlflow directory
##     This will clear existing Modeva datasets and experiments
## ----------------------------------------------------------------

# from modeva.utils.mlflow import clear_mlflow_home
# clear_mlflow_home()

Registry Hub¶

  • Low-code panel for managing dataset and experiment registry (You can come back to this panel from time to time).
  • Each time running Modeva, it requires authentication (Contact admin@modeva.ai for requesting the sequence number).
In [4]:
from modeva.dashboard.api import registry_hub
registry_hub()

✓ Auth code found in local storage.
✓ Auth code saved to local storage for future use.
Authenticating Modeva...

✓ License is active and valid.
✓ Authenticated successfully!
Out[4]:
Registry Hub
DataSet
Experiment
NameUpdated TimeDetailDelete
Demo-TwCredit2025-04-25 10:07:31
Demo-CalHouse2025-04-25 10:07:30
Demo-SimuCredit2025-04-25 10:07:30
Demo-BikeShare2025-04-25 10:07:29
Rows per page:
5
1-4 of 4

Data Loading¶

In [5]:
## --------------------------------------------------------
## Optional: create OOD dataset, save as CSV, then
##     upload as Extra data via registry_hub
## --------------------------------------------------------

# from modeva.data.utils.loading import load_builtin_data
# df = load_builtin_data("SimuCredit")

from modeva import DataSet
ds = DataSet()
ds.load_registered_data(name="Demo-SimuCredit")
df = ds.to_df()
df[(df['Gender'] == 0) & (df['Race'] == 0)].to_csv('SimuCredit_OOD1.csv', index=False)
df[(df['Gender'] == 1) & (df['Race'] == 1)].to_csv('SimuCredit_OOD2.csv', index=False)

# Go to registry_hub() above to upload OOD csv files as Extra to Demo-SimuCredit

New Experiment and Data Processing¶

In [6]:
from modeva.dashboard.experiment import Experiment

exp = Experiment(name='Exp20240901-SimuCredit')

## Optional - run exp.clear() to clear existing records in an existing experiment
# exp.clear()
In [7]:
# Load Demo-SimuCredit data
# exp.data_load('Demo-SimuCredit')

## Optional - View the 'main' dataset
exp.ds
Out[7]:
Mortgage Balance Amount Past Due Delinquency Inquiry Open Trade Utilization Gender Race Status
0 139734.22 2717.87 0.00 0 0 0 0.785162 0 0 0
1 243359.62 193.60 0.00 0 0 0 0.254759 0 0 1
2 187784.19 395.05 0.00 0 1 0 0.360995 0 0 1
3 594626.89 180.94 0.00 0 0 0 0.128144 0 1 1
4 166771.42 1241.13 0.00 0 0 0 0.702958 0 0 0
... ... ... ... ... ... ... ... ... ... ...
19995 226806.34 670.99 0.00 0 1 0 0.922122 0 1 1
19996 308625.65 3223.94 0.00 0 0 0 0.989716 1 1 1
19997 375035.34 133.05 131.15 1 0 0 0.092523 0 0 1
19998 165377.42 2256.07 0.00 0 0 0 0.630330 1 0 1
19999 299811.81 2420.01 1461.61 1 0 0 0.899019 1 1 0

20000 rows × 10 columns

In [8]:
exp.data_summary()
Out[8]:
Data Summary
​
Processed
​
main
Overall
Numeric
Categorical
​
Histogram
​
Attribute Count
samples20000
features10
numerical7
categorical3
mixed0
date0
duplicated0
missing cells0
missing cells (%)0
infinite cells0
infinite cells (%)0
In [9]:
exp.eda_2d()
Out[9]:
EDA 2D Charts
​
Processed
​
main
Settings
​
Mortgage
​
Status
​
None
​
Random
​
​
2
In [10]:
exp.eda_3d()
Out[10]:
EDA 3D Scatter
​
Processed
​
main
Settings
​
Mortgage
​
Balance
​
Status
​
None
​
Random
​
In [11]:
exp.eda_multivariate()
Out[11]:
EDA Multivariate
​
Processed
​
main
Correlation
PCA
UMAP
In [12]:
exp.data_process()
Out[12]:
Data Processing
​
Raw

Task Setting

​
Status
​
Classification
​

FeatureTypeUniqueMissing Imputation
Binning
Encoding
Scaling
Mortgage
numerical
199930----
Balance
numerical
191640----
Amount Past Due
numerical
82090----
Delinquency
numerical
60----
Inquiry
numerical
90----
Rows per page:
5
1-5 of 10

Model Training and Tuning¶

In [13]:
exp.model_train()
Out[13]:
Model Training
​
Processed
​
XGBoost
Model Parameters
​
​
​
​
​
​
​
​
​
Trained Model Leaderboard
Parameters Register
Registered Model Leaderboard
Model train AUC test AUC train ACC test ACC Parameters Delete
XGBoost0.85120.8470.77060.7735
XGBoost-Tuned0.86550.84480.77990.7692
In [14]:
exp.model_tune()
Out[14]:
Model Tuning
​
Processed
​
XGBoost
​
Grid
​
AUC
​
Tuning Parameters
​
max_depth
​
Linear
​
​
​
Fixed Parameters
​
n_estimators
​
​
Tuning Leaderboard
indexmax_depthn_estimatorsAUCRegister
131000.8383
011000.8287
251000.8251
371000.8137
491000.8083
Rows per page:
5
1-5 of 5
In [15]:
exp.model_leaderboard()
Out[15]:
Model Training
​
Processed
Registered Model Leaderboard
Modeltrain AUCtest AUCtrain ACCtest ACCParameterDelete
XGBoost0.85120.8470.77060.7735
XGBoost-Tuned0.86550.84480.77990.7692
Rows per page:
5
1-2 of 2

Model Explainability¶

In [16]:
exp.test_explainability()
Out[16]:
Model Explainability
​
Processed
​
test
​
XGBoost
Global
Local
Settings
​
Inherent Interpretability
​

Model Testing¶

In [17]:
exp.model_test()
Out[17]:
Model Test
​
Processed
​
test
​
XGBoost
Performance
Reliability
Robustness
Resilience
Settings
​
​
In [18]:
exp.test_weakness()
Out[18]:
Model Weakness
​
Processed
​
test
​
XGBoost
​
Settings
​
Performance
​
Uniform
​
​
AUC
​
Feature Segment Size AUC Threshold Weak
Inquiry[4.20, 5.60)300.847true
Amount Past Due[5008.50, 6678.00)300.847true
Balance[9845.10, 12304.75]60.40.847true
Amount Past Due[3339.00, 5008.50)230.66670.847true
Amount Past Due[1669.50, 3339.00)1200.70720.847true
Inquiry[2.80, 4.20)580.72440.847true
Utilization[0.60, 0.80)6150.74130.847true
Inquiry[1.40, 2.80)2090.74430.847true
Delinquency[3.00, 4.00)1740.74710.847true
Delinquency[0.00, 1.00)22500.75680.847true

Model Benchmarking¶

In [19]:
exp.model_compare()
Out[19]:
Model Comparison
​
Processed
​
test
​
 XGBoost  (+1)
Performance
Reliability
Robustness
Resilience
Settings
​
AUC

Factsheet and Report¶

In [20]:
exp.list_testsuite()
Out[20]:
Name Start Time Tags
0 Resilience-Result 2025-04-25 10:41:03 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
1 Robustness-Result 2025-04-25 10:41:02 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
2 Reliability-Result 2025-04-25 10:41:02 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
3 Performance-Result 2025-04-25 10:41:02 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
4 Local-Result 2025-04-25 10:40:54 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
5 Global-Result 2025-04-25 10:40:54 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
6 PCA-Result 2025-04-25 10:33:52 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
7 Correlation-Result 2025-04-25 10:33:51 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
8 EDA3D-Result 2025-04-25 10:33:47 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
9 EDA2D-Result 2025-04-25 10:33:40 {'dataset': 'Demo-SimuCredit_md_Exp20240901-Si...
In [21]:
exp.report()
Out[21]:

testsuite-generated Report: Exp20240901-SimuCredit

Resilience-Result

Data:
Demo-SimuCredit_md
Model:
XGBoost
Inputs:
{'dataset': 'test', 'method': 'worst-sample', 'metric': 'AUC', 'alphas': None, 'n_clusters': 10, 'random_state': 0}
index AUC
0.10
0.20
0.30.108096
0.40.320467
0.50.489588
0.60.612286
0.70.700268
0.80.76505
0.90.812749
1.00.847003

Robustness-Result

Data:
Demo-SimuCredit_md
Model:
XGBoost
Inputs:
{'dataset': 'test', 'metric': 'AUC', 'n_repeats': 10, 'perturb_features': None, 'perturb_method': 'quantile', 'noise_levels': [0.1, 0.2, 0.3, 0.4], 'threshold': 0.1, 'random_state': 0}
index 0.0 0.1 0.2 0.3 0.4
00.8470030.8392620.8321160.8204990.810964
10.8470030.8379070.8253540.8129140.800196
20.8470030.8389450.8272520.8175920.804479
30.8470030.8424060.8304840.8237310.81147
40.8470030.8414010.8344110.8219510.815282
50.8470030.8397630.8276420.8166020.807143
60.8470030.8384880.8295980.8207410.805865
70.8470030.840430.8300370.8211960.811785
80.8470030.8385170.8261330.8146230.807822
90.8470030.8380310.8258940.812240.800883

Reliability-Result

Data:
Demo-SimuCredit_md
Model:
XGBoost
Inputs:
{'train_dataset': 'test', 'test_dataset': 'test', 'test_size': 0.5, 'alpha': 0.1, 'max_depth': 5, 'width_threshold': 0.1, 'random_state': 0}
index Avg.Width Avg.Coverage
01.3290.9055

Performance-Result

Data:
Demo-SimuCredit_md
Model:
XGBoost
Inputs:
{'features': 'Status', 'use_prediction': False, 'dataset': 'test', 'sample_size': 2000, 'random_state': 0}
index AUC ACC F1 LogLoss Brier
train0.8512160.7706250.799190.4738890.156377
test0.8470030.77350.7992020.4812880.157991
GAP-0.0042130.0028750.0000120.0073990.001614

Local-Result

Data:
Demo-SimuCredit_md
Model:
XGBoost
Inputs:
{'dataset': 'test', 'sample_idx': 0}

Global-Result

Data:
Demo-SimuCredit_md
Model:
XGBoost
Inputs:
{'dataset': 'test', 'feature': 'Utilization'}

PCA-Result

Data:
Demo-SimuCredit_md
Model:
None
Inputs:
{'features': None, 'n_components': 5, 'dataset': 'main', 'sample_size': None, 'categorical_encoding': 'ordinal', 'random_state': 0}
index PC1 PC2 PC3 PC4 PC5
0-0.526211.537869-1.348023-0.8894940.241198
1-1.352178-0.755415-0.2459390.197904-0.104968
2-0.480909-0.828502-0.750361-0.331259-0.132894
3-1.454527-0.9874733.3416140.443364-0.139922
4-0.8060760.639831-1.108934-0.6521870.579872
5-1.160295-0.0670550.123406-0.3330440.347544
6-0.9964480.4782851.758346-0.18093-0.391355
70.1098910.407718-0.2294371.076270.203646
82.4325060.261122-1.3947090.075158-0.631031
9-0.8575560.5618631.005858-0.7212430.921994
100.1319252.0546750.827086-0.4801180.672351
111.0718160.859586-0.5900650.8916061.039925
12-1.147963-0.0616320.130431-0.060046-0.268612
13-0.4708550.138432-0.4260581.039255-0.663208
140.069752-1.6542250.2990821.7992830.640279

Correlation-Result

Data:
Demo-SimuCredit_md
Model:
None
Inputs:
{'features': None, 'dataset': 'main', 'method': 'pearson', 'sample_size': None, 'random_state': 0}
index Mortgage Balance Amount Past Due Delinquency Inquiry Open Trade Utilization Gender Race Status
Mortgage10.005831-0.0009860.010549-0.0026180.000380.0031760.1307640.1159650.129838
Balance0.00583110.4658370.0103220.0035350.0009860.5692530.126680.1122260.125015
Amount Past Due-0.0009860.46583710.3842520.2309510.16760.2604240.1091070.0972180.13266
Delinquency0.0105490.0103220.38425210.6140430.4535620.0031640.000210.0000480.097463
Inquiry-0.0026180.0035350.2309510.61404310.7376490.0025820.0005570.0001570.058305
Open Trade0.000380.0009860.16760.4535620.73764910.0003530.0004050.0003650.038481
Utilization0.0031760.5692530.2604240.0031640.0025820.00035310.1308230.115960.129884
Gender0.1307640.126680.1091070.000210.0005570.0004050.13082310.0042510.107363
Race0.1159650.1122260.0972180.0000480.0001570.0003650.115960.00425110.178891
Status0.1298380.1250150.132660.0974630.0583050.0384810.1298840.1073630.1788911

EDA3D-Result

Data:
Demo-SimuCredit_md
Model:
None
Inputs:
{'feature_x': 'Mortgage', 'feature_y': 'Balance', 'feature_z': 'Status', 'feature_color': None, 'dataset': 'main', 'sample_method': 'random', 'sample_size': 200, 'random_state': 0}

EDA2D-Result

Data:
Demo-SimuCredit_md
Model:
None
Inputs:
{'feature_x': 'Mortgage', 'feature_y': 'Status', 'feature_color': None, 'dataset': 'main', 'sample_method': 'random', 'sample_size': 200, 'smoother_order': 2, 'random_state': 0}
In [22]:
exp.export_report()
In [ ]: