KaoLEAF: Knowledge-Augmented Optimization and Learned Embeddings for Automated Feature Representation

Abstract

Auditable credit scoring with attention

We present KaoLEAF, an attention-based model for credit decisions on tabular applicant data. Numeric features are tokenised with learned periodic embeddings, arranged in a knowledge-based order, mixed locally by a gated convolution and globally by self-attention, and pre-trained by masked-feature modelling.

A temperature-calibrated deep ensemble yields a 300–850 score with five risk bands, and Integrated Gradients provide additive reason codes. On the public Loan Approval benchmark, KaoLEAF reaches a test AUC of 0.9825 and its fusion with gradient-boosted trees reaches 0.9852, ranking first among eleven models with an ECE of 0.0096. All results come from a single dataset and a single split.

Keywords: credit scoring, tabular deep learning, attention, calibration, explainability.

1Knowledge-ordered tokens. Periodic embeddings plus local and global mixing.

2Bank-ready output. Calibrated 300–850 score, five bands, policy kept outside the model.

3Checkable reasons. Integrated Gradients with completeness, cross-checked against SHAP.

Results

Results on 10,000 held-out applicants

Every model sees the same 24 features and the same split. The tree baselines were tuned and early-stopped on the same validation set.

Read this the way a reviewer would. KaoLEAF-Fusion takes first place. The standalone KaoLEAF ensemble sits behind the tuned gradient-boosting models, ahead of the plain FT-Transformer, the MLP, Random Forest, Extra Trees and logistic regression. The case for KaoLEAF is ranking competitive with boosting, plus a reusable embedding, calibrated scores and faithful explanations.

Table 1. Best value per column in green; KaoLEAF rows shaded. Source: kaoleaf_artifacts/leaderboard.csv.

Test ROC-AUC with 95 percent bootstrap intervals for all models — **Figure 1.** Test AUC with 95% bootstrap intervals on shared resamples. The top four overlap, so treat the leaders as close.

ROC, zoomed ROC and precision-recall curves — **Figure 2.** ROC, the low-false-positive region that matters for lending, and precision–recall.

Reliability diagram, KS curve and cumulative gains — **Figure 3.** Reliability, KS separation and cumulative gains for the calibrated ensemble.

Method

How KaoLEAF works

Twenty-four engineered features become tokens, ordered by financial meaning so that neighbours belong to the same block.

ProfileCredit behaviourCapacityRequest

24 tokensperiodic embedding: numbers become waves

Local gated convsemantic order: neighbours talk

Self-attention3 layers: everyone talks

Embedding zone risk fingerprint

Score 300–850bands A–E, calibrated and explained

Feature tokenisation with learned periodic embeddings

Numerics are quantile-mapped to ≈ N(0,1) on train only. Each feature learns its own frequencies.

$$\mathbf e_j=\mathrm{ReLU}\!\Big(W_j\big[\sin(2\pi \mathbf c_j\tilde x_j)\,;\,\cos(2\pi \mathbf c_j\tilde x_j)\big]\Big)+\tilde x_j\,\mathbf a_j+\mathbf b_j,\qquad \mathbf t_j=\mathbf e_j+\mathbf p_j$$ $$H^{(0)}=[\mathbf t_{\mathrm{cls}};\mathbf t_1;\dots;\mathbf t_{m+c}]\in\mathbb R^{(m+c+1)\times d}$$

Local operator over the knowledge-ordered tokens

A gated depthwise-separable convolution slides along the semantic order.

$$\tilde H=H+\mathrm{Drop}\big(\mathrm{GLU}(\mathrm{PW}(\mathrm{DW}_{k}(\mathrm{LN}(H))))\big)$$

Global dependency modelling

Pre-LayerNorm multi-head self-attention with a GEGLU feed-forward, repeated for L layers.

$$\mathrm{Attn}_h(H)=\mathrm{softmax}\!\Big(\tfrac{QK^\top}{\sqrt{d_h}}\Big)V,\qquad H'=\tilde H+\mathrm{MHA}(\mathrm{LN}\,\tilde H),\quad H''=H'+\mathrm{FFN}(\mathrm{LN}\,H')$$

Behaviour embedding and prediction

Attention pooling is concatenated with the [CLS] state to form the reusable embedding z.

$$\alpha=\mathrm{softmax}\!\big(H_{1:}\mathbf q/\sqrt d\big),\quad \mathbf z=\mathrm{GELU}\!\Big(W\big[\mathrm{LN}(h_{\mathrm{cls}})\,;\,\textstyle\sum_i\alpha_i h_i\big]\Big),\quad \hat p=\sigma\!\big(f(x)/T\big),\ f(x)=\mathbf w^\top\mathbf z+b$$

Two-stage learning and deep ensemble

Stage 1 reconstructs masked features without labels. Stage 2 fine-tunes with AdamW, warm-up plus cosine decay and early stopping on validation AUC.

$$\mathcal L_{1}=\tfrac{1}{|\mathcal M_{\mathrm{num}}|}\sum_{j\in\mathcal M_{\mathrm{num}}}(\hat x_j-\tilde x_j)^2+\overline{\mathrm{CE}}_{\mathrm{cat}},\qquad \mathcal L_{2}=\mathrm{BCE}\big(f(x),y\big),\qquad \bar f=\tfrac1S\sum_{s=1}^{S}f_s$$

Calibration and the KaoLEAF Score

Temperature scaling is fit on validation only. The fitted value is T* = 1.044.

$$T^\star=\arg\min_T\ \mathrm{NLL}\big(\sigma(\bar f/T)\big),\qquad \mathrm{Score}=\mathrm{round}\big(300+550\,\hat p\big),\qquad \text{bands } A\text{–}E \text{ at } \hat p\in\{0.2,0.4,0.6,0.8\}$$

Faithful explanations

Integrated Gradients start from the median applicant in quantile space and satisfy completeness. Categoricals use UNK-occlusion.

$$\mathrm{IG}_j(x)=(\tilde x_j-\tilde x^{0}_j)\!\int_0^1\!\partial_j f\big(\tilde x^{0}+\alpha(\tilde x-\tilde x^{0})\big)\,d\alpha,\qquad \sum_j \mathrm{IG}_j=f(x)-f(\tilde x^{0})$$

KaoLEAF-Fusion

A logistic stacker over the logits of KaoLEAF and the three strongest boosted models, fit on validation with the test set untouched.

$$\hat p_{\mathrm{fusion}}=\sigma\!\Big(\beta_0+\beta_{\mathrm{leaf}}\,\bar f+\textstyle\sum_{g\in\{\text{LGBM, Cat, XGB}\}}\beta_g\,\mathrm{logit}(\hat p_g)\Big)$$

What each component is worth

Single-seed ablation under an identical budget. Differences near 1e-3 AUC are inside seed noise; only the periodic embedding shows a consistent gap, so we claim only that.

Stage 1 and stage 2 training curves — **Figure 4.** Stage 1 masked-feature loss and stage 2 validation AUC across the five ensemble seeds.

Score

From probability to a score a bank can use

Representation, prediction and decision policy are separate layers. Thresholds and the hard rule for defaults_on_file live in policy, so a bank can change them without retraining.

Meet the demo applicant

29 yrsEmployedincome 52,000credit score 655DTI 0.31Personal Loanintent: Educationasks 15,000

Real output of the notebook's inference demo: one applicant, not a benchmark. Solid bars push toward approval, hatched bars against.

Band	Score range	Applicants	Share	Predicted	Observed approval
A · Very strong	740–850	4,632	46.3%	97.5%	97.8%
B · Strong	631–740	609	6.1%	70.4%	71.1%
C · Moderate	520–630	464	4.6%	50.3%	57.8%
D · Weak	410–520	504	5.0%	29.7%	33.9%
E · Very weak	300–410	3,791	37.9%	2.3%	2.7%

Observed approval rises monotonically from E to A and tracks the predicted rate in every band.

Score distribution, band calibration and portfolio share — **Figure 5.** Score distribution by outcome, band calibration and portfolio share.

Explainability

Explanations an auditor can check

Importance is reported per feature and per financial block, and cross-checked against a different model family.

Global importance by feature and block — **Figure 6.** Global importance: mean |Integrated Gradients|, grouped into profile, credit behaviour, capacity and request.

Reason codes for three applicants — **Figure 7.** Reason codes for the most confident approval, the most confident decline and a borderline case, where the bars nearly cancel and manual review applies.

KaoLEAF Integrated Gradients versus LightGBM SHAP — **Figure 8.** KaoLEAF versus LightGBM SHAP: Spearman ρ = 0.72.

t-SNE of raw features versus learned embedding — **Figure 9.** t-SNE of raw features versus the learned embedding, organised along a risk gradient.

ICE and partial dependence with monotonicity audit — **Figure 10.** ICE and partial-dependence curves for credit score, DTI, delinquencies and credit history, used for the monotonicity audit.

Data

Data and governance

KaoLEAF is benchmarked on the public Realistic Loan Approval Dataset (US and Canada) from Kaggle, created by parthpatel2130: 50,000 applicants, 55.0% approved. The same method is planned for GSB data, which cannot be made public. The pretrained model can be downloaded from 🤗 Hugging Face, and the dataset from 🤗 Hugging Face Datasets.

Approval rate versus key features — **Figure 11.** Approval rate by decile. DTI behaves like a threshold, which is where learned periodic embeddings help.

Slice and fairness audit — **Figure 12.** Slice audit by age group, occupation, product and intent.

Limits we state up front

The data is cross-sectional, so the local operator convolves over ordered financial features, not over time; the temporal encoder is the next step for longitudinal data. The dataset appears simulated, so results show method behaviour rather than bank performance. age is a protected attribute and a variant without it is supported. Pricing features may leak underwriting outcomes in production. Results come from one dataset and one split. The model is not validated for production lending.