Clustered Feature Importance: Eliminating the Substitution Trap in Financial Machine Learning

Executive Summary: If you load your trading chart with five different momentum indicators, they will all tell you the exact same thing. But when a machine learning model tests them, a dangerous blind spot emerges: the substitution trap. Because the indicators act as substitutes for one another, the algorithm assumes none of them matter—and deletes them all. Here is why this happens, how to fix it using indicator clustering, and how you can use this concept to clean up your own trading setups.

1. The Mountain Climber & The 5 Weather Apps

Imagine you are packing for a high-altitude mountain expedition. To be extra safe, you install five independent weather forecast apps on your smartphone.

On the morning of the climb, you unlock your phone. All five apps flash bright red warnings: Severe thunderstorm and torrential rain arriving in two hours.

Now imagine an over-analytical scientist looks at your phone and runs an experiment:

They uninstall App .
They check the forecast again. The remaining four apps still clearly warn of the thunderstorm.
The scientist concludes: "Deleting App caused zero loss in forecasting ability. Therefore, App is completely useless!"

Then the scientist repeats this test: they delete App , App , App , and App one by one. In every single test, the other apps covered for the one that was missing.

The scientist concludes that every single weather app is worthless and uninstalls all five. You hike up the mountain with zero warnings and get trapped in a flash flood.

How Standard Feature Selection Gets Confused (The Substitution Trap)
───────────────────────────────────────────────────────────────────
[RSI-14]      ──┐
[StochRSI]    ──┼──► All 5 Track Price Speed ──► Algorithm thinks none of them matter
[14D-Mom]     ──┤                                 (Measured importance drops to ~2% each)
[20D-Mom]     ──┤
[MACD Signal] ──┘                                 Result: Algorithm deletes all 5!

The Solution: Clustered Feature Importance (CFI)
───────────────────────────────────────────────────────────────────
┌────────────────────────────────────────────────────────┐
│  MOMENTUM FAMILY (Cluster Importance: 38% - VITAL!)    │ ──► Keeps the edge alive!
└────────────────────────────────────────────────────────┘
  ├── Test the entire family together as one unit
  └── Pick the single fastest, cleanest indicator to trade with

This exact scenario happens every day to traders and machine learning algorithms in financial markets. It is called The Substitution Trap.

2. Why Algorithms Get Tricked in Financial Markets

When developers build trading bots or machine learning models, they frequently feed in dozens of popular indicators:

14-period Relative Strength Index (RSI)
Stochastic RSI
14-day Rate of Change (ROC)
20-day Momentum
Exponential Moving Average (EMA) Spreads

Ask yourself: What are these indicators actually measuring?

They are all measuring the exact same thing: how fast the price is moving over time (velocity).

When a standard machine learning algorithm (like a Random Forest or Gradient Boosted Tree) tries to figure out which indicators are most useful, it tests them by scrambling them one at a time:

It scrambles the 14-day RSI.
But Stochastic RSI and 20-day Momentum are still intact, so the model's predictive accuracy barely drops.
The computer thinks: "Removing RSI didn't hurt my predictions at all. RSI must be noise."

The model repeats this process for all five momentum indicators. Instead of recognizing that momentum itself is the most powerful profit driver in the market, it dilutes importance across all five into tiny, meaningless slivers (e.g. 2% or 3% each).

The trader or data scientist looks at the report, sorts by importance, and deletes every indicator below a 5% threshold.

Just like the mountain climber deleting all five weather apps, the trader has just amputated their strongest trading edge from the model.

3. The Fix: Clustered Feature Importance (CFI)

The institutional solution is simple, elegant, and practical: Clustered Feature Importance (CFI).

Instead of testing indicators as isolated, lonely columns, you organize them into families based on what they actually measure.

The Three Core Indicator Families in Trading:

How Clustered Testing Works:

Instead of scrambling one indicator at a time, you scramble the entire family at once.

When the algorithm scrambles all five momentum indicators simultaneously, neither RSI nor Stochastic can cover for each other. The model's predictive accuracy drops off a cliff.

Immediately, the algorithm reveals the truth:

"The Momentum Family accounts for 38% of our strategy's predictive edge. It is non-negotiable."

4. The Trader's Playbook: What Should You Do on Your Charts?

You do not need a supercomputer to benefit from this principle. Every discretionary and systematic trader can apply this immediately to improve their decision-making:

1. Kill Chart Clutter (Stop Stacking Twins)

If you have RSI, Stochastic RSI, and MACD all open at the bottom of your screen, you do not have 3x confirmation. You have 3 copies of the exact same information taking up mental bandwidth.

2. Pick One Champion per Family

Once you identify an indicator family that works, pick the single cleanest, lowest-latency indicator in that family. For momentum, you might keep a standard 14-period RSI and delete the other four. You keep 100% of the information while cutting 80% of the noise.

3. Diversify Across Truly Different Families

Real edge comes from combining indicators from different families that do not substitute for one another:

A Structural Level tells you WHERE to look (e.g., a liquidity sweep below yesterday's low).
An Order Flow Signal tells you WHO is acting (e.g., aggressive taker absorption on CVD).
A Momentum Signal tells you WHEN price is turning.

When three different families agree, you have genuine institutional confluence—not duplicate noise.

5. Practical Python Implementation (Clean & Simple)

Here is a straightforward Python implementation showing how to group indicators into clusters and test them together as a family:

import numpy as np
import pandas as pd
from scipy.cluster.hierarchy import linkage, fcluster
from scipy.spatial.distance import squareform
from sklearn.metrics import roc_auc_score

def evaluate_indicator_families(model, X_train, y_train, X_test, y_test):
    """
    Groups correlated indicators into families and measures their collective edge.
    Prevents the substitution trap from hiding real alpha drivers.
    """
    # 1. Measure correlation between all indicators
    correlation_matrix = X_train.corr().abs().fillna(0)
    
    # 2. Convert correlation into distance (high correlation = close together)
    distance_matrix = np.sqrt(0.5 * (1.0 - correlation_matrix.values))
    np.fill_diagonal(distance_matrix, 0.0)
    
    # 3. Group indicators into families using hierarchical clustering
    tree = linkage(squareform(distance_matrix), method='ward')
    # Group into 3 to 5 logical families
    cluster_ids = fcluster(tree, t=3, criterion='maxclust')
    
    families = {}
    for idx, cluster_id in enumerate(cluster_ids):
        families.setdefault(cluster_id, []).append(X_train.columns[idx])
        
    # 4. Train baseline model and measure benchmark accuracy
    model.fit(X_train, y_train)
    baseline_score = roc_auc_score(y_test, model.predict_proba(X_test)[:, 1])
    
    # 5. Measure importance by scrambling each family AS A COMPLETE UNIT
    family_importance = {}
    for cluster_id, indicators in families.items():
        X_test_scrambled = X_test.copy()
        
        # Scramble all twin indicators in this family together
        for ind in indicators:
            X_test_scrambled[ind] = np.random.permutation(X_test_scrambled[ind].values)
            
        # Accuracy drop reveals the true value of the entire family
        scrambled_score = roc_auc_score(y_test, model.predict_proba(X_test_scrambled)[:, 1])
        family_importance[f"Family {cluster_id} ({', '.join(indicators)})"] = baseline_score - scrambled_score
        
    return pd.Series(family_importance).sort_values(ascending=False)

6. Key Takeaways

The Substitution Trap: If two or more indicators measure the same thing, traditional tests will falsely conclude that both are useless.
Cluster First, Prune Second: Group similar indicators into families, test the whole family at once, and only then choose the single best representative.
True Confluence Requires Independence: An edge is formed by pairing different types of information (e.g. Structure + Order Flow + Speed)—never by stacking five identical momentum oscillators.

🌐 Inspect Institutional Microstructure on Kingsmen Alpha

Track real-time Order Block footprints, taker CVD divergence, and systematic risk-parity execution models live across 1,740+ perpetual futures: 👉 Launch Kingsmen Alpha Workstation