Half my alerts were false positives. I raised my per-tenant training-sample floor from 200 to 1,000 — a 5x increase — and the false-positive rate stayed at 48.7%. Not "roughly 48.7." Not "within noise of 48.7." Exactly 48.7%, to three significant figures, across 30 post-deployment runs and 382 fresh alerts.

Volume was never the constraint. I want to tell you what was.

The setup

I run per-tenant IsolationForest models for M365 session-hijacking detection. Roughly 1,540 tenant-specific models plus one global fallback, refit weekly on a 30-day rolling window. The routing rule was the industry default: if a tenant has enough training samples, it gets its own model. Otherwise it falls back to the global one.

The intuition is obvious. A per-tenant model built on 40 sessions is going to be noisy. Bump the floor high enough, and the model has room to breathe. The tenants that can't clear the floor go to global, where they piggyback on everyone else's traffic. Everybody wins.

Except — after 133 hourly production runs, 48.7% of my model-layer alerts were coming out of a small number of "degenerate" per-tenant models. Models that emitted at most three distinct scores across the entire observation window. Modal score sitting right at the threshold. Every session in the tenant looked the same to the model, and every session in the tenant triggered an alert.

So I did the obvious thing.

The fix that changed nothing

Raised the sample-count floor from 200 to 1,000. Redeployed. Let it settle for 30 runs, 382 alerts of new post-deployment data.

Degenerate ratio pre-deploy: 48.7%.

Degenerate ratio post-deploy: 48.7%.

Not close to identical. Identical.

What actually happened underneath the aggregate: the top three offenders from the pre-deploy window fell below the raised floor and got demoted to global — which pushed their false-positive volume into the global model's alert stream instead of eliminating it. Meanwhile, two tenants that had been quietly degenerate all along, ranked just below the top offenders in the pre-deploy window, rose into the top-three slots. One of them — call it T-50cd — was scoring at 100% degeneracy: every single one of its predictions collapsed to a single constant value.

The set of degenerate tenants churned. The rate held steady.

The intervention did exactly the thing volume-based reasoning predicted it wouldn't do: it left the underlying rate untouched.

Why volume wasn't the constraint

I ran the diagnostics on the training matrices of the surviving offenders. Three metrics: how many features are effectively constant on that tenant's training set, the "effective rank" of the standardized matrix (a fancy way of asking how many independent directions the data actually populates), and the Shannon entropy of each discrete column.

Two of the offenders confirmed what I expected. Their training sets had 6–7 features stuck at zero variance. Effective rank was 4–7 on a 23-dimensional feature space, well below the global mean of 7.6. Five to six categorical columns took the same value across every single training row. These tenants weren't undertrained on volume. They were undertrained on diversity. All the sessions looked structurally identical to the model, so the model had nothing to build on.

I call this Mode A. Training-set diversity-undertraining. It's diagnosable at fit time from the training matrix alone.

Then I ran the third offender. T-8a43. The tenant with 9,526 training samples — a full order of magnitude above the raised floor.

Zero features below the variance floor. Zero zero-entropy categorical columns. Effective rank 10.02, inside the top decile of the entire per-tenant distribution.

By every diagnostic I had, T-8a43 was more diverse than the typical healthy tenant. And its per-tenant IForest produced near-constant scores in production anyway.

Mode B

This is the finding I didn't expect and the one that changes the architecture.

Diversity of the training set isn't sufficient. There's a second failure mode where the training data is fine, but something in how the IForest interacts with sub-sampling, or calibration, or the residual scoring inputs, produces a degenerate score surface anyway. I don't yet know which of those is the mechanism for T-8a43 — probably some combination. But the failure is real, it's reproducible, and it's invisible to any training-time diagnostic you can run on the training set alone.

Mode B is only detectable by watching the model's actual runtime output.

Which means: the routing layer needs three checks, not one.

  1. Sample count. The volume floor. Rejects tenants that literally don't have enough data. Necessary and cheap.
  2. Training-set diversity. Effective rank plus categorical-column entropy on the training matrix. Rejects Mode A tenants before you spend the compute to fit them. Fast; runs once per training cycle.
  3. Runtime score entropy. A rolling window of the model's actual scoring outputs. Demote to global fallback if unique-score counts drop below a threshold. Catches Mode B tenants, and catches any Mode A tenant that slipped past the training-time check.

None of the three checks subsume the others. Sample-count routing was the only one my deployment had. That's why the fix changed nothing: it acted on the constraint that wasn't binding.

Here's what the runtime check looks like, in eight lines: 

from collections import defaultdict, deque
import numpy as np

# Rolling window of each tenant's decision_function outputs
scores = defaultdict(lambda: deque(maxlen=1024))

def route(tenant_id, s, k_unique=5):
    scores[tenant_id].append(s)
    if np.unique(scores[tenant_id]).size < k_unique:
        return "demote_to_global"
    return "per_tenant"

That's the entire thing. Attach it to your per-tenant scoring loop, demote any tenant whose runtime output collapses to fewer than k_unique distinct values across a rolling window. Constant memory per tenant. Catches Mode B directly. Catches any Mode A tenant you missed at training time as a bonus. There is no argument against having it in your routing layer, and until I wrote the paper this piece is drawn from, I didn't.

The generalization

This is a routing-layer failure mode in multi-tenant unsupervised anomaly detection. It doesn't require IsolationForest specifically. Any per-tenant model that assumes the training distribution has meaningful support will produce this pattern when the support collapses on some tenants and not others. The dominant fix in the literature — better isolation geometry, EIF, SCiForest — targets a different problem. If your training set has rank 4 on a 23-dimensional space, no split geometry saves you. If your training set is diverse and your score surface still collapses, the split geometry isn't the problem either. The routing layer is the problem, and the routing layer is where the fix lives.

If you're running per-tenant anomaly detection at scale — MDR platforms, cloud security posture management, identity risk scoring — I'd bet you have both modes in your production data. You'll only see Mode A if you're looking at your training matrices. You'll only see Mode B if you're monitoring your models' score entropy at runtime.

Sample-count floors are cheap and defensible and completely insufficient. They pass every code review and catch none of the tenants that are actually broken.