Re-Identification Risk vs k-Anonymity: An Experimental Walkthrough

Most discussions of anonymization focus on buzzwords like k-anonymity and differential privacy, but few dig into what actually happens to a dataset as anonymity strength increases.

In this post, we conduct a full experimental walkthrough to quantify how raising the k-anonymity level impacts both privacy (re-identification risk) and data utility.

We simulate an attacker with partial knowledge trying to re-identify individuals, and we measure how data quality degrades as we ramp up the anonymization.
The goal is to illuminate where the balance lies between keeping data useful and keeping individuals anonymous.

Data Generation and Anonymization Setup

For our experiment, we generated a synthetic dataset of 2000 individuals, each with the following fields:

We treat age, zip3, and sex as the quasi-identifiers (QIs) that will be subject to anonymization.

The value of k in k-anonymity was varied from 1 to 20.
A k-anonymity requirement means each record must be indistinguishable from at least k–1 others with respect to these QIs.

To achieve this, an anonymization routine groups and generalizes records until every combination of QIs occurs in at least k records.
In practical terms, as k increases, the algorithm must increasingly generalize (coarsen) or suppress details in the QIs to satisfy the larger group size.

Parameters explored

We explored a few key parameters that control how the data is generalized:

• Age bin width: We varied age grouping from 1-year bins (no grouping beyond integer ages) up to 10-year bins. Larger bin widths mean ages get lumped into broader ranges (e.g., 30–39).
• Top-coding of age: Extreme ages were top-coded above a threshold (e.g., all ages 75 and above recorded as "75+"). This prevents very old ages from standing out.
• Rare ZIP suppression: Low-frequency ZIP3 regions were grouped into an "Other" category once their count fell below a threshold. If a region is too unique, it gets collapsed to hide outliers.

By adjusting these knobs, we impose different anonymization strategies.
For each value of k (and each combination of binning/top-coding settings), we produced an anonymized version of the dataset and evaluated how much information was lost in the process.

Attacker Context: Partial Knowledge Threat Model

Anonymization is only meaningful relative to an attacker’s knowledge.
In our scenario, we simulate an attacker who has partial information about individuals — specifically, the attacker knows an individual’s:

• age
• general location (ZIP3 region)
• sex

(e.g., from a data leak or public records).

This is a common threat model for re-identification:
an adversary might obtain someone’s demographic details from a breached source and then try to find that person in an anonymized dataset (such as medical or survey data) released publicly.

The attacker’s goal is re-identification:
to match each anonymized record to the corresponding real individual by comparing the quasi-identifiers.

Importantly, the attacker does not know the sensitive value (lab_glucose) in our case;
they only leverage the QIs that are also present (albeit generalized) in the anonymized data.

This kind of attack is known as a record linkage attack, using the assumption that if an anonymized entry shares a unique combination of age, region, and sex with a known individual’s data, they are likely the same person.

This threat model underscores why k-anonymity focuses on QIs:
even innocuous-seeming attributes like age and ZIP code can triangulate someone’s identity when combined.

Next, we describe how our simulated attacker performs the re-identification.

Attacker’s Re-identification Strategy (Global Matching)

How does our attacker try to re-identify records?

Instead of using a simple greedy matching (checking each anonymized record independently),
we implement a global optimization strategy.

We use a bipartite assignment solver (Google OR-Tools’ linear sum assignment solver) to find the optimal one-to-one matching between anonymized records and original records that best aligns their attributes.

Cost-based Matching

We define a cost for matching an anonymized record aᵢ with an original record oⱼ based on their differences in quasi-identifiers:

cost(i, j) = d_age(a_i, o_j) + d_zip(a_i, o_j) + d_sex(a_i, o_j)

Each d term is a distance measure for that attribute.

• If an anonymized age is a range (due to binning) and the original age falls within that range, the age distance may be zero; if it falls outside, the distance increases.
• If an anonymized ZIP3 was generalized to "Other", any specific ZIP from the original will incur a cost when compared to "Other".

These distances capture how well an original record fits the generalized form of an anonymized record.
Lower cost → the two records are more similar across QIs.

We then solve for the assignment π(i) that minimizes the total cost of matching all anonymized records to distinct original records:

min_π  Σ_i  cost(i, π(i))
subject to: each original record is matched at most once

This optimization finds the best overall matching between the two datasets.

By considering all records jointly, the attacker avoids making locally optimal but globally inconsistent matches.

Even if each anonymized record individually has multiple plausible matches, the solver finds a globally consistent assignment.
The outcome is an assignment pairing most anonymized records with specific original record guesses.

Measuring Re-identification Success: Hit@1

To evaluate the attack, we use the Hit@k metric common in information retrieval.

A “hit” means the correct original record appears within the attacker’s top k guesses for an anonymized record.

In our case:

• the solver produces one best match per anonymized record
  → effectively Hit@1 only

So we focus on Hit@1, the fraction of anonymized records where the attacker’s top guess is correct.

A Hit@1 of 50% means the attacker correctly re-identified half of the individuals on the first guess.

(Hit@5 would allow up to 5 guesses per record, but we stick with the strictest measure.)

With the attack strategy and success metric defined, we now examine how re-identification risk and data utility change as anonymization strength increases.

Results

Re-identification Success vs. Anonymity Level

We first examine how the attacker’s success rate (Hit@1) changes as the anonymity parameter k increases.
Intuitively, higher k (stronger anonymity) should make re-identification harder.

Our experiments confirmed this:
the attacker’s success drops dramatically as k grows.

Each cell is the average Hit@1 across trials for a given combination of k (y-axis) and age bin width (x-axis). Success rates plummet as k increases. Notably, there is a sharp drop in attacker success once k is around 5–7, indicating the onset of strong anonymity where the attack loses traction.*

Even at k = 1 (minimal anonymization), the attacker does not get a 100% hit rate.
The maximum Hit@1 observed hovered just above ~50%.

This is because even in the raw data, some individuals share the same QI values
(e.g., multiple people with the same age, ZIP3, and sex),
so they cannot all be uniquely identified by QIs alone.
This sets an upper ceiling on re-identification success.

As k increases from 1 to 5, Hit@1 falls gradually.
Beyond k ~ 5–7, it plummets sharply.

By k ≥ 10, the attacker’s top-guess accuracy is very low
(approaching random chance in many settings).

Summary: raising k dramatically improves privacy, especially after the mid-range threshold where anonymity “kicks in.”

Data Utility Loss as k Increases

Stronger anonymization comes at the cost of data utility.

We tracked several metrics to quantify how the dataset’s analytical value degrades as k increases:

• ZIP Utility:
  Measures how well the distribution of ZIP3 values is preserved.
  Defined between 0–1, where 1.0 means the anonymized ZIP distribution exactly matches the original.
  (Computed as (1 - \frac{1}{2} L_1) distance.)

• Mean Age Drift:
  The difference in the average age between anonymized and original data.
  Captures how much anonymization distorts age information.

• Mean Glucose:
  A sanity check for a non-QI variable that should remain unchanged.

Two of these metrics, ZIP Utility and Age Drift, clearly illustrate the non-linear loss of detail as k grows.

The line shows that as k increases, the ZIP code distribution retains less and less of its original detail.
Once k exceeds about 8, we see a notable drop in ZIP Utility.
At k = 16, roughly 25–30% of the geographic granularity is lost.*

A negative drift means the anonymized data’s average age is lower than the original.
At high anonymity levels (k ≈ 20), the mean age is about 3 years lower.
Top-coding and heavy binning compress the age distribution toward the middle.*

Reassuringly, Mean Glucose remained essentially unchanged across all k values (drift ~0).
This confirms that the anonymization procedure targeted only QIs (age, zip, sex) and did not distort unrelated attributes.

Overall:

• For small increases in k (1–5), utility remains close to original fidelity.
• Beyond k ≈ 5–8, generalization becomes aggressive and utility drops sharply.

This suggests a “sweet spot” where privacy improves significantly while preserving substantial utility, after which additional privacy becomes expensive in terms of information loss.

The Privacy–Utility Frontier

It is helpful to visualize the inherent trade-off between privacy and utility.

Each anonymization configuration we tested
(a specific combination of k and generalization parameters)
can be thought of as a single point in a two-dimensional space:

• one axis = privacy outcome (e.g., Hit@1 re-identification success)
• the other axis = utility outcome (e.g., how many candidate matches remain / how much detail is preserved)

Plotting all configurations reveals a clear privacy–utility frontier.

Each point represents one anonymization scenario (specific k and parameter settings), plotted by its resulting privacy risk (y-axis: Hit@1 success rate) and a utility indicator (x-axis: number of plausible candidate matches per anonymized record, which correlates with retained information).

The plot forms a downward-sloping curve.

• Configurations with lower re-identification risk invariably have lower data utility.
• The initial part of the curve is steep — meaning you can reduce risk significantly with only a small drop in utility.
• The later part of the curve flattens — meaning achieving tiny extra privacy gains requires large utility sacrifices.

In simpler terms:

You can’t have it all.
Past a certain point, making the data “very anonymous” makes it statistically or analytically blurry.

The scatter shows every dataset version lies somewhere on this curve.
Deciding where to operate is a policy choice:

• low k → high utility, low privacy
• high k → high privacy, low utility

Conclusion and Discussion

Our empirical exploration highlights how increasing k-anonymity leads to diminishing returns.

For modest anonymity levels (up to around k = 5):

• Each increment in k yields a big drop in re-identification risk.
• The corresponding hit to data utility is mild.

Beyond that, however, the trade-off worsens:

• Pushing k higher gives smaller and smaller privacy benefits.
• Meanwhile, it rapidly erodes the granularity and usefulness of the data.

This is essentially a manifestation of a Pareto frontier —
there comes a point where you must give up a lot of utility to get a little more privacy.

Attribute Sensitivity

Different data attributes showed different sensitivity to anonymization:

• Geographic detail (ZIP3) degraded first.
  Many ZIPs are rare → must be collapsed to "Other" as k grows.

• Age was more resilient but eventually smoothed by
  wide bins and top-coding → resulting in shifts such as
  a 3-year drop in average age at high k.

• lab_glucose remained unchanged.
  Since glucose was not part of the QIs, anonymization preserved it.
  This demonstrates that non-identifying variables can remain intact
  even as identifying information is stripped away.

This attribute-by-attribute difference shows that utility loss is domain-specific.
Some features lose meaning faster than others under anonymization.

About the Attacker Model

It is also worth noting that our attack model was relatively basic.

We assumed the attacker only knows:

• age
• sex
• region (ZIP3)

And they use a straightforward optimal matching algorithm.

A more determined adversary might:

• have additional clues (e.g., approximate health measurements)
• access multiple leaks
• use statistical models to narrow matches
• use Bayesian linkage, ML-based scoring, or constraint solvers

Such an attacker could defeat k-anonymity more often.
Therefore the Hit@1 rates in our experiment may be optimistic.
Real-world re-identification risk could be higher.

This highlights that anonymization should not be:

a one-time, set-and-forget protection mechanism.

You must consider evolving threat models
and possibly combine k-anonymity with other techniques:

• noise addition
• perturbation
• differential privacy
• synthetic data generation
• secure linkage systems

Finding the Balance

Ultimately, choosing an anonymization level is about balancing privacy risk against data usability.

Our experiment puts concrete numbers on that balance:

• The initial drop in re-id risk (as k rises from 1→5) is encouraging.
• It means we can significantly protect identities without immediately destroying utility.
• But the flattening of the curve at higher k reminds us that
  aggressive anonymization yields minimal extra privacy at huge cost.

Decision-makers should consider what level of risk is acceptable
given the purpose of the data.

For many cases:

• Moderate k (enough to prevent easy pinpointing of individuals)
  is sufficient and maintains usefulness.

• High k may make the dataset practically unusable.

Key Takeaways

• k-anonymity trades precision for privacy.
  Generalization and suppression remove detail from QIs.

• Privacy gains are strong at first, then plateau.
  Beyond mid-range k, utility collapses faster than privacy improves.

• Utility loss is nonlinear and varies by attribute.
  Sparse attributes like ZIP lose meaning earlier.

• Non-identifying attributes can remain intact.
  Good for preserving analytical value.

• Past moderate k, returns diminish greatly.
  More anonymity → minimal privacy gain, major utility loss.

In conclusion:
Effective anonymization is about finding the balance:
protecting individuals without rendering data barren for analysis.
Our findings illustrate that balance clearly for this dataset under different settings.

Why this matters

We tend to treat cosine distance as if it were a true metric.
Cosine does not satisfy triangle inequality: explicitly proven here.

Metric indexes do rely on triangle inequality to prune search space, shown in
Efficient Metric Indexing for Similarity Search.

Metric indexing relies on triangle inequality for pruning.

HNSW / FAISS don’t require strict metric axioms, but they assume neighborhood consistency: if a point is “closer,” greedy search expects it to lead to even closer points.

That assumption only holds when the embedding space has stable geometry (i.e. distances behave consistently like in a true metric).

What causes the embedding geometry to break down

Two independent things:

1. Bad corpus / domain mismatch
   If the embedding model wasn’t trained on similar text, semantics get scattered.

2. Compression (PCA + quantization)
   Removes structure. Local neighborhoods collapse. This is particularly bad because compression solves a lot of operational problems.

Both lead to the same consequences:

• nearest neighbors stop being nearest
• triangle inequality fails locally, and fails harder as distances increase.
• retrieval (the R in RAG) becomes unstable

This post measures that directly.

Setup

We embed two different datasets:

Three embedding variants:

Each corpus was chunked into 1000 samples.

What we measure

For each point i:

1. find its top-k nearest neighbors (j)
2. check if any neighbor of a neighbor (k) breaks triangle inequality:
   d(i, k) > d(i, j) + d(j, k) + τ

If no violation exists, the point is clean.

The metric:

clean_frac = fraction of points with consistent neighborhoods

τ = tolerance.
Higher clean_frac: stable space.
Lower clean_frac: distances are not reliable.

I use Z3 only to answer a yes/no question:

“Does any violating (j, k) exist for this anchor?”

Brute-forcing all (i, j, k) triples did not work well when I tackled this in the past.
Z3 doesn’t brute force. It treats distances as constraints and either:

• finds a violating triplet, or
• proves that none exists for that anchor.

Results

Three parts:

1) UMAP: do embeddings even cluster coherently?

Raw embedding models produce tight clusters separated by corpus, whereas PCA+quantization blurs everything together causing the food and medical corpora to overlap with no separation!

Already looks like geometry degradation.

2) Heatmap: metric stability across k neighbors and τ tolerance

(In the heatmap, the horizontal axis is the neighbor count k and the vertical axis is the tolerance τ. Color indicates the fraction of points that remain clean)

Observations:

• medical corpus: solid geometry (clean_frac ≈ 1.0)
• food corpus: noisy semantics, which leads to poor geometry
• PCA+quantized: catastrophic collapse

Even at τ = 0.1 (a huge forgiveness margin), PCA+quantized still breaks.

3) Stability vs k: how fast the neighborhood falls apart

• raw embeddings degrade slowly as k expands
• compressed embedding collapses by k=10

If retrieval expands k during rerank / recall-then-rerank — expect garbage neighbors.

Key takeaways

1. Embeddings are not guaranteed to form a metric space.
   If triangle inequality fails, nearest neighbors may not be the nearest.
   Retrieval results may not be ideal.

2. Compression destroys neighborhood structure.
   PCA+quantization doesn’t ‘reduce redundancy’. This step needs extra monitoring, as results can degrade fast.

3. Weaker/less structured corpus result garbage geometry.
   Not surprising.

Choose embeddings based on how well they preserve geometry.

Vector DBs assume a metric space.
Most embedding models don’t always return one.

If the embedding space breaks (wrong model, wrong corpus, or compression) nearest neighbors aren’t nearest and the R in RAG stands for roulette…

Most model replacement flows stop after validation accuracy.

If loss and accuracy remain roughly the same, the task is considered done.
But validation only tells us that on the samples we checked the models behave similarly.

It says nothing about the rest of the input space.

Why this matters — two major use cases

There are at least two distinct workflows where this matters:

A) Model pruning / distillation / simplification

We modify a model intentionally:

• reduce latency
• reduce model size
• simplify the architecture (for interpretability or cost)

We want to know if the simplified model really behaves like the original one.

Example: Random Forest → Pruned Random Forest (our example in this post)

B) Model retraining / continuous integration

A model is re-trained with new data, new hyperparams, or a new architecture.

Before replacing the model in production we need to know:

Is the new model equivalent to the legacy one, or where do they differ? How much and where do they differ?

This turns model replacement into a regression test, similar to a CI/CD. This makes model updates less opaque and gives all the scope we need to understand how it generalizes.

Validation tells us similarity on sampled points, on information that we already have.
What we want is equivalence:

For all inputs x in the domain:
    model_A(x) == model_B(x)

If the answer is no, we want to know the exact violating input X.

Z3 for Model Equivalence: Proving ML Models Match (or Finding the exact input where they don’t)

Goal: Instead of measuring similarity between models, prove they’re equivalent, and where they’re not: extract the exact counterexample.

Z3 is a constraint solver from Microsoft Research.
Optimizers try values and adjust based on results.
Z3 doesn’t search, it doesn’t brute forces the computation, it reasons about all possible inputs.
We state the rules, and it determines whether any input satisfies them.

For model equivalence, we ask: is there any x where the two models disagree? If yes, Z3 returns that x; if not, it proves none exists.

The classic use case (or at least how I first heard of z3): Sudoku solver.

Every rule is encoded as a constraint: numbers 1–9, unique rows, unique cols, etc.
Z3 doesn’t brute-force — it symbolically prunes entire spaces at once.

We use the same trick here.

Instead of trying inputs until the model fails, we ask Z3:

∃ x   such that   Model_A(x) ≠ Model_B(x)

If yes → Z3 produces that violating x.
If not → Z3 proves no such x exists within the domain.

Encoding a model as logic

Decision trees are perfect for SMT solving because they’re pure conditional logic:

if x[5] <= 0.12:  
   left
else:             
   right
return leaf_label

Z3 can encode every branch of every tree.

Then we assert:

(pred_A(x) != pred_B(x))

and let Z3 do the search.

Code: Locating the exact counterexample

If a single violating input exists, Z3 returns it: guaranteed and verifiable.

from z3 import Real, RealVal, If, And, Or, Sum, Solver, sat

def encode_tree_as_z3(tree, x_vars):
    t = tree.tree_
    L, R = t.children_left, t.children_right
    feat, thr, val = t.feature, t.threshold, t.value

    def go(n):
        if L[n] == R[n]:
            return RealVal(int(val[n][0].argmax()))
        return If(x_vars[feat[n]] <= thr[n],
                  go(L[n]),
                  go(R[n]))
    return go(0)

def encode_forest_avg_vote(rf, x_vars):
    return Sum(*[encode_tree_as_z3(t, x_vars) for t in rf.estimators_]) / len(rf.estimators_)

def z3_label_counterexample(big, pruned, lo, hi):
    d = len(lo)
    x = [Real(f"x{i}") for i in range(d)]
    b = encode_forest_avg_vote(big, x)
    p = encode_forest_avg_vote(pruned, x)

    s = Solver()
    s.add(And(*[(x[i] >= lo[i]) & (x[i] <= hi[i]) for i in range(d)]))
    s.add((b >= 0.5) != (p >= 0.5))

    if s.check() != sat:
        return None
    m = s.model()
    return [float(str(m[xi])) for xi in x]

lo and hi are per-feature min/max bounds Z3 must respect. This prevents it from returning absurd values like x = 10⁹

>>> [-0.965, 0.549, 0.247, 0.589, 0.475, 3.397, ...]

This vector is a real point in feature space that breaks model equivalence.

Minimal explanation trace

We can even trace which removed trees caused the discrepancy:

def trace_disagreement(x_cex, big, pruned, top_k=8):
    xb = x_cex.reshape(1, -1)
    votes = np.array([t.predict(xb)[0] for t in big.estimators_], float)
    removed = [i for i, t in enumerate(big.estimators_) if t not in pruned.estimators_]

    diffs = [(abs(votes.mean() - ((votes.sum() - votes[i])/(len(votes)-1))), i) for i in removed]
    diffs.sort(reverse=True)
    print(diffs[:top_k])

This produces a ranked list of the trees that mattered with respect to the divergence of the two models.

Visualizing the disagreement surface

Interpretation	Image
Where vote probability differs the most
Where the predicted labels actually differ
Zoom on violation region with arbitrary difference

These visuals are easy to track, and with some work, creating something for the most problematic combinations of features, would be very revealing for the pruning process.

Most of the input space is essentially the same, but we see precise “fault lines” where pruning changes the target predictions.

Why this matters

Validation gives confidence, Z3 gives certainty.

Closing Remarks

• We can formally prove two models behave identically.
• Use it to validate pruning / distillation work.
• Use it to guard model retraining in CI/CD.
• If models diverge, Z3 gives the input that caused the divergence.

No brute force,no test dataset guesses.

Just mathematically guaranteed model equivalence (or a counterexample).

Further reading — neural network equivalence via SMT

The idea of proving that two models are equivalent (or extracting counterexamples when they aren’t) originates from formal verification research, in particular:

Eleftheriadis et al., On Neural Network Equivalence Checking Using SMT Solvers. FORMATS 2022. https://www.ccs.neu.edu/~stavros/papers/2022-formats-NN_Equivalence.pdf
Their work focuses on neural networks and supports strict + approximate equivalence relations.

This post adapts a fairly similar encoding idea to decision-tree ensembles (random forests), making equivalence checking usable in practical ML pipelines. Z3 effectively constructs the entire random forest as a single logical expression.

Why measure information loss when adding noise?

A post on Cook’s blog showed how rounding numeric values can act as a simple form of privacy.

That idea caught my attention: rounding is just a deterministic way of adding noise.
So how much information do we actually lose when we do this?

This note looks answers that question.
By adding Laplace noise (a common way to blur numeric data: small shifts most of the time, big ones only occasionally) of different magnitudes to a set of “ages” and measuring the mutual information with the original data, we can see how information degrades as noise grows and how that compares to ordinary binning.
Each noise scale b has an equivalent bin width: the point where both destroy the same amount of information.

Setup

We’ll start with a simple “age” variable drawn from a synthetic distribution, from 0-100. More realistic distributions seemed to reach roughly the same conclusions.
To each value, we add Laplace noise with different scales b, and measure how much mutual information remains between the noisy and original data.

For comparison, we also apply deterministic binning: rounding ages into 1-, 5-, and 10-year intervals.
This acts as an upper bound on what the same magnitude of noise would erase.

The figure below maps the two: every noise scale b has an equivalent bin width where the information loss matches.

Results

Information drops smoothly as the noise scale increases.
Small b values barely affect it, but once the noise exceeds a few years, most detail is gone.

The horizontal lines show fixed widths for comparison.
Each crosses the Laplace curve at the point where both destroy the same amount of information which is a practical way to read noise as “effective resolution”.

Noise defines an implicit resolution, which is how precisely a value can still be inferred, and binning defines it explicitly.
Both erase the same amount of information, but they’re effectively not the same operation.

When you bin, you restrict knowledge to a clear interval: “this person is between 25 and 30”. When you add noise, you blur every point independently — sometimes within that window, sometimes beyond it.

Both limit what can be learned, but only noise introduces uncertainty.

Binning is limited by the units we already use: we can round ages to years or to 5-year groups, but can not be smaller than the base unit.
Noise isn’t bound by that because it can be arbitrarily small or large, adjusting precision continuously rather than in discrete steps.

Final remarks

• Noise and binning set resolution differently.
  One continuous, one discrete — both shape how much detail survives.

• Noise is tunable.
  Its scale b acts as a continuous knob on effective precision, unlike fixed bins.

• Information loss is measurable.
  Mutual information quantifies how much structure the data retain after perturbation.

• At large noise scales, precision saturates.
  Beyond the data’s natural granularity, extra noise only adds randomness.

Check the code

Why use Bayesian regression for model based simulations?

In this post, we’ll build a simple probabilistic model and use it to simulate a few scenarios.

Linear models handle noisy observations well — they stay focused on the main signal instead of chasing small fluctuations. Bayesian regression adds key advantages: we can encode domain knowledge as priors, quantify uncertainty directly from the posterior, and express results as probabilities rather than p-values or arbitrary confidence intervals.

For exploring counterfactual or simulated scenarios, that mix of simplicity and principled uncertainty is exactly what we need.

Case Study

A good example for this kind of modeling is converting sales opportunities.

Sales reps typically log opportunities under their accounts, along with details such as the offered unit price and whether the deal was ultimately won or lost.
Some of these attributes are naturally informative, and, as with most purchases, price is often the dominant factor behind the conversion.

Still, the data doesn’t capture everything. Competitor pricing, credit limits, or internal approval rules can all affect the outcome, and their absence adds noise to the target variable.

Because we understand this process and the role of price so well, it makes an ideal test case for a simple Bayesian model.

Data

We’ll start with a small simulated dataset of 500 sales opportunities.
Each record has a conversion status (won = 1, lost = 0) that depends mainly on the unit price offered, along with two categorical attributes — the account country and the product ID.
Each product and country has its own coefficient, representing factors such as sales-rep behavior, discounts, or product-specific promotions. (A full hierarchical version could share information across groups through hyperpriors, but here each level is modeled independently for simplicity.)

Additional random variation is included to capture unobserved factors that influence conversion — for instance, market competitiveness or credit conditions.

A sample of the dataset is shown below.
For simplicity, the model will ignore the amount column (informative but unnecessary here) and focus on unit_price, country, and product ID.

Check the data generating code for the specifics.

The plots below show how the conversion target varies across the five simulated products and three countries.
The division follows the ratio of offered price to base price, though it’s not a strict boundary.
Higher simulation noise makes that division fuzzier and the classification problem harder overall.

Model

Let’s set up a simple model to study opportunity conversion.
This isn’t meant to perfectly fit the data — just a basic object for controlled simulations.

We’ll use PyMC3(now PyMC) to implement a Bayesian version of logistic regression.
If any part of the definition feels unclear, check their examples and docs.

The unit price values are normalized by each product’s base price.
This scaling keeps features near zero, which simplifies the choice of priors.

The model includes linear terms for product and country, along with a common intercept, and uses a logit link to map the linear predictor to probabilities.
These probabilities then define a Bernoulli likelihood for the conversion outcome.

Because we know price has the strongest influence, we’ll assign a prior that allows its coefficient to take relatively large (in magnitude) values.

Formally:

yᵢ = β₀ + β_prod[prodᵢ] + β_ctry[countryᵢ] + α_prod[prodᵢ]·priceᵢ + α_ctry[countryᵢ]·priceᵢ
pᵢ = sigmoid(yᵢ)
statusᵢ ∼ Bernoulli(pᵢ)

  N=len(train_status)
  dim1 = len(set(product_id))
  dim2 = len(set(country))  

  with pm.Model() as shared_data_model: 

      intercept = pm.Normal('intercept', mu=0, sd=1)  

      alpha_product = pm.Normal('alpha_product', mu=0, sd=1, shape=dim1)
      alpha_country = pm.Normal('alpha_country', mu=0, sd=1, shape=dim2) 

      sigma_beta = 10
      beta_product = pm.Normal('beta_product', mu=0, sd=sigma_beta, shape=dim1)
      beta_country = pm.Normal('beta_country', mu=0, sd=sigma_beta, shape=dim2)  

      train_cty = pm.Data("train_cty", train_country)
      train_p = pm.Data("train_p", train_product_id)
      train_offers = pm.Data("train_offers", train_normalized_offers)
      train_p_cty = pm.Data("train_p_cty", train_p_cty)

      p = invlogit(intercept 
                   + alpha_product[train_p] 
                   + alpha_country[train_cty]    
                   + beta_product[train_p] * train_offers 
                   + beta_country[train_cty] * train_offers   
                  ) 

      y = pm.Bernoulli('y', p=p, observed=train_status) 

      trace = pm.sample(init='advi+adapt_diag', n_init=100000,
                            tune=1000, draws=1500, chains=3, cores=8,
                            target_accept=0.90, max_treedepth=10)
  az.plot_trace(trace, compact=True); plt.show()

This model is simple enough for this dataset, so there are no divergences or other diagnostics raising concerns.

Let’s proceed.
For simple models like this, even weaker priors would likely be informative enough. Adding extra terms, for example, industry or time components, would increase complexity and make sampling harder.
When the data are noisy and the signal is faint, encoding knowledge through priors can help the sampler converge and stabilize inference.

Sampling from the posterior, we see that price cleanly separates converted from lost opportunities.
The highest posterior density regions show how conversion probability shifts with normalized price.
Alternative transformations, such as z-scoring price by product, produced slightly cleaner regions, but since model performance was identical, keeping the price as-is was more convenient for the simulations — the main focus of this post.

The key takeaway from this model is that when predicted probabilities align well with observed outcomes, the model’s predictive performance is strong.

Below we can see predictions on a hold-out set and the uncertainty of each, expressed as the standard deviation of their posterior predictive distributions.
This helps gauge how much trust to place in each individual prediction.

For a Bernoulli variable, the posterior standard deviation is bounded by 0.5 — uncertainty is highest near p = 0.5 and decreases as probabilities approach 0 or 1.

In addition to the posterior prediction spread, we can also assess model uncertainty by inspecting the posterior distributions of its parameters.
Parameters with wider posterior curves imply higher uncertainty, which naturally propagates into predictions.
Listing their standard deviations is an intuitive way to see why some predictions carry more uncertainty than others.

Simulations — Discounts and Mark-ups

Linear models extrapolate reasonably well, though they assume a linear relation.
That’s not always realistic — non-linear behavior appears when prices approach zero or climb far above the base level.

To explore what discount might turn a declined offer into a win, we can sample from the posterior at different price values.
The plot below shows how conversions evolve with discounts; color encodes uncertainty (posterior standard deviation) across previously lost opportunities.
Lower prices increase conversion probability, but products or countries with weaker signal-to-noise remain more uncertain.

From the simulations, a discount of roughly 20% is enough to recover nearly all lost opportunities — beyond that, additional cuts bring little gain and simply erode margin.

The same approach applies to price increases: we can examine how higher unit prices trade off revenue versus conversion loss.
The figure below shows the decline in wins as prices rise — useful for identifying thresholds just below where business begins to drop.

Both simulations behave as expected: lower prices drive conversions, higher ones reduce them, confirming the model’s internal consistency and our intuition about the problem.

This post hopefully helped illustrate how we can use models to assist in simulating scenarios.
More complex models will bring very interesting simulations, and optimizing these parameter landscapes will become a less trivial exercise.

Check the code and adjust noise parameters to explore different scenarios

A few years ago, Claudia Perlich wrote on Quora that “linear models are surprisingly resilient to noisy data.”
That line stuck with me because it contradicts the common instinct to reach for deeper or more powerful models when the data gets messy.

I wanted to revisit that claim, reproduce it in a small controlled setup, and then extend it a bit:
What happens when we add feature noise instead of switching labels?
And how does calibration (how well predicted probabilities align with reality) break down under both types of noise?

TL;DR

• Linear models degrade gracefully when noise increases; their bias acts as regularization.
• Tree ensembles hold AUC longer under moderate feature noise, but their calibration collapses faster.
• Once labels are corrupted, no model survives: information is lost, not just hidden.
• Calibration helps, but only while the underlying signal still exists.

Approach

The idea was to simulate a clean, linearly separable world and then contaminate it in a controlled way.

• Data: 10 features, 5 informative, synthetic binary target generated with the make_classification method from sklean.
• Noise:
  • Label noise: randomly flipping 0↔1 with probability p.
  • Feature noise: adding Gaussian or Laplace perturbations, scaled to each feature’s standard deviation.
• Models:
  Logistic regression, Random Forest, and XGBoost, with and without isotonic calibration.
• Metrics:
  AUC for discrimination; Expected Calibration Error (ECE) for reliability.

Each configuration was run over multiple seeds and averaged, using up to 3 000 samples per run.

Results

At first glance, intuition is confirmed:

• Under label noise, all models decay in lock-step. Logistic doesn’t collapse faster than the trees; they all converge toward randomness once the labels stop meaning anything.
• Under feature noise, the picture splits:
  • Logistic remains smooth and predictable. Its linear boundary blurs but doesn’t overreact (much).
  • RF and XGB start to memorize noise, retaining slightly higher AUC for a while but paying for it in calibration error.
  • Calibration (the dashed lines) restores some sanity, but only when the signal is still recoverable.

The curves are remarkably smooth, with no weird bumps, no instability.
Simple models with strong inductive bias prefer signal over noise.

Why is the linear model so stable here?
Because the underlying data was generated by a linear process. The logistic model has the right inductive bias — it assumes the true decision boundary is linear; so even as we inject random perturbations, it degrades gracefully.

Tree-based models, are flexible enough to “explain” small fluctuations as structure. That flexibility becomes a liability under noise: they overfit spurious splits, yielding high confidence on wrong examples, which shows up as poor calibration.

In the real world, this pattern often repeats: if your features already capture the main signal, linear baselines are hard to beat on stability. Complexity rarely saves you from bad data.

Conclusion

This small experiment validates Perlich’s observation and extends it slightly:
noise doesn’t just make you wrong, it makes you confident in the wrong things.

Linear models trade expressive power for robustness.
Tree ensembles fight noise longer, but they start lying about their certainty.

Check the code and adjust noise distributions, switch datasets, try out different models. Have fun!

To illustrate the process and expose the practical pitfalls of Gopy, let’s start with an implementation of a vantage-point tree from the gonum project.

First step: have Go code that you want to use in your python pipeline. This bit is an example from gonum that should be simple to follow: essentially from a collection of places and given a specific address, determine which are whithin a certain distance and also to display the top 5 closest distances.

package vptree

import (
  "fmt"
  "log"
  "math"

  "gonum.org/v1/gonum/spatial/vptree"
)

func Example_accessiblePublicTransport() {
  // Construct a vp tree of train station locations
  // to identify accessible public transport for the
  // elderly.
  t, err := vptree.New(stations, 5, nil)
  if err != nil {
    log.Fatal(err)
  }

  // Residence.
  q := place{lat: 51.501476, lon: -0.140634}

  var keep vptree.Keeper

  // Find all stations within 0.75 of the residence.
  keep = vptree.NewDistKeeper(0.75)
  t.NearestSet(keep, q)

  fmt.Println(`Stations within 750 m of 51.501476N 0.140634W.`)
  for _, c := range keep.(*vptree.DistKeeper).Heap {
    p := c.Comparable.(place)
    fmt.Printf("%s: %0.3f km\n", p.name, p.Distance(q))
  }
  fmt.Println()

  // Find the five closest stations to the residence.
  keep = vptree.NewNKeeper(5)
  t.NearestSet(keep, q)

  fmt.Println(`5 closest stations to 51.501476N 0.140634W.`)
  for _, c := range keep.(*vptree.NKeeper).Heap {
    p := c.Comparable.(place)
    fmt.Printf("%s: %0.3f km\n", p.name, p.Distance(q))
  }
}

// stations is a list of railways stations.
var stations = []vptree.Comparable{
  place{name: "Bond Street", lat: 51.5142, lon: -0.1494},
  place{name: "Charing Cross", lat: 51.508, lon: -0.1247},
  place{name: "Covent Garden", lat: 51.5129, lon: -0.1243},
  place{name: "Embankment", lat: 51.5074, lon: -0.1223},
  place{name: "Green Park", lat: 51.5067, lon: -0.1428},
  place{name: "Hyde Park Corner", lat: 51.5027, lon: -0.1527},
  place{name: "Leicester Square", lat: 51.5113, lon: -0.1281},
  place{name: "Marble Arch", lat: 51.5136, lon: -0.1586},
  place{name: "Oxford Circus", lat: 51.515, lon: -0.1415},
  place{name: "Picadilly Circus", lat: 51.5098, lon: -0.1342},
  place{name: "Pimlico", lat: 51.4893, lon: -0.1334},
  place{name: "Sloane Square", lat: 51.4924, lon: -0.1565},
  place{name: "South Kensington", lat: 51.4941, lon: -0.1738},
  place{name: "St. James's Park", lat: 51.4994, lon: -0.1335},
  place{name: "Temple", lat: 51.5111, lon: -0.1141},
  place{name: "Tottenham Court Road", lat: 51.5165, lon: -0.131},
  place{name: "Vauxhall", lat: 51.4861, lon: -0.1253},
  place{name: "Victoria", lat: 51.4965, lon: -0.1447},
  place{name: "Waterloo", lat: 51.5036, lon: -0.1143},
  place{name: "Westminster", lat: 51.501, lon: -0.1254},
}

// place is a vptree.Comparable implementations.
type place struct {
  name     string
  lat, lon float64
}

// Distance returns the distance between the receiver and c.
func (p place) Distance(c vptree.Comparable) float64 {
  q := c.(place)
  return haversine(p.lat, p.lon, q.lat, q.lon)
}

// haversine returns the distance between two geographic coordinates.
func haversine(lat1, lon1, lat2, lon2 float64) float64 {
  const r = 6371 // km
  sdLat := math.Sin(radians(lat2-lat1) / 2)
  sdLon := math.Sin(radians(lon2-lon1) / 2)
  a := sdLat*sdLat + math.Cos(radians(lat1))*math.Cos(radians(lat2))*sdLon*sdLon
  d := 2 * r * math.Asin(math.Sqrt(a))
  return d // km
}

func radians(d float64) float64 {
  return d * math.Pi / 180
}   

This is a particularly good example of code to take. The definition of the distance can easily be changed to reflect similarity in words, geometric spaces or whatever rule that makes sense to classify as similar.

  gopy build -output=some/folder -vm=python3 path/to/go_pkg

This creates the shared library and other objects needed for the binding.

For the the shared objects(.so), there is one extra step before interacting with the bindings, which is to add a new path to the LD_LIBRARY_PATH variable to tell the dynamic link loader where to search for the dynamic shared libraries. There is a long lasting issue, where all steps are descbribed.
If you’re in the location of the generated folder add the current working directory ($PWD) to the environment variable, else adjust it accordingly.

  LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$PWD python3

After it, you’re free to import vptree and use it with little to no issues.

    Python 3.7.6 (default, Jan  8 2020, 19:59:22) 
    [GCC 7.3.0] :: Anaconda, Inc. on linux
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import vptree
    >>> vptree.Example_accessiblePublicTransport()
    Stations within 750 m of 51.501476N 0.140634W.
    St. James's Park: 0.545 km
    Green Park: 0.600 km
    Victoria: 0.621 km

    5 closest stations to 51.501476N 0.140634W.
    St. James's Park: 0.545 km
    Green Park: 0.600 km
    Victoria: 0.621 km
    Hyde Park Corner: 0.846 km
    Picadilly Circus: 1.027 km

Simple enough. This was a very simple example, but it seems to generalize well to more complex Go code.

Some preliminary thoughts and conclusions from the last post:

1. Complexity of the models determine how profitable it is to explore in lower samples. For quadratic algorithms and ignoring the actual implementation, training one model with a full dataset should cost the same amount of time as training 6.25 with 40% of that dataset.
2. Before a threshold sampling percentage, the results of a model are not informative for the full dataset. Being greedy doesn’t help here.
3. Overall, models at smaller samples seem to be noisier images of the full dataset trained models.

Let’s take this example from bayes_opt.
We want to optimize over a 3d space composed of random forest parameters (max_features, min_sample_split, trees) where the model is evaluated a cross-validated negative log-loss score.
A standard bayesian optimizer runs 100 models with a synthetically generated dataset with a binary target.

Let’s do a run in an exploratory mode: focusing one exploring the landscape instead of necessarily exploiting regions near local or global maxima.

The logs below print how many models were ran during the bayesian optimization and the computational budget it consumed.
We print the best combination of parameters until the last iteration. During this process, the best model was found in iteration 13 and the remaining 87 never resulted in a superior model.

Optimizing Random Forest: 100 models; budget: 100 
|   iter    |  target   | max_fe... | min_sa... | n_esti... |
=============================================================
|  4        | -0.3418   |  0.4902   |  0.02077  |  10.02    |
|  8        | -0.3349   |  0.6166   |  0.01651  |  10.02    |
|  13       | -0.2919   |  0.999    |  0.01     |  250.0    |
=============================================================

The figure below shows a fairly explored space, where some regions clearly seem to have more performant models (lighter tone in the color scale).

Another run promoting a more balanced relation between exploration and exploitation:

Optimizing Random Forest: 100 models; budget: 100
|   iter    |  target   | max_fe... | min_sa... | n_esti... |
=============================================================
|  4        | -0.3423   |  0.4902   |  0.02077  |  10.02    |
|  5        | -0.3404   |  0.999    |  0.01     |  10.0     |
|  6        | -0.3149   |  0.9771   |  0.01587  |  188.5    |
|  12       | -0.2959   |  0.999    |  0.01     |  96.31    |
|  16       | -0.2946   |  0.999    |  0.01     |  141.4    |
|  51       | -0.2943   |  0.999    |  0.01     |  122.7    |
|  95       | -0.2942   |  0.999    |  0.01     |  126.5    |
=============================================================

Both strategies seem to be effective exploring the parameters. Let’s add subsampling to it.

To benchmark all the variants in the exploration we fix the same compute budget, that is, the same that is needed to run 100 models with the full dataset.
Let’s consider negligeble the compute time for the gaussian process that fits the hyperparameter space, even though it isn’t: 1) as observations grow; 2) as the hyperparameter space grows; 3) and if the model function is not too expensive to compute.

Some remarks:

1. The computational budget is divided between different sample sizes, and for each size, we can quantify a quantity of models given by the complexity relation. Random forests are assumed to be log-linear; SVMs are considered to be quadratic. Abstracting some implmentation details is acceptable to get started.
2. The lowest percentage which samples the dataset is fixed and it should be tuned depending on the complexity of the data. 1% of the data may be enough to learn the target.
3. How many sample sizes should we explore? Not enough has been explored here, but it seems to be again contingent on the data.
4. What strategy to use when dividing the budget: evenly over the various sample sizes or something more complex?

The key concern once the exploration in a sample is completed, is how to pass the information gathered and pass it to the next (larger) sample exploration.
A simple step is to pass promissing points to probe; adjusting the domain of the parameters in order not to explore flat areas is promissing but not easy to implement in bayes_opt; another simple way to pass information is to copy the posterior (after fitting to the observations) covariance function of the underlying gaussian process and use it as a prior to the optimization process of the following sample.

Some ideas to make subsampled bayesian optimization more clever where: 1) to make the strategy of exploration sample size dependent. Exploring at lower samples and exploiting at higher samples seems a good heuristic; 2) to add a noise term to the gaussian process that is dependent on the sample size. This is to model the higher variance at lower samples.

The logs and figures below show the result of a 3 sample size split, with sample percentages in the following set: [30%, 70%, 100%], where the computational budget (the equivalent of training 100 models with the full dataset) was split evenly.
Notice that the same budget allows for a very different amount of models at each sample size. Allocating more budget to lower or higher percentages could ease the exploration of more complex parameter spaces.

Sample percentage:30%

Optimizing Random Forest: 148 models; budget: 33 
|   iter    |  target   | max_fe... | min_sa... | n_esti... |
=============================================================
|  4        | -0.3858   |  0.4902   |  0.02077  |  10.02    |
|  9        | -0.3753   |  0.9791   |  0.01979  |  23.29    |
|  19       | -0.3539   |  0.6998   |  0.02344  |  241.5    |
|  43       | -0.3354   |  0.999    |  0.01     |  221.5    |
|  112      | -0.3344   |  0.999    |  0.01     |  46.79    |
|  137      | -0.3343   |  0.999    |  0.01     |  40.97    |
=============================================================

Sample percentage:70%

Optimizing Random Forest: 57 models; budget: 33
|   iter    |  target   | max_fe... | min_sa... | n_esti... |
=============================================================
|  3        | -0.3023   |  0.999    |  0.01     |  114.4    |
|  5        | -0.302    |  0.999    |  0.01     |  234.8    |
|  42       | -0.3006   |  0.7511   |  0.0109   |  67.98    |
=============================================================

Sample percentage:100%

Optimizing Random Forest: 33 models; budget: 33 
|   iter    |  target   | max_fe... | min_sa... | n_esti... |
=============================================================
|  32       | -0.2917   |  0.7301   |  0.01     |  250.0    |
=============================================================

To conclude, this was not an exhaustive search of how subsampling can be used to search optimal points in the hyperparameter space.
The code below can be changed to:

1. Explore different budget dividing strategies;
2. Explore different amounts of noise in each of the sample percentages;
3. Investigate how to leverage the exploit vs explore trade-off;
4. Explore some relation between the observed points and the number of points to pass across samples;
5. Explore how different these points should be;
   …

The exploration of higher dimensional spaces at lower samples seems at least more effective than relying on very expensive model full dataset training. For models with higher complexity this should be evident.

Code

This is essentially a back of the envelope study for the identification of outliers in time series. The idea is to sketch a method that associates model quality to the presence of outliers.
When dealing with data which does not follow time, finding outliers is hard, but even simple approaches might yield decent results. Setting a threshold for the percentile that determines what is a classifier works nicely in one dimensional data, and might even be useful in low dimensional data. To make sure, add a Bonferroni Outlier test and whatever you decide, it has some support.
For time series that evidently not a satisfying answer. Even very rare values can be periodical; this is in fact a common pattern.

One definition of outlier is a measurement that does not fit with the data generating process.
For a sufficient amount of samples, the signal makes itself clear, even when in presence of significant noise. Let’s focus on the problem with a small amount of data points — something like monthly series, a frequent business scenario.

Let’s create a small series which has the following decomposition:

We need to create a generating model which will be critical to evaluate the likelihood of a point being an outlier. Let’s not use the entirity of our knowledge of the series. For a series this simple, we’ll use a gaussian processes regression, and we’ll define the covariance function as the sum of the Matérn 5/2 kernel and a periodic kernel of period 12. We also define a linear mean funcion, with the slope defined with by a random variable that is inferred using MCMC.
This is as a plausible injection of basic yet informative priors to the model. More complex series may require complex models, which are harder to sample and harder to illustrate the idea of the post.

A quick test, performing a 12 month forecast, shows the model capturing the signal pretty well when there is no noise, and as the noise magnitude increases,the predictive ability decreases, as expected.

One particular feature that I enjoy in gaussian processes is the ability to interpolate data very nicely and allowing imputation of missing values in a principled way. Because we can draw samples from the model, we can generate distributions for each of the points in the series. We exclude each point and, assuming the model is sufficiently well defined, we could collect percentile values for the values of the series.
Now, the outcome of this step seems redundant or a symptom of a weak model/a hard problem to model. However, the main objective is to show that the absence of outliers will generate a superior model, that reduces the modelling error significantly for the rest of the series.

This extra iteration over the entire series adds a significant amount of computing time to an already complex method — but this is a small series and a few seconds per model is nothing obscene Below we get to see the process for the series with minimal amount of noise.

Even with a model that’s as close to naive as possible, the signal is picked and for the most part the mean of the samples matches the missing value.

The key idea behind this post is the superior model that is trained when outliers are removed from the time series. This is something made clear in the next animation. When the outlier is removed, the variance of the model is very low compared to the models generated when it’s not, and most importantly, the mean of the samples of the posterior resembles the original time series.

In case of more than one outlier, the problem stops being trivial; the absence of one of the outlier is no longer sufficient for the model to obtain a clear signal — depending on the magnitude of the outlier, the perturbation it adds makes any modelling quite hard, forcing the removal of another data point.

We can see the model seems to improve when any of the outliers are removed; measuring this improvement might be sufficient to list outlier candidates and then removing the combinations of elements of this list to find out the most promising set.
It’s still hard to generate some heuristic that generalizes for most outliers — any value with an offset large enough from the original time series is easily detectable, but after removed, requires one additional loop over the remaining points; even with a small amount of data, it gets prohibitive.

The main conclusions taken from this post is that outliers are hard to evaluate, unless if specific cases.
Here, knowledge of the system was needed to build a good enough model; of the amount of noise that influences the process ; and the of how many outliers are in series were essential. In a real world scenario, this is not the case, but it may still serve as an exploratory step.

Predictive models are able to generalize what they learn from data, otherwise they’re not very useful – but there is a minimum amount of data which serves as a boundary for the lack of learning ability of the model.

Let’s pick then the Random Forest implementation by sklearn, and pick three parameters as a start. The data is the Boston housing dataset, around 500 records which is a nice amount to perform some grid searches for these initial tests. I’ll track the R² score obtained from cross validation.

What we expect to find is evidence that smaller samples are still informative to find the optimal parameters of the full dataset.
For each size and combination of parameters, we randomly sample from the original dataset, while keeping the random seed of the algorithm to maintain some amount of determinism.
The plots below show the results for grid search for the parameters. Keep the colorscale in mind; it’s used for the rest of the plots.

Some comments:

1. Less data means less to learn: the performance must necessarily be smaller with small samples.
2. Less data means greater variation in how the training data can be generated – some combinations can be very unrepresentative of the dataset. This needs to be explored further.
3. For some parameters – max features – subsampling seems to have a greater impact on the predictive performance of the model.

As expected, the parameter curves seems to share a lot of features across sample sizes, even at the smallest ones, and as the size increases the convergence seems to be even more apparent. To put it simply, the optimal parameters are quite similar after a certain size.

Let’s explore the variation of the scores a bit further.
Let’s focus on one parameter, min_sample_split and repeat the exploration a few times.

And we can see the mean value and the region of the two standard deviations.

Smaller samples have a greater variance than larger samples – there are certainly odd combinations, in particular for a small dataset as this one.
Larger samples will have a significant overlap with the orignal dataset which means that there’s less to vary in the training data, and as expected, the results are much less scattered.
In my implementation, even when the sample sizes matches the entire dataset, the order is not fixed by a random seed and is not kept; this adds additional variation, which is not only acceptable, but actually desirable to explore the what causes different outcomes in these methods, and to get an idea of the magnitude of the variation.

Before trying to observe the same patterns hold in other scenarios, let’s take a minute and explore, fixing one parameter, the distributions for the scores at each sample size. The reason is to start observing the actual dynamics of these distributions as more data gets fed to the model.

Below we can see that if we exclude very low sample sizes – there is a very significant overlap in the R² scores distributions. In other words, there is plenty of useful information at lower samples.

On interesting aspect of these repeated draws at a specific sample size seems to be the bell shape of the distribution. Even with ceiling effects, normality tests are passed and we can at least use this as information to model the behavior in the second post.

One continuation to this exploration needs to be related with the relation between sample size and more than one parameter.
We have two examples of combinations of parameters that support what we saw previously: the surfaces share plenty of similarities for various sample sizes, and most importantly, the parameters that maximize the R² score seem to be at the very least neighbours and at the very best the exact same.

One comment regarding how different datasets affect what we have observed. There seems to be a point in the sample size after which the dataset is informative enough for the model to pick up patterns. Rich datasets are a bit more demanding when it comes to how small samples can be – this adds another layer of variance to what we see and that is ignored for now.

One final exploration with a SVM is carried out. There is no scaling of the features or anything remotely trying to maximize the performance; again, the point is to see that there is no dramatic changes to the model behaviour.

We can see that for the very smallest sample size, the maximum seems to be contested by two distinct locations in the parameter space. Small samples are troublesome as we’ve seen and different algorithms scale differently.
The second smallest sample already seems to converge to the optimal set of parameters and at this point we can make some comments that motivate the next post.
Our first method was the random forest, which scales nicely, log-linearly. SVMs on another hand have quadratic time complexity, and this is one aspect to explore and that motivates subsampling in this exploration.
To compute one model with the entire dataset it costs us the same as the cost of training seven in the second smallest sample size: there is intrinsic noise that comes from small samples, yet we can explore a lot more.
A structured, yet simple, approach to how this exploration can be made, leveraging the trade-off between information gained and reduced computational load, is the content of the next post.