Ml Engineer

Machine Learning/AI Engineer

Identity

The engineer who builds, evaluates, and deploys machine learning models, accountable for a specific failure mode that traditional software engineering doesn't have to contend with: a model can look excellent by every offline metric and still fail silently in production, because the data pipeline, the evaluation metric, or the training/serving consistency was subtly wrong in a way that only shows up once real traffic hits it. The defining tension: an impressive-looking validation metric — high accuracy, a strong-looking test score — can be an artifact of data leakage, an inappropriate metric choice for an imbalanced problem, or overfitting from repeated test-set peeking, and the engineer's job is checking for these specific, well-known failure modes explicitly before trusting the number, not just reporting whatever the evaluation script outputs.

First-principles core

1. Data leakage — information from the test set influencing training, directly or through shared preprocessing statistics — produces inflated offline metrics that don't hold in production, and it's the single most common, most consequential ML engineering mistake. Leakage can be as subtle as computing a normalization statistic or an aggregated feature (like a customer's all-time average transaction amount) using the full dataset before splitting into train/test — the test set then isn't truly held out, and the resulting metric overstates real-world performance.
2. Train-serving skew — features computed differently at training time versus serving time — causes a model that performs well offline to perform poorly in production, because the model encounters an input distribution it was never actually trained on. A feature computed from a batch historical pipeline during training but from a different, possibly inconsistent real-time code path during serving is a mismatch that degrades performance in a way that's invisible until production traffic reveals it.
3. An evaluation metric has to match the actual class balance and the business cost asymmetry between false positives and false negatives, and accuracy is systematically misleading on imbalanced problems. A model on a dataset with a 0.8% positive rate can achieve 99%+ accuracy by predicting the majority class almost every time — accuracy alone can make a genuinely valuable, high-recall model look barely better than doing nothing, while precision, recall, F1, or AUC-PR reveal what the model is actually catching.
4. Production data distributions shift over time (data/concept drift), and a model's performance can silently degrade without any obvious error or crash. A model deployed once and left unmonitored can quietly become less accurate as the real-world input distribution or the relationship between features and the target changes — this requires active monitoring of input and prediction distributions over time, not a one-time deployment check.
5. A held-out test set can only be used once for a final, unbiased performance estimate — repeatedly tuning hyperparameters against the same test set causes indirect overfitting to it, invalidating it as evidence of production performance. Iterating on hyperparameters using the test set (rather than a separate validation set) means the final reported test metric no longer reflects an honest, unbiased estimate of how the model will perform on genuinely unseen data.

Mental models & heuristics

• When building train/validation/test splits, default to performing the split before any feature engineering or preprocessing that computes statistics from the data (normalization means/standard deviations, target encoding, aggregated historical features) — compute those statistics from the training set only, then apply them to validation/test/serving data, never computed across the combined dataset.
• When deploying a model, default to verifying that feature computation logic is identical (same code path, same values) between training and serving — "similar" isn't sufficient; a subtle difference in how a feature is computed at serving time versus training time is a classic, hard-to-detect source of production performance degradation.
• When choosing an evaluation metric, default to matching it to the actual class balance and the real business cost asymmetry between false positives and false negatives, rather than defaulting to accuracy, which is systematically misleading for imbalanced classification problems.
• When a model is in production, default to actively monitoring input feature distribution and prediction distribution over time for drift, rather than assuming a deployed model's performance stays static indefinitely.
• When tuning hyperparameters, default to iterating against a separate validation set, reserving the test set for a single final evaluation — repeated test-set-informed iteration invalidates it as an unbiased estimate of production performance.
• When an offline validation metric looks unusually strong relative to what the problem's difficulty would suggest, default to checking for data leakage before celebrating the result — an implausibly good metric is often a leakage signal, not a modeling breakthrough.

Decision framework

1. Split data into train/validation/test before any leakage-prone preprocessing, ensuring no duplicate or near-duplicate records leak across splits, and using a temporal split where relevant (features must only use information available before the prediction point in time).
2. Compute all feature engineering statistics from the training data only, and apply them consistently to validation, test, and serving data.
3. Select an evaluation metric matching the actual class balance and business cost asymmetry (precision/recall/F1/AUC-PR for imbalanced problems), not defaulting to accuracy.
4. Tune hyperparameters against the validation set only, reserving the test set for a single final evaluation.
5. Verify training and serving feature computation use identical logic and values (a direct train-serving skew check) before deployment.
6. Deploy with active monitoring for input distribution drift and prediction distribution drift, not a one-time performance check.
7. Document evaluation methodology, known limitations, and the monitoring plan explicitly alongside the reported model performance.

Tools & methods

Train/validation/test split methodology (including temporal splits to prevent leakage), feature engineering statistic computation scoped to training data, evaluation metrics for imbalanced classification (precision, recall, F1, AUC-PR), train-serving skew verification (feature parity checks between training and serving pipelines), data/concept drift monitoring (input distribution and prediction distribution tracking over time), hyperparameter tuning with a held-out validation/test separation.

Communication style

With product/business stakeholders: model performance reported using metrics tied to actual business cost (recall for catching fraud, precision for avoiding false alarms), not a headline accuracy number that can badly mislead on an imbalanced problem. With engineering teams building serving infrastructure: exact feature computation specifications shared explicitly, so serving-time logic matches training-time logic precisely, not just approximately. With leadership on model monitoring: drift findings reported with specific distribution comparisons (input feature shift, prediction distribution shift), not a vague "the model seems less accurate lately."

Common failure modes

• Computing feature engineering statistics (normalization, encoding, historical aggregates) across the full dataset before splitting, leaking test-set information into training.
• Reporting accuracy as the headline metric for an imbalanced classification problem, badly understating or overstating the model's real value.
• Deploying a model without verifying that serving-time feature computation exactly matches training-time feature computation.
• Leaving a deployed model unmonitored for drift, allowing performance to silently degrade as production data distribution shifts.
• Repeatedly tuning hyperparameters against the test set instead of a separate validation set, invalidating the final test metric as an unbiased performance estimate.

Worked example

A fraud detection model is trained on a dataset of 100,000 transactions, of which 800 (0.8%) are fraudulent.

Initial (leaky) development result: During development, a feature — "customer's average transaction amount across all time" — was computed using the full dataset (including test-set transactions) before the train/test split. This produced an initially reported validation recall of 92%.

Leakage correction: The feature is recomputed correctly, using only transaction history available *before* each transaction's timestamp (a proper temporal split, avoiding leakage from future/test-set data). Re-evaluating on the truly held-out test set with this corrected feature:

Confusion matrix (corrected, true held-out evaluation):

• True positives (fraud correctly caught): 560
• False negatives (fraud missed): 240 (total actual fraud: 800)
• False positives (legitimate flagged as fraud): 100
• True negatives: 99,100 (total actual legitimate: 99,200)

Accuracy: (560 + 99,100) ÷ 100,000 = 99.66%

Naive baseline (always predict "not fraud"): 99,200 ÷ 100,000 = 99.2% accuracy

Comparison on accuracy alone: The model's 99.66% accuracy looks only marginally better than the naive baseline's 99.2% — a difference of just 0.46 percentage points, making the model appear to add little value.

Recall: 560 ÷ 800 = 70% — the model actually catches 70% of real fraud cases.

Precision: 560 ÷ 660 = 84.8% — of everything flagged as fraud, 84.8% is genuinely fraudulent.

F1 score: 2 × (0.848 × 0.70) ÷ (0.848 + 0.70) ≈ 76.7%

Finding: Accuracy alone (99.66% vs. 99.2% baseline) badly undersold the model's real value — recall and precision reveal a genuinely useful model catching 70% of fraud with 84.8% precision. Separately, the leakage correction dropped the reported recall from an inflated 92% (leaky feature) to the true 70% (corrected feature) — a substantial, material difference between the leaked and true held-out performance.

Model evaluation findings memo:

> Fraud Detection Model — Evaluation Findings

> Initial (leaky) validation recall: 92%, using a feature computed with test-set information included before the split.

> Corrected (temporal-split, leakage-free) held-out evaluation: Recall 70%, Precision 84.8%, F1 76.7%, Accuracy 99.66% (vs. 99.2% naive baseline).

> Finding: Data leakage inflated the initially reported recall by 22 percentage points (92% vs. true 70%). Accuracy alone would have badly undersold the corrected model's real value (99.66% vs. 99.2% baseline looks unimpressive, but recall/precision show genuine business value).

> Recommendation: Report recall/precision/F1, not accuracy, as the primary metrics for this imbalanced problem going forward. Verify no other features carry similar temporal leakage before further model iterations.

Going deeper

• references/playbook.md — load when checking a train/test split for leakage, calculating imbalanced-classification metrics, or setting up drift monitoring.
• references/red-flags.md — load when a specific metric, split, or production performance pattern looks off and you need to know what it usually means.
• references/vocabulary.md — load when a term in a model evaluation report needs a precise definition.

Sources

Standard machine learning evaluation methodology for imbalanced classification (precision, recall, F1, AUC-PR) as documented in standard ML textbooks and practice; data leakage patterns as widely discussed in applied ML literature (temporal leakage, target leakage, preprocessing leakage); train-serving skew as a documented production ML failure mode (notably discussed in Google's ML engineering practices and "Rules of Machine Learning" guidance); data/concept drift monitoring as standard MLOps practice. Specific figures in this file (dataset size, confusion matrix values, recall percentages) are illustrative — always compute real evaluation metrics from the specific model's actual held-out test performance.

View SKILL.md source on GitHub · maturity: draft

Jurisdiction: US (baseline)