MLOps Reference Pipeline

Most machine-learning portfolio work stops at a model in a notebook, which answers the least interesting question: whether an estimator can be fit at all. The harder and more production-relevant question is whether every stage around it — validation, feature engineering, training, serving, monitoring, release — can be built, tested, documented and operated as one reliable system. This project set out to answer that, under a real constraint: no cloud account was available, so every production service had to be substituted with a local equivalent rather than skipped.

The pipeline runs as a sequence of independently testable stages. Validation checks the raw data against a rule registry and emits an audit report; feature engineering produces processed features plus a persisted preprocessor artifact; training logs parameters, metrics and artifacts to MLflow and writes an evaluation report and manifest. A FastAPI service loads those artifacts and serves predictions, checking a manifest hash on startup so training-serving skew fails loudly instead of silently. Docker packages that service, docker-compose brings it up alongside Prometheus and Grafana, and GitHub Actions runs install, tests, image build and a Trivy scan on every push.

Local equivalents instead of a blocked project

With no cloud account, the choice was to stall or to substitute. Local artifact storage stands in for object storage, Docker for managed compute, Prometheus and Grafana for a hosted metrics service, and GitHub Actions for a managed pipeline. Each substitution keeps the engineering concept intact while being honest that the deployment target is local, and the architecture notes record how each piece maps to its cloud counterpart.

Imbalance-aware metrics over accuracy

Accuracy is close to meaningless on a heavily imbalanced label — a model that predicts the majority class every time scores well while learning nothing. Hardening the evaluation step around metrics that account for imbalance was a correctness decision about the pipeline, and it turned out to be the decision that exposed the dataset problem below.

A manifest-hash check at startup

The serving container verifies that the artifacts it loaded match the ones training produced. Training-serving skew is the kind of failure that otherwise surfaces as quietly wrong predictions rather than an error, so the service refuses to start on a mismatch instead of serving from a mismatched preprocessor.

Staying on the rejected dataset

Once the data was shown to be unlearnable, an alternative public dataset with genuine signal was evaluated. It was rejected: it shared no columns with the current schema, so adopting it meant rebuilding the validation rules, feature configuration and imbalance handling around a different problem shape — discarding a finished pipeline to obtain a working classifier. The project stayed put by decision rather than by default, and reframed the deliverable instead.

Scanning as a gate, not a checkbox

Trivy runs against the built image inside CI, so vulnerabilities are found on the artifact that would actually ship. Most findings were closed by targeted dependency upgrades and a base-image bump; the one blocked by an upstream constraint was documented as backlog rather than forced with an unsafe pin.

Validation as evidence, not assertion

The validation stage runs a reusable rule registry and writes JSON audit reports, including per-column analysis of whether missing values are missing at random or not, so imputation choices are backed by evidence about the data rather than convention.

Feature pipeline with leakage tests

Feature engineering is a scikit-learn pipeline persisted as an artifact so training and serving share the exact same transformation. Its tests specifically cover leakage — the failure where information unavailable at prediction time reaches the model and flatters every offline metric.

MLflow as the model history

Every training run logs its parameters, metrics and artifacts to a project-local MLflow store. Without a formal registry, that run history is what makes past models comparable and retrievable rather than overwritten by the next run.

An observability stack that was actually run

Prometheus scrapes the service's metrics endpoint and Grafana visualises them, with the whole stack brought up through docker-compose and verified end to end rather than left as configuration that has never been executed.

Accuracy said the model was fine on a heavily imbalanced fraud label, while the model had in fact learned nothing — it predicted zero fraud cases at every threshold. That only surfaced once the evaluation step was hardened with imbalance-aware metrics, which put the model's ranking ability at indistinguishable from chance. The obvious next assumption would have been a bug in the feature or training code. Instead the dataset itself was put on trial, with four independent checks: Pearson correlation for linear relationships, mutual information for nonlinear ones, decile and categorical fraud-rate spread for high-risk zones, and a row-order gap analysis for generation artefacts. Every one fell short of its threshold for meaningful signal, and the row-order analysis showed no clustering, periodicity or drift. The conclusion was that the fraud label had been generated independently of every other column in the file — not weakly learnable, but unlearnable by construction.

Every stage was built, tested and verified end to end: CI runs green through install, tests, image build and the vulnerability scan, and the compose stack serves predictions while Prometheus scrapes it and Grafana renders the result. There is deliberately no working classifier. The dataset cannot support one, and the deliverable is a pipeline that establishes that with evidence and says so, rather than a model tuned until its metrics look presentable. Structural validation passing and the data being useful turned out to be two different claims, and the pipeline is what made the difference visible.

Well-formed is not the same as useful

The raw data passed every structural validation check it was given — types, ranges, nulls, schema. None of that says anything about whether the label can be predicted from the features. Those are separate questions needing separate checks, and a pipeline that only asks the first will happily train a model on noise.

The metric you choose decides what you can see

The same model looked acceptable under accuracy and obviously broken under imbalance-aware metrics. The evaluation step isn't reporting overhead bolted on at the end — it's the instrument that determines which failures are visible at all, and choosing it badly hides exactly the failures that matter most.

A pivot isn't free

A dataset with stronger properties isn't automatically the right move. The alternative had real signal but zero schema overlap, so switching meant rebuilding the finished validation, feature and imbalance work around a different problem. Framing the deliverable honestly is what made staying defensible, rather than a compromise quietly glossed over.

Security findings need triage, not just fixes

Scanning the image produced a real backlog rather than a clean bill of health. Most items closed through targeted upgrades and a newer base image, but one was blocked by an upstream dependency's constraints — recording it as known and accepted is a more honest engineering position than forcing an unsafe pin to turn the badge green.