01 · What this was
A production system, not a notebook
Most portfolio ML projects stop at a Jupyter notebook and an accuracy number. This one didn't: it ran as six scheduled Cloud Run jobs against two public APIs, wrote to a partitioned BigQuery warehouse, retrained a LightGBM model daily on a rolling two-year window, logged every run to a self-hosted MLflow server, and served predictions through a public Streamlit dashboard — all provisioned by Terraform, all deployed by Cloud Build on every push to main.
The forecasting target: consumption 24 hours ahead, at 15-minute resolution, for each of France's 12 metropolitan regions — sourced from RTE's public éCO2mix feed and enriched with Open-Meteo weather.
It was archived on 2026-08-15 — not because it broke, but because it had finished doing its job. The write-up below is the honest record: what was built, what it actually achieved in production, and what I'd change if it kept running.
02 · Architecture
Six jobs, one image, three BigQuery datasets
A single Docker image served every batch job; JOB_MODULE picked the entry point. Cloud Scheduler staggered them across the early-morning Paris window so each stage's inputs were ready before the next one ran.
03 · Data & features
Twelve features, computed once in SQL
Everything is computed in a single BigQuery round-trip — lags, rolling windows, and calendar flags — then cast to a fixed-category pd.Categorical for region so train and inference always agree on encoding.
holidays library04 · Model benchmark
LightGBM shipped. XGBoost scored better.
Benchmarked with skore's EstimatorReport / ComparisonReport against two naive persistence baselines, on an identical 20,555-row validation split (2026-03-04 → 2026-03-22).
| Model | R² | RMSE (MW) | Fit time | Predict time | Status |
|---|---|---|---|---|---|
| LightGBM | 0.956 | 397.6 | 6.08 s | 0.60 s | Deployed |
| XGBoost | 0.966 | 348.2 | 16.81 s | 0.29 s | Benchmarked only |
| Lag 24h baseline | 0.910 | 570.6 | <1 ms | <1 ms | Reference |
| Lag 168h baseline | 0.921 | 534.1 | <1 ms | <1 ms | Reference |
XGBoost beat the deployed LightGBM model on every accuracy metric — 12% lower RMSE, 1 point higher R² — while also predicting faster. LightGBM was the early choice for simpler categorical handling and was never revisited. If this pipeline kept running, swapping the estimator is the single highest-leverage change available — no new features, no new data, just a different tree library.
A separate experiment isolated the effect of the two features that were built but never shipped:
| LightGBM variant | R² | RMSE (MW) | Feature set |
|---|---|---|---|
| Base | 0.956 | 397.6 | 12 features |
| Extended | 0.960 | 381.3 | + rolling_48h_mean, rolling_168h_std |
05 · Production record
The last numbers before shutdown
Rolling 7-day error, pulled directly from elec_ml.metrics on the day the infrastructure was retired — the true production number, not a validation-set estimate.
| Region | MAE (MW) | p95 (MW) | p99 (MW) |
|---|---|---|---|
| France (total) | 2,784.3 | 8,659.2 | 9,907.1 |
| Normandie | 140.2 | 474.6 | 532.6 |
| Centre-Val de Loire | 158.6 | 414.7 | 556.9 |
| Bretagne | 143.4 | 486.8 | 549.1 |
| Pays de la Loire | 222.9 | 706.2 | 826.6 |
| Occitanie | 255.2 | 764.7 | 993.2 |
| Provence-Alpes-Côte d'Azur | 257.6 | 697.7 | 984.2 |
| Grand Est | 281.6 | 798.9 | 1,106.5 |
| Hauts-de-France | 293.5 | 865.3 | 1,008.1 |
| Nouvelle-Aquitaine | 374.3 | 892.3 | 1,112.3 |
| Île-de-France | 331.1 | 1,196.4 | 1,367.2 |
| Auvergne-Rhône-Alpes | 500.4 | 1,083.5 | 1,457.5 |
| Bourgogne-Franche-Comté | 502.9 | 932.7 | 993.6 |
France total MAE of 2,784 MW against an average national demand of ~42,838 MW over the same week — roughly 6.5% relative error, 24 hours ahead, in mid-August.
06 · Retrospective
What I'd do differently
- → Ship XGBoost.Benchmarked better on every metric, in this same repo, and was never promoted to production. The gap between "validated in a notebook" and "deployed" is the most common failure mode in ML systems, and this project is its own example of it.
- → Recalibrate the MAE alert.The production alert fires above 400 MW rolling France-wide MAE — a threshold set early, before demand-scale drift. Live MAE settled around 2,700–2,800 MW, meaning the alert was almost certainly firing continuously rather than flagging real degradation. A threshold needs revisiting once you have enough production history to know what "normal" looks like.
-
→
Ship the two extended features.
rolling_48h_meanandrolling_168h_stdwere built, tested, and measured a 4% RMSE improvement — then left out of the production feature set because the daily retrain window was never re-validated with them live. - → Drift detection was deliberately skipped.PSI/KS tests were considered and rejected: with a daily retrain and genuine seasonal variance in electricity demand, a naive drift test would flag the correct seasonal signal as data drift. The right version of this needs a seasonally-aware baseline — not a reason to skip it forever, just not a one-week job.
- → No automated retrain-on-degradation.Rejected on purpose: a runaway retrain loop during a seasonal transition (a heatwave, a cold snap) could chase noise instead of signal. Daily retrain on a fixed schedule was the deliberately boring, safer choice.
07 · Stack
Everything it ran on
| Layer | Choice | Why |
|---|---|---|
| Compute | Cloud Run Jobs + Services | Scale to zero — no idle cost on a portfolio budget |
| Storage | BigQuery + GCS | Serverless, generous free tier, partition expiry for automatic retention |
| Orchestration | Cloud Scheduler | Managed cron — no Airflow to operate for six jobs |
| Experiment tracking | MLflow, self-hosted | Portable, no vendor lock-in; SQLite-on-GCS avoided a Cloud SQL bill |
| IaC | Terraform | Every GCP resource reproducible from source |
| CI/CD | Cloud Build | Push to main → build, deploy, smoke-test, all six jobs plus two services |
08 · Status
Decommissioned, not deleted
Every job, every Terraform file, the modeling notebook, and 38 tests are still in the repository. The infrastructure that ran it — Cloud Run jobs, the public dashboard, MLflow, BigQuery — was torn down on 2026-08-15 to stop billing on a project that had already answered its own question. The code is the artifact now.