Machine Learning - The Continuous Improvement Cycle

The million-dollar model that never shipped

In October 2006, Netflix made the data science world an offer it couldn't refuse. Beat Cinematch, the company's in-house recommendation algorithm, by 10% and walk away with $1 million. Netflix released a dataset of roughly 100 million movie ratings and thousands of teams from around the world got to work.

It took almost three years. In September 2009 a team called BellKor's Pragmatic Chaos crossed the 10% line and claimed the prize. Their winning solution was a blend of hundreds of models stacked on top of each other, a remarkable piece of engineering and a genuine leap forward for the field.

Here is the part of the story that rarely gets told. Netflix never put the winning blend into production. A few years later its engineering team explained that they had taken two of the simpler algorithms developed along the way and built them into their systems, but the extra accuracy of the final ensemble did not justify the engineering effort it would have taken to run it at scale. By then the business had also changed underneath the competition: people were streaming instead of mailing back DVDs, and what Netflix needed to predict had changed with them.

This is the most important lesson in machine learning, and it's why this layer of the AI Strategy Framework is called the continuous improvement cycle. A model is not a trophy. A simple model running in production, learning from real users and improving every month, will beat a perfect model sitting in a notebook every time.

Start without machine learning

Last time we finished with a Gold dataset, trusted dashboards and a set of engineered features ready for training. It's tempting to reach straight for the most sophisticated algorithm you can find. Resist it.

Martin Zinkevich's "Rules of Machine Learning", a guide written from Google's experience, opens with a rule that surprises most people: don't be afraid to launch a product without machine learning. A heuristic built from your analytics, like "recommend the most-watched shows in the genre this person watches most", is cheap, transparent and often surprisingly good. It also does something critical: it gives you a baseline. Every model you build afterwards has to beat it, in production, on a metric the business cares about.

The fourth rule is just as useful: keep the first model simple and get the infrastructure right. The first model's job is not to be brilliant. Its job is to prove that data can flow from your systems into training, that predictions can flow back into the product, and that you can measure what happens next.

The anatomy of the cycle

Every successful machine learning program I've seen runs the same loop:

1. Frame the decision. Write down the business decision the model will improve and the metric that defines success. Back in 2019 I wrote about measuring success in machine learning models; the lesson still holds. Accuracy is not a business metric. Minutes watched, orders placed and dollars saved are.
2. Train and validate. Split your data into training, validation and test sets so you are honest about how the model performs on data it has never seen. Compare everything to your baseline.
3. Deploy carefully. Release the model to a small slice of traffic first, sometimes called a canary, and compare it to the current approach with an A/B test.
4. Monitor. Watch both the model metrics and the business metric. Watch the inputs too. When the data feeding a model starts to look different from the data it was trained on, performance will drift.
5. Learn and retrain. Feed new outcomes back into the Gold dataset, retrain on a schedule or when drift is detected, and start the loop again.

Notice that only one of those five steps is "training". The rest is everything around it.

The model is the smallest part

In 2015 a group of Google engineers published a paper called "Hidden Technical Debt in Machine Learning Systems". Its most famous diagram shows a real-world machine learning system as a collection of boxes: data collection, data verification, feature extraction, configuration, serving infrastructure, monitoring. In the middle sits a tiny box labelled "ML Code".

That diagram is the whole AI Strategy Framework in one picture. The reason this series spent five parts on vision, collection, storage, transformation and analysis before getting to models is that those layers are where machine learning programs succeed or fail. When the foundation is solid, building and retraining models becomes routine. When it isn't, every model is a heroic one-off that decays the moment its creator moves on.

Feedback loops are the product

Netflix is again the best example of what happens when you take the cycle seriously. Its teams have written about how even the artwork shown for a title is personalized and tested, with different images shown to different members and the winners chosen by what people actually go on to watch. Every interaction creates new data, the new data improves the next model, and the improved model creates a better experience that generates more interaction.

This is the flywheel every company wants, and it only exists when four things are true:

• Outcomes are captured. You record not just what the model predicted, but what happened next.
• Data flows back automatically. New outcomes land in the Gold dataset without someone exporting a spreadsheet.
• Retraining is cheap. A model can be retrained and redeployed in hours, not months.
• Someone owns it. A named person is accountable for the model's performance after launch, not just for building it.

The discipline that makes this possible is usually called MLOps: version control for data and models, automated testing, repeatable deployments and monitoring. It isn't glamorous, but it is what turns a data science project into a capability.

When models go wrong

Models fail quietly. A broken website throws an error; a degrading model keeps producing confident answers that are slightly worse each week. Plan for the common failure modes before you launch:

• Data drift: the inputs change. A new product line, a new customer segment or a change in an upstream system shifts the data away from what the model learned.
• Concept drift: the relationship itself changes. The pandemic broke countless demand forecasting models overnight because the past stopped describing the future.
• Feedback distortion: the model influences the data it later learns from. A recommender that only shows popular titles will only ever learn that popular titles are popular.
• Silent breakage: a feature stops being populated, gets filled with defaults, and the model carries on regardless.

Each of these is caught the same way: monitoring inputs and outcomes, alerting on unusual changes, and keeping the baseline running so you always have something to compare against.

Keep it simple

1. Launch the baseline first. A transparent heuristic in production teaches you more than a sophisticated model in development, and it gives every future model a bar to clear.
2. Tie every model to one business metric. If you can't say which number moves when the model improves, you aren't ready to build it.
3. Budget for the loop, not the launch. Plan the monitoring, retraining and ownership before you start training. The first version is the beginning of the work, not the end.

Wrapping Up

Netflix paid a million dollars for an algorithm and chose not to run it, and it was the right call. The company understood something that took the rest of the industry years to learn: the value of machine learning lives in the cycle, not in any single model. Get the loop running, keep it simple, and let each turn of the wheel make the next one easier.

In the final part of the series we climb to the top of the pyramid: AI and deep learning. A great deal has changed since I first sketched this framework, and the pinnacle looks very different today than it did then. The foundation beneath it, as you'll see, matters more than ever.