Model reproducibility: the same result in your notebook and in production

What it means for a model to be reproducible

Reproducible means that, given the same inputs, the model returns the same result regardless of where it runs. That is the condition that turns a validated model into a trusted one: what the risk team approved is exactly what decides on the portfolio.

In the engine this rests on three pieces defined once: the environment the model runs in, the data contract it receives and returns, and the initial validation against a set of known cases.

• Same inputs, same result, on any execution.
• What was validated and what decides are the same version.
• The verification is recorded, not left to anyone's memory.

The environment is pinned

The first pillar is that the model does not run on an environment shifting underneath it. Before deploying, you pin the Python version and the exact dependency versions it was trained with, along with how the model was serialized and the inference method to be invoked.

Pinning exact versions rather than ranges is what prevents a library update from moving results without anyone touching the policy. If the model needs a package that is not in the environment, confirm it with the uFlow team before proceeding instead of assuming it will be available.

• Declared Python version.
• Exact dependency versions, not ranges.
• Inference method and serialization format defined.

The data contract, declared once

The second pillar is the contract: what the model receives and what it returns. You declare input variables and their order, data types, the preprocessing to apply and the output variables.

Two details usually left implicit in the notebook deserve attention. Feature order, because a model will not warn you if they arrive shuffled: it simply returns a different number. And null handling, because an empty field that meant zero during training is not the same as a null in production. Declaring it in the flow keeps behavior consistent.

• Input variables, with their order and types.
• Explicit preprocessing, not inherited from the notebook.
• Null handling defined in the flow.
• Output variables the policy will use.

Start-up validation: control set and tolerance

The third pillar is evidence. Before publishing, you run a controlled set of cases whose local results you already know through the engine and compare the outputs. The acceptable tolerance is defined beforehand, because in floating-point arithmetic a minimal difference can be expected and a large one never is.

If any case falls outside tolerance, the review is short and orderly: Python and library versions, serialization, feature order and types, null handling, rounding, preprocessing and, when the model has a random component, the seed. It is a checklist, not an investigation.

• Control set with known local results.
• Tolerance declared before comparing.
• Review guided by a short, always identical list.

The evidence model governance needs

For a regulated institution, reproducibility is not a technical convenience but part of the model file. The model risk team needs to show that the implemented model behaves like the validated one, and that any later change was recorded.

The engine contributes that part: every version of the model and of the policy is stored, you can roll back to a previous one under control, and each decision keeps the rules applied and the data queried. With that, validation stops being a loose document and becomes tied to the version that actually decides.

• Versioning of the model and of the policy invoking it.
• Controlled rollback to a previous version.
• Traceability of every decision, with rules and data applied.

Comparing two versions on real traffic

Once reproduced, the new model can coexist with the current one in a champion/challenger setup: the incumbent keeps deciding while the new one is evaluated in parallel on a contained slice, until results justify the switch.

It is the natural complement to reproducibility. First you verify the model behaves as it did in the lab; then you measure whether it decides better than the one in production.

Checklist before publishing

The eight points worth closing and documenting:

• Python version.
• Exact dependency versions.
• Model serialization format.
• Inference method.
• Input variables, with order and types.
• Preprocessing and null handling.
• Output variables.
• Control set and accepted tolerance.