Evaluate
This section explains how to evaluate predictions of a Deployment.
Prediction Evaluation
Evaluating a prediction means overwriting, or correcting, the model's output. This can be used as a reference, or to be used as training data for a next iteration of the model.
Predictions are evaluated using a prediction log's ID. These can be retrieved using the prediction log methods.
from deeploy import CreateEvaluation
deployment_id = "example"
prediction_log_id = "example"
# Example of a disagreement during the evaluation
evaluation_input_disagree: CreateEvaluation = {
"agree": False,
"desired_output": { "predictions": [True] },
"comment": "Example evaluation from the Python Client",
}
evaluation_disagree = client.evaluate(deployment_id, prediction_log_id, evaluation_input_disagree)
# Example of an agreement during the evaluation
evaluation_input_agree: CreateEvaluation = {
"agree": True,
"comment": "Example evaluation from the Python Client",
}
evaluation_agree = client.evaluate(deployment_id, prediction_log_id, evaluation_input_agree)