Skip to main content
Version: 1.37

Create a feedback loop with evaluations

Deeploy is built on the fundamental belief that providing clear explanations of deployed models' prediction processes is crucial for both present understanding and future reference. As such, each deployment within Deeploy is equipped with an endpoint that enables the collection of feedback from experts or end-users. This feedback is solicited and recorded on a per-prediction basis, allowing for comprehensive evaluation.

Evaluate a prediction​

We assume you have already created a Deployment and made inference calls through the UI, API, or Python client. Use the Deeploy API or Python client to add evaluations.

tip

You need the request log ID and prediction log ID to connect an evaluation to a prediction. Both IDs can be found in the output of your inference call.

Consult the evaluation section within the Swagger documentation.

Update an evaluation​

You have the option to adjust the evaluation of a prediction. For instructions, consult the aforementioned documentation.

Monitor evaluations​

Monitor evaluation for a Deployment on the Evaluation tab on your Deployment’s Monitoring page. Specifically, you can monitor:

  • Disagreement ratio Assess the extent to which evaluators are in disagreement with the predictions made by the model.
  • Disagreement per class Identify the specific areas within the outcomes where the highest number of disagreements occur.

Read monitoring a Deployment for more details.

View evaluation explanations​

View the evaluation of a specific prediction by clicking on a prediction on the Predictions tab within a Deployment. Scroll down in the Prediction modal to view:

  • the token used for the evaluation
  • the desired response
  • the explanation