Skip to main content
Version: Cloud

Add metadata to a Repository

The metadata.json can be used to provide metadata about your Deployments. The metadata.json can be provided in the root of your Repository (or the selected contract folder via the contractPath attribute), like the structure below.

repository (or contract path)
|__ metadata.json
...

Types of metadata are supported, of which an example can be found in our Public Scikit-Learn Census Repository:

  • features: An array of objects to provide information about the features in the outcome of your model. A feature consists of a name of type string. Optionally, it is possible to specify an observedMin and observedMax of type number, and a distribution. Extract this information from your training data (or another baseline, e.g. a specific timeframe of your production data). The order of provided features corresponds with the order of the values. E.g. if 10 values are expected, 10 features can be provided, where the first feature corresponds with the first value. features are used to:
    • Display feature names in SHAP explainers
    • Display boundaries in the Input Validation graph, and threshold suggestions when creating input validation alerts
  • predictionClasses: An object with keys and values to provide information about the predictions of your model. Every key in the object provides a category name for the corresponding model outcome. In the example, a model outcome of true means the person is credit worthy, so the key is "credit worthy". This category name is then used to provide labels to our monitoring graph, and will also be displayed in the Deployment details.
  • problemType: The key problemType expects a string with value classification or regression. This value is used to calculate the right performance metric based on the type of machine learning problem you model is solving. If the problemType is "classification", the performance is calculated as accuracy. If the problemType is "regression", the performance is calculated as RMSE.
  • exampleInput: An example of a json input that can be consumed by the model. The exampleInput can be used as a default option to interact with your model in the Test page of the Deployment.
  • inputTensorShape: The tensor dimensions of the json input that can be consumed by the model. More information about the tensor shape can be found in the TensorFlow Documentation.
  • exampleOutput: An example of a json output that can be given back by the model.
  • outputTensorShape: The tensor dimensions of the json output that can be given back by the model. More information about the tensor shape can be found in the TensorFlow Documentation.
  • customId: A custom ID that can be attached to the request, used to handle and filter all the predictions in a better way. Find out more about custom ID's.
  • customRequestFormat: Used to optionally state custom mapping format of input (for custom dockers and external deployments). More Info
  • customResponseFormat: Used to optionally state custom mapping format of output (for custom dockers and external deployments). More Info

Example metadata.json file​

A full example can be found here: example-sklearn-census.

{
  "predictionClasses": {
    "Credit worthy": true,
    "Not credit worthy": false
  },
  "problemType": "classification",
  "exampleInput": {
    "instances": [
      [
        51.0,
        0.0,
        13.0,
        2.0,
        4.0,
        1.0,
        0.0,
        1.0,
        12250.0,
        500.0,
        40.0,
        21.0
      ]
    ]
  },
  "inputTensorShape": "(1, 12)",
  "exampleOutput": {
    "predictions": [
      true
    ]
  },
  "outputTensorShape": "(1)",
  "customId": "clientId",
    "features": [
    {
      "name": "Age",
      "observedMin": 18,
      "observedMax": 96,
      "distribution": [
      {
          "start": "17.0",
          "end": "18.32",
          "count": "762"
      },
      {
          "start": "18.32",
          "end": "19.65",
          "count": "574"
      },
      {
          "start": "19.65",
          "end": "20.98",
          "count": "601"
      },
      ]
    },
    {
      "name": "Work class",
      "observedMin": 0,
      "observedMax": 7
    },
    {
      "name": "Education",
      "observedMin": 0,
      "observedMax": 15
    },
    {
      "name": "Maritual status",
      "observedMin": 0,
      "observedMax": 6
    },
    {
      "name": "Occupation",
      "observedMin": 0,
      "observedMax": 13
    },
    {
      "name": "Relationship",
      "observedMin": 0,
      "observedMax": 5
    },
    {
      "name": "Capital gain",
      "observedMin": 93,
      "observedMax": 129014
    },
    {
      "name": "Capital loss",
      "observedMin": 0,
      "observedMax": 14194
    },
    {   
      "name": "Hours per week",
      "observedMin": 3,
      "observedMax": 64
    },
    {
      "name": "Native country",
      "observedMin": 0,
      "observedMax": 40
    }
  ],
}

Custom mapping​

The metadata.json file supports custom request and response mapping through the customRequestFormat and customResponseFormat keys. With these mappings Deeploy supports all request and response formats for external Deployments and managed custom Docker deployments, splitting up the request in multiple prediction logs and leveraging Deeploy's full monitoring capabilities.

How custom mapping works​

Custom mapping allows you to specify how to navigate your custom model input and output structure (JSON) (up to max 3 levels deep) to identify model input and output.

Benefits of custom mapping​

  • Enables monitoring capabilities on custom docker and external deployments
  • Allows splitting of batch inference request into different prediction logs.
  • Preserves your custom API format while working with Deeploy
Note

For any mapping without the <array> keyword, the mapping should point to array of inputs/outputs that can be split into individual input/output.

Request mapping examples​

Custom keys​

This is the case when a sub level of keys has array of inputs. In case there is only 1 input this array should still exist with 1 input value.

:::Note In case individual inputs are property of array of objects. See case using <array>.

"customRequestFormat": "key1.key2"

Applied to:

{
  "key1": {
    "key2": [[51.0, 0.0, 13.0, 2.0], [31.0, 0.0, 13.0, 2.0]]
  }
}

This would be split into separate prediction logs as:

{
  "key1": {
    "key2": [[51.0, 0.0, 13.0, 2.0]]
  }
}
{
  "key1": {
    "key2": [[31.0, 0.0, 13.0, 2.0]]
  }
}

Custom keys in object array​

This is the case when at a sub level of keys there is object array and individual inputs are a property of the objects.

<array> is the restricted keyword to recognize object array.

"customRequestFormat": "key1.<array>.key2"

Applied to:

{
  "key1": [
    {
      "key2": [51.0, 0.0, 13.0, 2.0]
    },
    {
      "key2": [31.0, 0.0, 13.0, 2.0]
    }
  ]
}

This would split into seperate requests as:

{
  "key1": [
    {
      "key2": [51.0, 0.0, 13.0, 2.0]
    }
  ]
}
{
  "key1": [
    {
      "key2": [31.0, 0.0, 13.0, 2.0]
    }
  ]
}

Response mapping examples​

Similarly, response mapping allows Deeploy to properly interpret custom output formats to recognize outputs of model. This is important for enabling monitoring capabilities in Deeploy such as tracking drift, calculating performance metrics, alerting etc.

"customResponseFormat": "out1"

Applied to model response:

{
  "out1": [true, false]
}

This would be processed as separate outputs:

{
  "out1": [true]
}
{
  "out1": [false]
}