GPU Support
Using GPUs for your deployments is currently only available for Enterprise subscriptions.
Deeploy enables using GPUs for your deployments. Make sure to have set up available GPU nodes during installation. At the advanced configuration options select a node that has a GPU available to automatically claim a GPU for that component.
Selecting a GPU node means the node will solely be available for the component (model, transformer or explainer) that selected that node. Sharing a GPU or running other workloads on that same GPU node is unsupported.
Model frameworks
When you select a GPU node and use the Pytorch or Tensorflow framework for your model then it will be served with the GPU version of the model serving framework. Note that this does require your tensors to be loaded to the GPU device. See this Pytorch handler.py as example. If you use a custom docker and want to make use of the GPU, you need to install CUDA and cuDNN in your docker image. Make sure they are compatible with each other and the selected GPU. Check out this compatibility matrix for an overview.
Explainer and Transformer frameworks
Using a GPU for your explainer is supported only for integrated explainers (Pytorch and Tensorflow) and custom docker. For transformers only custom docker is supported.