--torchcompile=tensorrt

[andravin]

The pytorch-image-models benchmark.py --torchcompile option works great with the PyTorch nightly builds and the default inductor backend. I inferred that tensorrt is also a valid backend, though the torch.compile documentation does not mention any backends besides the default; but --torchcompile=tensorrt yields an error when I use it with any model ..

Running inference benchmark on resnet50d for 40 steps w/ input size (3, 224, 224) and batch size 128.
/home/andrew/.local/lib/python3.8/site-packages/torch/jit/_check.py:181: UserWarning: The TorchScript type system doesn't support instance-level annotations on empty non-base types in `__init__`. Instead, either 1) use a type annotation in the class body, or 2) wrap the type in `torch.jit.Attribute`.
  warnings.warn("The TorchScript type system doesn't support "
ERROR: "tensorrt raised Exception: Please convert all Tensors to FakeTensors first or instantiate FakeTensorMode with 'allow_non_fake_inputs'. Found in aten.convolution.default(*(FakeTensor(FakeTensor(..., device='meta', size=(128, 3, 224, 224), dtype=torch.float16), cuda:0), Parameter containing:
tensor([[[[ 6.2988e-02,  7.7698e-02,  1.4087e-01],
          [ 4.1534e-02,  1.8262e-01, -1.1276e-02],
          [ 2.2018e-02,  8.1406e-03, -1.2115e-01]],
          
          ...

I suspect this is a PyTorch issue, but I also believe that many users would like any easy way to use tensorrt.

Has anybody made this work? Or do you use a different workflow to export models to tensorrt?

[dvando]

I have the same issue here, did you solve it?

[andravin]

No, I did not get the torch inductor tensorrt backend to work. As far as I can tell, it is not actually supported.

The preferred method to export a PyTorch model to tensorrt seems to be Torch-TensorRT.