Convert Stable Diffusion Inpainting model to FP16 with FP32 inputs

[jdp8]

Feature request

I've used this script to convert models to ONNX in FP16 format but maintaining the FP32 inputs. One of the models that I converted was Stable Diffusion 2 Inpainting to FP16 and tried to use it in ONNX Runtime and ONNX Runtime Web but it doesn't give me the expected results in either engine. I also converted the model with the Optimum conversion script to FP32 and this model gives me the expected result in ONNX Runtime. Results are shown below:

Input Image:

Mask Image:

Correct Onnx Runtime Output (converted with Optimum script):

Incorrect Onnx Runtime Output (converted with Stable-Diffusion-ONNX-FP16 script):

Incorrect Onnx Runtime Web Output (converted with Stable-Diffusion-ONNX-FP16 script):

I've also used the Optimum conversion script to convert the model to FP16 and this worked but the inputs are expected to be FP16. This datatype does not exist in JavaScript (specifically, Float16Array) and therefore cannot be used in ONNX Runtime Web.

With that being said, is it possible to convert a model to FP16 but leaving the inputs as FP32 in order for the UNET to be less than 2 GB?

Motivation

I would like to run Stable Diffusion Inpainting in ONNX Runtime Web and for the UNET to be less than 2GB. The FP16 model that I have at the moment gives me an output that is not as expected in ONNX Runtime and ONNX Runtime Web. So far, only the Optimum models give me a correct output in ONNX Runtime but I would like to use this in ONNX Runtime Web.

Your contribution

I am willing to contribute to this change given some guidance. Not sure how difficult it would be but I believe it would be similar to how it's implemented in the script mentioned beforehand.