Supported file format

ST Edge AI Core can import the following quantized model formats:

• Quantized TensorFlow Lite models generated by a post-training or quantization-aware process. Calibration is performed by the TensorFlow Lite framework, primarily through the TFLite Converter, which exports a TensorFlow Lite file.
• Quantized ONNX models based on either the operator-oriented (QOperator) or tensor-oriented (QDQ, Quantize and DeQuantize) format. The first format depends on the supported QOperators (see the “QLinearXXX* operators”) while the second is more generic. The DeQuantizeLinear(QuantizeLinear(tensor)) operators are inserted between the original operators (in float) to simulate the quantization and dequantization process. Both formats can be generated with the ONNX runtime module.

For performance reasons related to peak memory usage and processing capabilities, the deployed kernels support only quantized weights and activations (“integer-only quantization” mode). If this pattern is not detected by the optimization and rendering passes of the code generator, it falls back to a floating-point implementation of the operator. This fallback involves inserting QUANTIZE/DEQUANTIZE operators, a process known as fake or simulated quantization. Consequently:

• The dynamic quantization approach, which determines quantization parameters during inference, is not supported. If this type of model is provided, a fully float32 implementation is generated.
• Mixed or hybrid models are supported.

The analyze, validate, and generate commands can be used without limitations.

$ stedgeai analyze -m <quantized_model_file>.tflite --target stm32
$ stedgeai validate -m <quantized_model>.tflite --target stm32 -vi test_data.npz
$ stedgeai analyze -m <quantized_model_file>.onnx --target stm32

Privileged quantization scheme

Quantized tensors ([1], [2])

ST Edge AI Core supports 8-bit integer arithmetic (int8 or uint8) for quantized tensors, following the convention used by Google for quantized models ([2]). Each real number r is represented as a function of the quantized value q, a scale factor (an arbitrary positive real number), and a zero point. This quantization scheme is an affine mapping from the integer value q to the real value r. The zero point has the same integer C type as the quantized data.

Precision depends on the scale factor, and quantized values are linearly distributed around the zero point value. Unlike floating-point representation, the resolution is constant across the tensor.

Supported/recommended methods

• Posttraining quantization: https://www.tensorflow.org/lite/performance/post_training_quantization
  
• Quantization aware training: https://www.tensorflow.org/model_optimization/guide/quantization/training

“Integer only” method

The following code snippet illustrates the recommended TFLiteConverter options to enforce a full integer scheme, that is, post-training quantization for all operators, including the input and output tensors.

def representative_dataset_gen():
  data = tload(...)

  for _ in range(num_calibration_steps):
    # Get sample input data as a numpy array in a method of your choosing.
    input = get_sample(data)
    yield [input]

converter = tf.lite.TFLiteConverter.from_saved_model(<saved_model_dir>)
# converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.representative_dataset = representative_dataset_gen
# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# This ensures that if any ops can't be quantized, the converter throws an error
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
# These set the input and output tensors to int8
converter.inference_input_type = tf.int8  # or tf.uint8
converter.inference_output_type = tf.int8  # or tf.uint8
quant_model = converter.convert()

# Save the quantized file
with open(<tflite_quant_model_path>, "wb") as f:
    f.write(quant_model)
...

“Full integer quantization” method

Because ST Edge AI Core supports mixed models, the full integer quantization method can be used. However, it is preferable to enable the TensorFlow Lite ops (TFLITE_BUILTINS), because only a limited number of TensorFlow operators are supported (see the supported [TFLITE] operators).

...
converter = tf.lite.TFLiteConverter.from_saved_model(<saved_model_dir>)
# converter = tf.lite.TFLiteConverter.from_keras_model(model)
converter.representative_dataset = representative_dataset_gen
# This enables quantization
converter.optimizations = [tf.lite.Optimize.DEFAULT]
# This optional option, ensures that TF Lite operators are used.
converter.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS]
# These set the input and output tensors to float32
converter.inference_input_type = tf.float32  # or tf.uint8, tf.int8
converter.inference_output_type = tf.float32  # or tf.uint8, tf.int8
quant_model = converter.convert()
...

TensorFlow lite, OPTIMIZE_FOR_SIZE option support

Post-quantization TensorFlow lite script (TFLiteConverter, TF 1.15) allows to generate a weight-only quantized file ('OPTIMIZE_FOR_SIZE' option). This simplest scheme (also called “hybrid” quantization) allows to reduce the size of the generated file (~by 4). Only the weights are quantized from floating-point to 8 bits of precision. At inference, weights are converted from 8 bits of precision to floating-point and computed using floating-point kernels.

This quantization scheme is not supported by ST Edge AI Core, in particular by the C-inference engine and operator implementations (network runtime library), mainly because of the devices resource constraints. Additional RAM memory to cache the uncompressed parameters would be requested to reduce the latency. If this model is imported, the parameters are converted to floating-point before code generation. Only the full 8b integer quantization of weights and activations scheme is supported.

Quantize and DeQuantize (QDQ) format

To quantize an ONNX model, use the quantization services provided by the ONNX Runtime module. The tensor-oriented (QDQ, Quantize and DeQuantize) format is recommended. As illustrated in the following figure, the additional DeQuantizeLinear(QuantizeLinear(tensor)) operators inserted between the original operators are automatically detected and removed so the associated optimized integer kernels can be deployed.

Notes

• The quantize process automatically merges Batch Normalization operators. The model can also be optimized for inference before quantization; the ONNX Simplifier can be used for this purpose.

    $ python -m onnxruntime.quantization.preprocess --input model.onnx --output model-infer.onnx

• Because the deployed kernels primarily use channel-last (NHWC) data format, and to preserve the original input data representation, a ‘Transpose’ operator is added in the generated graph. Note that, depending on the target, this operation can be performed by software and the cost may not be negligible. To improve this behavior and also to take account for application constraints, advanced options can be used to specify the expected input or output data layout (refer to “How to change the I/O data type or layout (NHWC vs NCHW)” article for more details)

• By default, the I/O data type (float32) of the original ONNX model is not preserved but optimized to align with the activation type used by the quantization scheme. Advanced options can be used to specify the expected data type. As illustrated in the following figure, the option “–input-data-type float32” is used to insert a quantizer operation and preserve the original data type. (Refer to “How to change the I/O data type or layout (NHWC vs NCHW)” article for more details)

Supported/recommended methods/options

“Static Quantization” method

The following code snippet illustrates a typical Python script (posttraining quantization) to quantize an NN model processing images (classifier or object-detector applications). It is based on the end-to-end example from the “Quantize ONNX Models” article. Note that only the inputs (images used for the calibration) are requested and the associated data can be transposed (hwc to chw) to be conform to the expected input data representation.

import numpy
import onnxruntime
from onnxruntime.quantization import QuantFormat, QuantType, StaticQuantConfig, quantize, CalibrationMethod
from onnxruntime.quantization import CalibrationDataReader
from PIL import Image

input_model_path = 'my_model.onnx'
output_model_path = 'my_model_quantized.onnx'

calibration_dataset_path = '/path/to/data/for/calibration'

def _preprocess_images(images_folder: str, height: int, width: int):
    """
    Load a batch of images and preprocess them..
    """
    image_names = os.listdir(images_folder)
    batch_filenames = image_names
    unconcatenated_batch_data = []

    for image_name in batch_filenames:
        image_filepath = images_folder + "/" + image_name
        pillow_img = Image.new("RGB", (width, height))
        pillow_img.paste(Image.open(image_filepath).resize((width, height)))
        input_data = numpy.float32(pillow_img) - numpy.array(
            [123.68, 116.78, 103.94], dtype=numpy.float32
        )
        nhwc_data = numpy.expand_dims(input_data, axis=0)
        nchw_data = nhwc_data.transpose(0, 3, 1, 2)  # ONNX Runtime standard
        unconcatenated_batch_data.append(nchw_data)
    batch_data = numpy.concatenate(
        numpy.expand_dims(unconcatenated_batch_data, axis=0), axis=0
    )
    return batch_data

class XXXDataReader(CalibrationDataReader):
    def __init__(self, calibration_image_folder: str, model_path: str):
        self.enum_data = None

        # Use inference session to get input shape.
        session = onnxruntime.InferenceSession(model_path, None)
        (_, _, height, width) = session.get_inputs()[0].shape

        # Convert image to input data
        self.nhwc_data_list = _preprocess_images(
            calibration_image_folder, height, width)
      
        self.input_name = session.get_inputs()[0].name
        self.datasize = len(self.nhwc_data_list)

    def get_next(self):
        if self.enum_data is None:
            self.enum_data = iter(
                [{self.input_name: nhwc_data} for nhwc_data in self.nhwc_data_list]
            )
        return next(self.enum_data, None)

    def rewind(self):
        self.enum_data = None

dr = XXXDataReader(
        calibration_dataset_path, input_model_path
    )

conf = StaticQuantConfig(
    calibration_data_reader=dr,
    quant_format=QuantFormat.QDQ,
    calibrate_method=CalibrationMethod.MinMax,
    optimize_model=True,
    activation_type=QuantType.QInt8,
    weight_type=QuantType.QInt8,
    # nodes_to_exclude=['resnetv17_dense0_fwd', ..],
    # nodes_to_quantize=['resnetv17_dense0_fwd', ..],
    per_channel=True)
      
quantize(infer_model,
    output_model_path, conf)

Note that for a quick evaluation in term of inference time and memory footprint, the XXXDataReader object can be updated to generate the fake image with the random data.

import numpy

class XXXDataReader(CalibrationDataReader):
    def __init__(self, calibration_image_folder: str, model_path: str):
        self.enum_data = None

        # Use inference session to get input shape.
        session = onnxruntime.InferenceSession(model_path, None)
        (_, chnannel, height, width) = session.get_inputs()[0].shape

        # Generate the random data in the half-open interval [0.0, 1.0).
        self.nhwc_data_list = [np.random.random_sample((1, chnannel, height, width)).astype(np.float32)
                               for i in range(20)]

        self.input_name = session.get_inputs()[0].name
        self.datasize = len(self.nhwc_data_list)

    def get_next(self):
        if self.enum_data is None:
            self.enum_data = iter(
                [{self.input_name: nhwc_data} for nhwc_data in self.nhwc_data_list]
            )
        return next(self.enum_data, None)

    def rewind(self):
        self.enum_data = None

Requested ONNX Opset

Models must use opset 7 or later to be quantized. Models with an opset below 10 must be reconverted to ONNX from their original framework using a later opset. To enable advanced optimizations such as batch-normalization folding, opset 21 is recommended.

import onnx

original_model_path = 'original_model.onnx'
new_model_path = 'new_model.onnx'

new_opset = 21

onnx_model = onnx.load(original_model_path)
converted_model = onnx.version_converter.convert_version(onnx_model, new_opset)
onnx.save(converted_model, new_model_path)

ONNX Simplifier

• https://github.com/daquexian/onnx-simplifier

Before quantizing an ONNX model, you can simplify it to produce a more efficient inference model for quantization or deployment.

$ onnxsim my_model.onnx my_simplified_model.onnx --overwrite-input-shape 1,3,224,224

Known limitations

• ONNX QuantizeLinear and DequantizeLinear are supported only when the axis attribute corresponds to the channel dimension. In other cases, the generated code may be incorrect without an explicit error from ST Edge AI Core. If the following assertion is raised during model validation, quantize the ONNX model per tensor (per_channel=False) as a possible workaround.

Assertion failed: n_channel_in == ( (ai_shape_dimension*)((((&((p_tensor_weights)->shape))))->data) )[(0x0)],
file <root_file_location>\layers_conv2d_stm32_integer.c, line 3628