Overview

This article explains how to use the 'stm_ai_runner' Python package, also known as 'ai_runner', to profile and validate a deployed C-model. As illustrated in the following figure, the model can be deployed either on a target device or on the host. The AiRunner object provides a simple and unified interface for inference and profiling, allowing users to inject data, execute inference, and retrieve predictions.

The 'stm_ai_runner' Python package is also integrated into the ST Edge Core AI CLI (used by the ‘validate’ command). But it can also be used independently to extend the default validation process. End-users (data scientists or ML/AI designers) can update (with minor adaptations) its classical validation Python scripts to validate the deployed model with the real dataset and metrics.

Multiple back-ends are supported, in this article two main configurations are considered:

• Execution on host. Through the 'generate' command, the user has the possibility to create a shared library (or DLL, using the ‘–dll’ option) containing the specialized c-files. This shared library exports the embedded C-API functions (legacy API or st-ai API) which are bound in the Python environment to export a common API.

• Execution on a physical target. The specialized files are linked with a generic embedded test application (also called the aiValidation application). On the host side, a simple message-based protocol on top of a serial protocol exposes a set of services for discovering and using the deployed models.

Setting up a work environment

Following Python packages should be installed in a Python 3.x environment to use the 'stm_ai_runner' package it is recommended to use a virtual environment.

protobuf<3.21
tqdm
colorama
pyserial
numpy

To be able to import the 'stm_ai_runner' package, set the 'PYTHONPATH' environment variable:

export PYTHONPATH=$STEDGEAI_CORE_DIR/scripts/ai_runner:$PYTHONPATH

%STEDGEAI_CORE_DIR% represents the root location where the ST Edge AI Core components are installed, typically in a path like "<tools_dir>/STEdgeAI/2.1/".

$ stedgeai generate -m <model_path> --target stm32 --c-api st-ai --dll 
...

 Generated files (6)
 ------------------------------------------------------------------------------------------
 <workspace-directory-path>\inspector_network\workspace\generated\network_data.h
 <workspace-directory-path>\inspector_network\workspace\generated\network_data.c
 <workspace-directory-path>\inspector_network\workspace\generated\network_details.h
 <workspace-directory-path>\inspector_network\workspace\generated\network.h
 <workspace-directory-path>\inspector_network\workspace\generated\network.c
 <workspace-directory-path>\inspector_network\workspace\lib\libai_network.dll

Creating txt report file <output-directory-path>\network_generate_report.txt

$ stedgeai validate -m <model_path> --target stm32 --mode target -d file:st_ai_ws

$ python $STEDGEAI_CORE_DIR/scripts/ai_runner/examples/checker.py

$ stedgeai validate -m <model_path> --target stm32 --mode target -d serial

$ python $STEDGEAI_CORE_DIR/scripts/ai_runner/examples/checker.py -d serial

Getting started - Minimal script

Following code shows a minimal script to perform a model inference (with random input data) running on a physical target and to display the profiling information.

import sys
import argparse

from stm_ai_runner import AiRunner

desc = 'serial'

# create AiRunner object
runner = AiRunner()
# connection
runner.connect(desc)
# display and retrieve model info (optional)
runner.summary()
model_info: dict = runner.get_info()
input_details: list[dict] = runner.get_input_infos()  # = model_info['inputs']
output_details: list[dict] = runner.get_output_infos()  # = model_info['outputs']
# generate the random input data
inputs: list[np.ndarray] = runner.generate_rnd_inputs(batch_size=2)
# perform the inference
mode: AiRunner.Mode = AiRunner.Mode.PER_LAYER
outputs, profiler = runner.invoke(inputs, mode=mode)
# display the profiling info
runner.print_profiling(inputs, profiler, outputs)
# disconnect
runner.disconnect()

This excerpt is part of the '$STEDGEAI_CORE_DIR/scripts/ai_runner/examples' folder (see 'minimal.py' and 'checker.py' files).

AiRunner API

The '$STEDGEAI_CORE_DIR/scripts/ai_runner/examples' folder provides different simple scripts using the AiRunner API.

import sys
from stm_ai_runner import AiRunner

desc=...

runner = AiRunner()
runner.connect(desc)

if not runner.is_connected:
    print('No c-model available, use the --desc/-d option to specifiy a valid path/descriptor')
    print(f' {runner.get_error()}')
    sys.exit(1)
...

available_models: list[str] = runner.names
print(available_models)
# ['network0', 'network1', ...]

runner = AiRunner()
runner.connect(desc)
...
session: AiRunnerSession = runner.session('network_2')

model_info: dict = runner.get_info()  # equivalent to runner.get_info(available_models[0])

model_inputs: list[dict] = runner.get_input_infos()  # equivalent to runner.get_input_infos(available_models[0])
model_outputs: list[dict] = runner.get_output_infos()  # equivalent to runner.get_output_infos(available_models[0])

# perform the inference
mode: AiRunner.Mode = AiRunner.Mode.PER_LAYER
outputs, profiler = runner.invoke(inputs, mode=mode)

Services

summary()

The 'summary(name: Optional[str] = None)' method displays a summary of the information provided by the 'get_info(name: Optional[str] = None)' method.

runner.summary()

generate_rnd_inputs()

The 'AiRunner.generate_rnd_inputs(name: Optional[str])' method is a helper service allowing to generate the input data for a given model. val parameter is used to set the range of the data which are uniformly distributed over a specific interval [low, high[. Default is [-1.0, 1.0[ for the floating-point type.

inputs: Union[np.ndarray, List[np.ndarray]] = runner.generate_rnd_inputs(name='network', batch_size=2)

print_profiling()

The 'print_profiling(inputs, profiler, outputs)' method displays a summary of the profiling information returned by the ‘invoke()’ method.

# perform the inference
outputs, profiler = runner.invoke(inputs)
# display the profiling info
runner.print_profiling(inputs, profiler, outputs)

Examples

Location: %STEDGEAI_CORE_DIR%/script/ai_runner/example/

• checker.py provides an example to use the ai_runner module including the profiling information

  # Try to load the shared library located in the default location:  ./st_ai_ws.
  # It displays a summary and performs two inferences with the random data.
  $ python checker.py
  
  # As previously, but it performs a connection with a STM32 board (auto-detect mode) 
  $ python checker.py -d serial
  
  # Set the expected COM port and baudrate 
  $ python checker.py -d serial:COM6:115200

• tflite_test.py provides a typical example to compare the outputs of the generated C-model against the predictions from the tf.lite.Interpreter.

• mnist provides a complete example with two scripts allowing to train a model (train.py) and to test (test.py) with the generated c-model.