Setting up a work environment

A prerequisite to get started using ST Edge AI Core CLI is to install the newest version of ST Edge AI Core CLI, which is described in the “Installer” article.

%STEDGEAI_CORE_DIR% indicates the root location where the ST Edge AI Core components are installed.

• Python 3.9+ environment.
• STM32CubeProgrammer supporting the STM32N6 devices (STM32CubeProg).
• One of the following toolchains or IDEs for STM32 Arm® Cortex®-M-based MCUs must be installed.
  • STMicroelectronics - STM32CubeIDE version 1.15.1 or later (STM32CubeIDE).
  • IAR Systems - IAR Embedded Workbench® IDE - Armv9.30.1+ (www.iar.com/iar-embedded-workbench).
  • GNU Arm Embedded Toolchain (Arm GNU Toolchain).
• STM32N6 development board. STM32N6570-DK board is recommended. A Nucleo development board can be used, but the UCs are limited because no external RAM is available.

The `$STEDGEAI_CORE_DIR/scripts/N6_scripts/config.json' file should be updated to indicate the paths to find the external tools. Note that in this lab, the 'config_n6l.json' file is not used.

Generate the specialized c-files

The following command generates specialized configuration C-files with minimal default options. When no argument is provided to the '--st-neural-art' option, a default compilation profile file is used, and the associated "default" setting is considered. This profile uses the mandatory ST Neural-ART compiler options and a memory-pool descriptor file allowing the use of all available memory on the board. Only the first megabyte is reserved for the application, while the external flash is used to store the weights/parameters of the deployed model.

• Default compilation profile file: $STEDGEAI_CORE_DIR/Utilities/windows/targets/stm32/resources/neural_art.json.
  
• Memory-pool descriptor file: $STEDGEAI_CORE_DIR/Utilities/windows/targets/stm32/resources/mpools/stm32n6.mpool.

$> stedgeai generate -m mobilenet_v2_0.35_224_fft_int8.tflite --target stm32n6 --st-neural-art

Generated files are stored in the default 'st_ai_output' folder. The default file-prefix 'network_' and c-name suffix 'Default' are used. The '-n/--name' option (respectively '-o/--output' option) can be used to override the c-name suffix (respectively output folder).

<user_n6_getting_started>\st_ai_output\mobilenet_v2_0.35_224_fft_int8_OE_3_1_0.onnx
<user_n6_getting_started>\st_ai_output\mobilenet_v2_0.35_224_fft_int8_OE_3_1_0_Q.json
<user_n6_getting_started>\st_ai_output\network.c
<user_n6_getting_started>\st_ai_output\network_atonbuf.xSPI2.raw
<user_n6_getting_started>\st_ai_output\network_generate_report.txt

• The intermediate files (<model_file_name>_OE_3_1_0[_Q].[onnx | json]) are also provided. They can be used later with the ST Neural-ART compiler.
  
• The 'network_generate_report.txt' provides the main information about the imported model and how it is deployed. A summary about the usage of the memory, used options, epoch types..

The following command uses the minimal profile enhanced with the optimization for speed.

$> stedgeai generate -m mobilenet_v2_0.35_224_fft_int8.tflite --target stm32n6 --st-neural-art profile-allmems--O3

Available tool built-in profiles are described in the “ST Neural-ART compiler primer” article.

Build and load the built-in test application

The following command without options can be used to:

• Prepare the compiler environment (iar or gcc, see "compiler_type" from `$STEDGEAI_CORE_DIR/scripts/N6_scripts/config.json' file).
• Update the C-project with the generated specialized files.
  • Location: $STEDGEAI_CORE_DIR/Projects/STM32N6570-DK/Applications/NPU_Validation/.
• Convert the raw memory initializer files to an Intel-hex format to be used by the flasher application
• Build the c-project.
  • Calling makefile entry point for gcc-based project.
  • Calling the IAR project file for iar-based project.
• Load the application and the memory initializers - Note that the firmware image is loaded through a debug session. The board should be configured in dev mode.
• Start the application.

$> python $STEDGEAI_CORE_DIR/scripts/N6_scripts/n6_loader.py

XXX -- Preparing compiler IAR
XXX -- Setting a breakpoint in main.c at line 127 (before the infinite loop)
XXX -- Copying network.c to project: <user_n6_getting_started>\st_ai_output\network.c -> $PROJECT\X-CUBE-AI\App\network.c
XXX -- Extracting information from the c-file
XXX -- Converting memory files in results/<model>/generation/ to Intel-hex with proper offsets
XXX -- arm-none-eabi-objcopy.exe --change-addresses 0x70000000 -Ibinary -Oihex network_atonbuf.xSPI2.raw network_atonbuf.xSPI2.hex
XXX -- Resetting the board...
XXX -- Patching macro file for automatically loading Intel-Hex files upon breakpoint
XXX -- Flashing memory xSPI2 -- 623.969kB
XXX -- Flashing memory xSPI2 -- return code OK
XXX -- Dumping macro file
XXX -- Building project (conf= N6-DK)
XXX -- Compilation successful
XXX -- Running the program & loading internal memories
XXX -- Loading memories successful
XXX -- Start operation achieved successfully

At this stage, the board is ready to receive a command from the host machine (serial COM). AiRunner package or “validate” command of the ST Edge AI CLI can be used.

$> python $STEDGEAI_CORE_DIR/scripts/N6_scripts/n6_loader.py --build-config N6-DK-USB

Evaluate the performance (profiling)

The following command allows performing a quick profiling of the deployed model (refer to “How to use the AiRunner package” article for more details). Ten samples randomly generated are used to perform the inferences.

$> export PYTHONPATH=$STEDGEAI_CORE_DIR/scripts/ai_runner
$> python $STEDGEAI_CORE_DIR/scripts/ai_runner/examples/checker.py -d serial:921600 --perf-only -b 10

The first part of the log shows the main information of the deployed model and the used runtime including the device settings.

The next part shows the inference time by epoch. Note that the MCU/CPU cycles are used to report the execution time of the different epoch phases. In the context of the used profiling application, the MCU clock is proportional to the NPU clock.

Evaluate the accuracy

$> stedgeai validate -m mobilenet_v2_0.35_224_fft_int8.tflite --target stm32n6 --mode target -d serial:921600

Computing the metrics...
...
  Confusion matrix (axis=-1) - 5 classes (10 samples)
  ---------------------------------
  C0        0    .    .    .    .
  C1        .    0    .    .    .
  C2        .    .    0    .    .
  C3        .    .    .    0    .
  C4        .    1    .    .    9

 Evaluation report (summary)
 ----------------------------------- ... -----------------------------------------------------------
 Output       acc      rmse          ...    nse        cos        tensor
 ----------------------------------- ... -----------------------------------------------------------
 X-cross #1   90.00%   0.086785592   ...    0.865483   0.962875   'nl_69', 10 x int8(1x5), m_id=[69]
 ----------------------------------- ... -----------------------------------------------------------

$> stedgeai validate -m mobilenet_v2_0.35_224_fft_int8.tflite --target stm32n6 --mode target -d serial:921600 \
     -vi input_20_images.npy

Computing the metrics...
...
  Confusion matrix (axis=-1) - 5 classes (20 samples)
  ---------------------------------
  C0        5    .    .    .    .
  C1        .    3    .    .    .
  C2        .    .    3    .    .
  C3        .    .    .    5    .
  C4        .    .    .    .    4

 Evaluation report (summary)
 ----------------------------------- ... -----------------------------------------------------------
 Output       acc       rmse         ...    nse        cos        tensor
 ----------------------------------- ... -----------------------------------------------------------
 X-cross #1   100.00%   0.023515496  ...    0.996440   0.998586   'nl_69', 20 x int8(1x5), m_id=[69]
 ----------------------------------- ... -----------------------------------------------------------

Enable the epoch controller mode

$> cp -r $STEDGEAI_CORE_DIR/Utilities/windows/targets/stm32/resources/mpools/ .
$> cp $STEDGEAI_CORE_DIR/Utilities/windows/targets/stm32/resources/neural_art.json .

  ..
  "allmems--O3-ec" : {
    "memory_pool": "./mpools/stm32n6.mpool",
    "options": "--native-float --mvei --cache-maintenance --Ocache-opt --enable-virtual-mem-pools 
                --Os --optimization 3 --enable-epoch-controller"
  },
  ..

$> stedgeai generate -m mobilenet_v2_0.35_224_fft_int8.tflite --target stm32n6 --st-neural-art allmems--O3-ec@neural_art.json

<user_n6_getting_started>\st_ai_output\mobilenet_v2_0.35_224_fft_int8_OE_3_1_0.onnx
<user_n6_getting_started>\st_ai_output\mobilenet_v2_0.35_224_fft_int8_OE_3_1_0_Q.json
<user_n6_getting_started>\st_ai_output\network.c
<user_n6_getting_started>\st_ai_output\network_atonbuf.xSPI2.raw
<user_n6_getting_started>\st_ai_output\network_ecblobs.h
<user_n6_getting_started>\st_ai_output\network_generate_report.txt

ECBLOB_CONST_SECTION
static const uint64_t _ec_blob_1 [] =
{
  0x000094edca057a7aUL, 0x0000005d0000003dUL, 0x5c00004200802241UL, 0x1500006008000000UL,
  0x000700005c027c00UL, 0x000040005c070043UL, 0x5c0c00430000ffe0UL, 0x1041204200000000UL,
...

$> python $STEDGEAI_CORE_DIR/scripts/N6_scripts/n6_loader.py

...
XXX -- Copying network.c to project: <user_n6_getting_started>\st_ai_output\network.c -> $PROJECT\X-CUBE-AI\App\network.c
XXX -- Copying network_ecblobs.h to project: <user_n6_getting_started>\st_ai_output\network_ecblobs.h -> $PROJECT\X-CUBE-AI\App\network_ecblobs.h
...
XXX -- Start operation achieved successfully

$> export PYTHONPATH=$STEDGEAI_CORE_DIR/scripts/ai_runner
$> python $STEDGEAI_CORE_DIR/scripts/ai_runner/examples/checker.py -d serial:921600 --perf-only -b 10

Change the I/O data format

To facilitate the deployment of a given model, the I/O data format can be modified to meet the firmware constraints.

$> stedgeai generate -m mobilenet_v2_0.35_224_fft_int8.tflite --target stm32n6 \
    --st-neural-art allmems--O3-ec@neural_art.json --output-data-type int8

$> stedgeai generate -m <model-path-onnx-qdq.onnx> --target stm32n6 --st-neural-art <st-neural-art-profile>
         --inputs-ch-position chlast --input-data-type uint8

External tools (config.json)

The location of the required tools is done in a file called 'config.json' which must be located in the same directory of the 'n6_loader.py' script. An example of such a file is given below. This file is in a json-format that allows comments.

Location: $STEDGEAI_CORE_DIR/scripts/N6_scripts/config.json

{
  // Set Compiler_type to either gcc or iar
  "compiler_type": "gcc",
  // Set Compiler_binary_path to your bin/ directory where IAR or GCC can be found
  //     If "Compiler_type" == gcc, then gdb_server_path shall point to where ST-LINK_gdbserver.exe can be found
  "gdb_server_path": "C:/cubeIDE/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.stlink-gdb-server.win32_2.1.100.202309131603/tools/bin",
  "gcc_binary_path": "C:/cubeIDE/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.gnu-tools-for-stm32.11.3.rel1.win32_1.1.100.202309151323/tools/bin/",
  "iar_binary_path": "C:/IAR/IAR9.30.1/common/bin/",
  // Full path to arm-none-eabi-objcopy.exe
  "objcopy_binary_path": "C:/cubeIDE/STM32CubeIDE/plugins/com.st.stm32cube.ide.mcu.externaltools.gnu-tools-for-stm32.11.3.rel1.win32_1.1.100.202309151323/tools/bin/arm-none-eabi-objcopy.exe",
  // Cube Programmer binary path
  "cubeProgrammerCLI_binary_path": "C:/CubeProgN6/bin/STM32_Programmer_CLI.exe",
  // make path if not present in $PATH
  "make_binary_path": "C:/make/bin/make.exe",
  // Cube IDE path to use automatic detection of (gdb_server, gcc, objcopy, cubeProgrammer, make)
  "cubeide_path":"C:/Users/foobar/TOOLS/STM32CubeIDE_x.y.z/STM32CubeIDE"
}

Omitting a line for a tool (or commenting it) results in the tool searching in the system path as a last resort (e.g it is possible not to write the "make_binary_path" if 'make.exe' is in the system path)

In this section, either gcc or iar toolchain can be selected:

• For IAR users:
  • Set "compiler_type" to "iar"
  • Set "iar_binary_path" with the path containing IAR binaries (ending in bin)
• For GCC users:
  • Set "compiler_type" to "gcc"
  • Set "gcc_binary_path" with the path containing arm-none-eabi-gcc binaries (ending in bin)
  • Set "gdb_server_path" to the path containing stlink gdbserver (ending in bin)

The file can be then shortened to something like:

{
  "compiler_type": "gcc",
  "cubeide_path":"C:/Users/foobar/TOOLS/STM32CubeIDE_x.y.z/STM32CubeIDE"
}

or

{
  "compiler_type": "iar",
  "iar_binary_path": "C:/IAR/IAR9.30.1/common/bin/",
  "cubeide_path":"C:/Users/foobar/TOOLS/STM32CubeIDE_x.y.z/STM32CubeIDE"
}

Test application (config_n6l.json)

To use the n6_loader.py script, the location of the C-project used for validation, the location of the generated specialized files, and various options can be provided through a specific project configuration file: config_n6l.json. This file is in a json-format that allows comments.

// This file is for configuring N6 loader, the util used to copy STEdgeAI outputs into a project,
// compile the project, and load the results on the board.
{
    // The 2lines below are _only used if you call n6_loader.py ALONE (memdump is optional and will be the parent dir of network.c by default)
    "network.c": "C:/st_ai_output/network.c",
    //"memdump_path": "C:/Users/foobar/CODE/stm.ai/stm32ai_output",
    // Location of the "validation" project  + build config name to be built (if applicable)
    "project_path": "C:/Projects/NPU_Validation",
    // If using the NPU_Validation project, valid build_conf names are "N6-DK", "N6-DK-USB", "N6-Nucleo", "N6-Nucleo-USB"
    "project_build_conf": "N6-DK",
    // Skip programming weights to earn time (but lose accuracy) -- useful for performance tests
    "skip_external_flash_programming": false,
    "skip_ram_data_programming": false
}

This file configures the behaviour of the n6_loader.py script.

• "network.c" shall indicate where to find the specialized “.c” file that was output from the STEdgeAI generate call.
• "project_path" shall indicate where the “NPU_Validation” is located on the user’s computer.
• "project_build_config" shall be kept to “N6-DK” when using a Discovery Kit board. (And see below for explanations about N6-DK-USB)
• "skip_external_flash_programming" and "skip_ram_data_programming" allow the user to prevent programming part of the data on board. The values for this field are true and false.

When everything is set up, the syntax is:

python n6_loader.py --n6-loader-config ./config_n6l.json

When not provided, all the configuration fields are filled automatically by default values that consider that the script is in its installation location, with the NPU_Validation project in its default location,..

Basic checklist when an issue occurs when following the steps above

• Ensure that the board is properly powered up (The discovery-kit board is a bit power-hungry and needs a good USB port and a good USB cable)
• Ensure the switches on the board are properly set up (the only switches being SW1 and SW2) + the jumper JP2 on the discovery-kit.
• Ensure all the software/firmware used come from the same delivery of tools (the backwards-compatibility is not ensured and using, e.g., old validation projects will most likely result in full failure)
• Ensure that no errors have occurred during any of the steps:
  • Generation
    • Check that the STM32N6-NeuralArt options have been correctly set --target stm32n6 --st-neural-art ...
    • Check that there was no error at all
  • n6_loader
    • Check the toolchain used is correct (configured in the config.json file)
    • Check the files copied are correct (source / destination found in the config_n6l.json file)
    • Check that there is no error like “command not found”
    • If applicable, check that the external flash programming is done without errors
    • Check that the compilation goes through without errors (or check compile.log)
    • Check that there is a success message at the end
    • Extra, set skip_external_flash_programming and skip_ram_data_programming to “false” in config_n6l.json file
  • Validation
    • Ensure that the connection speed has been specified on the command line: --desc serial:921600
    • For connection issues, see below

Cannot connect to the board / connection issues

When doing a validation on target, it is possible that some connection issues appear. This can be due to multiple causes, but the main ones are

• Another software is currently using the UART (for example, terminal software connected to the ST-Link UART)
  • Disconnect the other UART snooper.
• Multiple ST boards are connected on the computer, and the automatic discovery fails (invalid firmware)
• OR some other peripherals are connected to the computer and leads to issues when doing automatic discovery of ST-Link (infinite loop, with no timeout when trying to connect to the board)
  • Help stedgeai application to find the correct port, --desc serial:921600 asks the validation to be done over UART, with a speed of 921600 bps (as set up by the default validation project). To force the use of a given port, the syntax is as follows (for example, using windows, and forcing communication on COM17) --desc serial:COM17:921600.

Timeout (50000 ms) during validation

Such a timeout signals an inference that has been started properly but that hangs forever. Though this should never happen, double check the checklist above, and report a problem with the files used as inputs, the command lines used and the output generated files.

Validating my model takes forever…