NPU compiler location

‘atonn’ designates the name of the executable.

%STEDGEAI_CORE_DIR%\Utilities\<host_os>\atonn

%STEDGEAI_CORE_DIR% indicates the location where the ST Edge AI Core components are installed.

Mandatory options

Recommended options

Some options are recommended, to allow for greater performances, or better user experience in most of the cases:

• The --enable-virtual-mem-pools option is important to allow the compiler to “fuse”/“merge” multiple memory pools that are adjacent. The default behavior is not to allocate buffers that spread over multiple pools, this option removes this constraint. The resulting buffer allocation is usually more space-efficient (and allows for a greater use of internal memories to handle activation buffers).
• The -Os option provides usually good results without inference speed reduction.
• The --enable-epoch-controller option is required to enable the epoch controller unit and to generate the “command stream” as C-array tables (part of the network_ecblobs.h file).

Useful options

• Each optimization can be done one-by-one. Using -A,--Oauto explores all combinations possible and assess which one seems to provide the “best” that is, fastest results
• Adding --d-auto n (with 0<n<3: provides debug information about automatic optimization done by -A,--Oauto)
• When using -A,--Oauto, --Omax-ca-pipe 0 should be added to ensure that the exploration of the configurations is optimal.

Introduction

Memory-pool descriptor files are used to give the NPU compiler a vision of the memory characteristics and mapping that is free to use for the compiler on the embedded system.

When the NPU compiler can use two memory locations to fit, for example, an activation buffer, it decides which location is ‘the best’ based on estimated data access time. For this allocation algorithm to perform well, it is then required to describe the memory with as much accuracy as possible.
The example mpool files are written for STM32N6 and contain descriptions for each internal memory cuts + external memories accessed through XSPI:

• Disposable AXISRAM is definitely slower than NPU-RAMs. The NPU compiler prefers using NPU-RAMs to allocate buffers and uses the AXISRAM if the NPU-RAMs are full.
• External memories are even slower than AXISRAM. NPU compiler allocates data in external memories as a last-resort.

Additional characteristics can be specified for each memory region defined in the memory pool file:

• rights: Read/write accesses are required for each defined region. Read-only regions contain weights, while write regions contain activations and, as a last resort, weights if read-only pools are full.
• constants_preferred: A cue can be added for each region to advise the NPU compiler to preferentially use a given pool for storing constants (weights).. This overrides speed considerations addressed above.
• chacheable: Cacheability can be enabled for each region.

Finally, if applicable, the cache structure and characteristics should be defined in the mpool file, this gives the NPU compiler all the information it needs to optimize as much as possible the inference time.

Example

Below is an example of a memory-pool file used as a default configuration in the STEdgeaAI tool. The syntax of the mpool file is strict JSON: no comments are allowed.

This example can be used as a base for variations, as it contains the mandatory section of such a file:

• memory: Definition of the memory characteristics described above
  • cacheinfo: cache characteristics
  • mem_file_prefix: prefix for the memory initializer files generated by the NPU compiler.
  • mempools: the memory-pools description

{
    "memory": {
        "cacheinfo": [
            {
                "nlines": 512,
                "linesize": 64,
                "associativity": 8,
                "bypass_enable": 1,
                "prop": { "rights": "ACC_WRITE",  "throughput": "MID",   "latency": "MID", "byteWidth": 8,
                          "freqRatio": 2.50, "read_power": 13.584, "write_power": 12.645 }
            }
        ],
        "mem_file_prefix": "atonbuf",
        "mempools": [
            {
                "fname": "AXIFLEXMEM",
                "name":  "flexMEM",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "MID",  "latency": "MID", "byteWidth": 8,
                            "freqRatio": 2.50, "read_power": 9.381,  "write_power": 8.569 },
                "offset": { "value": "0x34000000", "magnitude":  "BYTES" },
                "size":   { "value": "0",        "magnitude": "KBYTES" }
            },
            {
                "fname": "AXISRAM1",
                "name":  "cpuRAM1",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "MID",  "latency": "MID", "byteWidth": 8,
                            "freqRatio": 2.50, "read_power": 16.616, "write_power": 14.522 },
                "offset": { "value": "0x34064000", "magnitude":  "BYTES" },
                "size":   { "value": "0",        "magnitude": "KBYTES" }
            },
            {
                "fname": "AXISRAM2",
                "name":  "cpuRAM2",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "MID",  "latency": "MID", "byteWidth": 8,
                            "freqRatio": 2.50, "read_power": 17.324, "write_power": 15.321 },
                "offset": { "value": "0x34100000", "magnitude":  "BYTES" },
                "size":   { "value": "1024",       "magnitude": "KBYTES" }
            },
            {
                "fname": "AXISRAM3",
                "name":  "npuRAM3",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "HIGH", "latency": "LOW", "byteWidth": 8,
                            "freqRatio": 1.25, "read_power": 18.531, "write_power": 16.201 },
                "offset": { "value": "0x34200000", "magnitude":  "BYTES" },
                "size":   { "value": "448",        "magnitude": "KBYTES" }
            },
            {
                "fname": "AXISRAM4",
                "name":  "npuRAM4",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "HIGH", "latency": "LOW", "byteWidth": 8,
                            "freqRatio": 1.25, "read_power": 18.531, "write_power": 16.201 },
                "offset": { "value": "0x34270000", "magnitude":  "BYTES" },
                "size":   { "value": "448",        "magnitude": "KBYTES" }
            },
            {
                "fname": "AXISRAM5",
                "name":  "npuRAM5",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "HIGH", "latency": "LOW", "byteWidth": 8,
                            "freqRatio": 1.25, "read_power": 18.531, "write_power": 16.201 },
                "offset": { "value": "0x342e0000", "magnitude":  "BYTES" },
                "size":   { "value": "448",        "magnitude": "KBYTES" }
            },
            {
                "fname": "AXISRAM6",
                "name":  "npuRAM6",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "HIGH", "latency": "LOW", "byteWidth": 8,
                            "freqRatio": 1.25, "read_power": 19.006, "write_power": 15.790 },
                "offset": { "value": "0x34350000", "magnitude":  "BYTES" },
                "size":   { "value": "448",        "magnitude": "KBYTES" }
            },
            {
                "fname": "xSPI1",
                "name":  "hyperRAM",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_WRITE", "throughput": "MID", "latency": "HIGH", "byteWidth": 2,
                            "freqRatio": 5.00, "cacheable": "CACHEABLE_ON","read_power": 380, "write_power": 340.0,
                            "constants_preferred": "true" },
                "offset": { "value": "0x90000000", "magnitude":  "BYTES" },
                "size":   { "value": "32",         "magnitude": "MBYTES" }
            },
            {
                "fname": "xSPI2",
                "name":  "octoFlash",
                "fformat": "FORMAT_RAW",
                "prop":   { "rights": "ACC_READ",  "throughput": "MID", "latency": "HIGH", "byteWidth": 1,
                            "freqRatio": 6.00, "cacheable": "CACHEABLE_ON", "read_power": 110, "write_power": 400.0,
                            "constants_preferred": "true" },
                "offset": { "value": "0x70000000", "magnitude":  "BYTES" },
                "size":   { "value": "64",         "magnitude": "MBYTES" }
            }
        ]
    }
}

Syntax description and main options

Virtual mem-pools considerations

The --enable-virtual-mem-pools allows the compiler to merge contiguous memory pool into a single, virtual pool.

In the example above, the pools cpuRAM2, npuRAM3-npuRAM6 are all contiguous (that is, the end of cpuRAM2 is the start of npuRAM3). When specifying --enable-virtual-mem-pools, the NPU compiler considers all those pools as a single one, and thus will, if needed to allocate buffers that span on multiple “concrete” pools.
This is, in a way, equivalent to create only one large entry in the memory pool file that would merge all the first memory pools.

Using --enable-virtual-mem-pools enables the user to conveniently use the same .mpool file to fit different models with different memory requirements.
This, however comes at the cost that if merged memory characteristics are very different, the resulting virtual pool inherits the worst characteristics of them, and thus it may result in suboptimal allocation of buffers. (When allocating buffers to the virtual memory pool, the underlying concrete pools characteristics are not considered.)

To overcome this issue, it might be useful to add holes (for example, 1 byte) between ranges of memory that have very different characteristics, this prevents the compiler from merging heterogenous pools.

--st-neural-art STR

Argument syntax: empty or <profile>[@<compilation-profiles-file-path>]

Set a profile from a compilation profile file. This option is mandatory to enable the passes allowing to analyze/generate a model.

Front-end options

The following options impact the generated files.

Compilation Profiles JSON File

The following JSON snippet provides a typical compilation profiles json file. This is not strict JSON format. The comments are allowed and start with //.

{
    "Globals": {
    // This section is required but can be empty
    },
    "Profiles": {
        // Minimal options required for the generated file to work
        "minimal" : {
            "memory_pool": "./mpools/stm32n6.mpool",
            "options": "--native-float --mvei --cache-maintenance --Ocache-opt"
        },
        // Advised minimal options
        "default" : {
            "memory_pool": "./mpools/stm32n6.mpool",
            "options": "--native-float --mvei --cache-maintenance --Ocache-opt --enable-virtual-mem-pools --Os"
        },
        "test" : {
            "memory_pool": "./mpools/stm32n6.mpool",
            "options": "--native-float --mvei --cache-maintenance --Ocache-opt --enable-virtual-mem-pools --optimization 1"
        },
        "test2" : {
            "memory_pool": "./my_memory_pools/stm32n6_test.mpool",
            "options": "--native-float --mvei --cache-maintenance --Ocache-opt --enable-virtual-mem-pools\
                        --optimization 1 --Oauto-sched --enable-epoch-controller"
        }
    }
}

This file has two sections: "Globals" and "Profiles".

• The "Globals" section is used for debug purposes. It is optional, and can be removed from the profile file.
• The "Profiles" section allows defining multiple named profiles that are selectable through the '--st-neural-art' option of the ST Edge AI CLI.

A profile is a combination of:

• A path to a memory-pool descriptor file (.mpool file).
• A list of options to be passed to the NPU compiler.
• Optional: A path to a machine description file (.mdesc file) - by default, the stm32n6 description file is used.

Generic recommendations

• When using small-enough models, try to refrain from using external memories: any transaction with external memory is time-consuming!
• Remember that --enable-virtual-mem-pools, though convenient, may have massive disruptive effects on inference time.
  • When in optimization steps, try to remove this option from the compiler’s parameters.

NPU compiler options

• Experiment with options, a lot of options can be added and may provide successful (or not) effects.
• After a successful --Oauto compilation for a model, note down the “best” options used, and report them in the arguments of the compiler at next compilation, this will save compilation time during development.

Memory-pools

• Carving memory for performance-improvement of a model is a time-consuming activity.
  • Start with basic memory pools with e.g. virtual memory pools enabled.
  • Check if it seems possible to fit the whole model in internal memories.
    • If the firmware is small-enough, it is possible to evolve from the example mpool by adding space before the AXISRAM2 pool (do not forget to remove this memory space in the linker script)
  • When trying to optimize memory usage/allocation for speed, try to disjoint virtually-merged-pools (by adding small holes between them).
    • Ultimately, try not to use virtual-memory-pools.
• For development purposes
  • An external RAM can be used easily instead of an external flash (for example, by setting the size of the external flash to 0 and adding constants_preferred=true to external ram characteristics) if both types of memories are available.
  • This prevents wear of the flash, will be quicker to load weights on the board, etc.
    • Be aware that this can make the inference faster (external RAM on discovery-kit board can use hexa-SPI vs. octo-SPI for external Flash)