Perform model validation on desktop

To start the validation process of the model, the same JSON file used for code generation can be reused. Because the operator is mapped on a Python function, the model can be run directly and generated using ST Edge AI Core known operators list.

Perform model validation on target

To perform a validation on target, the implementation has to be generated and flashed on the target, then the validation process can be executed using the custom JSON file already used for the code generation and the validation on desktop processes.

Perform code generation

This is the most complex usecase supported today in ST Edge AI Core tool. To perform a code generation in this case you will need to create a JSON config file like the Simple Custom Layer usecase. But because the layer can be anything you can define, you will need to specify a python file containing your custom Layer Python implementation (the tensorflow.keras.layers.Layer inherited class) and the C implementation of the custom layer in the JSON file.

Things here can be done in two phases:

• In a first phase you indicate only the python layer definition to ST Edge AI Core so that the tool can run the model internally and generate network files for known layers. The tool will also generate an empty template file associated with your custom layer that you need to fill with your own C implementation.
• In a second phase when your C file is ready, you re-start ST Edge AI Core with the Python and C file specified in the JSON. The code should be generated correctly in the standard output folder.

For the above example we need to create first a JSON file with the following content to indicate the python file containing the implementation of MyCustomLayer. The ‘python’ key can be specified as absolute path or relative to the JSON file as follow:

{
    "MyCustomLayer": {
        "python": "MyCustomLayerImplementation.py"
    }
}

The first key “MyCustomLayer” is the name of the custom Layer used (name of the Python class) and MyCustomLayerImplementation.py is the name of the python file containing MyCustomLayer implementation defined above.

We can perform the first phase by calling the ST Edge AI Core tool as follow:

$ stm32ai generate ~/Desktop/my_model_with_custom.h5 --custom ~/Desktop/my_model_config_file.json

The result should indicates the generated files

Generated files (6)
----------------------------------------------------------------------------
 stm32ai_output\network_custom_layers.c
 stm32ai_output\network_config.h
 stm32ai_output\network.h
 stm32ai_output\network.c
 stm32ai_output\network_data.h
 stm32ai_output\network_data.c

We can see that we have a network_custom_layers.c generated file. This is our empty template file and it should like this :

void custom_init_MyCustomLayer(ai_layer* layer)
{
  ai_layer_custom* l = ai_layer_custom_get(layer);
  ai_tensor* t_in0 = ai_layer_get_tensor_in(l, 0);
  ai_tensor* t_out0 = ai_layer_get_tensor_out(l, 0);
  /*USER CODE BEGINS HERE*/
  /*USER CODE ENDS HERE*/
  ai_layer_custom_release(layer);
}

/* Layer Forward Function #0 */
void custom_forward_MyCustomLayer(ai_layer* layer)
{
  ai_layer_custom* l = ai_layer_custom_get(layer);
  ai_tensor* t_in0 = ai_layer_get_tensor_in(l, 0);
  ai_tensor* t_out0 = ai_layer_get_tensor_out(l, 0);
  /*USER CODE BEGINS HERE*/
  /*USER CODE ENDS HERE*/
  ai_layer_custom_release(layer);
}

The template generated is filled with two functions, one to perform initialization before the network is run and the second to define the forward function to use for inferences. These functions will be called with an ai_layer parameter which allows you to retrieve information about input/output tensors and information.

Let’s now implement the C functions. The current implementation of the custom layer does not allow you to get the weights and parameter directly from the generated data array by ST Edge AI Core. This is your responsibility to set the weights and parameters in your custom layer implementation. For this example we have used Netron to get the parameters and trained weights of the custom layer.

The next section of code show you how MyCustomLayer is implemented in C:

void custom_forward_MyCustomLayer(ai_layer* layer)
{
  ai_layer_custom* l = ai_layer_custom_get(layer);
  ai_tensor* t_in0 = ai_layer_get_tensor_in(l, 0);
  ai_tensor* t_out0 = ai_layer_get_tensor_out(l, 0);
  
  /*USER CODE BEGINS HERE*/
  if(l->id == AI_LAYER_CUSTOM_MY_CUSTOM_LAYER_ID) // The first instance of MyCustomLayer
  {
    const int factor = 3; // Parameter used for the first instance of MyCustomLayer in Python
    const ai_float param2 = 0.5009999871253967; // The trained weights of the first instance of MyCustomLayer 
      
    ai_float *d_in = ai_tensor_get_data(t_in0).float32; // Data of the input tensor
    ai_float *d_out = ai_tensor_get_data(t_out0).float32; // Data of the output tensor
      
    for(int i = 0; i < ai_tensor_get_data_size(t_in0); i++){
      d_out[i] = d_in[i] * factor * param2; // Performing the layer's operation for each data
    }
  }
  
  if(l->id == AI_LAYER_CUSTOM_MY_CUSTOM_LAYER_1_ID) // The second instance of MyCustomLayer
  {
    const int factor = 5; // Parameter used for the second instance of MyCustomLayer in Python
    const ai_float param2 = 0.5009999871253967; // The trained weights of the second instance of MyCustomLayer
    
    ai_float *d_in = ai_tensor_get_data(t_in0).float32; // Data of the input tensor
    ai_float *d_out = ai_tensor_get_data(t_out0).float32; // Data of the output tensor
    
    for(int i = 0; i < ai_tensor_get_data_size(t_in0); i++){
      d_out[i] = d_in[i] * factor * param2; // Performing the layer's operation for each data
    }
  }
  /*USER CODE ENDS HERE*/
  ai_layer_custom_release(layer);
}

In the implementation of MyCustomLayer (which is simple enough), no specific computation before running was necessary, so the init function will stay empty. We can see here multiple interesting points:

• We are using layer IDs to get which layer is being computed. Because we have two instances of our custom layer with two different parameters (and weights), we need to identify which layer is being infered. This is done by fetching the layer ID “l->id” and doing different computation in the different cases. Your layers IDs are available by C defines in the template file. You can also get a better view of the different layers IDs by running the analyze command on your model with ST Edge AI Core.

• The layer parameters that were set in the python model need to be set manually:

  model.add(MyCustomLayer(3)) ==> const int factor = 3;
  [...]
  model.add(MyCustomLayer(5)) ==> const int factor = 5;

• The layer trained weights need to be set manually:

  const ai_float param2 = 0.5009999871253967;
  [...]
  const ai_float param2 = 0.5009999871253967;

  Note: Here my weights param2 have the same values but in a more concrete usecase and through backpropagation these values would have been different.

• Pointers to tensor in/out data are obtained via ai_tensor_get_data(…) and allow you to iterate though your different values. Once you have those pointers you can get the input values, do some computation and modify the output values, which represent the inference of your layer.

Once your C implementation is complete, you are 90% done. You can then complete the JSON config file by adding your C file to it:

{
    "MyCustomLayer": {
        "python": "MyCustomLayerImplementation.py",
        "c": "network_custom_layers.c"
    }
}

You are now able to perform code generation of your model through ST Edge AI Core, all your model files will be generated in the standard output folder.

Perform model validation on desktop

To perform validation on a model containing custom layers, you will need to follow the steps of the previous chapter Perform code generation which describes 95% of the process.

Prerequisites :

• Python file containing the implementation of your custom layer.
  
• C file containing your custom layer implementation
  
• Your JSON file with python and c fields pointing to correct files

To perform a validation on desktop, once your prerequisites are OK, you can use ST Edge AI Core by specifying your model and your JSON file as follow:

$ stm32ai validate ~/Desktop/my_model_with_custom.h5 --custom ~/Desktop/my_model_config_file.json

There is not big changes compared to a standard model validation.

Perform model validation on target

To perform validation on target on a model containing custom layers through ST Edge AI Core, you need to follow the steps of the previous chapter Perform code generation which describes 95% of the process.

Prerequisites :

• Python file containing the implementation of your custom layer.
• C file containing your custom layer implementation
• Your JSON file with python and c fields pointing to correct files

To perform a validation on target, once your prerequisites are OK, you can use ST Edge AI Core by specifying your model, the mode set to ‘stm32’ and your JSON file as follow:

$ stm32ai validate ~/Desktop/my_model_with_custom.h5 --custom ~/Desktop/my_model_config_file.json --mode stm32

There is not big changes compared to a standard model on target validation.