Skip to content

Commit

Permalink
Updated arduino/nicla_vision LABS part
Browse files Browse the repository at this point in the history
These chapters from the LABS part have been edited:
setup.qmd
image_classification.qmd
object_detection.qmd
kws.qmd
motion_classification.qmd
  • Loading branch information
hzeljko committed Mar 2, 2025
1 parent f8282ef commit f9ad586
Show file tree
Hide file tree
Showing 151 changed files with 491 additions and 298 deletions.

Large diffs are not rendered by default.

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/MFCC.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/code_ide.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/dataset.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/deploy.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/ei_MFCC.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/feat_expl.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/files.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/impulse.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/model.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/pa_block.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/pers_ass.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/phone.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/test.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/upload.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/yes.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/jpg/yes_no.jpg
Binary file modified contents/labs/arduino/nicla_vision/kws/images/png/hey_google.png
Binary file modified contents/labs/arduino/nicla_vision/kws/images/png/image17.png
132 changes: 80 additions & 52 deletions contents/labs/arduino/nicla_vision/kws/kws.qmd

Large diffs are not rendered by default.

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion contents/labs/arduino/nicla_vision/nicla_vision.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@

These labs provide a unique opportunity to gain practical experience with machine learning (ML) systems. Unlike working with large models requiring data center-scale resources, these exercises allow you to directly interact with hardware and software using TinyML. This hands-on approach gives you a tangible understanding of the challenges and opportunities in deploying AI, albeit at a tiny scale. However, the principles are largely the same as what you would encounter when working with larger systems.

![Nicla Vision. Source: Arduino](./images/jpg/nicla_vision_quarter.jpeg)
![Nicla Vision. Source: Arduino](./images/jpg/nicla_vision_quarter.jpeg){width=70%}

## Pre-requisites

Expand Down

Large diffs are not rendered by default.

Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image14.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image15.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image19.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image2.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image20.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image22.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/jpg/image29.jpg
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image1.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image10.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image12.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image16.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image17.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image21.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image25.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image27.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image28.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image3.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image4.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image5.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image6.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image7.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image8.png
Binary file modified contents/labs/arduino/nicla_vision/setup/images/png/image9.png
123 changes: 81 additions & 42 deletions contents/labs/arduino/nicla_vision/setup/setup.qmd
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,8 @@ bibliography: setup.bib

The [Arduino Nicla Vision](https://docs.arduino.cc/hardware/nicla-vision) (sometimes called *NiclaV*) is a development board that includes two processors that can run tasks in parallel. It is part of a family of development boards with the same form factor but designed for specific tasks, such as the [Nicla Sense ME](https://www.bosch-sensortec.com/software-tools/tools/arduino-nicla-sense-me/) and the [Nicla Voice](https://store-usa.arduino.cc/products/nicla-voice?_gl=1*l3abc6*_ga*MTQ3NzE4Mjk4Mi4xNjQwMDIwOTk5*_ga_NEXN8H46L5*MTY5NjM0Mzk1My4xMDIuMS4xNjk2MzQ0MjQ1LjAuMC4w). The *Niclas* can efficiently run processes created with TensorFlow Lite. For example, one of the cores of the NiclaV runs a computer vision algorithm on the fly (inference), while the other executes low-level operations like controlling a motor and communicating or acting as a user interface. The onboard wireless module allows the management of WiFi and Bluetooth Low Energy (BLE) connectivity simultaneously.

![](images/jpg/image29.jpg)
\noindent
![](images/jpg/image29.jpg){width="70%" fig-align="center"}

## Hardware

Expand All @@ -26,11 +27,12 @@ The central processor is the dual-core [STM32H747,](https://content.arduino.cc/a

- TensorFlow Lite

![](images/jpg/image22.jpg)
\noindent
![](images/jpg/image22.jpg){width="60%" fig-align="center"}

### Memory

Memory is crucial for embedded machine learning projects. The NiclaV board can host up to 16 MB of QSPI Flash for storage. However, it is essential to consider that the MCU SRAM is the one to be used with machine learning inferences; the STM32H747 is only 1MB, shared by both processors. This MCU also has incorporated 2MB of FLASH, mainly for code storage.
Memory is crucial for embedded machine learning projects. The NiclaV board can host up to 16 MB of QSPI Flash for storage. However, it is essential to consider that the MCU SRAM is the one to be used with machine learning inferences; the STM32H747 is only 1 MB, shared by both processors. This MCU also has incorporated 2 MB of FLASH, mainly for code storage.

### Sensors

Expand All @@ -46,11 +48,13 @@ Memory is crucial for embedded machine learning projects. The NiclaV board can h

Start connecting the board (*microUSB*) to your computer:

![](images/jpg/image14.jpg)
\noindent
![](images/jpg/image14.jpg){width="70%" fig-align="center"}

Install the Mbed OS core for Nicla boards in the Arduino IDE. Having the IDE open, navigate to `Tools > Board > Board Manager`, look for Arduino Nicla Vision on the search window, and install the board.

![](images/jpg/image2.jpg)
\noindent
![](images/jpg/image2.jpg){width="80%" fig-align="center"}

Next, go to `Tools > Board > Arduino Mbed OS Nicla Boards` and select `Arduino Nicla Vision`. Having your board connected to the USB, you should see the Nicla on Port and select it.

Expand All @@ -60,33 +64,40 @@ Next, go to `Tools > Board > Arduino Mbed OS Nicla Boards` and select `Arduino N

On Arduino IDE, go to `Examples > PDM > PDMSerialPlotter`, open and run the sketch. Open the Plotter and see the audio representation from the microphone:

![](images/png/image9.png)
\noindent
![](images/png/image9.png){width="80%" fig-align="center"}

> Vary the frequency of the sound you generate and confirm that the mic is working correctly.
### Testing the IMU

Before testing the IMU, it will be necessary to install the LSM6DSOX library. For that, go to Library Manager and look for LSM6DSOX. Install the library provided by Arduino:

![](images/jpg/image19.jpg)
\noindent
![](images/jpg/image19.jpg){width="95%" fig-align="center"}

Next, go to `Examples > Arduino_LSM6DSOX > SimpleAccelerometer` and run the accelerometer test (you can also run Gyro and board temperature):

![](images/png/image28.png)
\clearpage
\noindent
![](images/png/image28.png){width="70%" fig-pos="htb" fig-align="center"}

### Testing the ToF (Time of Flight) Sensor

As we did with IMU, it is necessary to install the VL53L1X ToF library. For that, go to Library Manager and look for VL53L1X. Install the library provided by Pololu:

![](images/jpg/image15.jpg)
\noindent
![](images/jpg/image15.jpg){width="80%" fig-align="center"}

Next, run the sketch [proximity_detection.ino](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Arduino-IDE/proximity_detection/proximity_detection.ino):

![](images/png/image12.png)
\noindent
![](images/png/image12.png){width="70%" fig-align="center"}

On the Serial Monitor, you will see the distance from the camera to an object in front of it (max of 4m).
On the Serial Monitor, you will see the distance from the camera to an object in front of it (max of 4 m).

![](images/jpg/image13.jpg)
\noindent
![](images/jpg/image13.jpg){width="70%" fig-align="center"}

### Testing the Camera

Expand All @@ -100,72 +111,89 @@ OpenMV IDE is the premier integrated development environment with OpenMV Cameras

Go to the [OpenMV IDE page](https://openmv.io/pages/download), download the correct version for your Operating System, and follow the instructions for its installation on your computer.

![](images/png/image21.png)
\noindent
![](images/png/image21.png){width="70%" fig-align="center"}

The IDE should open, defaulting to the helloworld_1.py code on its Code Area. If not, you can open it from `Files > Examples > HelloWord > helloword.py`

![](images/png/image7.png)
\noindent
![](images/png/image7.png){width="75%" fig-align="center"}

Any messages sent through a serial connection (using print() or error messages) will be displayed on the **Serial Terminal** during run time. The image captured by a camera will be displayed in the **Camera Viewer** Area (or Frame Buffer) and in the Histogram area, immediately below the Camera Viewer.

> Before connecting the Nicla to the OpenMV IDE, ensure you have the latest bootloader version. Go to your Arduino IDE, select the Nicla board, and open the sketch on `Examples > STM_32H747_System STM32H747_manageBootloader`. Upload the code to your board. The Serial Monitor will guide you.
After updating the bootloader, put the Nicla Vision in bootloader mode by double-pressing the reset button on the board. The built-in green LED will start fading in and out. Now return to the OpenMV IDE and click on the connect icon (Left ToolBar):

![](images/jpg/image23.jpg)
\noindent
![](images/jpg/image23.jpg){width="65%" fig-align="center"}

A pop-up will tell you that a board in DFU mode was detected and ask how you would like to proceed. First, select `Install the latest release firmware (vX.Y.Z)`. This action will install the latest OpenMV firmware on the Nicla Vision.

![](images/png/image10.png)
\noindent
![](images/png/image10.png){width="65%" fig-align="center"}

You can leave the option `Erase internal file system` unselected and click `[OK]`.

Nicla's green LED will start flashing while the OpenMV firmware is uploaded to the board, and a terminal window will then open, showing the flashing progress.

![](images/png/image5.png)
\noindent
![](images/png/image5.png){width="70%" fig-align="center"}

Wait until the green LED stops flashing and fading. When the process ends, you will see a message saying, "DFU firmware update complete!". Press `[OK]`.

![](images/png/image1.png)
\noindent
![](images/png/image1.png){width="45%" fig-align="center"}

A green play button appears when the Nicla Vison connects to the Tool Bar.

![](images/jpg/image18.jpg)
\noindent
![](images/jpg/image18.jpg){width="70%" fig-align="center"}

Also, note that a drive named "NO NAME" will appear on your computer.:
Also, note that a drive named "NO NAME" will appear on your computer.

![](images/png/image3.png)
\noindent
![](images/png/image3.png){width="80%" fig-align="center"}

Every time you press the `[RESET]` button on the board, it automatically executes the *main.py* script stored on it. You can load the [main.py](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/main.py) code on the IDE (`File > Open File...`).

![](images/png/image16.png)
\noindent
![](images/png/image16.png){width="70%" fig-align="center"}

> This code is the "Blink" code, confirming that the HW is OK.
For testing the camera, let's run *helloword_1.py*. For that, select the script on `File > Examples > HelloWorld > helloword.py`,

When clicking the green play button, the MicroPython script (*hellowolrd.py*) on the Code Area will be uploaded and run on the Nicla Vision. On-Camera Viewer, you will start to see the video streaming. The Serial Monitor will show us the FPS (Frames per second), which should be around 14fps.

![](images/png/image6.png)
\noindent
![](images/png/image6.png){width="80%" fig-align="center"}

Here is the [helloworld.py](http://helloworld.py/) script:

``` python
# Hello World Example 2
#
# Welcome to the OpenMV IDE! Click on the green run arrow button below to run the script!
# Welcome to the OpenMV IDE! Click on the green run
# arrow button below to run the script!

import sensor, image, time

sensor.reset() # Reset and initialize the sensor.
sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565 (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # Set frame size to QVGA (320x240)
sensor.skip_frames(time = 2000) # Wait for settings take effect.
clock = time.clock() # Create a clock object to track the FPS.
sensor.reset() # Reset and initialize
# the sensor.
sensor.set_pixformat(sensor.RGB565) # Set pixel format to RGB565
# (or GRAYSCALE)
sensor.set_framesize(sensor.QVGA) # Set frame size to
# QVGA (320x240)
sensor.skip_frames(time = 2000) # Wait for settings take
# effect.
clock = time.clock() # Create a clock object
# to track the FPS.

while(True):
clock.tick() # Update the FPS clock.
img = sensor.snapshot() # Take a picture and return the image.
img = sensor.snapshot() # Take a picture and return
# the image.
print(clock.fps())
```

Expand Down Expand Up @@ -193,27 +221,32 @@ Edge Impulse officially supports the Nicla Vision. So, for starting, please crea

- Open the zip file on your computer and select the uploader corresponding to your OS:

![](images/png/image17.png)
\noindent
![](images/png/image17.png){width="75%" fig-align="center"}

- Put the Nicla-Vision on Boot Mode, pressing the reset button twice.

- Execute the specific batch code for your OS for uploading the binary *arduino-nicla-vision.bin* to your board.

Go to your project on the Studio, and on the `Data Acquisition tab`, select `WebUSB` (1). A window will pop up; choose the option that shows that the `Nicla is paired` (2) and press `[Connect]` (3).

![](images/png/image27.png)
\noindent
![](images/png/image27.png){width="75%" fig-align="center"}

In the *Collect Data* section on the `Data Acquisition` tab, you can choose which sensor data to pick.

![](images/png/image25.png)
\noindent
![](images/png/image25.png){width="75%" fig-align="center"}

For example. `IMU data`:

![](images/png/image8.png)
\noindent
![](images/png/image8.png){width="75%" fig-align="center"}

Or Image (`Camera`):

![](images/png/image4.png)
\noindent
![](images/png/image4.png){width="75%" fig-align="center"}

And so on. You can also test an external sensor connected to the `ADC` (Nicla pin 0) and the other onboard sensors, such as the microphone and the ToF.

Expand All @@ -225,15 +258,18 @@ The shield has 14 Grove connectors: five single analog inputs (A0-A5), one doubl

> Note that all 17 Nicla Vision pins will be connected to the Shield Groves, but some Grove connections remain disconnected.
![](images/jpg/image20.jpg)
\noindent
![](images/jpg/image20.jpg){width="45%" fig-align="center"}

This shield is MKR compatible and can be used with the Nicla Vision and Portenta.

![](images/jpg/image26.jpg)
\noindent
![](images/jpg/image26.jpg){width="55%" fig-align="center"}

For example, suppose that on a TinyML project, you want to send inference results using a LoRaWAN device and add information about local luminosity. Often, with offline operations, a local low-power display such as an OLED is advised. This setup can be seen here:

![](images/jpg/image11.jpg)
\noindent
![](images/jpg/image11.jpg){width="70%" fig-align="center"}

The [Grove Light Sensor](https://wiki.seeedstudio.com/Grove-Light_Sensor/) would be connected to one of the single Analog pins (A0/PC4), the [LoRaWAN device](https://wiki.seeedstudio.com/Grove_LoRa_E5_New_Version/) to the UART, and the [OLED](https://arduino.cl/producto/display-oled-grove/) to the I2C connector.

Expand All @@ -260,14 +296,16 @@ while(True):

To verify that the UART is working, you should, for example, connect another device as the Arduino UNO, displaying "Hello Word" on the Serial Monitor. Here is the [code](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Arduino-IDE/teste_uart_UNO/teste_uart_UNO.ino).

![](images/jpg/image24.jpg)
\noindent
![](images/jpg/image24.jpg){width="50%" fig-align="center"}

Below is the *Hello World code* to be used with the I2C OLED. The MicroPython SSD1306 OLED driver (ssd1306.py), created by Adafruit, should also be uploaded to the Nicla (the ssd1306.py script can be found in [GitHub](https://github.com/Mjrovai/Arduino_Nicla_Vision/blob/main/Micropython/ssd1306.py)).

``` python
# Nicla_OLED_Hello_World - By: marcelo_rovai - Sat Sep 30 2023

#Save on device: MicroPython SSD1306 OLED driver, I2C and SPI interfaces created by Adafruit
#Save on device: MicroPython SSD1306 OLED driver,
# I2C and SPI interfaces created by Adafruit
import ssd1306

from machine import I2C
Expand All @@ -290,8 +328,9 @@ Finally, here is a simple script to read the ADC value on pin "PC4" (Nicla pin A
import pyb
from time import sleep

adc = pyb.ADC(pyb.Pin("PC4")) # create an analog object from a pin
val = adc.read() # read an analog value
adc = pyb.ADC(pyb.Pin("PC4")) # create an analog object
# from a pin
val = adc.read() # read an analog value

while (True):

Expand Down

0 comments on commit f9ad586

Please sign in to comment.