Get the model-zoo model 1
In the same directory of tpu-mlir_xxxx.tar.gz (tpu-mlir’s release package), use the following command to clone the model-zoo project:
$ git clone --depth=1 https://github.com/sophgo/model-zoo
$ cd model-zoo
$ git lfs pull --include "*.onnx,*.jpg,*.JPEG" --exclude=""
$ cd ../
If you have cloned model-zoo, you can execute the following command to synchronize the model to the latest state:
$ cd model-zoo
$ git pull
$ git lfs pull --include "*.onnx,*.jpg,*.JPEG" --exclude=""
$ cd ../
This process downloads a large amount of data from GitHub. Due to differences in specific network environments, this process may take a long time.
If you get the model-zoo test package provided by SOPHGO, you can
do the following to create and set up the model-zoo. After completing
this step, go directly to the next section Get the tpu-perf tool.
$ mkdir -p model-zoo
$ tar -xvf path/to/model-zoo_.tar.bz2 --strip-components=1 -C model-zoo
Test process
Unzip the SDK and create a Docker container
Execute the following command in the tpu-mlir_xxxx.tar.gz directory (note that tpu-mlir_xxxx.tar.gz and
model-zoo needs to be at the same level):
$ tar zxf tpu-mlir_xxxx.tar.gz
$ docker pull sophgo/tpuc_dev:latest
$ docker run --rm --name myname -v $PWD:/workspace -it sophgo/tpuc_dev:latest
After running the command, it will be in a Docker container.
Set environment variables and install tpu-perf
Complete setting the environment variables needed to run the tests with the following command:
$ cd tpu-mlir_xxxx
$ source envsetup.sh
There will be no prompts after the process ends. Then install tpu-perf with the following command:
$ pip3 install ../tpu_perf-x.x.x-py3-none-manylinux2014_x86_64.whl
Run the test
Compile the model
confg.yaml in model-zoo configures the test content of the SDK. For example, the configuration file for resnet18 is model-zoo/vision/classification/resnet18-v2/config.yaml .
Execute the following command to run all test samples:
$ cd ../model-zoo
$ python3 -m tpu_perf.build --mlir -l full_cases.txt
The following models are compiled:
* efficientnet-lite4
* mobilenet_v2
* resnet18
* resnet50_v2
* shufflenet_v2
* squeezenet1.0
* vgg16
* yolov5s
After the command is finished, you will see the newly generated output folder (where the test output is located).
Modify the properties of the output folder to make it accessible to systems outside of Docker.
$ chmod -R a+rw output
Configure SOC device
Note: If your device is a PCIE board, you can skip this section directly.
The performance test only depends on the libsophon runtime environment, so after packaging models, compiled in the toolchain compilation environment, and model-zoo, the performance test can be carried out in the SOC environment by tpu_perf. However, the complete model-zoo as well as compiled output contents may not be fully copied to the SOC since the storage on the SOC device is limited. Here is a method to run tests on SOC devices through linux nfs remote file system mounts.
First, install the nfs service on the toolchain environment server “host system”:
$ sudo apt install nfs-kernel-server
Add the following content to /etc/exports (configure the shared directory):
/the/absolute/path/of/model-zoo *(rw,sync,no_subtree_check,no_root_squash)
Where * means that everyone can access the shared directory. Moreover, it
can be configured to be accessible by a specific network segment or IP, such as:
/the/absolute/path/of/model-zoo 192.168.43.0/24(rw,sync,no_subtree_check,no_root_squash)
Then execute the following command to make the configuration take effect:
$ sudo exportfs -a
$ sudo systemctl restart nfs-kernel-server
In addition, you need to add read permissions to the images in the dataset directory:
chmod -R +r path/to/model-zoo/dataset
Install the client on the SOC device and mount the shared directory:
$ mkdir model-zoo
$ sudo apt-get install -y nfs-common
$ sudo mount -t nfs :/path/to/model-zoo ./model-zoo
In this way, the test directory is accessible in the SOC environment. The rest of the SOC test operation is basically the same as that of PCIE. Please refer to the following content for operation. The difference in command execution position and operating environment has been explained in the execution place.
Run the test
Running the test needs to be done in an environment outside Docker (it is assumed that you have installed and configured the 1684X device and driver), so you can exit the Docker environment:
$ exit
Run the following commands under the PCIE board to test the performance of the generated bmodel.
$ pip3 install ./tpu_perf-*-py3-none-manylinux2014_x86_64.whl
$ cd model-zoo
$ python3 -m tpu_perf.run --mlir -l full_cases.txt
Note: If multiple SOPHGO accelerator cards are installed on the host, you can
specify the running device of tpu_perf by adding --devices id when using
tpu_perf. Such as:
$ python3 -m tpu_perf.run --devices 2 --mlir -l full_cases.txt
The SOC device uses the following steps to test the performance of the generated bmodel.
Download the latest tpu-perf, tpu_perf-x.x.x-py3-none-manylinux2014_aarch64.whl, from https://github.com/sophgo/tpu-perf/releases to the SOC device and execute the following operations:
$ pip3 install ./tpu_perf-x.x.x-py3-none-manylinux2014_aarch64.whl
$ cd model-zoo
$ python3 -m tpu_perf.run --mlir -l full_cases.txt
After that, performance data is available in output/stats.csv, in which the running time, computing resource utilization, and bandwidth utilization of the relevant models are recorded.