14. Appendix.04: Model-zoo test
14.1. Configure the system environment
If you are using Docker for the first time, use the methods in Environment Setup to install and configure Docker. At the same time, git-lfs will be used in this chapter. If you use git-lfs for the first time, you need execute the following commands for installation and configuration in the user’s own system (not in Docker container).
$ curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
$ sudo apt-get install git-lfs
14.2. Get the model-zoo model
In your working directory, get the model-zoo test package from the SDK package provided by SOPHGO, then create and set up model-zoo as follows:
$ mkdir -p model-zoo
$ tar -xvf path/to/model-zoo_<date>.tar.bz2 --strip-components=1 -C model-zoo
The directory structure of model-zoo is as follows:
├── config.yaml
├── requirements.txt
├── dataset
├── harness
├── output
└── ...
config.yaml: contains generic configuration: dataset directory, model root directory, etc., as well as some reused parameters and commands
requirements.txt: contains python dependencies for model-zoo.
dataset: directory contains the datasets of the models in modelzoo, which will be called by tpu_perf as plugins.
output: directory will be used to store the compiled output bmodel and some intermediate data.
The other directories contain information and configuration for each model. The directory corresponding to each model has a config.yaml file, which configures the model’s name, path and FLOPs, dataset production parameters, and the model’s quantization compilation commands.
14.3. Prepare the runtime environment
Install the dependencies needed to run model-zoo on your system (outside of the Docker container):
# for ubuntu operating system
$ sudo apt install build-essential
$ sudo apt install python3-dev
$ sudo apt install -y libgl1
# for centos operating system
$ sudo yum install make automake gcc gcc-c++ kernel-devel
$ sudo yum install python-devel
$ sudo yum install mesa-libGL
# accuracy tests require the following operations to be performed, performance tests can be performed without, it is recommended to use Anaconda to create a virtual environment of python 3.7 or above
$ cd path/to/model-zoo
$ pip3 install -r requirements.txt
In addition, tpu hardware needs to be invoked for performance and accuracy tests, so please install the runtime environment for the TPU hardware.
14.4. Configure SOC device
Note: If your device is a PCIE board, you can skip this section directly.
The performance test only depends on the runtime environment for the TPU hardware, so after packaging models, compiled in the toolchain compilation environment, and model-zoo, the performance test can be carried out in the SOC environment by tpu_perf. However, the complete model-zoo as well as compiled output contents may not be fully copied to the SOC since the storage on the SOC device is limited. Here is a method to run tests on SOC devices through linux nfs remote file system mounts.
First, install the nfs service on the toolchain environment server “host system”:
$ sudo apt install nfs-kernel-server
Add the following content to /etc/exports (configure the shared directory):
/the/absolute/path/of/model-zoo *(rw,sync,no_subtree_check,no_root_squash)
Where * means that everyone can access the shared directory. Moreover, it
can be configured to be accessible by a specific network segment or IP, such as:
/the/absolute/path/of/model-zoo 192.168.43.0/24(rw,sync,no_subtree_check,no_root_squash)
Then execute the following command to make the configuration take effect:
$ sudo exportfs -a
$ sudo systemctl restart nfs-kernel-server
In addition, you need to add read permissions to the images in the dataset directory:
$ chmod -R +r path/to/model-zoo/dataset
Install the client on the SOC device and mount the shared directory:
$ mkdir model-zoo
$ sudo apt-get install -y nfs-common
$ sudo mount -t nfs <IP>:/path/to/model-zoo ./model-zoo
In this way, the test directory is accessible in the SOC environment. The rest of the SOC test operation is basically the same as that of PCIE. Please refer to the following content for operation. The difference in command execution position and operating environment has been explained in the execution place.
14.5. Prepare dataset
14.5.1. ImageNet
Download ImageNet 2012 Dataset 。
After unzipping, move the data under Data/CLS_LOC/val to a directory like model-zoo:
$ cd path/to/sophon/model-zoo
$ mkdir -p dataset/ILSVRC2012/ILSVRC2012_img_val
$ mv path/to/imagenet-object-localization-challenge/Data/CLS_LOC/val dataset/ILSVRC2012/ILSVRC2012_img_val
# It is also possible to map the dataset directory to dataset/ILSVRC2012/ILSVRC2012_img_val through the soft link ln -s
14.5.2. COCO (optional)
If the precision test uses the coco dataset (networks trained with coco such as yolo), please download and unzip it as follows:
$ cd path/to/model-zoo/dataset/COCO2017/
$ wget http://images.cocodataset.org/annotations/annotations_trainval2017.zip
$ wget http://images.cocodataset.org/zips/val2017.zip
$ unzip annotations_trainval2017.zip
$ unzip val2017.zip
14.5.3. Vid4 (optional)
If you need precision test on BasicVSR, please download and unzip the Vid4 dataset as follows:
$ pip3 install gdown
$ cd path/to/model-zoo/dataset/basicvsr/
$ gdown https://drive.google.com/open?id=1ZuvNNLgR85TV_whJoHM7uVb-XW1y70DW --fuzzy
$ unzip -o Vid4.zip -d eval
14.6. Prepare the toolchain compilation environment
It is recommended to use the toolchain software in a docker environment, see Base environment configuration to install Docker. and execute the following commands in your working directory (the directory which model-zoo is located) to create a Docker container:
$ docker pull sophgo/tpuc_dev:v3.4
$ docker run --name myname -v $PWD:/workspace -it sophgo/tpuc_dev:v3.4
If you want to keep the container after it exits, simply remove the --rm parameter:
$ docker run --name myname -v $PWD:/workspace -it sophgo/tpuc_dev:v3.4 --rm
After running the command, it will be in a Docker container. You can the latest tpu-mlir wheel installation package from the SDK package provided by SOPHGO, such as tpu_mlir-*-py3-none-any.whl. Install tpu_mlir in the Docker container:
$ pip install tpu_mlir-*-py3-none-any.whl[all]
14.7. Install tpu-perf tool
Get the latest tpu-perf wheel installer from the SDK package provided by SOPHGO. For example, tpu_perf-x.x.x-py3-none-manylinux2014_x86_64.whl .
You need to install tpu-perf both inside and outside of Docker:
# go to Docker and install tpu-perf
$ pip3 install path/to/tpu_perf-x.x.x-py3-none-manylinux2014_x86_64.whl
14.8. Model performance and accuracy testing process
14.8.1. Compile the model
The model compilation process needs to be done within Docker, where tpu_mlir and tpu_perf need to be installed as described above.
confg.yaml in model-zoo configures the test content of the SDK. For example, the configuration file for resnet18 is model-zoo/vision/classification/resnet18-v2/mlir.config.yaml .
Execute the following command to compile the resnet18-v2 model:
$ cd ../model-zoo
$ python3 -m tpu_perf.build --target BM1684X --mlir vision/classification/resnet18-v2/mlir.config.yaml
where the --target is used to specify the processor model, which currently supports BM1684 , BM1684X , BM1688 , BM1690 and CV186X .
Execute the following command to compile all the high priority test samples:
$ cd ../model-zoo
$ python3 -m tpu_perf.build --target BM1684X --mlir -l full_cases.txt --priority_filter high
Full compilation may require reserving more than 2T of space, please adjust according to actual conditions. The --clear_if_success parameter can be used to delete intermediate files after successful compilation, saving space.
The following high priority models will be compiled (Due to continuous additions of models in the model-zoo, only a partial list of models is provided here):
* efficientnet-lite4
* mobilenetv2
* resnet18-v2
* resnet50-v2
* shufflenet_v2
* squeezenet1.0
* vgg16
* yolov5s
* ...
After the command is finished, you will see the newly generated output folder. This compilation result can be used for performance and accuracy testing without recompilation. But you need modify the properties of the output folder to make it accessible to systems outside of Docker:
$ chmod -R a+rw output
14.8.2. Performance test
Running the test needs to be done in an environment outside Docker, it is assumed that you have installed and configured the runtime environment for the TPU hardware, so you can exit the Docker environment:
$ exit
PCIE board
Run the following commands under the PCIE board to test the performance of the generated high priority model bmodel :
$ pip3 install path/to/tpu_perf-x.x.x-py3-none-manylinux2014_x86_64.whl
$ cd model-zoo
$ python3 -m tpu_perf.run --target BM1684X --mlir -l full_cases.txt --priority_filter high
where the --target is used to specify the processor model, which currently supports BM1684 , BM1684X , BM1688 , BM1690 and CV186X .
Note: If multiple SOPHGO accelerator cards are installed on the host, you can
specify the running device of tpu_perf by adding --devices id when using
tpu_perf. Such as:
$ python3 -m tpu_perf.run --target BM1684X --devices 2 --mlir -l full_cases.txt --priority_filter high
SOC device
The SOC device uses the following steps to test the performance of the generated high priority model bmodel.
Get the latest tpu-perf wheel installer from the SDK package provided by SOPHGO. For example, tpu_perf-x.x.x-py3-none-manylinux2014_aarch64.whl, then transfer the file to the SOC device and execute the following operations:
$ pip3 install path/to/tpu_perf-x.x.x-py3-none-manylinux2014_aarch64.whl
$ cd model-zoo
$ python3 -m tpu_perf.run --target BM1684X --mlir -l full_cases.txt --priority_filter high
Output results
After that, performance data is available in output/stats.csv, in which the running time, computing resource utilization, and bandwidth utilization of the relevant models are recorded. The performance test results for resnet18-v2 as follows:
name,prec,shape,gops,time(ms),mac_utilization,ddr_utilization,processor_usage
resnet18-v2,FP32,1x3x224x224,3.636,6.800,26.73%,10.83%,3.00%
resnet18-v2,FP16,1x3x224x224,3.636,1.231,18.46%,29.65%,2.00%
resnet18-v2,INT8,1x3x224x224,3.636,0.552,20.59%,33.20%,3.00%
resnet18-v2,FP32,4x3x224x224,14.542,26.023,27.94%,3.30%,3.00%
resnet18-v2,FP16,4x3x224x224,14.542,3.278,27.73%,13.01%,2.00%
resnet18-v2,INT8,4x3x224x224,14.542,1.353,33.59%,15.46%,2.00%
14.8.3. Precision test
Running the test needs to be done in an environment outside Docker, it is assumed that you have installed and configured the runtime environment for the TPU hardware, so you can exit the Docker environment:
$ exit
Run the following commands under the PCIE board to test the precision of the generated high priority model bmodel :
$ pip3 install path/to/tpu_perf-x.x.x-py3-none-manylinux2014_x86_64.whl
$ cd model-zoo
$ python3 -m tpu_perf.precision_benchmark --target BM1684X --mlir -l full_cases.txt --priority_filter high
where the --target is used to specify the processor model, which currently supports BM1684 , BM1684X , BM1688 , BM1690 and CV186X .
Note: If multiple SOPHGO accelerator cards are installed on the host, you can
specify the running device of tpu_perf by adding --devices id when using
tpu_perf. Such as:
$ python3 -m tpu_perf.precision_benchmark --target BM1684X --devices 2 --mlir -l full_cases.txt --priority_filter high
Specific parameter descriptions can be obtained with the following commands:
$ python3 -m tpu_perf.precision_benchmark --help
The output precision data is available in output/topk.csv . The precision results for resnet18-v2:
name,top1,top5
resnet18-v2-FP32,69.68%,89.23%
resnet18-v2-INT8,69.26%,89.08%