TPU-MLIR Quick Start

Legal Notices

Copyright © SOPHGO 2024. All rights reserved.

No part or all of the contents of this document may be copied, reproduced or transmitted in any form by any organization or individual without the written permission of the Company.

Attention

All products, services or features, etc. you purchased is subject to SOPHGO’s business contracts and terms. All or part of the products, services or features described in this document may not be covered by your purchase or use. Unless otherwise agreed in the contract, SOPHGO makes no representations or warranties (including express and implied) regarding the contents of this document. The contents of this document may be updated from time to time due to product version upgrades or other reasons. Unless otherwise agreed, this document is intended as a guide only. All statements, information and recommendations in this document do not constitute any warranty, express or implied.

Technical Support

Address:

Building 1, Zhongguancun Integrated Circuit Design Park (ICPARK), No. 9 Fenghao East Road, Haidian District, Beijing

Postcode:

100094

URL:

https://www.sophgo.com/

Email:

sales@sophgo.com

Tel:

010-57590723

Release Record

Version	Release date	Explanation
v1.18.0	2025.05.01	YOLO series adds automatic mixed precision setting; Added SmoothQuant option for run_calibration; New one-click compilation script for LLM
v1.17.0	2025.04.03	Significant improvement in LLM model compilation speed; TPULang supports PPL operator integration; Fixed random error issue with Trilu bf16 on Mars3
v1.16.0	2025.03.03	TPULang ROI_Extractor support; Einsum supports abcde,abfge->abcdfg pattern; LLMC adds Vila model support
v1.15.0	2025.02.05	Added LLMC quantization support; Address boundary check in codegen; Fixed several comparison issues
v1.14.0	2025.01.02	Added post-processing fusion for yolov8/v11; Support for Conv3D stride > 15; Improved FAttention accuracy
v1.13.0	2024.12.02	Streamlined Release package; Performance optimization for MaxPoolWithMask training operator; Added support for large RoPE operators
v1.12.0	2024.11.06	tpuv7-runtime cmodel integration; BM1690 multi-core LayerGroup optimization; Support for PPL backend operator development
v1.11.0	2024.09.27	Added soc mode for BM1688 tdb; bmodel supports fine-grained merging; Fixed several performance degradation issues
v1.10.0	2024.08.15	Added yolov10 support; New quantization tuning section; Optimized tpu-perf log output
v1.9.0	2024.07.16	BM1690 added 40 model regression tests; New quantization algorithms: octav, aciq_guas and aciq_laplace
v1.8.0	2024.05.30	BM1690 supports multi-core MatMul operator; TPULang supports input/output order specification; tpuperf removes patchelf dependency
v1.7.0	2024.05.15	CV186X dual-core changed to single-core; BM1690 testing process aligned with BM1684X; Support for gemma/llama/qwen models
v1.6.0	2024.02.23	Added Pypi release format; Support for user-defined Global operators; Added CV186X processor platform support
v1.5.0	2023.11.03	More Global Layer support for multi-core parallel processing
v1.4.0	2023.09.27	System dependencies upgraded to Ubuntu22.04; Added BM1684 Winograd support
v1.3.0	2023.07.27	Added manual floating-point operation region specification; Added supported frontend framework operator list; Added NNTC vs TPU-MLIR quantization comparison
v1.2.0	2023.06.14	Adjusted mixed quantization examples
v1.1.0	2023.05.26	Added post-processing using intelligent deep learning processor
v1.0.0	2023.04.10	PyTorch support, added section for PyTorch model conversion
v0.8.0	2023.02.28	Added pre-processing using intelligent deep learning processor
v0.6.0	2022.11.05	Added section for mixed precision operation process
v0.5.0	2022.10.20	Added model-zoo specification to test all models within
v0.4.0	2022.09.20	Caffe support, added section for Caffe model conversion
v0.3.0	2022.08.24	TFLite support, added section for TFLite model conversion
v0.2.0	2022.08.02	Added chapter for running test samples in SDK
v0.1.0	2022.07.29	Initial release, supports resnet/mobilenet/vgg/ssd/yolov5s with yolov5s as example case

Table of contents

• 1. TPU-MLIR Introduction
• 2. Environment Setup
  • 2.1. Basic Environment Configuration
  • 2.2. Install tpu_mlir
  • 2.3. Install the dependency of tpu_mlir
• 3. Compile the ONNX model
  • 3.1. Install tpu_mlir
  • 3.2. Prepare working directory
  • 3.3. ONNX to MLIR
  • 3.4. MLIR to F16 bmodel
  • 3.5. MLIR to INT8 bmodel
    • 3.5.1. Calibration table generation
    • 3.5.2. Compile to INT8 symmetric quantized model
  • 3.6. Effect comparison
  • 3.7. Model performance test
    • 3.7.1. Install the libsophon
    • 3.7.2. Check the performance of BModel
• 4. Compile the Torch Model
  • 4.1. Install tpu_mlir
  • 4.2. Prepare working directory
  • 4.3. TORCH to MLIR
  • 4.4. MLIR to F16 bmodel
  • 4.5. MLIR to INT8 bmodel
    • 4.5.1. Calibration table generation
    • 4.5.2. Compile to INT8 symmetric quantized model
  • 4.6. Effect comparison
• 5. Compile the Caffe model
  • 5.1. Install tpu_mlir
  • 5.2. Prepare working directory
  • 5.3. Caffe to MLIR
  • 5.4. MLIR to F32 bmodel
  • 5.5. MLIR to INT8 bmodel
    • 5.5.1. Calibration table generation
    • 5.5.2. Compile to INT8 symmetric quantized model
• 6. Compile the TFLite model
  • 6.1. Install tpu_mlir
  • 6.2. Prepare working directory
  • 6.3. TFLite to MLIR
  • 6.4. MLIR to bmodel
• 7. Quantization and optimization
  • 7.1. TPU-MLIR Full Int8 Symmetric Quantization
    • 7.1.1. run_calibration Process Introduction
    • 7.1.2. run_calibration Parameter Introduction
    • 7.1.3. The Use of run_calibration Parameters Introduction
  • 7.2. Overview of TPU-MLIR Mixed Precision Quantization
  • 7.3. pattern-match
    • 7.3.1. YOLO Series Automatic Mixed Precision Method
    • 7.3.2. Transformer Series Automatic Mixed Precision Method
  • 7.4. 1. search_qtable
    • 7.4.1. Install tpu_mlir
    • 7.4.2. Prepare working directory
    • 7.4.3. Accuracy test of float anf int8 models
    • 7.4.4. To Mix Precision Model
  • 7.5. 2. run_sensitive_layer
    • 7.5.1. Install tpu_mlir
    • 7.5.2. Prepare working directory
    • 7.5.3. Accuracy test of float anf int8 models
    • 7.5.4. To Mix Precision Model
  • 7.6. 3. fp_forward
• 8. Use Tensor Computing Processor for Preprocessing
  • 8.1. Model Deployment Example
    • 8.1.1. Deploy to BM168x
    • 8.1.2. Deploy to CV18xx
• 9. Use Tensor Computing Processor for Postprocessing
  • 9.1. Add detection post_process(YOLOv5s)
    • 9.1.1. Prepare working directory
    • 9.1.2. ONNX to MLIR
    • 9.1.3. MLIR to Bmodel
    • 9.1.4. Bmodel Verification
  • 9.2. Add segmentation post_processing(YOLOv8s_seg)
    • 9.2.1. Prepare working directory
    • 9.2.2. ONNX to MLIR
    • 9.2.3. MLIR to Bmodel
    • 9.2.4. Bmodel Verification
• 10. Compile LLM Model
  • 10.1. Overview
  • 10.2. Command-Line Arguments
  • 10.3. Example Usage
• 11. Appendix.01: Reference for converting model to ONNX format
  • 11.1. PyTorch model to ONNX
    • 11.1.1. Step 0: Create a working directory
    • 11.1.2. Step 1: Build and save the model
    • 11.1.3. Step 2: Export ONNX model
  • 11.2. TensorFlow model to ONNX
    • 11.2.1. Step 0: Create a working directory
    • 11.2.2. Step 1: Prepare and convert the model
  • 11.3. PaddlePaddle model to ONNX
    • 11.3.1. Step 0: Install openssl-1.1.1o
    • 11.3.2. Step 1: Create a working directory
    • 11.3.3. Step 2: Prepare the model
    • 11.3.4. Step 3: Convert the model
• 12. Appendix.02: CV18xx Guidance
  • 12.1. Compile yolov5 model
    • 12.1.1. Install tpu_mlir
    • 12.1.2. Prepare working directory
    • 12.1.3. ONNX to MLIR
    • 12.1.4. MLIR to BF16 Model
    • 12.1.5. MLIR to INT8 Model
    • 12.1.6. Result Comparison
  • 12.2. Merge cvimodel Files
    • 12.2.1. Step 0: generate the cvimodel for batch 1
    • 12.2.2. Step 1: generate the cvimodel for batch 2
    • 12.2.3. Step 2: merge the cvimodel of batch 1 and batch 2
    • 12.2.4. Step 3: use the cvimodel through the runtime interface
    • 12.2.5. Overview:
  • 12.3. Compile and Run the Runtime Sample
    • 12.3.1. 1) Run the provided pre-build samples
    • 12.3.2. 2) Cross-compile samples
    • 12.3.3. 3) Run samples in docker environment
  • 12.4. FAQ
    • 12.4.1. Model transformation FAQ
    • 12.4.2. Model performance evaluation FAQ
    • 12.4.3. Common problems of model deployment
    • 12.4.4. Others
• 13. Appendix.03: BM168x Guidance
  • 13.1. Merge bmodel Files
    • 13.1.1. Step 0: generate the bmodel for batch 1
    • 13.1.2. Step 1: generate the bmodel for batch 2
    • 13.1.3. Step 2: merge the bmodel of batch 1 and batch 2
    • 13.1.4. Overview:
• 14. Appendix.04: Model-zoo test
  • 14.1. Configure the system environment
  • 14.2. Get the model-zoo model
  • 14.3. Prepare the runtime environment
  • 14.4. Configure SOC device
  • 14.5. Prepare dataset
    • 14.5.1. ImageNet
    • 14.5.2. COCO (optional)
    • 14.5.3. Vid4 (optional)
  • 14.6. Prepare the toolchain compilation environment
  • 14.7. Install tpu-perf tool
  • 14.8. Model performance and accuracy testing process
    • 14.8.1. Compile the model
    • 14.8.2. Performance test
    • 14.8.3. Precision test
• 15. Appendix.05：TPU Profile Tool Guidance
  • 15.1. Compile Bmodel
  • 15.2. Generate Raw Profile Data
  • 15.3. Visualize Profile Data
• 16. Appendix.06: TDB Guidance
  • 16.1. Preparatory work
  • 16.2. Start TDB
  • 16.3. TDB command summary
  • 16.4. TDB usage process
  • 16.5. TDB function description
    • 16.5.1. next feature
    • 16.5.2. breakpoint feature
    • 16.5.3. info feature
    • 16.5.4. print feature
    • 16.5.5. watchpoint feature
    • 16.5.6. py feature
  • 16.6. BModel Disassembler
  • 16.7. BModel Checker
• 17. Appendix.07: Supported Operations
  • 17.1. List of operators currently supported by TPU-MLIR