TPU-MLIR Technical Reference Manual

Legal Notices

Copyright © SOPHGO 2025. All rights reserved.

No part or all of the contents of this document may be copied, reproduced or transmitted in any form by any organization or individual without the written permission of the Company.

Attention

All products, services or features, etc. you purchased is subject to SOPHGO’s business contracts and terms. All or part of the products, services or features described in this document may not be covered by your purchase or use. Unless otherwise agreed in the contract, SOPHGO makes no representations or warranties (including express and implied) regarding the contents of this document. The contents of this document may be updated from time to time due to product version upgrades or other reasons. Unless otherwise agreed, this document is intended as a guide only. All statements, information and recommendations in this document do not constitute any warranty, express or implied.

Technical Support

Address:

Building 1, Zhongguancun Integrated Circuit Design Park (ICPARK), No. 9 Fenghao East Road, Haidian District, Beijing

Postcode:

100094

URL:

https://www.sophgo.com/

Email:

sales@sophgo.com

Tel:

010-57590723

Release Record

Version	Release date	Explanation
v1.18.0	2025.05.01	YOLO series adds automatic mixed precision setting; Added SmoothQuant option for run_calibration; New one-click compilation script for LLM
v1.17.0	2025.04.03	Significant improvement in LLM model compilation speed; TPULang supports PPL operator integration; Fixed random error issue with Trilu bf16 on Mars3
v1.16.0	2025.03.03	TPULang ROI_Extractor support; Einsum supports abcde,abfge->abcdfg pattern; LLMC adds Vila model support
v1.15.0	2025.02.05	Added LLMC quantization support; Address boundary check in codegen; Fixed several comparison issues
v1.14.0	2025.01.02	Added post-processing fusion for yolov8/v11; Support for Conv3D stride > 15; Improved FAttention accuracy
v1.13.0	2024.12.02	Streamlined Release package; Performance optimization for MaxPoolWithMask training operator; Added support for large RoPE operators
v1.12.0	2024.11.06	tpuv7-runtime cmodel integration; BM1690 multi-core LayerGroup optimization; Support for PPL backend operator development
v1.11.0	2024.09.27	Added soc mode for BM1688 tdb; bmodel supports fine-grained merging; Fixed several performance degradation issues
v1.10.0	2024.08.15	Added yolov10 support; New quantization tuning section; Optimized tpu-perf log output
v1.9.0	2024.07.16	BM1690 added 40 model regression tests; New quantization algorithms: octav, aciq_guas and aciq_laplace
v1.8.0	2024.05.30	BM1690 supports multi-core MatMul operator; TPULang supports input/output order specification; tpuperf removes patchelf dependency
v1.7.0	2024.05.15	CV186X dual-core changed to single-core; BM1690 testing process aligned with BM1684X; Support for gemma/llama/qwen models
v1.6.0	2024.02.23	Added Pypi release format; Support for user-defined Global operators; Added CV186X processor platform support
v1.5.0	2023.11.03	More Global Layer support for multi-core parallel processing
v1.4.0	2023.09.27	System dependencies upgraded to Ubuntu22.04; Added BM1684 Winograd support
v1.3.0	2023.07.27	Added manual floating-point operation region specification; Added supported frontend framework operator list; Added NNTC vs TPU-MLIR quantization comparison
v1.2.0	2023.06.14	Adjusted mixed quantization examples
v1.1.0	2023.05.26	Added post-processing using intelligent deep learning processor
v1.0.0	2023.04.10	PyTorch support, added section for PyTorch model conversion
v0.8.0	2023.02.28	Added pre-processing using intelligent deep learning processor
v0.6.0	2022.11.05	Added section for mixed precision operation process
v0.5.0	2022.10.20	Added model-zoo specification to test all models within
v0.4.0	2022.09.20	Caffe support, added section for Caffe model conversion
v0.3.0	2022.08.24	TFLite support, added section for TFLite model conversion
v0.2.0	2022.08.02	Added chapter for running test samples in SDK
v0.1.0	2022.07.29	Initial release, supports resnet/mobilenet/vgg/ssd/yolov5s with yolov5s as example case

Table of Contents

• 1. TPU-MLIR Introduction
• 2. Environment Setup
  • 2.1. Code Download
  • 2.2. Docker Configuration
  • 2.3. ModelZoo (Optional)
  • 2.4. Compilation
• 3. User Interface
  • 3.1. Model Conversion Process
    • 3.1.1. Support for Image Input
    • 3.1.2. Support for Multiple Inputs
    • 3.1.3. Support for INT8 Symmetric and Asymmetric
    • 3.1.4. Support for Mixed Precision
    • 3.1.5. Support for Quantized TFLite Models
    • 3.1.6. Support for Caffe Models
    • 3.1.7. Support LLM models
  • 3.2. Introduction to Tool Parameters
    • 3.2.1. model_transform.py
    • 3.2.2. run_calibration.py
    • 3.2.3. model_deploy.py
    • 3.2.4. llm_convert.py
    • 3.2.5. model_runner.py
    • 3.2.6. npz_tool.py
    • 3.2.7. visual.py
    • 3.2.8. mlir2graph.py
    • 3.2.9. gen_rand_input.py
    • 3.2.10. model_tool
• 4. Overall Design
  • 4.1. Layered
  • 4.2. Top Pass
  • 4.3. Tpu Pass
  • 4.4. Other Passes
• 5. Front-end Conversion
  • 5.1. Main Work
  • 5.2. Workflow
  • 5.3. Example
• 6. Quantization
  • 6.1. Basic Concepts
    • 6.1.1. Asymmetric Quantization
    • 6.1.2. Symmetric Quantization
  • 6.2. Scale Conversion
  • 6.3. Quantization derivation
    • 6.3.1. Convolution
    • 6.3.2. InnerProduct
    • 6.3.3. Add
    • 6.3.4. AvgPool
    • 6.3.5. LeakyReLU
    • 6.3.6. Pad
    • 6.3.7. PReLU
• 7. Calibration
  • 7.1. General introduction
  • 7.2. Introduction to the Default Process
  • 7.3. Calibration data screening and preprocessing
    • 7.3.1. Screening Principles
    • 7.3.2. Input format and preprocessing
  • 7.4. Quantization Threshold Algorithm Implementation
    • 7.4.1. KLD Algorithm
    • 7.4.2. Auto-tune Algorithm
    • 7.4.3. Octav Algorithm
    • 7.4.4. Aciq Algorithm
  • 7.5. optimization algorithms Implementation
    • 7.5.1. sq Algorithm
    • 7.5.2. we Algorithm
    • 7.5.3. bc Algorithm
    • 7.5.4. search_threshold Algorithm
    • 7.5.5. search_qtable Algorithm
  • 7.6. Example: yolov5s calibration
  • 7.7. visual tool introduction
• 8. Lowering
  • 8.1. Basic Process
  • 8.2. Mixed precision
• 9. SubNet
• 10. LayerGroup
  • 10.1. Basic Concepts
  • 10.2. BackwardH
  • 10.3. Dividing the Mem Cycle
  • 10.4. LMEM Allocation
  • 10.5. Divide the optimal Group
• 11. GMEM Allocation
  • 11.1. 1. Purpose
  • 11.2. 1. Principle
    • 11.2.1. 2.1. GMEM allocation in weight tensor
    • 11.2.2. 2.2. GMEM allocation in global neuron tensors
• 12. CodeGen
  • 12.1. Main Work
  • 12.2. Workflow
  • 12.3. BM168X and Related classes in TPU-MLIR
  • 12.4. Backend Function Loading
  • 12.5. Backend store_cmd
• 13. MLIR Definition
  • 13.1. Top Dialect
    • 13.1.1. Operations
• 14. Accuracy Validation
  • 14.1. Introduction
    • 14.1.1. Objects
    • 14.1.2. Metrics
    • 14.1.3. Datasets
  • 14.2. Validation Interface
  • 14.3. Validation Example
    • 14.3.1. mobilenet_v2
    • 14.3.2. yolov5s
• 15. Quantzation aware training
  • 15.1. Basic Principles
  • 15.2. tpu-mlir QAT implementation scheme and characteristics
    • 15.2.1. Main body flow
    • 15.2.2. Features of the Scheme
  • 15.3. Installation Method
    • 15.3.1. Install with setup package
    • 15.3.2. Install from source
  • 15.4. Basic Steps
    • 15.4.1. Step 1: Interface import and model prepare
    • 15.4.2. Step 2: Calibration and quantization training
    • 15.4.3. Step 3: Export tuned fp32 model
    • 15.4.4. Step 4: Initiate the training
  • 15.5. Use Examples-resnet18
  • 15.6. Tpu-mlir QAT test environment
    • 15.6.1. Adding a cfg File
    • 15.6.2. Modify and execute run_eval.py
  • 15.7. Use Examples-yolov5s
  • 15.8. Use Examples-bert
• 16. TpuLang Interface
  • 16.1. Main Work
  • 16.2. Work Process
  • 16.3. Operator Conversion Example
  • 16.4. Tpulang API usage method
    • 16.4.1. Initialization
    • 16.4.2. Prepare the tensors
    • 16.4.3. Build the graph
    • 16.4.4. compile
    • 16.4.5. deinit
    • 16.4.6. deploy
• 17. Custom Operators
  • 17.1. Overview
  • 17.2. Custom Operator Addition Process
    • 17.2.1. Add TpuLang Custom Operator
    • 17.2.2. Add Caffe Custom Operator
  • 17.3. Custom Operator Example
    • 17.3.1. Example of TpuLang
    • 17.3.2. Example of Caffe
  • 17.4. Custom AP(Application Processor) Operator Adding Process
    • 17.4.1. TpuLang Custom AP Operator Adding
  • 17.5. Custom AP(Application Processor) Operator Example
    • 17.5.1. TpuLang Example
• 18. Implementing Backend Operators with PPL
  • 18.1. How to Write and Call Backend Operators
  • 18.2. PPL Workflow in TPU-MLIR
• 19. final.mlir Truncation Method
  • 19.1. final.mlir structure introduction
  • 19.2. final.mlir Truncation Process
• 20. Superior MaskRCNN Interface Guidance
  • 20.1. MaskRCNN Basic
    • 20.1.1. Fast Block Segmentation Methods
  • 20.2. Superior MaskRCNN
  • 20.3. Quick Start
    • 20.3.1. Prepare Your Yaml
    • 20.3.2. Block Unit Test
  • 20.4. New Frontend Interface API
    • 20.4.1. [Step 1] Run model_transform
    • 20.4.2. [Step 2] Generate Input Data
    • 20.4.3. [Step 3] Run model_deploy
  • 20.5. IO_MAP Guidance
    • 20.5.1. [Step-1] Describe Block Interface
    • 20.5.2. [Step-2] Describe IO_MAP
  • 20.6. Generate IO_MAP
  • 20.7. mAP Inference
• 21. LLMC Guidance
  • 21.1. TPU-MLIR weight-only quantization
  • 21.2. llmc_tpu
    • 21.2.1. Environment Setup
    • 21.2.2. tpu Directory
    • 21.2.3. Operating Steps
• 22. Appendix.01: Migrating from NNTC to tpu-mlir
  • 22.1. ONNX to MLIR
  • 22.2. Make a quantization calibration table
  • 22.3. Generating int8 models
• 23. Appendix 02: Basic Elements of TpuLang
  • 23.1. Tensor
  • 23.2. Tensor Preprocessing (Tensor.preprocess)
  • 23.3. Scalar
  • 23.4. Control Functions
    • 23.4.1. Initialization Function
    • 23.4.2. compile
    • 23.4.3. Deinitialization
    • 23.4.4. Reset Default Graph
    • 23.4.5. Get Current Default Graph
    • 23.4.6. Reset Graph
    • 23.4.7. Rounding Mode
  • 23.5. Operator
    • 23.5.1. NN/Matrix Operator
    • 23.5.2. Base Element-wise Operator
    • 23.5.3. Element-wise Compare Operator
    • 23.5.4. Activation Operator
    • 23.5.5. Data Arrange Operator
    • 23.5.6. Sort Operator
    • 23.5.7. Shape About Operator
    • 23.5.8. Quant Operator
    • 23.5.9. Up/Down Scaling Operator
    • 23.5.10. Normalization Operator
    • 23.5.11. Vision Operator
    • 23.5.12. Select Operator
    • 23.5.13. Preprocess Operator
    • 23.5.14. Transform Operator
    • 23.5.15. Transform Operator
    • 23.5.16. Transform Operator
    • 23.5.17. Transform Operator