TPU-MLIR Technical Reference Manual

Release Record

Version	Release date	Explanation
v1.28.0	2026.04.14	Support Qwen3.5, Qwen3-ASR, PaddleOCR-VL and lfm2-vl in llm_convert; LLM multi-device pipeline (PP) and batch-size support; New WeightDeduplicate pass to merge identical weights; yolov26 post-processing fusion and new llm_model memory analysis tool
v1.27.0	2026.02.04	FP8 CUDA inference; Mixing per-token/per-channel and group-size dynamic quantization; LoRA and dynamic shape support multi-core; Module hash dump/load for cache reuse; TPULang adds RotPosEmb and cumsum interfaces
v1.26.0	2025.12.25	CUDA inference framework; Dynamic group quantization for BM1684X; 4K-aligned bmodel format (model_tool can refresh existing bmodels); Qwen3-VL multi-ViT and LightStereo support; LLM compressed-tensors mode
v1.25.0	2025.11.28	Qwen3-VL support BM1690E support
v1.24.0	2025.10.30	MINICPMV4 support Supports peak system and device memory estimation for bmodel
v1.23.0	2025.09.30	GLM4.1V support BM1688 Conv and MatMul support W4A8 quantization
v1.22.0	2025.08.31	PPL supports dynamic compilation llm_analyse supports performance estimation for large models
v1.21.0	2025.07.31	BM1688 supports yolov8 post-processing bmodel_checker supports replacing wrong output as reference data
v1.20.0	2025.06.30	Support IO_RELOC; Deconv3D INT8 bugfix; BatchNorm and Conv backward operators support 128 batch training
v1.19.0	2025.05.30	Support AWQ and GPTQ models; Deconv3D F16, F32 bugfix
v1.18.0	2025.05.01	YOLO series adds automatic mixed precision setting; Added SmoothQuant option for run_calibration; New one-click compilation script for LLM
v1.17.0	2025.04.03	Significant improvement in LLM model compilation speed; TPULang supports PPL operator integration; Fixed random error issue with Trilu bf16 on CV184X
v1.16.0	2025.03.03	TPULang ROI_Extractor support; Einsum supports abcde,abfge->abcdfg pattern; LLMC adds Vila model support
v1.15.0	2025.02.05	Added LLMC quantization support; Address boundary check in codegen; Fixed several comparison issues
v1.14.0	2025.01.02	Added post-processing fusion for yolov8/v11; Support for Conv3D stride > 15; Improved FAttention accuracy
v1.13.0	2024.12.02	Streamlined Release package; Performance optimization for MaxPoolWithMask training operator; Added support for large RoPE operators
v1.12.0	2024.11.06	tpuv7-runtime cmodel integration; Support for PPL backend operator development
v1.11.0	2024.09.27	Added SoC mode for BM1688 tdb; bmodel supports fine-grained merging; Fixed several performance degradation issues
v1.10.0	2024.08.15	Added yolov10 support; New quantization tuning section; Optimized tpu-perf log output
v1.9.0	2024.07.16	New quantization algorithms: octav, aciq_guas and aciq_laplace
v1.8.0	2024.05.30	TPULang supports input/output order specification; tpuperf removes patchelf dependency
v1.7.0	2024.05.15	CV186X dual-core changed to single-core; Support for gemma/llama/qwen models
v1.6.0	2024.02.23	Added Pypi release format; Support for user-defined Global operators; Added CV186X processor platform support
v1.5.0	2023.11.03	More Global Layer support for multi-core parallel processing
v1.4.0	2023.09.27	System dependencies upgraded to Ubuntu22.04; Added BM1684 Winograd support
v1.3.0	2023.07.27	Added manual floating-point operation region specification; Added supported frontend framework operator list; Added NNTC vs TPU-MLIR quantization comparison
v1.2.0	2023.06.14	Adjusted mixed quantization examples
v1.1.0	2023.05.26	Added post-processing using intelligent deep learning processor
v1.0.0	2023.04.10	PyTorch support, added section for PyTorch model conversion
v0.8.0	2023.02.28	Added pre-processing using intelligent deep learning processor
v0.6.0	2022.11.05	Added section for mixed precision operation process
v0.5.0	2022.10.20	Added model-zoo specification to test all models within
v0.4.0	2022.09.20	Caffe support, added section for Caffe model conversion
v0.3.0	2022.08.24	TFLite support, added section for TFLite model conversion
v0.2.0	2022.08.02	Added chapter for running test samples in SDK
v0.1.0	2022.07.29	Initial release, supports resnet/mobilenet/vgg/ssd/yolov5s with yolov5s as example case

Table of Contents

• 1. TPU-MLIR Introduction
• 2. Environment Setup
  • 2.1. Code Download
  • 2.2. Docker Configuration
  • 2.3. ModelZoo (Optional)
  • 2.4. Compilation
  • 2.5. Code Development
• 3. User Interface
  • 3.1. Model Conversion Process
    • 3.1.1. Support for Image Input
    • 3.1.2. Support for Multiple Inputs
    • 3.1.3. Support for INT8 Symmetric and Asymmetric
    • 3.1.4. Support for Mixed Precision
    • 3.1.5. Support for Quantized TFLite Models
    • 3.1.6. Support for Caffe Models
    • 3.1.7. Support LLM models
  • 3.2. Introduction to Tool Parameters
    • 3.2.1. model_transform.py
    • 3.2.2. run_calibration.py
    • 3.2.3. model_deploy.py
    • 3.2.4. llm_convert.py
    • 3.2.5. llm_analyse.py
    • 3.2.6. model_runner.py
    • 3.2.7. npz_tool.py
    • 3.2.8. visual.py
    • 3.2.9. mlir2graph.py
    • 3.2.10. gen_rand_input.py
    • 3.2.11. model_tool
    • 3.2.12. mlir_cut
• 4. Overall Design
  • 4.1. Layered
  • 4.2. Top Pass
  • 4.3. Tpu Pass
  • 4.4. Other Passes
• 5. Front-end Conversion
  • 5.1. Main Work
  • 5.2. Workflow
  • 5.3. Example
• 6. Quantization
  • 6.1. Basic Concepts
    • 6.1.1. Asymmetric Quantization
    • 6.1.2. Symmetric Quantization
  • 6.2. Scale Conversion
  • 6.3. Quantization derivation
    • 6.3.1. Convolution
    • 6.3.2. InnerProduct
    • 6.3.3. Add
    • 6.3.4. AvgPool
    • 6.3.5. LeakyReLU
    • 6.3.6. Pad
    • 6.3.7. PReLU
• 7. Calibration
  • 7.1. General introduction
  • 7.2. Introduction to the Working Environment
  • 7.3. Introduction to the Default Process
  • 7.4. Calibration data screening and preprocessing
    • 7.4.1. Screening Principles
    • 7.4.2. Input format and preprocessing
  • 7.5. Quantization Threshold Algorithm Implementation
    • 7.5.1. KLD Algorithm
    • 7.5.2. Auto-tune Algorithm
    • 7.5.3. Octav Algorithm
    • 7.5.4. Aciq Algorithm
  • 7.6. optimization algorithms Implementation
    • 7.6.1. sq Algorithm
    • 7.6.2. smc Algorithm
    • 7.6.3. we Algorithm
    • 7.6.4. bc Algorithm
    • 7.6.5. search_threshold Algorithm
    • 7.6.6. search_qtable Algorithm
    • 7.6.7. mix_search and fast_search Option
  • 7.7. Example: yolov5s calibration
  • 7.8. visual tool introduction
• 8. Lowering
  • 8.1. Basic Process
  • 8.2. Mixed precision
• 9. SubNet
• 10. LayerGroup
  • 10.1. Basic Concepts
  • 10.2. BackwardH
  • 10.3. Dividing the Mem Cycle
  • 10.4. LMEM Allocation
  • 10.5. Divide the optimal Group
• 11. GMEM Allocation
  • 11.1. 1. Purpose
  • 11.2. 1. Principle
    • 11.2.1. 2.1. GMEM allocation in weight tensor
    • 11.2.2. 2.2. GMEM allocation in global neuron tensors
• 12. CodeGen
  • 12.1. Main Work
  • 12.2. Workflow
  • 12.3. BM168X and Related classes in TPU-MLIR
  • 12.4. Backend Function Loading
  • 12.5. Backend store_cmd
• 13. MLIR Definition
  • 13.1. Top Dialect
    • 13.1.1. Operations
• 14. Accuracy Validation
  • 14.1. Introduction
    • 14.1.1. Objects
    • 14.1.2. Metrics
    • 14.1.3. Datasets
  • 14.2. Validation Interface
  • 14.3. Validation Example
    • 14.3.1. mobilenet_v2
    • 14.3.2. yolov5s
• 15. Quantzation aware training
  • 15.1. Basic Principles
  • 15.2. tpu-mlir QAT implementation scheme and characteristics
    • 15.2.1. Main body flow
    • 15.2.2. Features of the Scheme
  • 15.3. Installation Method
    • 15.3.1. Install with setup package
    • 15.3.2. Install from source
  • 15.4. Basic Steps
    • 15.4.1. Step 1: Interface import and model prepare
    • 15.4.2. Step 2: Calibration and quantization training
    • 15.4.3. Step 3: Export tuned fp32 model
    • 15.4.4. Step 4: Initiate the training
  • 15.5. Use Examples-resnet18
  • 15.6. Tpu-mlir QAT test environment
    • 15.6.1. Adding a cfg File
    • 15.6.2. Modify and execute run_eval.py
  • 15.7. Use Examples-yolov5s
  • 15.8. Use Examples-bert
• 16. TpuLang Interface
  • 16.1. Main Work
  • 16.2. Work Process
  • 16.3. Operator Conversion Example
  • 16.4. Tpulang API usage method
    • 16.4.1. Initialization
    • 16.4.2. Prepare the tensors
    • 16.4.3. Build the graph
    • 16.4.4. compile
    • 16.4.5. deinit
    • 16.4.6. deploy
• 17. Custom Operators
  • 17.1. Overview
  • 17.2. Custom Operator Addition Process
    • 17.2.1. Add TpuLang Custom Operator
    • 17.2.2. Add Caffe Custom Operator
  • 17.3. Custom Operator Example
    • 17.3.1. Example of TpuLang
    • 17.3.2. Example of Caffe
  • 17.4. Custom AP(Application Processor) Operator Adding Process
    • 17.4.1. TpuLang Custom AP Operator Adding
  • 17.5. Custom AP(Application Processor) Operator Example
    • 17.5.1. TpuLang Example
• 18. Implementing Backend Operators with PPL
  • 18.1. How to Write and Call Backend Operators
  • 18.2. PPL Workflow in TPU-MLIR
• 19. final.mlir Truncation Method
  • 19.1. final.mlir structure introduction
  • 19.2. final.mlir Truncation Process
• 20. Superior MaskRCNN Interface Guidance
  • 20.1. MaskRCNN Basic
    • 20.1.1. Fast Block Segmentation Methods
  • 20.2. Superior MaskRCNN
  • 20.3. Quick Start
    • 20.3.1. Prepare Your Yaml
    • 20.3.2. Block Unit Test
  • 20.4. New Frontend Interface API
    • 20.4.1. [Step 1] Run model_transform
    • 20.4.2. [Step 2] Generate Input Data
    • 20.4.3. [Step 3] Run model_deploy
  • 20.5. IO_MAP Guidance
    • 20.5.1. [Step-1] Describe Block Interface
    • 20.5.2. [Step-2] Describe IO_MAP
  • 20.6. Generate IO_MAP
  • 20.7. mAP Inference
• 21. LLMC Guidance
  • 21.1. TPU-MLIR weight-only quantization
  • 21.2. llmc_tpu
    • 21.2.1. Environment Setup
    • 21.2.2. tpu Directory
    • 21.2.3. Operating Steps
• 22. Analyse TPU Performance with TPU Profile
  • 22.1. 1. Software and Hardware Framework of TPU
  • 22.2. 1. Compile Bmodel
  • 22.3. 1. Generate Profile Binary Data
  • 22.4. 4. Visualize Profile Data
  • 22.5. 5. Analyse the Result
    • 22.5.1. 5.1 Overall Introduction
    • 22.5.2. 5.2 Global Layer
    • 22.5.3. 5.3 Local Layer Group
  • 22.6. 6. Summary
• 23. Appendix.01: Migrating from NNTC to tpu-mlir
  • 23.1. ONNX to MLIR
  • 23.2. Make a quantization calibration table
  • 23.3. Generating int8 models
• 24. Appendix 02: Basic Elements of TpuLang
  • 24.1. Tensor
  • 24.2. Tensor Preprocessing (Tensor.preprocess)
  • 24.3. Scalar
  • 24.4. Control Functions
    • 24.4.1. Initialization Function
    • 24.4.2. compile
    • 24.4.3. compile_f32
    • 24.4.4. Deinitialization
    • 24.4.5. Reset Default Graph
    • 24.4.6. Get Current Default Graph
    • 24.4.7. Reset Graph
    • 24.4.8. Rounding Mode
  • 24.5. Operator
    • 24.5.1. NN/Matrix Operator
    • 24.5.2. Base Element-wise Operator
    • 24.5.3. Element-wise Compare Operator
    • 24.5.4. Activation Operator
    • 24.5.5. Data Arrange Operator
    • 24.5.6. Sort Operator
    • 24.5.7. Shape About Operator
    • 24.5.8. Quant Operator
    • 24.5.9. Up/Down Scaling Operator
    • 24.5.10. Normalization Operator
    • 24.5.11. Vision Operator
    • 24.5.12. Select Operator
    • 24.5.13. Preprocess Operator
    • 24.5.14. Transform Operator