13. MLIR Definition

This chapter introduces the definition of each element of MLIR, including Dialect, Interface, etc.

13.1. Top Dialect

13.1.1. Operations

13.1.1.1. AddOp

Brief intro:

Add operation, \(Y = coeff_0 * X_0 + coeff_1 * X_1\)

Input:

inputs: tensor array, corresponding to 2 or more input tensors

Output:

output: tensor

Attributes:

do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number
coeff: the coefficient corresponding to each tensor, 1.0 by default

Output:

output: tensor

Interface:

None

Example:

%2 = "top.Add"(%0, %1) {do_relu = false} : (tensor<1x3x27x27xf32>, tensor<1x3x27x27xf32>) -> tensor<1x3x27x27xf32> loc("add")

13.1.1.2. AvgPoolOp

Brief intro:

Perform average pooling on the input tensor, \(S=\frac{1}{width\ *\ height}\sum_{i,j}a_{ij}\), where \(width\) and \(height\) represent the width and height of the kernel_shape. \(\sum_{i,j}a_{ij}\) means to sum the kernel_shape. A sliding window of a given size will sequentially pool the input tensor

Input:

input: tensor

Output:

output: tensor

Attributes:

kernel_shape: controls the size of the sliding window
strides: step size, controlling each step of the sliding window
pads: controls the shape of the padding
pad_value: padding content, constant, 0 by default
count_include_pad: whether the result needs to count the pads filled
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%90 = "top.AvgPool"(%89) {do_relu = false, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]} : (tensor<1x256x20x20xf32>) -> tensor<1x256x20x20xf32> loc("resnetv22_pool1_fwd_GlobalAveragePool")

13.1.1.3. Depth2SpaceOp

Brief intro:

Depth to space operation, \(Y = Depth2Space(X)\)

Input:

inputs: tensor

Output:

output: tensor

Attributes:

block_h: tensor block size of h dimension, i64 type
block_w: tensor block size of w dimension, i64 type
is_CRD: column-row-depth. If true, the data is arranged in the depth direction according to the order of HWC, otherwise it is CHW, bool type
is_inversed: if true, the shape of the result is: \([n, c * block_h * block_w, h / block_h, w / block_w]\), otherwise it is: \([n, c / (block_h * block_w), h * block_h, w * block_w]\), bool type

Output:

output: tensor

Interface:

None

Example:

%2 = "top.Depth2Space"(%0) {block_h = 2, block_w = 2, is_CRD = true, is_inversed = false} : (tensor<1x8x2x3xf32>) -> tensor<1x2x4x6xf32> loc("add")

13.1.1.4. BatchNormOp

Brief intro:

Perform Batch Normalization on a 4D input tensor. More details on batch normalization can be found in the paper “Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift .

The specific calculation formula is as follows:

\[y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta\]

Input:

input: 4D input tensor
mean: mean of the input tensor
variance: variance of the input tensor
gamma: \(\gamma\) tensor in the formula, can be None
beta: \(\beta\) tensor in the formula, can be None

Output:

output: tensor

Attributes:

epsilon: constant \(\epsilon\) in formula, 1e-05 by default
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%5 = "top.BatchNorm"(%0, %1, %2, %3, %4) {epsilon = 1e-05, do_relu = false} : (tensor<1x3x27x27xf32>, tensor<3xf32>, tensor<3xf32>, tensor<3xf32>, tensor<3xf32>) -> tensor<1x3x27x27xf32> loc("BatchNorm")

13.1.1.5. CastOp

(To be implemented)

13.1.1.6. ClipOp

Brief intro:

Constrain the given input to a certain range

Input:

input: tensor

Output:

output: tensor

Attributes:

min: the lower limit
max: the upper limit

Output:

output: tensor

Interface:

None

Example:

%3 = "top.Clip"(%0) {max = 1%: f64,min = 2%: f64} : (tensor<1x3x32x32xf32>) -> tensor<1x3x32x32xf32> loc("Clip")

13.1.1.7. ConcatOp

Brief intro:

Concatenates the given sequence of tensors in the given dimension. All input tensors either have the same shape (except the dimension to be concatenated) or are all empty.

Input:

inputs: tensor array, corresponding to 2 or more input tensors

Output:

output: tensor

Attributes:

axis: the subscript of the dimension to be concatenated
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%2 = "top.Concat"(%0, %1) {axis = 1, do_relu = false} : (tensor<1x3x27x27xf32>, tensor<1x3x27x27xf32>)  -> tensor<1x6x27x27xf32> loc("Concat")

13.1.1.8. ConvOp

Brief intro:

Perform 2D convolution operation on the input tensor.

In simple terms, the size of the given input is \((N, C_{\text{in}}, H, W)\). The output \((N, C_{\text{out}}, H_{ \text{out}}, W_{\text{out}})\) is calculated as:

\[\text{out}(N_i, C_{\text{out}_j}) = \text{bias}(C_{\text{out}_j}) + \sum_{k = 0}^{C_{\text{in}} - 1} \text{weight}(C_{\text{out}_j}, k) \star \text{input}(N_i, k),\]

where \(\star\) is a valid cross-correlation operation, \(N\) is the batch size, \(C\) is the number of channels, \(H, W\) is the input image height and width.

Input:

input: tensor
filter: parameter tensor. The shape is

\((\text{out\_channels}, \frac{\text{in\_channels}}{\text{groups}}, \text{kernel\_size[0]}, \text{kernel\_size[1]})\)

bias: learnable bias tensor with the shape of \((out\_channels)\)

Output:

output: tensor

Attributes:

kernel_shape: the size of the convolution kernel
strides: strides of convolution
pads: the number of layers to add 0 to each side of the input
group: the number of blocked connections from the input channel to the output channel, the default is 1
dilations: the spacing between convolution kernel elements, optional
inserts: optional
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%2 = "top.Conv"(%0, %1) {kernel_shape = [3, 5], strides = [2, 1], pads = [4, 2]} : (tensor<20x16x50x100xf32>, tensor<33x3x5xf32>)  -> tensor<20x33x28x49xf32> loc("Conv")

13.1.1.9. DeconvOp

Brief intro:

Perform a deconvolution operation on the input tensor.

Input:

input: tensor
filter: parameter tensor. The shape is

\((\text{out\_channels}, \frac{\text{in\_channels}}{\text{groups}}, \text{kernel\_size[0]}, \text{kernel\_size[1]})\)

bias: learnable bias tensor with the shape of \((out\_channels)\)

Output:

output: tensor

Attributes:

kernel_shape: the size of the convolution kernel
strides: strides of convolution
pads: the number of layers to add 0 to each side of the input
group: the number of blocked connections from the input channel to the output channel, the default is 1
dilations: the spacing between convolution kernel elements, optional
inserts: optional
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%2 = "top.Deconv"(%0, %1) {kernel_shape = (3, 5), strides = (2, 1), pads = (4, 2)} : (tensor<20x16x50x100xf32>, tensor<33x3x5xf32>)  -> tensor<20x33x28x49xf32> loc("Deconv")

13.1.1.10. DivOp

Brief intro:

Division operation, \(Y = X_0 / X_1\)

Input:

inputs: tensor array, corresponding to 2 or more input tensors

Output:

output: tensor

Attributes:

do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number
multiplier: the multiplier for quantization, the default is 1
rshift: right shift for quantization, 0 by default

Output:

output: tensor

Interface:

None

Example:

%2 = "top.Div"(%0, %1) {do_relu = false, relu_limit = -1.0, multiplier = 1, rshift = 0} : (tensor<1x3x27x27xf32>, tensor<1x3x27x27xf32>) -> tensor<1x3x27x27xf32> loc("div")

13.1.1.11. InputOp

(To be implemented)

13.1.1.12. LeakyReluOp

Brief intro:

Apply the LeakyRelu function on each element in the tensor. The function can be expressed as: f(x) = alpha * x for x < 0, f(x) = x for x >= 0

Input:

input: tensor

Output:

output: tensor

Attributes:

alpha: the coefficients corresponding to each tensor

Output:

output: tensor

Interface:

None

Example:

%4 = "top.LeakyRelu"(%3) {alpha = 0.67000001668930054 : f64} : (tensor<1x32x100x100xf32>) -> tensor<1x32x100x100xf32> loc("LeakyRelu")

13.1.1.13. LSTMOp

Brief intro:

Perform the LSTM operation of the RNN

Input:

input: tensor

Output:

output: tensor

Attributes:

filter: convolution kernel
recurrence: recurrence unit
bias: parameter of LSTM
initial_h: Each sentence in LSTM will get a state after the current cell. The state is a tuple(c, h), where h=[batch_size, hidden_size]
initial_c: c=[batch_size, hidden_size]
have_bias: whether to set bias, the default is false
bidirectional: set the LSTM of the bidirectional loop, the default is false
batch_first: whether to put the batch in the first dimension, the default is false

Output:

output: tensor

Interface:

None

Example:

%6 = "top.LSTM"(%0, %1, %2, %3, %4, %5) {batch_first = false, bidirectional = true, have_bias = true} : (tensor<75x2x128xf32>,tensor<2x256x128xf32>, tensor<2x256x64xf32>, tensor<2x512xf32>, tensor<2x2x64xf32>, tensor<2x2x64xf32>) -> tensor<75x2x2x64xf32> loc("LSTM")

13.1.1.14. LogOp

Brief intro:

Perform element-wise logarithm on the given input tensor

Input:

input: tensor

Output:

output: tensor

Attributes:

None

Output:

output: tensor

Interface:

None

Example:

%1 = "top.Log"(%0) : (tensor<1x3x32x32xf32>) -> tensor<1x3x32x32xf32> loc("Log")

13.1.1.15. MaxPoolOp

Brief intro:

Perform max pool on the given input tensor

Input:

input: tensor

Output:

output: tensor

Attributes:

kernel_shape: controls the size of the sliding window
strides: step size, controlling each step of the sliding window
pads: controls the shape of the padding
pad_value: padding content, constant, 0 by default
count_include_pad: whether the result needs to count the pads filled
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%8 = "top.MaxPool"(%7) {do_relu = false, kernel_shape = [5, 5], pads = [2, 2, 2, 2], strides = [1, 1]} : (tensor<1x256x20x20xf32>) -> tensor<1x256x20x20xf32> loc("resnetv22_pool0_fwd_MaxPool")

13.1.1.16. MatMulOp

Brief intro:

2D matrix multiplication operation, \(C = A * B\)

Input:

input: tensor: matrix of size m*k
right: tensor: matrix of size k*n

Output:

output: tensor: matrix of size m*n

Attributes:

bias: the bias_scale will be calculated according to the bias during quantization (can be empty)
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Output:

output: tensor

Interface:

None

Example:

%2 = "top.MatMul"(%0, %1) {do_relu = false, relu_limit = -1.0} : (tensor<3x4xf32>, tensor<4x5xf32>) -> tensor<3x5xf32> loc("matmul")

13.1.1.17. MulOp

Brief intro:

multiplication operation, \(Y = X_0 * X_1\)

Input:

inputs: tensor array, corresponding to 2 or more input tensors

Output:

output: tensor

Attributes:

do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number
multiplier: the multiplier for quantization, the default is 1
rshift: right shift for quantization, default is 0

Output:

output: tensor

Interface:

None

Example:

%2 = "top.Mul"(%0, %1) {do_relu = false, relu_limit = -1.0, multiplier = 1, rshift = 0} : (tensor<1x3x27x27xf32>, tensor<1x3x27x27xf32>) -> tensor<1x3x27x27xf32> loc("mul")

13.1.1.18. MulConstOp

Brief intro:

Multiply with a constant, \(Y = X * Const_Val\)

Input:

inputs: tensor

Output:

output: tensor

Attributes:

const_val: constants of type f64
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Output:

output: tensor

Interface:

None

Example:

%1 = arith.constant 4.7 : f64
%2 = "top.MulConst"(%0) {do_relu = false, relu_limit = -1.0} : (tensor<1x3x27x27xf64>, %1) -> tensor<1x3x27x27xf64> loc("mulconst")

13.1.1.19. PermuteOp

Brief intro:

Change the tensor layout. Change the order of tensor data dimensions, and rearrange the input tensor according to the given order

Input:

inputs: tensor array, tensor of any types

Attributes:

order: the order in which tensors are rearranged

Output:

output: rearranged tensor

Interface:

None

Example:

%2 = "top.Permute"(%1) {order = [0, 1, 3, 4, 2]} : (tensor<4x3x85x20x20xf32>) -> tensor<4x3x20x20x85xf32> loc("output_Transpose")

13.1.1.20. ReluOp

Brief intro:

Performs the ReLU function on each element in the input tensor, if the limit is zero, the upper limit is not used

Input:

input: tensor

Output:

output: tensor

Attributes:

relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Output:

output: tensor

Interface:

None

Example:

%1 = "top.Relu"(%0) {relu_limit = 6.000000e+00 : f64} : (tensor<1x3x32x32xf32>) -> tensor<1x3x32x32xf32> loc("Clip")

13.1.1.21. ReshapeOp

Brief intro:

Reshape operator, which returns a tensor of the given shape with the same type and internal values as the input tensor. Reshape may operate on any row of the tensor. No data values will be modified during the reshaping process

Input:

input: tensor

Output:

output: tensor

Attributes:

None

Interface:

None

Example:

%133 = "top.Reshape"(%132) : (tensor<1x255x20x20xf32>) -> tensor<1x3x85x20x20xf32> loc("resnetv22_flatten0_reshape0_Reshape")

13.1.1.22. ScaleOp

Brief intro:

Scale operation \(Y = X * S + B\), where the shape of X/Y is [N, C, H, W], and the shape of S/B is [1, C, 1, , 1].

Input:

input: tensor
scale: the magnification of the input
bias: the bias added after scaling

Output:

output: tensor

Attributes:

do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Interface:

None

Example:

%3 = "top.Scale"(%0, %1, %2) {do_relu = false} : (tensor<1x3x27x27xf32>, tensor<1x3x1x1xf32>, tensor<1x3x1x1xf32>) -> tensor<1x3x27x27xf32> loc("Scale")

13.1.1.23. SigmoidOp

Brief intro:

The activation function, which maps elements in the tensor to a specific interval, [0, 1] by default. The calculation method is:

\[Y = \frac{scale}{1 + e^{-X}} + bias\]

Input:

inputs: tensor array, tensor of any types

Attributes:

scale: the magnification of the input, 1 by default
bias: default is 0

Output:

output: tensor

Interface:

None

Example:

%2 = "top.Sigmoid"(%1) {bias = 0.000000e+00 : f64, scale = 1.000000e+00 : f64} : (tensor<1x16x64x64xf32>) -> tensor<1x16x64x64xf32> loc("output_Sigmoid")

13.1.1.24. SiLUOp

Brief intro:

The activation function, \(Y = \frac{X}{1 + e^{-X}}\) or \(Y = X * Sigmoid(X)\)

Input:

input: tensor array, tensor of any types

Attributes:

None

Output:

output: tensor

Interface:

None

Example:

%1 = "top.SiLU"(%0) : (tensor<1x16x64x64xf32>) -> tensor<1x16x64x64xf32> loc("output_Mul")

13.1.1.25. SliceOp

Brief intro:

Tensor slice, slicing each dimension of the input tensor according to the offset and step size in the offset and steps arrays to generate a new tensor

Input:

input: tensor array, tensor of any types

Attributes:

offset: an array for storing slice offsets. The index of the offset array corresponds to the dimension index of the input tensor
steps: an array that stores the step size of the slice. The index of the steps array corresponds to the index of the input tensor dimension

Output:

output: tensor

Interface:

None

Example:

%1 = "top.Slice"(%0) {offset = [2, 10, 10, 12], steps = [1, 2, 2, 3]} : (tensor<5x116x64x64xf32>) -> tensor<3x16x16x8xf32> loc("output_Slice")

13.1.1.26. SoftmaxOp

Brief intro:

For the input tensor, the normalized index value is calculated on the dimension of the specified axis. The calculation method is as follows:

\[\sigma(Z)_i = \frac{e^{\beta{Z_i}}}{\sum_{j=0}^{K-1}{e^{\beta{Z_j}}}},\]

where \(\sum_{j=0}^{K-1}{e^{\beta{Z_j}}}\) does the exponential summation on the axis dimension. j ranges from 0 to K-1 and K is the size of the input tensor in the axis dimension.

For example, the size of the input tensor is \((N, C, W, H)\), and the Softmax is calculated on the channel of axis=1. The calculation method is:

\[Y_{n,i,w,h} = \frac{e^{\beta{X_{n,i,w,h}}}}{\sum_{j=0}^{C-1}{e^{\beta{X_{n,j,w,h}}}}}\]

Input:

input: tensor array, tensor of any types

Attributes:

axis: dimension index, which is used to specify the dimension to perform softmax. It can take the value from [-r, r-1], where r is the number of dimensions of the input tensor. When axis is negative, it means the reverse order dimension
beta: The scaling factor for the input in the tflite model, invalid for non-tflite models, 1.0 by default.

Output:

output: the tensor on which the softmax is performed.

Interface:

None

Example:

%1 = "top.Softmax"(%0) {axis = 1 : i64} : (tensor<1x1000x1x1xf32>) -> tensor<1x1000x1x1xf32> loc("output_Softmax")

13.1.1.27. SqueezeOp

Brief intro:

Crop the input tensor with the specified dimension and return the cropped tensor

Input:

input: tensor

Output:

output: tensor

Attributes:

axes: specifies the dimension to be cropped. 0 represents the first dimension and -1 represents the last dimension

Interface:

None

Example:

%133 = "top.Squeeze"(%132) {axes = [-1]} : (tensor<1x255x20x20xf32) -> tensor<1x255x20xf32> loc(#loc278)

13.1.1.28. UpsampleOp

Brief intro:

Upsampling op, upsampling the input tensor nearest and returning the tensor

Input:

tensor

Attributes:

scale_h: the ratio of the height of the target image to the original image
scale_w: the ratio of the width of the target image to the original image
do_relu: whether to perform Relu operation on the result, False by default
relu_limit: specify the upper limit value if doing Relu. There is no upper limit if it is a negative number

Output:

output: tensor

Interface:

None

Example:

%179 = "top.Upsample"(%178) {scale_h = 2 : i64, scale_w = 2 : i64} : (tensor<1x128x40x40xf32>) -> tensor<1x128x80x80xf32> loc("268_Resize")

13.1.1.29. WeightOp

Brief intro:

The weight op, including the reading and creation of weights. Weights will be stored in the npz file. The location of the weight corresponds to the tensor name in npz.

Input:

None

Attributes:

None

Output:

output: weight Tensor

Interface:

read: read weight data, the type is specified by the model
read_as_float: convert the weight data to float type for reading
read_as_byte: read the weight data in byte type
create: create weight op
clone_bf16: convert the current weight to bf16 and create a weight Op
clone_f16: convert the current weight to f16 and create a weight Op

Example:

%1 = "top.Weight"() : () -> tensor<32x16x3x3xf32> loc("filter")