Improve AArch64 depthwise convolution through smlal/smlal2 intrinsic

[giuseros]

• Added an intrinsic to load a single int16x8 vector and produce two
  int32x4 output vectors through smlal/smlal2 instructions

• Changed the NHWC depthwise schedule to accomodate the aforementioned
  intrinsic

Change-Id: I347c3bf98fa8dd87057304dcda0d78e558424c57

[giuseros]

cc @u99127 @anijain2305

[FrozenGene]

I like this change. May I ask two questions?

1. could we make smlal / smlal2 work on convolution too?
2. If we don't use tensorize, do we have method to handle this? For example, add optimization of custom LLVM pass or low detail analyze on tir or llvm ir?

[giuseros]

Hi @FrozenGene
Thanks for your reply!

1. Yes we can, but there is a issue. It's very hard to have the two quantized strategies (int8 + smull/sadalp vs int16 + smlal/smlal2) available to the auto-tuner at the same time (you already saw this post, I will reply there to your comment). In the case of depthwise, we only have the int16 strategy, so the issue does not arise.
2. Nice you asked, because I am trying to prototype the idea of a "tir pass" where this tensorization (and possibly every arm tensorization in tensor_intrin.py) happens. My hope is to make it work more or less like vectorize. One of the main reasons to have this is to enable Ansor support

[FrozenGene]

Hi @FrozenGene
Thanks for your reply!

1. Yes we can, but there is a issue. It's very hard to have the two quantized strategies (int8 + smull/sadalp vs int16 + smlal/smlal2) available to the auto-tuner at the same time (you already saw this post, I will reply there to your comment). In the case of depthwise, we only have the int16 strategy, so the issue does not arise.
2. Nice you asked, because I am trying to prototype the idea of a "tir pass" where this tensorization (and possibly every arm tensorization in tensor_intrin.py) happens. My hope is to make it work more or less like vectorize. One of the main reasons to have this is to enable Ansor support

For ansor, we could make ansor support tensorize (like TensorCore we need tensorize too). However, if we could done it in the llvm / tir, we will make ansor support it easily. If we could lift it as generic pass, I think it will bring benifit to other hardware platform too.

[mbaret]

LGTM now. I agree with @FrozenGene that this would be best implemented nearer the LLVM codegen for reusability, but I think this is a good start and demonstrates a worthwhile benefit to performance.

[giuseros]

Hi @mbaret , thanks for the review!

@FrozenGene , any update on this?

[FrozenGene]

@giuseros @mbaret Thanks, merged now