[FIX][CI] hotfix check_grad perf regression

[altanh]

see #8579

Lifted the interpreter executor evaluation out of a hot loop in check_grad. Additionally, modified the interpreter executor to only create one Interpreter per expression evaluation (rather than each time the evaluated expression closure is invoked). Also fixed a bug that was in some code for grouped_conv2d on ARM CPU.

cc @tqchen

[junrushao]

Thanks for the fix!

[altanh]

it turns out the problem goes a bit deeper with the Interpreter executor, please wait to merge until we settle on a complete solution

[altanh]

cc @Wheest author of #6137 regarding the fix I added for grouped_conv2d default config on ARM cpu, please check that it's correct

[tqchen]

@altanh your change might have triggered other errors. Let us skip the interepreter in the gradient checkers and focus on running VM instead

[altanh]

@altanh your change might have triggered other errors. Let us skip the interepreter in the gradient checkers and focus on running VM instead

I saw the errors, I'm trying one more quick fix and if it doesn't work I will open an alternative PR which disables the interpreter in check_grad

[leandron]

This is merged now. Thanks @altanh, @tqchen, @junrushao1994, @jcf94, @mbrookhart and @jroesch!

[Wheest]

Thanks @altanh, the ARM grouped conv change LGTM. How did you discover it, if it wasn't caught by the tests in CI?

[altanh]

Thanks @altanh, the ARM grouped conv change LGTM. How did you discover it, if it wasn't caught by the tests in CI?

Good question... I was debugging the test locally and encountered the error with a basic pytest invocation. Perhaps the ARM test wasn't being run, or the default config wasn't being used on CI? Definitely weird