[RELAY][PASS] Bind, FoldConstant

[tqchen]

• Refactor type_subt->Bind(overloaded name) and move to type_functor.cc
• Add support of Bind which bind params(free vars) to certain expression
  • This is useful to bind parameters as a constant into the IR during inference.
• Support FoldConstant
  • This is useful for folding contents that can be pre-computed, in nnvmv1 it was called precompute_prune.
• Support constant node in graph runtime backend.
  • constants will be generated as params dict.

The general flow now becomes:

• bind params into IR as constant
• Run optimizations, including constant folding
• In graph runtime codegen, the constants nodes are then lifted out as params

[tqchen]

cc @yzhliu @zhiics @ZihengJiang @jroesch @MarisaKirisame @PariksheetPinjari909 please review

[MarisaKirisame]

add returns in pydoc above

[MarisaKirisame]

Optional[Dict[str, NDArray]]

[MarisaKirisame]

I did a quick pass of the code, LGTM
one major issue with all the pass(not just this one, but dead code elim too) we write is they do not respect effect or recursion, and we have to have them respect it. I think we can talk about this issue in PLDI.

[tqchen]

The side effect is checked by is_stateful, I think the folding pass should work with functions of recursion, by ignoring recursive calls. But indeed they don't unroll the recursion if that is what you mean

[MarisaKirisame]

@tqchen tuplegetitem(tuplenode(...)) will just take the expr
but if some expr in tuplenode went into infinite recursion, or has effect, it will change the behavior of the program

[tqchen]

I see, indeed, we will need side effect check pass and possibly combine dead code elim into folding. For now it is fine as the program is side-effect free

[jroesch]

Looks good to me we can follow up on those issues afterwards.