[VM][DMLC] Lower memory usage when loading and dumping weights

[AndrewZhaoLuo]

Right now there is a bad pattern in VM executable where when loading weights, we load serialized representation in memory, and then deserialize off the in-memory store without progressively freeing memory.

This is bad because if our weights take up ~ 5GB, then the serialized representation in memory takes up 5GB and the deserialized representation will take ~ 5 GB too. This means peak memory use for using the VM for execution is 2 * the size of the weight models.

This is bad, especially with some of the larger models out there today.

This fixes thing by using a stream from disk, and depending on the standard C file interface to buffer things for performant results.

Some before and after graphs though loading and benchmarking a model with ~5GB weights:

Before:

After:

This is a draft since:

• I've only tested loading weights, but we can see similar savings in other similar streams.
• We need to make a decision on DMLC stream interface. The main issue is that a lot of existing code depends on DMLC stream interface, but DMLC itself is a header only library. We only have access to in-memory streams in the current state. The way I have gotten around this is by implementing a simple class.
• We need to decide best way forward. The one in this PR is simple, though technically duplicates some code from DMLC core lib
• Alternatives are including DMLC as dependency, adding to DMLC functionality and pulling those things changes, or get rid of DMLC stream interface entirely
• This one is the simplest which is why I will do this for the draft.

[tvm-bot]

Thanks for contributing to TVM! Please refer to the contributing guidelines https://tvm.apache.org/docs/contribute/ for useful information and tips. Please request code reviews from Reviewers by @-ing them in a comment.

• No users to tag found in teams: vm, dmlc _{See #10317 for details}

_{Generated by tvm-bot}

[AndrewZhaoLuo]

cc @tqchen @jwfromm

[tqchen]

The approach of having overload file support util is fine, one thing is that it would needs to be part of the runtime folder as it is simple enough.

Given most of the cases are on GPU, having ability to be able to load one array into CPU, copy into GPU then immediately free that CPU array can also be effective.

[AndrewZhaoLuo]

@tqchen thanks for the comments.

PTAL, ready for review.

[tqchen]

Thanks @AndrewZhaoLuo , one minor comment