[BYOC][CONTRIB] Vitis-AI codegen integration by anilmartha · Pull Request #6343 · apache/tvm

anilmartha · 2020-08-26T11:20:25Z

This PR implements the Vitis-AI codegen using the BYOC flow and enables us to offload subgraphs to FPGA DPU accelerators (cloud/edge).

Below are the features added as part of this PR

Annotate the graph for the given Vitis-AI DPU (Deep Learning Processing Unit) target.
During codegen phase, convert the relay subgraph into PyXIR and save XGraph.
Vitis-AI runtime supports SaveToBinary and LoadFromBinary. We save the XGraph it in our own format and serialize it in the Module by keeping track of the path to the files.
Tests include a complete resnet18 model test partly offloaded to PyXIR for DPU acceleration. However, we don't have access to an FPGA instance in the CI docker environment and therefore the offloaded subgraph is just executed on CPU.

This PR depends on following Vitis-AI CI contribution PR.
The RFC for this PR can be found from here.

This work is co-authored by Jorn Tuyls jornt@xilinx.com @jtuyls, Elliott Delaye elliott@xilinx.com @edelaye and Sumit Nagpal sumitn@xilinx.com @sumitn-xilinx.

comaniac · 2020-08-26T18:32:19Z

cc @tmoreau89 @zhiics @masahi @tqchen @liangfu

mbaret

A general comment would be that it might be helpful if the Python-side code was more extensively documented.

mbaret · 2020-09-08T14:39:17Z

Just as a general comment, it's useful if you can mark conversations as 'resolved' once you've made the changes so it's easy to see where conversations are still taking place.

tmoreau89 · 2020-09-22T17:26:21Z

@zhiics @comaniac @anilmartha I'd like to revive this PR so we can consider merging it in. As @mbaret pointed, please mark your comments as resolved if they have been addressed. If anything is still outstanding, please make it clear what you'd like @anilmartha to change.

comaniac · 2020-09-22T17:38:57Z

@tmoreau89 I think we are waiting for more commits and responses from @anilmartha. I've just resolved the comments that should not be the blockers. The blockers now can be summarized as follows:

@anilmartha is working on serialization of xgraph and will add it in this PR itself ([BYOC][CONTRIB] Vitis-AI codegen integration #6343 (comment)).
Organize Xilinx artifacts in a better way ([BYOC][CONTRIB] Vitis-AI codegen integration #6343 (comment), [BYOC][CONTRIB] Vitis-AI codegen integration #6343 (comment)).
Question about 2 subgaph support ([BYOC][CONTRIB] Vitis-AI codegen integration #6343 (comment)).
Rename in test ([BYOC][CONTRIB] Vitis-AI codegen integration #6343 (comment)).

jtuyls · 2020-09-22T18:00:02Z

@zhiics @comaniac @anilmartha I'd like to revive this PR so we can consider merging it in. As @mbaret pointed, please mark your comments as resolved if they have been addressed. If anything is still outstanding, please make it clear what you'd like @anilmartha to change.

@tmoreau89 @comaniac We are currently working on following pieces to be added based on the comments:

Serialization of Vitis-AI build info into the TVM binary stream instead of using the previous directory based approach, see comment. This is currently being verified internally. This should also address organizing Vitis-AI artifacts comment 1 and comment 2.
Add additional codegen tests based on comment
Address comment on 2 subgraphs test
Address renaming comment

We will add these pieces shortly and will tag reviewers to have a look.

tqchen · 2020-09-22T18:52:26Z

please also double check the merging, as i noticed a few files get removed

comaniac

LGTM. Just some nits. Please fix the CI errors.

zhiics

just a few minor comments. Please fix the CI.

@tmoreau89, @mbaret, @jtuyls please take another look.

zhiics · 2020-10-20T18:10:38Z

+namespace runtime {
+
+/*! \brief The target Vitis-AI accelerator device */
+TVM_REGISTER_PASS_CONFIG_OPTION("relay.ext.vitis_ai.options.target", String);


I think we are kind of relying on the PASS_CONFIG too much. It's okay to keep it for now. We will need to figure out a better way to add these configure since they are not really related to passes.

mbaret

This is looking much better, approved.

tmoreau89

Thanks; nice work LGTM

zhiics · 2020-10-21T21:50:59Z

@anilmartha please fix the CI error. Do we need to update the docker image? If that is the case, we need to bump up the image version as well.

anilmartha · 2020-10-22T09:57:22Z

@zhiics
The CI will pass once we have updated CI with latest PyXIR version. For the same raised #6716.

…ources

anilmartha · 2020-11-06T10:03:34Z

we have made some small changes to make the tests pass.
@zhiics @comaniac @tmoreau89 @mbaret Could you have another look?

zhiics · 2020-11-06T17:13:31Z

Thanks everyone. This is now merged!

* [BYOC][CONTRIB] VITIS-AI integration * Remove environment related files * Update vitis_ai.rst * Add review changes * Remove new lines and note frame in vitis_ai.rst * use sys.exit * Add condition for vitis_ai runtime exec function * remove unused graph_json * correct indentation * use code python instead of bash * Rename VITISAI.cmake to VitisAI.cmake * use relay.ext.vitis_ai.options.build_dir in comparison * Re-add deleted docker related files * Make use of PyXIR XGraph and RuntimeModule serialization & refactor flow * Fix linter errors * Fix linter errors * Address sphinx warnings * Add infertype to fix Vitis-AI annotation test * Renaming util to utils * Add Vitis-AI flag to config.cmake file * Move vitis-ai config options to compiler sources instead of runtime sources * Fix clang-format errors Co-authored-by: Anil Martha <anil.martha@xilinx.com> Co-authored-by: anilm (generated by with_the_same_user script) <anilm@xhdabidk40.xilinx.com> Co-authored-by: Jorn Tuyls <jornt@xilinx.com>

[BYOC][CONTRIB] VITIS-AI integration

8903b1a

mbaret reviewed Aug 28, 2020

View reviewed changes

Comment thread python/tvm/relay/op/contrib/vitis_ai.py

mbaret reviewed Aug 28, 2020

View reviewed changes

comaniac requested changes Aug 29, 2020

View reviewed changes

Anil Martha and others added 3 commits August 31, 2020 23:31

Remove environment related files

5eebecb

Update vitis_ai.rst

9dce161

Add review changes

fb18b17

comaniac requested changes Sep 3, 2020

View reviewed changes

Anil Martha added 2 commits September 6, 2020 23:00

Remove new lines and note frame in vitis_ai.rst

0b57227

use sys.exit

969faf6

zhiics requested changes Sep 8, 2020

View reviewed changes

Anil Martha and others added 9 commits September 8, 2020 10:36

Add condition for vitis_ai runtime exec function

680f373

remove unused graph_json

20015ae

correct indentation

fb9d3f2

use code python instead of bash

4352c7b

Rename VITISAI.cmake to VitisAI.cmake

3c45378

use relay.ext.vitis_ai.options.build_dir in comparison

dea81b5

Re-add deleted docker related files

832fae0

Merge branch 'master' into vitis-ai-integration-merge-master

48d17c6

Merge branch 'master' into vitis-ai-integration-merge-master

cc0d0f5

comaniac added status: need update need update based on feedbacks status: review in progress labels Sep 25, 2020

comaniac mentioned this pull request Sep 25, 2020

[RFC] v0.7 Release Planning #6421

Closed

Make use of PyXIR XGraph and RuntimeModule serialization & refactor flow

2a845fd

comaniac approved these changes Sep 25, 2020

View reviewed changes

Comment thread python/tvm/contrib/target/vitis_ai.py Outdated

Comment thread python/tvm/contrib/target/vitis_ai.py Outdated

Comment thread src/runtime/contrib/vitis_ai/vitis_ai_runtime.cc Outdated

Anil Martha and others added 2 commits September 28, 2020 06:40

Fix linter errors

287f285

Address sphinx warnings

dcdcf5c

tqchen changed the base branch from master to main October 11, 2020 18:21

Anil Martha and others added 2 commits October 14, 2020 07:25

Merge branch 'main' into vitis-ai-integration

dd07779

Merge branch 'main' into vitis-ai-integration

f615cda

anilmartha mentioned this pull request Oct 20, 2020

Update CI Vitis-AI PyXIR version to 0.1.3 #6716

Closed

Add infertype to fix Vitis-AI annotation test

128115e

zhiics reviewed Oct 20, 2020

View reviewed changes

mbaret approved these changes Oct 20, 2020

View reviewed changes

tmoreau89 approved these changes Oct 20, 2020

View reviewed changes

zhiics approved these changes Oct 21, 2020

View reviewed changes

anilmartha and others added 6 commits November 2, 2020 20:05

Merge branch 'main' into vitis-ai-integration

97354a3

Renaming util to utils

07c84d9

Add Vitis-AI flag to config.cmake file

753322e

Move vitis-ai config options to compiler sources instead of runtime s…

bdf2bbd

…ources

Fix clang-format errors

4cf8838

Merge branch 'main' into vitis-ai-integration

807816d

zhiics merged commit 3bfe6d3 into apache:main Nov 6, 2020

zhiics added status: accepted and removed status: need update need update based on feedbacks status: review in progress labels Nov 6, 2020

junrushao mentioned this pull request Nov 1, 2021

Apache TVM v0.8 Release Note Candidate #9416

Closed

Conversation

anilmartha commented Aug 26, 2020

Uh oh!

comaniac commented Aug 26, 2020

Uh oh!

Uh oh!

mbaret left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mbaret commented Sep 8, 2020

Uh oh!

tmoreau89 commented Sep 22, 2020

Uh oh!

comaniac commented Sep 22, 2020

Uh oh!

jtuyls commented Sep 22, 2020

Uh oh!

tqchen commented Sep 22, 2020

Uh oh!

comaniac left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

zhiics left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

zhiics Oct 20, 2020

Choose a reason for hiding this comment

Uh oh!

mbaret left a comment

Choose a reason for hiding this comment

Uh oh!

tmoreau89 left a comment

Choose a reason for hiding this comment

Uh oh!

zhiics commented Oct 21, 2020

Uh oh!

anilmartha commented Oct 22, 2020

Uh oh!

anilmartha commented Nov 6, 2020

Uh oh!

zhiics commented Nov 6, 2020