predict API for dynamic batch

[Duum]

Can I use the python mxnet_predict of Amalgamation for dynamic batch.
The API shows I must specify the input data shape when I create the Predictor.
But it not make sense when using it in ocr task, because I do not know how many words image i can split out from a big image.

[shuokay]

I don't think mxnet support dynamic batch. One possible solution is:

1. Set a fixed batch size, for example 10.
2. Split your data and make sure batch size is 10
3. Pad the batch which size is small than 10
4. Feed every batch to your model and also recover the result of the padded batch carefully.

[Duum]

thanks @shuokay
It make sense when mxnet bind the array shape in the initialize of the Predictor if it want to avoid to allocate memory dynamically. But I think it is fussy in deploying some task .
Could mxnet predict API provide a (-1,channel,height,width) option for dynamic batch?

[shuokay]

It is not easy to implement a dynamic batch with symbolic programming. One possible way is create several symbols and bind data accordingly when Forward, but the overhead is too high compared to divide&pad method.

PS: We have adopted the divide&pad method and so far it works well.

[yajiedesign]

This issue is closed due to lack of activity in the last 90 days. Feel free to reopen if this is still an active issue. Thanks!