cudnn Dropout reproducibility

[eric-haibin-lin]

https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/dropout-inl.h#L491-L493

The seed of CUDNN dropout does not respect the random seed in mxnet

[mxnet-label-bot]

Hey, this is the MXNet Label Bot.
Thank you for submitting the issue! I will try and suggest some labels so that the appropriate MXNet community members can help resolve it.
Here are my recommended labels: Cuda, Bug

[zachgk]

I did a quick recursive grep and found several other usages of the rand function. Here is the full list I found (some of these may be correct):

• https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/dropout-inl.h#L95
• https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/dropout-inl.h#L491
• https://github.com/apache/incubator-mxnet/blob/master/src/operator/random/sampler.h#L110
• https://github.com/apache/incubator-mxnet/blob/master/src/operator/rnn-inl.h#L1551
• https://github.com/apache/incubator-mxnet/blob/master/src/operator/rnn_impl.h#L159
• https://github.com/apache/incubator-mxnet/blob/master/src/operator/rnn_impl.h#L1004
• https://github.com/apache/incubator-mxnet/blob/master/src/operator/rnn_impl.h#L1891

[eric-haibin-lin]

@ptrendx @DickJC123 just FYI

[larroy]

So all those places should be seeded propperly correct?

[eric-haibin-lin]

random/sampler.* and rnn_impl.* are for other operators, not dropout. But yes, in general those should also be fixed, and deserve separate github issues

[eric-haibin-lin]

below is a test case:

import mxnet as mx

a = mx.nd.ones((10,10), ctx=mx.gpu())
dropout = mx.gluon.nn.Dropout(0.5)
seed = 1234

mx.random.seed(seed)

with mx.autograd.record():
    b = dropout(a)

mx.random.seed(seed)

with mx.autograd.record():
    c = dropout(a)

print(b)
print(c)
mx.test_utils.assert_almost_equal(b.asnumpy(), c.asnumpy())

[eric-haibin-lin]

we need to delay the initialization of the seed in https://github.com/apache/incubator-mxnet/blob/master/src/operator/nn/dropout-inl.h#L497
request a random resource: https://github.com/apache/incubator-mxnet/blob/e260f13499a6c84737a97aae09e5921ad8fdd4fd/src/operator/numpy/random/np_choice_op.cc#L71
get a random number generator: https://github.com/apache/incubator-mxnet/blob/e260f13499a6c84737a97aae09e5921ad8fdd4fd/src/operator/numpy/random/np_choice_op.h#L166
do sampling with the generator: https://github.com/apache/incubator-mxnet/blob/e260f13499a6c84737a97aae09e5921ad8fdd4fd/src/operator/numpy/random/np_choice_op.h#L179

[leezu]

Reopen as fix is reverted due to performance issue.