This repository was archived by the owner on Nov 17, 2023. It is now read-only.
Description As follow up on #16532
I found out rnn operator with dropout result cannot be fixed with fixed seed on CPU.#16532
@with_seed()
def test_rnn_with_seed():
info = np.iinfo(np.int32)
seed = np.random.randint(info.min, info.max)
_test_rnn(seed, mx.cpu())
_test_rnn(seed, mx.gpu())
def _test_rnn(seed, ctx):
data = mx.nd.ones((5, 3, 10), ctx=ctx)
rnn = mx.gluon.rnn.RNN(100, 3, dropout=0.5)
rnn.initialize(ctx=ctx)
mx.random.seed(seed)
with mx.autograd.record():
result1 = rnn(data)
mx.random.seed(seed)
with mx.autograd.record():
result2 = rnn(data)
# dropout on gpu should return same result with fixed seed
assert_almost_equal(result1.asnumpy(), result2.asnumpy())
Current workaround is DO NOT use dropout in rnn and manually append a dropout layer after rnn, the following works:
def _test_rnn(seed, ctx):
data = mx.nd.ones((5, 3, 10), ctx=ctx)
rnn = mx.gluon.rnn.RNN(100, 3, dropout=0.)
rnn.initialize(ctx=ctx)
dropout = mx.gluon.nn.Dropout(0.5)
with mx.autograd.record():
result1 = rnn(data)
mx.random.seed(seed)
o1 = dropout(result1)
with mx.autograd.record():
result2 = rnn(data)
mx.random.seed(seed)
o2 = dropout(result2)
# dropout on gpu should return same result with fixed seed
assert_almost_equal(o1.asnumpy(), o2.asnumpy())
Reactions are currently unavailable
As follow up on #16532
I found out rnn operator with dropout result cannot be fixed with fixed seed on CPU.
GPU (cudnn) is fine after PR #16532
Current workaround is DO NOT use dropout in rnn and manually append a dropout layer after rnn, the following works: