Solve the bug of dead lock when restore state

[CFAndy]

Fix the start iter_ for slave solver when the root solver restore state from binary file .
This will cause deadlock when using NCCL

[CFAndy]

After doing the testAll, the root solver will enter into barrier wait state.
At first, I though it was a NCCL issue. But it turned out to be a bug of Caffe.

[fucevin]

I met this issue every time. But this bug doesn't impact digits, why?

[drnikolaev]

@ChenFengAndy could you please check 0.16 branch?

[CFAndy]

|===============================+======================+======================|
| 0 TITAN X (Pascal) Off | 0000:02:00.0 Off | N/A |
| 23% 27C P8 9W / 250W | 7159MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 1 TITAN X (Pascal) Off | 0000:03:00.0 Off | N/A |
| 23% 23C P8 10W / 250W | 9723MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 2 TITAN X (Pascal) Off | 0000:82:00.0 Off | N/A |
| 23% 28C P8 9W / 250W | 9723MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
| 3 TITAN X (Pascal) Off | 0000:83:00.0 Off | N/A |
| 23% 26C P8 9W / 250W | 9723MiB / 12189MiB | 0% Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 11240 C /root/caffe/build/tools/caffe 7151MiB |
| 1 11240 C /root/caffe/build/tools/caffe 9715MiB |
| 2 11240 C /root/caffe/build/tools/caffe 9715MiB |
| 3 11240 C /root/caffe/build/tools/caffe 9715MiB |

I0606 14:26:59.799461 18501 sgd_solver.cpp:108] Iteration 234960, lr = 0.053008
I0606 14:27:21.995661 18501 solver.cpp:441] Iteration 235000, Testing net (#0)
I0606 14:28:05.918299 18499 data_reader.cpp:140] Restarting data pre-fetching

[CFAndy]

still there

[drnikolaev]

Thanks, reopening. Was this state file created in 0.15 release?

[CFAndy]

caffe-0.16+cuDNNv6

[CFAndy]

do you want me to test my fix on 0.16 or you can fix it?

[CFAndy]

create a fix for 0.16 #350.

[drnikolaev]

Fixed in v0.16.2