Retry on mongos operation failures

[durran]

This adds additional read retry scenarios on OperationFailures coming from a mongos only. This is because the ping that is executed against a running mongos always returns ready no matter what the state of the cluster behind it is, and the normal server selection/socket retry will not cover this scenario.

1. Adds a max_read_retries configuration option that is only applicable to sharded clusters when specific OperationFailures are raised.
2. Adds a read_retry_interval configuration option to set the wait time between retries.

Note that we can only look at the error messages themselves, and not the codes. The codes have changed between server versions and some are not relevant to the same scenario anymore, such as the commented 15988 - which was previously an "error querying server" in mongos but in 3.0 is a "FieldPath field names may not be empty strings", which is not a retryable error.

We don't scan the cluster in these scenarios as the mongos is up and running, and the scan would be a waste of a network call do to the ping reasons stated above.

[jonhyman]

"could not get last error" and "connection attempt failed" are others I think you should add here.

[durran]

Added.

[jonhyman]

Thank you for writing this. As mentioned, I'll do a series of failover tests on Tuesday and will report back.

[jonhyman]

Success! You need to merge in #693 to this, but then it's fantastic. We did various stepdowns and kill -9 of primaries and secondaries and it recovered well.

[durran]

Great!