Describe the feature:
The current transaction recovery feature of JanusGraph (see Transaction recovery) lacks to important features:
- the recovery process cannot be started multiple times (only if the graph is closed between the starts)
- the recovery process does not give information if a secondary failure that was attempted to be fixed was indeed fixed or not
To overcome the above limitations, our proposal contains two changes:
- provide a new
startRecurringTransactionRecovery() method that supports executing the recovery process multiple times without closing the graph
- next to success and failure counter of
getStatistics() method, provide also a third counter that expresses how many transaction could not be fixed from those that initially failed (and was attempted to be fixed during recovery).
Describe a specific use case for the feature:
In our setup, JanusGraph is running with Cassandra storage and ElasticSearch mixed index backend. We ingest data into our graph daily overnight. From time to time, ElasticSearch becomes unavailable during ingestion and the newly ingested data is not indexed by the mixed index backend. This eventually results in missing records when queries utilize mixed index backend.
To make our ingestion process more robust and reliable, we would like to run the Transaction recovery process every night after the ingestion to make sure that whatever is stored in the primary storage, it is also get indexed. If something was still not indexed after running the transaction recovery process, we need to be able to mark the ingestion process failed so manual intervention can be performed to fix any inconsistency between the primary and secondary persistence.
Describe the feature:
The current transaction recovery feature of JanusGraph (see Transaction recovery) lacks to important features:
To overcome the above limitations, our proposal contains two changes:
startRecurringTransactionRecovery()method that supports executing the recovery process multiple times without closing the graphgetStatistics()method, provide also a third counter that expresses how many transaction could not be fixed from those that initially failed (and was attempted to be fixed during recovery).Describe a specific use case for the feature:
In our setup, JanusGraph is running with Cassandra storage and ElasticSearch mixed index backend. We ingest data into our graph daily overnight. From time to time, ElasticSearch becomes unavailable during ingestion and the newly ingested data is not indexed by the mixed index backend. This eventually results in missing records when queries utilize mixed index backend.
To make our ingestion process more robust and reliable, we would like to run the Transaction recovery process every night after the ingestion to make sure that whatever is stored in the primary storage, it is also get indexed. If something was still not indexed after running the transaction recovery process, we need to be able to mark the ingestion process failed so manual intervention can be performed to fix any inconsistency between the primary and secondary persistence.