Writing match log to kafka is not robust

[yihuang]

I think current way of writing to kafka is not robust, in case of match engine crash, some log messages not yet written into kafka successfully may lost.

[greensheng]

We have considered the case of process crash. Through the snapshot mechanism to ensure that any log will not be lost, when the process is restarted, a snapshot recovery will be done. Therefore, there will be no data loss, but duplicate logs will be generated.

[yihuang]

Thanks for your reply and work. Let me elaborate a bit:
1、Do you ensure that when snapshotting, all the messages produced to kafka get acked?
2、AFAIK kafka producer doesn't guarantee message order, does that impact the following processes.

[greensheng]

Thanks for your reply and work. Let me elaborate a bit:
1、Do you ensure that when snapshotting, all the messages produced to kafka get acked?
2、AFAIK kafka producer doesn't guarantee message order, does that impact the following processes.

1，Yes, We use synchronous write to submit logs, writing 1000 logs at a time.When we save the snapshot, we will first check whether the snapshot.logSeq has been submitted to Kafka.
2，Only one partition is used for each log topic, so the sequence can be guaranteed.

[yihuang]

2, The retry behavior in the producer side will change the order of messages.

[greensheng]

2, The retry behavior in the producer side will change the order of messages.

Log producer is synchronous and single thread, and there will be no concurrent submission.

[yihuang]

https://stackoverflow.com/questions/49007592/kafka-message-ordering-in-partition-while-producer-retry

[greensheng]

I see what you mean. I read the source code of Kafka go. Retrying really leads to the order not being guaranteed.

At present, if the log consumer finds the log.seq discontinuity, it will terminate the process.

Do you think enable. idempotence can solve this problem?

[yihuang]

Yes, enable.idempotence = true suppose to solve this, if kafka go is a binding to librdkafka.