chore(inputs.kinesis_consumer): Replace consumer library by own implementation

[srebhan]

Summary

The current implementation of the kinesis consumer relies on the github.com/harlow/kinesis-consumer library. This library seems not to be maintained anymore. Furthermore, the mentioned library does not harmonize well with Telegraf when it comes to storing shard iterators in a DynamoDB as Telegraf wants the iterators to represent the latest metric delivered and the library wants to store the last record consumed.

To overcome these issues and to reduce the number of reads from the DynamoDB in case it is configured this PR implements the consumer itself and now only relies on the AWS Golang SDK v2.

This PR has been tested in a toy scenario including

• consuming a stream without checkpointing iterators in DynamoDB
• consuming a stream with checkpointing iterators in DynamoDB
• resuming a not-fully consumed stream with iterators from DynamoDB
• resharding in Kinesis with both removing and adding shards
• consuming with blocked outputs to run into metrics exceed max_undelivered_messages.

When enabling the tracing log-level for this plugin, details on the consumption process are logged.

Checklist

• No AI generated code was used in this PR

Related issues

[mstrandboge]

@srebhan Could you rebase this PR?

[telegraf-tiger]

Download PR build artifacts for linux_amd64.tar.gz, darwin_arm64.tar.gz, and windows_amd64.zip.
Downloads for additional architectures and packages are available below.

☺️ This pull request doesn't significantly change the Telegraf binary size (less than 1%)

📦 Click here to get additional PR build artifacts

Artifact URLs

DEB	RPM	TAR GZ	ZIP
amd64.deb	aarch64.rpm	darwin_amd64.tar.gz	windows_amd64.zip
arm64.deb	armel.rpm	darwin_arm64.tar.gz	windows_arm64.zip
armel.deb	armv6hl.rpm	freebsd_amd64.tar.gz	windows_i386.zip
armhf.deb	i386.rpm	freebsd_armv7.tar.gz
i386.deb	ppc64le.rpm	freebsd_i386.tar.gz
mips.deb	riscv64.rpm	linux_amd64.tar.gz
mipsel.deb	s390x.rpm	linux_arm64.tar.gz
ppc64el.deb	x86_64.rpm	linux_armel.tar.gz
riscv64.deb		linux_armhf.tar.gz
s390x.deb		linux_i386.tar.gz
		linux_mips.tar.gz
		linux_mipsel.tar.gz
		linux_ppc64le.tar.gz
		linux_riscv64.tar.gz
		linux_s390x.tar.gz

[andriyfedorov]

@srebhan This PR appears to have introduced an issue with the Kinesis consumer, at least in our environment. It operates correctly with Kinesis streams that have a relatively small number of shards but stops consuming messages when the shard count reaches approximately 20 or more. No errors are reported — the consumer simply stops receiving data.

The last known working version of Telegraf is v1.33.3.