Iceberg Kafka Connector experiences a constant hanging lag for low-volume topics

[fenil25]

Apache Iceberg version

1.7.1 (latest release)

Query engine

Kafka Connect

Please describe the bug 🐞

I have iceberg kafka-connect sink and when the volume of records in topic is low, iceberg connector doesn't consume any records.
One would see a constant lag and iceberg will never produce any records and commit to the table. Once, we have enough records, all the records will be pushed through and committed to the table.

example -

/bin/kafka-consumer-groups.sh --bootstrap-server cdc-kafka-prod.internal.ec2.tinyspeck.com:9092 --describe --group cg-control-payments_group
Consumer group 'cg-control-payments_group' has no active members.

TOPIC                    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
prod.aux2.payments_group 0          45              46              1               -               -               -
prod.aux2.payments_group 1          41              42              1               -               -               -
prod.aux2.payments_group 2          35              36              1               -               -               -
prod.aux2.payments_group 3          32              33              1               -               -               -

./bin/kafka-consumer-groups.sh --bootstrap-server cdc-kafka-prod.internal.ec2.tinyspeck.com:9092 --describe --group cg-control-payments_group
Consumer group 'cg-control-payments_group' has no active members.

TOPIC                    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID
prod.aux2.payments_group 0          45              46              1               -               -               -
prod.aux2.payments_group 1          41              42              1               -               -               -
prod.aux2.payments_group 2          35              36              1               -               -               -
prod.aux2.payments_group 3          32              33              1               -               -               -

I am observing this constant lag of 1 for all partition since a day now. No commit has been made to the append-only table. I confirmed that the record exists in the topic and is not expired. fetch.min.bytes is also set to default value of 1.
When new records flow through the topic, all the records are committed to the iceberg table. But, this comes with a lot of unnecessary delay.

Willingness to contribute

• I can contribute a fix for this bug independently
• I would be willing to contribute a fix for this bug with guidance from the Iceberg community
• I cannot contribute a fix for this bug at this time