Unstable record consumption with version 0.2

• Left experiment shows stable record consumption with v0.1
• Right experiments show a massive increase in consumed messages with v0.2, exactly every 5 minutes.

changed milestone to %0.2.0

added Bug label

Version 0.1 with the current (v0.2) UC images and a Kieker/Avro adapter provides the same results as v0.1

When using version 0.1 with the current (v0.2) UC images and the current (v0.2) workload generator, the problem occurs again.

Reason for this issue is most likely due to Kafka's internal cleanup. The Kafka broker config log.retention.check.interval.ms (which is set to 5 minutes per default) controls how often expired records are deleted. Since we set log retention time to 10 seconds, every 5 minutes all record which are older than 10 seconds are deleted. This causes the consumer to skip a lot of offsets, resulting in a significant increase in the lag.

We have yet to explore why this happens only with the new workload generator.

Producer config is the same for both workload generators (v0.1 and v0.2), except usage of schema registry. So this seems not causing the issue.

Next step to investigate: Usage of schema registry

Clean up takes place despite of using Kieker or Avro/SchemaRegistry records.

The issue arise due to a bug in AbstractWorkloadGenerator.java#L101, where the message (and thus its timestamp) is only generated once instead of a new message, whenever it should be sent.

assigned to @she

This issue is resolved with 1ff31ed0

closed