Decouple Theodolite and Kafka
We aim for establishing Theodolite as cloud-native scalability benchmarking tool, not restricted to stream processing. While in most parts, it should already be possible to benchmark arbitrary cloud-native applications, some adjustments are still required:
- Installing Kafka with Helm should remain possible to simplify Theodolite's usage (i.e., nothing to do here)
- Confluent's Kafka charts are maintained quite bad and cause a lot of problems (#179 (closed), #74 (closed)). It might better to
- either switch to another chart (e.g., Bitnami)
- or use a Kafka operator (e.g. Strimzi)
- When using an operator, we could create topics etc. easily as Kubernetes resources, meaning that we wouldn't have to do much here
- When not using an operator or using middlewares, which are not provided as operators (maybe ZooKeeper for Samza), we need to find another mechanism for "set up" and "tear down" actions. Best and most "cloud-native-ish" would probably be to use containers, pod, or jobs for that. (see #270 (closed))
- Additionally, we might want to make middleware resources (e.g., for Kafka) long-running, i.e., lasting for the entire execution lifespan and not to be re-created with each SLO experiment. Main reason for that the time required by, e.g., Kafka to start-up. A possible solution is let benchmarks declare a list of infrastructure Kubernetes resources. (see #272 (closed))
Edited by Sören Henning