Merge branch 'main' into add-dropped-records-slos

CITATION.cff

@@ -11,7 +11,7 @@ title: Theodolite
 version: "0.8.2"
 repository-code: "https://github.com/cau-se/theodolite"
 license: "Apache-2.0"
-doi: "10.1016/j.bdr.2021.100209"
+doi: "10.1007/s10664-022-10162-1"
 preferred-citation:
   type: article
   authors:
@@ -21,9 +21,9 @@ preferred-citation:
     - family-names: Hasselbring
       given-names: Wilhelm
       orcid: "https://orcid.org/0000-0001-6625-4335"
-  doi: "10.1016/j.bdr.2021.100209"
-  journal: "Big Data Research"
-  month: 7
-  title: "Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures"
-  volume: 25
-  year: 2021
+  doi: "10.1007/s10664-022-10162-1"
+  journal: "Empirical Software Engineering"
+  month: 8
+  title: "A Configurable Method for Benchmarking Scalability of Cloud-Native Applications"
+  volume: 27
+  year: 2022

README.md

@@ -4,7 +4,7 @@
 
 > A theodolite is a precision optical instrument for measuring angles between designated visible points in the horizontal and vertical planes.  -- <cite>[Wikipedia](https://en.wikipedia.org/wiki/Theodolite)</cite>
 
-Theodolite is a framework for benchmarking the horizontal and vertical scalability of cloud-native applications.
+Theodolite is a framework for benchmarking the horizontal and vertical scalability of cloud-native applications in Kubernetes.
 
 ## Quickstart
 
@@ -23,8 +23,21 @@ Documentation on Theodolite itself as well as regarding its benchmarking method
 * The source code of Theodolite's SLO checkers are located in [`slo-checker`](slo-checker).
 * The documentation, which is hosted on [theodolite.rocks](https://www.theodolite.rocks), is located in [`docs`](docs).
 
+## Contributing
+
+We are happy to accept any kind of contributions to Theodolite.
+This includes reporting any issues you find using Theodolite, bug fixes and improvements as well as integrating your research within the project.
+
+See our website to [start contributing](https://www.theodolite.rocks/development/).
+
 ## How to Cite
 
 If you use Theodolite, please cite
 
-> Sören Henning and Wilhelm Hasselbring. (2021). Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures. Big Data Research, Volume 25. DOI: [10.1016/j.bdr.2021.100209](https://doi.org/10.1016/j.bdr.2021.100209). arXiv:[2009.00304](https://arxiv.org/abs/2009.00304).
+> Sören Henning and Wilhelm Hasselbring. “A Configurable Method for Benchmarking Scalability of Cloud-Native Applications”. In: *Empirical Software Engineering* 27. 2022. DOI: [10.1007/s10664-022-10162-1](https://doi.org/10.1007/s10664-022-10162-1).
+
+When referring to our stream processing benchmarks, please cite
+
+> Sören Henning and Wilhelm Hasselbring. “Theodolite: Scalability Benchmarking of Distributed Stream Processing Engines in Microservice Architectures”. In: *Big Data Research* 25. 2021. DOI: [10.1016/j.bdr.2021.100209](https://doi.org/10.1016/j.bdr.2021.100209). arXiv:[2009.00304](https://arxiv.org/abs/2009.00304).
+
+See our website for a [list of publications](https://www.theodolite.rocks/publications.html) directly related to Theodolite.
\ No newline at end of file

buildimages/README.md

+# Theodolite Build Images
+
+This directory contains some Dockerfiles for images required for Theodolite build infrastructure.

codemeta.json

@@ -14,7 +14,7 @@
     "relatedLink": [
         "https://www.theodolite.rocks"
     ],
-    "referencePublication": "https://doi.org/10.1016/j.bdr.2021.100209",
+    "referencePublication": "https://doi.org/10.1007/s10664-022-10162-1",
     "programmingLanguage": [
         "Kotlin",
         "Java",

docs/Gemfile.lock

@@ -239,7 +239,7 @@ GEM
       jekyll-seo-tag (~> 2.1)
     minitest (5.15.0)
     multipart-post (2.1.1)
-    nokogiri (1.13.9-x86_64-linux)
+    nokogiri (1.13.10-x86_64-linux)
       racc (~> 1.4)
     octokit (4.22.0)
       faraday (>= 0.9)
@@ -248,7 +248,7 @@ GEM
     pathutil (0.16.2)
       forwardable-extended (~> 2.6)
     public_suffix (4.0.7)
-    racc (1.6.0)
+    racc (1.6.1)
     rainbow (3.1.1)
     rb-fsevent (0.11.1)
     rb-inotify (0.10.1)

docs/development/index.md

 ---
-title: Development
+title: Contributing
 has_children: true
 nav_order: 10
 ---
+
+# Contributing
+
+Theodolite is open-source research software. We welcome everyone to contribute to this project.
+Contributions are not limited to code contributions, instead we welcome and recognize everything concerning:
+
+* Raising issues, questions and suggestions for using Theodolite
+* Fixing bugs or implementing new features
+* Improving the documentation
+* Using Theodolite as part of your (not necessarily scientific) research
+* Reporting on your scalability evaluations with Theodolite
+
+## Start Contributing
+
+If you have bug reports, feature requests, questions or suggestions, you may create a [GitHub issue](https://github.com/cau-se/theodolite/issues) or directly [contact Theodolite's maintainers](../project-info).
+You can also create a [GitHub pull request](https://github.com/cau-se/theodolite/pulls) if you have already implemented bug fixes and improvements.
+
+If you would like to get more involved in Theodolite's project development and maintenance, you may contact us as well so we can set you up an account for [our internal GitLab](../project-info#project-management).
+
+## Internal Project Structure
+
+Theodolite is organized as a monorepo containing multiple largely independent modules in subdirectories.
+See the project's [`README.md`](https://github.com/cau-se/theodolite/blob/main/README.md#project-structure) for an overview of all modules.
+Each module directory provides a dedicated `README.md` file describing how to build, test, package,... the corresponding module.

docs/development/release-process.md

 ---
 title: Release Process
 has_children: false
-parent: Development
+parent: Contributing
 nav_order: 1
 ---
 

docs/project-info.md

@@ -15,7 +15,20 @@ You might also want to raise an issue on [GitHub](http://github.com/cau-se/theod
 
 ## Project Management
 
-Theodolite's internal development including issue boards, merge requests and extensive CI pipelines is tracked in our [internal GitLab](https://git.se.informatik.uni-kiel.de/she/theodolite). We provide a public mirror on GitHub, [cau-se/theodolite](http://github.com/cau-se/theodolite), where we are also happy to welcome issues and pull requests.
+Theodolite's internal development including issue boards, merge requests and extensive CI pipelines is tracked in our [internal GitLab](https://git.se.informatik.uni-kiel.de/she/theodolite).
+While all internal development is publicly accessible, contributing requires an account to be set up.
+To ease contribution, we provide a public mirror on GitHub, [cau-se/theodolite](http://github.com/cau-se/theodolite), where we are also happy to welcome issues and pull requests.
+Also releases are published via GitHub. See the following table for an overview:
+
+| Project management | Public GitHub | Internal GitLab |
+|:---|:---|:---|
+| Source code | [GitHub](https://github.com/cau-se/theodolite) | [GitLab](https://git.se.informatik.uni-kiel.de/she/theodolite) |
+| Issue Tracking | [GitHub Issues](https://github.com/cau-se/theodolite/issues) | [GitLab Issues](https://git.se.informatik.uni-kiel.de/she/theodolite/-/issues) |
+| Pull/Merge requests | [GitHub Pull requests](https://github.com/cau-se/theodolite/pulls) | [GitLab Merge requests](https://git.se.informatik.uni-kiel.de/she/theodolite/-/merge_requests) | 
+| Roadmap | | [GitLab Milestones](https://git.se.informatik.uni-kiel.de/she/theodolite/-/milestones) | 
+| CI/CD pipelines | | [GitLab CI/CD](https://git.se.informatik.uni-kiel.de/she/theodolite/-/pipelines) | 
+| Releases   | [GitHub Releases](https://github.com/cau-se/theodolite/releases) | [GitLab Releases](https://git.se.informatik.uni-kiel.de/she/theodolite/-/releases) | 
+| Container images | [GitHub Packages](https://github.com/orgs/cau-se/packages?repo_name=theodolite) | |
 
 ## Contributors
 

theodolite-benchmarks/README.md

@@ -4,6 +4,16 @@ Theodolite comes with a set of 4 benchmarks for event-driven microservices, whic
 The benchmarks are based on typical use cases for stream processing and named: UC1, UC2, UC3 and UC4.
 Additionally, we include a load generator for each benchmark.
 
+## Project organization
+
+All benchmark implementations are organized in a Gradle multi-project. See the [`settings.gradle`](settings.gradle) file for an overview of subprojects and how they are organized.
+We also use Gradle convention plugins, organized in [`buildSrc`](buildSrc), for sharing build configuration among subprojects.
+
+Additionally, this directory contains:
+
+* *Theodolite* Benchmark definitions for all benchmarks in [`definitions`](definitions).
+* Docker Compose files to assist in local development and to run smoke tests in [`docker-test`](docker-test).
+
 ## Building and packaging the benchmarks
 
 All benchmarks can be built with:

theodolite-benchmarks/beam-commons/src/main/java/rocks/theodolite/benchmarks/commons/beam/kafka/EventTimePolicy.java

@@ -10,8 +10,8 @@ import rocks.theodolite.benchmarks.commons.model.records.ActivePowerRecord;
 /**
  * TimeStampPolicy to use event time based on the timestamp of the record value.
  */
-public class EventTimePolicy
-    extends TimestampPolicy<String, ActivePowerRecord> {
+public class EventTimePolicy extends TimestampPolicy<String, ActivePowerRecord> {
+
   protected Instant currentWatermark;
 
   public EventTimePolicy(final Optional<Instant> previousWatermark) {
@@ -19,7 +19,6 @@ public class EventTimePolicy
     this.currentWatermark = previousWatermark.orElse(BoundedWindow.TIMESTAMP_MIN_VALUE);
   }
 
-
   @Override
   public Instant getTimestampForRecord(final PartitionContext ctx,
       final KafkaRecord<String, ActivePowerRecord> record) {

theodolite-benchmarks/beam-commons/src/main/java/rocks/theodolite/benchmarks/commons/beam/kafka/KafkaGenericReader.java

@@ -9,7 +9,7 @@ import org.apache.beam.sdk.values.PCollection;
 import org.apache.kafka.common.serialization.Deserializer;
 
 /**
- * Simple {@link PTransform} that read from Kafka using {@link KafkaIO}.
+ * Simple {@link PTransform} that reads from Kafka using {@link KafkaIO}.
  *
  * @param <K> Type of the Key.
  * @param <V> Type of the Value.
@@ -17,10 +17,11 @@ import org.apache.kafka.common.serialization.Deserializer;
 public class KafkaGenericReader<K, V> extends PTransform<PBegin, PCollection<KV<K, V>>> {
 
   private static final long serialVersionUID = 2603286150183186115L;
+
   private final PTransform<PBegin, PCollection<KV<K, V>>> reader;
 
   /**
-   * Instantiates a {@link PTransform} that reads from Kafka with the given Configuration.
+   * Instantiates a {@link PTransform} that reads from Kafka with the given configuration.
    */
   public KafkaGenericReader(
       final String bootstrapServer,
@@ -30,13 +31,12 @@ public class KafkaGenericReader<K, V> extends PTransform<PBegin, PCollection<KV<
       final Class<? extends Deserializer<V>> valueDeserializer) {
     super();
 
-    // Check if boostrap server and inputTopic are defined
+    // Check if the boostrap server and the input topic are defined
     if (bootstrapServer.isEmpty() || inputTopic.isEmpty()) {
       throw new IllegalArgumentException("bootstrapServer or inputTopic missing");
     }
 
-    this.reader =
-        KafkaIO.<K, V>read()
+    this.reader = KafkaIO.<K, V>read()
         .withBootstrapServers(bootstrapServer)
         .withTopic(inputTopic)
         .withKeyDeserializer(keyDeserializer)

theodolite-benchmarks/definitions/uc3-beam-flink/uc3-beam-flink-benchmark-operator.yaml

@@ -81,6 +81,13 @@ spec:
           resource: "uc3-load-generator-deployment.yaml"
           properties:
             loadGenMaxRecords: "150000"
+    - typeName: "AggregationDurationDays"
+      patchers:
+        - type: "EnvVarPatcher"
+          resource: "jobmanager-deployment.yaml"
+          properties:
+            container: "jobmanager"
+            variableName: "AGGREGATION_DURATION_DAYS"
   slos:
     - name: "lag trend"
       sloType: "lag trend"

theodolite-benchmarks/definitions/uc3-beam-samza/uc3-beam-samza-benchmark-operator.yaml

@@ -70,6 +70,13 @@ spec:
           resource: "uc3-load-generator-deployment.yaml"
           properties:
             loadGenMaxRecords: "150000"
+    - typeName: "AggregationDurationDays"
+      patchers:
+        - type: "EnvVarPatcher"
+          resource: "uc3-beam-samza-deployment.yaml"
+          properties:
+            container: "uc3-beam-samza"
+            variableName: "AGGREGATION_DURATION_DAYS"
   slos:
     - name: "lag trend"
       sloType: "lag trend"

theodolite-benchmarks/definitions/uc3-flink/uc3-flink-benchmark-operator.yaml

@@ -81,6 +81,13 @@ spec:
           resource: "uc3-load-generator-deployment.yaml"
           properties:
             loadGenMaxRecords: "150000"
+    - typeName: "AggregationDurationDays"
+      patchers:
+        - type: "EnvVarPatcher"
+          resource: "jobmanager-deployment.yaml"
+          properties:
+            container: "jobmanager"
+            variableName: "AGGREGATION_DURATION_DAYS"
   slos:
     - name: "lag trend"
       sloType: "lag trend"

theodolite-benchmarks/definitions/uc3-hazelcastjet/uc3-hazelcastjet-benchmark-operator.yaml

@@ -56,6 +56,13 @@ spec:
           resource: "uc3-load-generator-deployment.yaml"
           properties:
             loadGenMaxRecords: "150000"
+    - typeName: "AggregationDurationDays"
+      patchers:
+        - type: "EnvVarPatcher"
+          resource: "uc3-hazelcastjet-deployment.yaml"
+          properties:
+            container: "uc-application"
+            variableName: "AGGREGATION_DURATION_DAYS"
   slos:
     - name: "lag trend"
       sloType: "lag trend"

theodolite-benchmarks/definitions/uc3-kstreams/uc3-kstreams-benchmark-operator.yaml

@@ -61,6 +61,13 @@ spec:
           resource: "uc3-load-generator-deployment.yaml"
           properties:
             loadGenMaxRecords: "150000"
+    - typeName: "AggregationDurationDays"
+      patchers:
+        - type: "EnvVarPatcher"
+          resource: "uc3-kstreams-deployment.yaml"
+          properties:
+            container: "uc-application"
+            variableName: "AGGREGATION_DURATION_DAYS"
   slos:
     - name: "lag trend"
       sloType: "lag trend"

theodolite-benchmarks/docker-test/uc4-beam-flink/docker-compose.yml

@@ -67,6 +67,8 @@ services:
           parallelism.default: 1
           state.backend: rocksdb
           state.checkpoints.dir: file:///data/flink/checkpoints
+        - GRACE_PERIOD_MS=5000
+        - TRIGGER_INTERVAL_SECONDS=1
       depends_on:
         - schema-registry
         - kafka

theodolite-benchmarks/docker-test/uc4-beam-flink/test.sh

@@ -2,7 +2,7 @@
 
 until docker-compose exec -T kcat kcat -L -b kafka:9092 -t output -J | jq -r '.topics[0].partitions | length' | grep "\b3\b"; do sleep 5s; done
 
-docker-compose exec -T kcat kcat -C -b kafka:9092 -t output -s key=s -s value=avro -r http://schema-registry:8081 -f '%k:%s\n' -c 2000 |
+docker-compose exec -T kcat kcat -C -b kafka:9092 -t output -s key=s -s value=avro -r http://schema-registry:8081 -f '%k:%s\n' -c 300 |
     tee /dev/stderr |
     awk -F ':' '!/^%/ {print $1}' |
     sort |

theodolite-benchmarks/docker-test/uc4-beam-samza/docker-compose.yml

@@ -52,6 +52,8 @@ services:
       KAFKA_BOOTSTRAP_SERVERS: kafka:9092
       SCHEMA_REGISTRY_URL: http://schema-registry:8081
       ENABLE_METRICS: "false"
+      GRACE_PERIOD_MS: 5000
+      TRIGGER_INTERVAL_SECONDS: 1
   load-generator: 
     image: ghcr.io/cau-se/theodolite-uc4-workload-generator:${THEODOLITE_TAG:-latest}
     depends_on:

theodolite-benchmarks/docker-test/uc4-beam-samza/test.sh

@@ -2,7 +2,7 @@
 
 until docker-compose exec -T kcat kcat -L -b kafka:9092 -t output -J | jq -r '.topics[0].partitions | length' | grep "\b3\b"; do sleep 5s; done
 
-docker-compose exec -T kcat kcat -C -b kafka:9092 -t output -s key=s -s value=avro -r http://schema-registry:8081 -f '%k:%s\n' -c 2000 |
+docker-compose exec -T kcat kcat -C -b kafka:9092 -t output -s key=s -s value=avro -r http://schema-registry:8081 -f '%k:%s\n' -c 300 |
     tee /dev/stderr |
     awk -F ':' '!/^%/ {print $1}' |
     sort |