Migrate Beam benchmark implementation

Co-authored-by: Jan Bensien stu128012@mail.uni-kiel.de

This merge request will fix #147 (closed). It is based on the existing implementation in the https://git.se.informatik.uni-kiel.de/stu128012/theodolite/-/tree/beam_current_theodolite fork but has to tackle the following requirements:

• Migrate the individual benchmarks with their new enumeration
  • Migrate UC1
  • Migrate UC2 (f.k.a. UC3)
  • Migrate UC3 (f.k.a. UC4)
  • Migrate UC4 (f.k.a. UC2)
• Migrate to Titan Control Center Commons 0.1.0 (Avro records and Schema Registry are already used)
• Fix code quality issues if any
• Support Docker builds (already available in the base branch)
• Build and push Docker images in CI pipeline
• Update Docker Compose files in docker-test
• Share code among Beam runners (either create subprojects or dedicated Gradle tasks or dedicated Dockerfiles or environment variables or ...)
  • Current approach is to use different subprojects. Other approaches would increase the complexity and is currently not possible because of:
    • https://issues.apache.org/jira/browse/BEAM-9830
    • https://stackoverflow.com/questions/61441333/conflict-with-runner-dependencies-in-beam
    • build beam commons project (for builing configurations, common PT transformations)
    • outsource uc1-4 pipelines into single projects
    • outsource uc1-4 beam samza into single projects
    • outsource uc1-4 beam flink into single projects
• Clean up Gradle build files
  • Build Gradle plugins:
    • Build beam commons plugin
    • Build beam flink plugin
    • Build beam samza plugin
  • Check if same versions of dependecies are used (slf4, etc)
• Provide corresponding Kubernetes benchmark definitions -> Moved to #281 (closed)
• Solve #278 (closed)
• Fix #293 (closed)
• Solve #291 (not part of this MR)
• Make samza configurable through environment variables
• Resolve ./gradlew build --warning-mode all
• @she Image names in Docker Compose files must be replaced when merging and pushing images

Depends on !188 (merged)

theodolite-benchmarks/docker-test/uc4-beam-flink/docker-compose.yml

@@ -31,14 +31,6 @@ services:
     environment:
       SCHEMA_REGISTRY_HOST_NAME: schema-registry
       SCHEMA_REGISTRY_KAFKASTORE_CONNECTION_URL: 'zookeeper:2181'
-  benchmark:
-    image: uc4-beam-flink:latest
-    depends_on:
-      - schema-registry
-      - kafka
-    environment:
-      KAFKA_BOOTSTRAP_SERVERS: kafka:9092
-      SCHEMA_REGISTRY_URL: http://schema-registry:8081
   load-generator: 
     image: ghcr.io/cau-se/theodolite-uc4-workload-generator:latest
     depends_on:
@@ -51,3 +43,29 @@ services:
       SCHEMA_REGISTRY_URL: http://schema-registry:8081
       NUM_SENSORS: 4
       NUM_NESTED_GROUPS: 4
+  benchmark-jobmanager:
+      image: uc4-beam-flink:latest
+      ports:
+        - "8080:8081"
+      command: standalone-job --job-classname application.Uc4BeamFlink
+      environment:
+        - PARALLELISM=1
+        - KAFKA_BOOTSTRAP_SERVERS=kafka:9092
+        - SCHEMA_REGISTRY_URL=http://schema-registry:8081
+        - |
+          FLINK_PROPERTIES=
+          jobmanager.rpc.address: benchmark-jobmanager
+          parallelism.default: 1
+      depends_on:
+        - schema-registry
+        - kafka
+  benchmark-taskmanager:
+      image: uc4-beam-flink:latest
+      command: taskmanager
+      environment:
+        - |
+          FLINK_PROPERTIES=
+          jobmanager.rpc.address: benchmark-jobmanager
+      depends_on:
+        - schema-registry
+        - kafka