Migrate Beam benchmark implementation
Co-authored-by: Jan Bensien stu128012@mail.uni-kiel.de
This merge request will fix #147 (closed). It is based on the existing implementation in the https://git.se.informatik.uni-kiel.de/stu128012/theodolite/-/tree/beam_current_theodolite fork but has to tackle the following requirements:
-
Migrate the individual benchmarks with their new enumeration -
Migrate UC1 -
Migrate UC2 (f.k.a. UC3) -
Migrate UC3 (f.k.a. UC4) -
Migrate UC4 (f.k.a. UC2)
-
-
Migrate to Titan Control Center Commons 0.1.0 (Avro records and Schema Registry are already used) -
Fix code quality issues if any -
Support Docker builds (already available in the base branch) -
Build and push Docker images in CI pipeline -
Update Docker Compose files in docker-test -
Share code among Beam runners (either create subprojects or dedicated Gradle tasks or dedicated Dockerfiles or environment variables or ...) - Current approach is to use different subprojects.
Other approaches would increase the complexity and is currently not possible because of:
- https://issues.apache.org/jira/browse/BEAM-9830
- https://stackoverflow.com/questions/61441333/conflict-with-runner-dependencies-in-beam
-
build beam commons project (for builing configurations, common PT transformations) -
outsource uc1-4 pipelines into single projects -
outsource uc1-4 beam samza into single projects -
outsource uc1-4 beam flink into single projects
- Current approach is to use different subprojects.
Other approaches would increase the complexity and is currently not possible because of:
-
Clean up Gradle build files -
Build Gradle plugins: -
Build beam commons plugin -
Build beam flink plugin -
Build beam samza plugin
-
-
Check if same versions of dependecies are used (slf4, etc)
-
-
Provide corresponding Kubernetes benchmark definitions -> Moved to #281 (closed) -
Solve #278 (closed) -
Fix #293 (closed) -
Solve #291 (not part of this MR) -
Make samza configurable through environment variables -
Resolve ./gradlew build --warning-mode all -
@she Image names in Docker Compose files must be replaced when merging and pushing images
Depends on !188 (merged)
Edited by Sören Henning
Merge request reports
Activity
Filter activity
added modul/benchmarks label
added 1 commit
- 5c2b370b - Add migrated uc4-beam-samza to settings.gradle
added 1 commit
- a0bb0d9b - uc3-beam-flink Rm cs + sp warnings but pmd remain
added 1 commit
- 72d2186b - Externalize DuplicateAsFlatMap uc4-beam-samza
added 1 commit
- b9407d77 - Fix more cs, pmd, spotbugs warnings and errors in uc4-beam-samza
added 9 commits
-
b9407d77...3f1b94c2 - 8 commits from branch
she:master - cb48d6b5 - Merge branch 'master' into feature/147-add-beam-implementations
-
b9407d77...3f1b94c2 - 8 commits from branch
Please register or sign in to reply