Migrate Beam benchmark implementation
-
Review changes -
-
Download -
Patches
-
Plain diff
Merged
Lorenz Boguhn requested to merge stu203404/theodolite:feature/147-add-beam-implementations into master
Co-authored-by: Jan Bensien stu128012@mail.uni-kiel.de
This merge request will fix #147 (closed). It is based on the existing implementation in the https://git.se.informatik.uni-kiel.de/stu128012/theodolite/-/tree/beam_current_theodolite fork but has to tackle the following requirements:
-
Migrate the individual benchmarks with their new enumeration -
Migrate UC1 -
Migrate UC2 (f.k.a. UC3) -
Migrate UC3 (f.k.a. UC4) -
Migrate UC4 (f.k.a. UC2)
-
-
Migrate to Titan Control Center Commons 0.1.0 (Avro records and Schema Registry are already used) -
Fix code quality issues if any -
Support Docker builds (already available in the base branch) -
Build and push Docker images in CI pipeline -
Update Docker Compose files in docker-test
-
Share code among Beam runners (either create subprojects or dedicated Gradle tasks or dedicated Dockerfiles or environment variables or ...) - Current approach is to use different subprojects.
Other approaches would increase the complexity and is currently not possible because of:
- https://issues.apache.org/jira/browse/BEAM-9830
- https://stackoverflow.com/questions/61441333/conflict-with-runner-dependencies-in-beam
-
build beam commons project (for builing configurations, common PT transformations) -
outsource uc1-4 pipelines into single projects -
outsource uc1-4 beam samza into single projects -
outsource uc1-4 beam flink into single projects
- Current approach is to use different subprojects.
Other approaches would increase the complexity and is currently not possible because of:
-
Clean up Gradle build files -
Build Gradle plugins: -
Build beam commons plugin -
Build beam flink plugin -
Build beam samza plugin
-
-
Check if same versions of dependecies are used (slf4, etc)
-
-
Provide corresponding Kubernetes benchmark definitions -> Moved to #281 (closed) -
Solve #278 (closed) -
Fix #293 (closed) -
Solve #291 (not part of this MR) -
Make samza configurable through environment variables -
Resolve ./gradlew build --warning-mode all -
@she Image names in Docker Compose files must be replaced when merging and pushing images
Depends on !188 (merged)
Edited by Sören Henning
Merge request reports
Compare and
- version 76eee35715
- version 75cf673c2e
- version 74c0ff9842
- version 738f3c0c91
- version 721068dfed
- version 71befcf2e1
- version 708d2dd54a
- version 6937300817
- version 680689e593
- version 67a68e5c6c
- version 667db18708
- version 6596e671a5
- version 6407e5b705
- version 63430e158e
- version 625da264ed
- version 6113f311b9
- version 60dcc167a4
- version 591a6a0917
- version 58b516d4ab
- version 5729dba57a
- version 56df125ad6
- version 55f5a9367f
- version 54c69d3578
- version 535e3059f6
- version 52fae58895
- version 51802d396d
- version 505a400162
- version 49f56fc513
- version 48b6d305b4
- version 47039f458d
- version 4659888afb
- version 45f0370c7e
- version 449409548c
- version 435406b743
- version 42c8633798
- version 41f0fe3fe4
- version 40c17d9ab8
- version 39ebc42fe7
- version 382be456a4
- version 37777ad136
- version 362e825d40
- version 352c0bdaa2
- version 34034c17c7
- version 3370492a50
- version 32d24d68fb
- version 316f24de85
- version 3000c0c259
- version 29a7bbf1c9
- version 28786b874b
- version 271e6586cf
- version 26e275c86f
- version 2599313bef
- version 240c707a65
- version 23f55de283
- version 22f55de283
- version 21f0cb5638
- version 2060863606
- version 193235b855
- version 18dede63aa
- version 1786d4732c
- version 1666ae4c77
- version 15cb48d6b5
- version 14b9407d77
- version 1372d2186b
- version 12b32a44dd
- version 11cf32613b
- version 10e76dea6e
- version 91350835e
- version 880741887
- version 7a0bb0d9b
- version 65c2b370b
- version 5dd7c2ea5
- version 4c8034d6c
- version 385f1f108
- version 226020df8
- version 158059c9a
- master (base)
- latest version93ec2862165 commits,
- version 76eee35715164 commits,
- version 75cf673c2e163 commits,
- version 74c0ff9842162 commits,
- version 738f3c0c91161 commits,
- version 721068dfed160 commits,
- version 71befcf2e1159 commits,
- version 708d2dd54a158 commits,
- version 6937300817157 commits,
- version 680689e593155 commits,
- version 67a68e5c6c154 commits,
- version 667db18708153 commits,
- version 6596e671a5149 commits,
- version 6407e5b705147 commits,
- version 63430e158e145 commits,
- version 625da264ed144 commits,
- version 6113f311b9141 commits,
- version 60dcc167a4140 commits,
- version 591a6a0917139 commits,
- version 58b516d4ab138 commits,
- version 5729dba57a137 commits,
- version 56df125ad6136 commits,
- version 55f5a9367f135 commits,
- version 54c69d3578133 commits,
- version 535e3059f6130 commits,
- version 52fae58895127 commits,
- version 51802d396d121 commits,
- version 505a400162120 commits,
- version 49f56fc513119 commits,
- version 48b6d305b4117 commits,
- version 47039f458d114 commits,
- version 4659888afb110 commits,
- version 45f0370c7e109 commits,
- version 449409548c104 commits,
- version 435406b743102 commits,
- version 42c863379899 commits,
- version 41f0fe3fe498 commits,
- version 40c17d9ab897 commits,
- version 39ebc42fe796 commits,
- version 382be456a495 commits,
- version 37777ad13691 commits,
- version 362e825d4090 commits,
- version 352c0bdaa287 commits,
- version 34034c17c783 commits,
- version 3370492a5081 commits,
- version 32d24d68fb77 commits,
- version 316f24de8576 commits,
- version 3000c0c25975 commits,
- version 29a7bbf1c972 commits,
- version 28786b874b69 commits,
- version 271e6586cf68 commits,
- version 26e275c86f67 commits,
- version 2599313bef66 commits,
- version 240c707a6565 commits,
- version 23f55de28330 commits,
- version 22f55de28330 commits,
- version 21f0cb563832 commits,
- version 206086360630 commits,
- version 193235b85529 commits,
- version 18dede63aa24 commits,
- version 1786d4732c23 commits,
- version 1666ae4c7722 commits,
- version 15cb48d6b520 commits,
- version 14b9407d7719 commits,
- version 1372d2186b18 commits,
- version 12b32a44dd17 commits,
- version 11cf32613b16 commits,
- version 10e76dea6e15 commits,
- version 91350835e14 commits,
- version 88074188713 commits,
- version 7a0bb0d9b12 commits,
- version 65c2b370b11 commits,
- version 5dd7c2ea510 commits,
- version 4c8034d6c9 commits,
- version 385f1f1088 commits,
- version 226020df85 commits,
- version 158059c9a1 commit,
Compare changes
- Side-by-side
- Inline
Files
84Loading