Migrate Beam benchmark implementation
-
Review changes -
-
Download -
Patches
-
Plain diff
Merged
Lorenz Boguhn requested to merge stu203404/theodolite:feature/147-add-beam-implementations into master
Co-authored-by: Jan Bensien stu128012@mail.uni-kiel.de
This merge request will fix #147 (closed). It is based on the existing implementation in the https://git.se.informatik.uni-kiel.de/stu128012/theodolite/-/tree/beam_current_theodolite fork but has to tackle the following requirements:
-
Migrate the individual benchmarks with their new enumeration -
Migrate UC1 -
Migrate UC2 (f.k.a. UC3) -
Migrate UC3 (f.k.a. UC4) -
Migrate UC4 (f.k.a. UC2)
-
-
Migrate to Titan Control Center Commons 0.1.0 (Avro records and Schema Registry are already used) -
Fix code quality issues if any -
Support Docker builds (already available in the base branch) -
Build and push Docker images in CI pipeline -
Update Docker Compose files in docker-test
-
Share code among Beam runners (either create subprojects or dedicated Gradle tasks or dedicated Dockerfiles or environment variables or ...) - Current approach is to use different subprojects.
Other approaches would increase the complexity and is currently not possible because of:
- https://issues.apache.org/jira/browse/BEAM-9830
- https://stackoverflow.com/questions/61441333/conflict-with-runner-dependencies-in-beam
-
build beam commons project (for builing configurations, common PT transformations) -
outsource uc1-4 pipelines into single projects -
outsource uc1-4 beam samza into single projects -
outsource uc1-4 beam flink into single projects
- Current approach is to use different subprojects.
Other approaches would increase the complexity and is currently not possible because of:
-
Clean up Gradle build files -
Build Gradle plugins: -
Build beam commons plugin -
Build beam flink plugin -
Build beam samza plugin
-
-
Check if same versions of dependecies are used (slf4, etc)
-
-
Provide corresponding Kubernetes benchmark definitions -> Moved to #281 (closed) -
Solve #278 (closed) -
Fix #293 (closed) -
Solve #291 (not part of this MR) -
Make samza configurable through environment variables -
Resolve ./gradlew build --warning-mode all -
@she Image names in Docker Compose files must be replaced when merging and pushing images
Depends on !188 (merged)
Edited by Sören Henning
Merge request reports
Compare and
Show latest version
- version 76eee35715
- version 75cf673c2e
- version 74c0ff9842
- version 738f3c0c91
- version 721068dfed
- version 71befcf2e1
- version 708d2dd54a
- version 6937300817
- version 680689e593
- version 67a68e5c6c
- version 667db18708
- version 6596e671a5
- version 6407e5b705
- version 63430e158e
- version 625da264ed
- version 6113f311b9
- version 60dcc167a4
- version 591a6a0917
- version 58b516d4ab
- version 5729dba57a
- version 56df125ad6
- version 55f5a9367f
- version 54c69d3578
- version 535e3059f6
- version 52fae58895
- version 51802d396d
- version 505a400162
- version 49f56fc513
- version 48b6d305b4
- version 47039f458d
- version 4659888afb
- version 45f0370c7e
- version 449409548c
- version 435406b743
- version 42c8633798
- version 41f0fe3fe4
- version 40c17d9ab8
- version 39ebc42fe7
- version 382be456a4
- version 37777ad136
- version 362e825d40
- version 352c0bdaa2
- version 34034c17c7
- version 3370492a50
- version 32d24d68fb
- version 316f24de85
- version 3000c0c259
- version 29a7bbf1c9
- version 28786b874b
- version 271e6586cf
- version 26e275c86f
- version 2599313bef
- version 240c707a65
- version 23f55de283
- version 22f55de283
- version 21f0cb5638
- version 2060863606
- version 193235b855
- version 18dede63aa
- version 1786d4732c
- version 1666ae4c77
- version 15cb48d6b5
- version 14b9407d77
- version 1372d2186b
- version 12b32a44dd
- version 11cf32613b
- version 10e76dea6e
- version 91350835e
- version 880741887
- version 7a0bb0d9b
- version 65c2b370b
- version 5dd7c2ea5
- version 4c8034d6c
- version 385f1f108
- version 226020df8
- version 158059c9a
- master (base)
- latest version93ec2862165 commits,
- version 76eee35715164 commits,
- version 75cf673c2e163 commits,
- version 74c0ff9842162 commits,
- version 738f3c0c91161 commits,
- version 721068dfed160 commits,
- version 71befcf2e1159 commits,
- version 708d2dd54a158 commits,
- version 6937300817157 commits,
- version 680689e593155 commits,
- version 67a68e5c6c154 commits,
- version 667db18708153 commits,
- version 6596e671a5149 commits,
- version 6407e5b705147 commits,
- version 63430e158e145 commits,
- version 625da264ed144 commits,
- version 6113f311b9141 commits,
- version 60dcc167a4140 commits,
- version 591a6a0917139 commits,
- version 58b516d4ab138 commits,
- version 5729dba57a137 commits,
- version 56df125ad6136 commits,
- version 55f5a9367f135 commits,
- version 54c69d3578133 commits,
- version 535e3059f6130 commits,
- version 52fae58895127 commits,
- version 51802d396d121 commits,
- version 505a400162120 commits,
- version 49f56fc513119 commits,
- version 48b6d305b4117 commits,
- version 47039f458d114 commits,
- version 4659888afb110 commits,
- version 45f0370c7e109 commits,
- version 449409548c104 commits,
- version 435406b743102 commits,
- version 42c863379899 commits,
- version 41f0fe3fe498 commits,
- version 40c17d9ab897 commits,
- version 39ebc42fe796 commits,
- version 382be456a495 commits,
- version 37777ad13691 commits,
- version 362e825d4090 commits,
- version 352c0bdaa287 commits,
- version 34034c17c783 commits,
- version 3370492a5081 commits,
- version 32d24d68fb77 commits,
- version 316f24de8576 commits,
- version 3000c0c25975 commits,
- version 29a7bbf1c972 commits,
- version 28786b874b69 commits,
- version 271e6586cf68 commits,
- version 26e275c86f67 commits,
- version 2599313bef66 commits,
- version 240c707a6565 commits,
- version 23f55de28330 commits,
- version 22f55de28330 commits,
- version 21f0cb563832 commits,
- version 206086360630 commits,
- version 193235b85529 commits,
- version 18dede63aa24 commits,
- version 1786d4732c23 commits,
- version 1666ae4c7722 commits,
- version 15cb48d6b520 commits,
- version 14b9407d7719 commits,
- version 1372d2186b18 commits,
- version 12b32a44dd17 commits,
- version 11cf32613b16 commits,
- version 10e76dea6e15 commits,
- version 91350835e14 commits,
- version 88074188713 commits,
- version 7a0bb0d9b12 commits,
- version 65c2b370b11 commits,
- version 5dd7c2ea510 commits,
- version 4c8034d6c9 commits,
- version 385f1f1088 commits,
- version 226020df85 commits,
- version 158059c9a1 commit,
Compare changes
- Side-by-side
- Inline
Files
15Loading