Skip to content
Snippets Groups Projects
Commit 888c85a4 authored by Benedikt Wetzel's avatar Benedikt Wetzel
Browse files

Merge branch 'read-exp-id-from-file' of... 

Merge branch 'read-exp-id-from-file' of git.se.informatik.uni-kiel.de:wetzel/spesb into read-exp-id-from-file
parents f1aaeb2d f2733ede
Branches
No related tags found
4 merge requests!159Re-implementation of Theodolite with Kotlin/Quarkus,!157Update Graal Image in CI pipeline,!138Load execution ID from file,!83WIP: Re-implementation of Theodolite with Kotlin/Quarkus
This commit is part of merge request !138. Comments created here will be created in the context of that merge request.
Show whitespace changes
Inline Side-by-side
slope-evaluator/README.md
+ 1
− 1
View file @ 888c85a4
......@@ -8,7 +8,7 @@ For development:
uvicorn main:app --reload
```
Build the docker image:
## Build the docker image:
```sh
docker build . -t theodolite-evaluator
......
theodolite-quarkus/README.md
+ 71
− 13
View file @ 888c85a4
# theodolite-quarkus project
# Theodolite-quarkus project
This project uses Quarkus, the Supersonic Subatomic Java Framework.
......@@ -6,7 +6,8 @@ If you want to learn more about Quarkus, please visit its website: https://quark
## Running the application in dev mode
You can run your application in dev mode that enables live coding using:
You can run your application in dev mode using:
```shell script
./gradlew quarkusDev
```
......@@ -14,13 +15,16 @@ You can run your application in dev mode that enables live coding using:
## Packaging and running the application
The application can be packaged using:
```shell script
./gradlew build
```
It produces the `theodolite-quarkus-1.0.0-SNAPSHOT-runner.jar` file in the `/build` directory.
Be aware that it’s not an _über-jar_ as the dependencies are copied into the `build/lib` directory.
It produces the `theodolite-quarkus-1.0.0-SNAPSHOT-runner.jar` file in the `/build` directory. Be aware that it’s not
an _über-jar_ as the dependencies are copied into the `build/lib` directory.
If you want to build an _über-jar_, execute the following command:
```shell script
./gradlew build -Dquarkus.package.type=uber-jar
```
......@@ -30,11 +34,13 @@ The application is now runnable using `java -jar build/theodolite-quarkus-1.0.0-
## Creating a native executable
You can create a native executable using:
```shell script
./gradlew build -Dquarkus.package.type=native
```
Or, if you don't have GraalVM installed, you can run the native executable build in a container using:
```shell script
./gradlew build -Dquarkus.package.type=native -Dquarkus.native.container-build=true
```
......@@ -44,20 +50,73 @@ You can then execute your native executable with:
If you want to learn more about building native executables, please consult https://quarkus.io/guides/gradle-tooling.
# RESTEasy JAX-RS
## Build docker images
For the jvm version use:
```shell script
./gradlew build
docker build -f src/main/docker/Dockerfile.jvm -t theodolite-quarkus-jvm .
```
For the native image version use:
```shell script
./gradlew build -Dquarkus.package.type=native
docker build -f src/main/docker/Dockerfile.native -t theodolite-quarkus-native .
```
## Execute docker images:
<p>A Hello World RESTEasy resource</p>
Remember to set the environment variables first.
Guide: https://quarkus.io/guides/rest-json
Jvm version:
## Build and afterwards run the application in Docker container
```shell script
docker run -i --rm theodolite-quarkus-jvm
```
Native image version:
```build_jvm.sh``` to build the jvm version
```shell script
docker run -i --rm theodolite-quarkus-native
```
```build_native.sh``` to build the native image graal version
## Environment variables
## Install Detekt Code analysis Plugin
**Production:** (Docker-Container)
| Variables name | Default value |Usage |
| -----------------------------|:----------------------------------:| ------------:|
| `NAMESPACE` | `default` |Determines the namespace of the Theodolite will be executed in. Used in the KubernetesBenchmark|
| `THEODOLITE_EXECUTION` | `./config/BenchmarkExecution.yaml`|The complete path to the benchmarkExecution file. Used in the TheodoliteYamlExecutor. |
| `THEODOLITE_BENCHMARK_TYPE` | `./config/BenchmarkType.yaml` |The complete path to the benchmarkType file. Used in the TheodoliteYamlExecutor.|
| `THEODOLITE_APP_RESOURCES` | `./config` |The path under which the yamls for the resources for the subexperiments are found. Used in the KubernetesBenchmark|
| `MODE` | `yaml-executor` | Defines the mode of operation: either `yaml-executor` or `operator`|
**Development:** (local via Intellij)
When running Theodolite from within IntelliJ via
[Run Configurations](https://www.jetbrains.com/help/idea/work-with-gradle-tasks.html#gradle_run_config), set the *
Environment variables* field to:
```
NAMESPACE=default;THEODOLITE_BENCHMARK=./../../../../config/BenchmarkType.yaml;THEODOLITE_APP_RESOURCES=./../../../../config;THEODOLITE_EXECUTION=./../../../../config/BenchmarkExecution.yaml;MODE=operator
```
Alternative:
``` sh
export NAMESPACE=default
export THEODOLITE_BENCHMARK=./../../../../config/BenchmarkType.yaml
export THEODOLITE_APP_RESOURCES=./../../../../config
export THEODOLITE_EXECUTION=./../../../../config/BenchmarkExecution.yaml
export MODE=operator
./gradlew quarkusDev
```
#### Install Detekt Code analysis Plugin
Install https://plugins.jetbrains.com/plugin/10761-detekt
......@@ -66,7 +125,6 @@ Install https://plugins.jetbrains.com/plugin/10761-detekt
- Check Enable Detekt
- Specify your detekt configuration and baseline file (optional)
-> detekt issues will be annotated on-the-fly while coding
**ingore Failures in build:** add
......
theodolite-quarkus/config/README.md 0 → 100644
+ 194
− 0
View file @ 888c85a4
## The Benchmark Object
The *benchmark* object defines all static components of an execution of a benchmark with Theodolite.
An exapmle for a benchmark object is given in [example-benchmark-yaml-resource](example-benchmark-yaml-resource.yaml).
A **Benchmark** is a [*standard tool for the competitive evaluation and comparison of competing systems or components according to specific characteristics, such as performance, dependability, or security*](https://doi.org/10.1145/2668930.2688819). In Theodolite, we have [specification-based benchmarks](https://doi.org/10.1145/2668930.2688819), or at least something very close to that. That is, our benchmarks are architectural descriptions---in our case---[of typical use cases of stream processing in microservices](https://doi.org/10.1016/j.bdr.2021.100209) (e.g. our UC1). Hence, we don't really have a piece of software, which represents a benchmark. We only have implementations of benchmarks, e.g. an implementation of UC1 with Kafka Streams. For simplification, we call these *benchmark implementations* simply *benchmarks*.
```yaml
name: String
appResource:
- String
...
loadGenResource:
- String
...
resourceTypes:
- typeName: String
patchers:
- type: String
resources: String
<Patcher Arguments> ...
...
loadTypes:
- typeName: String
patchers:
- type: String
resources: String
<Patcher Arguments> ...
...
kafkaConfig:
bootstrapServer: String
topics:
- name: String
numPartitions: UnsignedInt
replicationFactor: UnsignedInt
- name: String
removeOnly: bool
...
```
The properties have the following definitions:
* **name**: The name of the *benchmark*
* **appResource**: A list of file names that reference Kubernetes resources that are deployed on the cluster for the system under test (SUT).
* **loadGenResources**: A list of file names that reference Kubernetes resources that are deployed on the cluster for the load generator.
* **resourceTypes**: A list of resource types that can be scaled for this *benchmark*. For each resource type the concrete values are defined in the *execution* object. Each resource type has the following structure:
* **typeName**: Name of the resource type.
* **patchers**: List of [patchers](#Patchers) used to scale this resource type. Each patcher has the following structure:
* **type**: Type of the [patcher](#Patchers). The concrete types can be looked up in the list of [patchers](#Patchers).
* **resources**: Specifies the Kubernetes resource to be patched.
* **Patcher Arguments**: (Optional) Patcher specific additional arguments.
* **loadTypes**: A list of load types that can be scaled for this *benchmark*. For each load type the concrete values are defined in the *execution* object. Each load type has the following structure:
* **typeName**: Name of the load type.
* **patchers**: List of patchers used to scale * **resourceTypes**: A list of resource types that can be scaled for this *benchmark*. For each resource type the concrete values are defined in the *execution* resource object.Each resource type has the following structure:
* **typeName**: Name of the resource type.
* **patchers**: List of patchers used to scale this resource type. Each patcher has the following structure:
* **type**: Type of the Patcher. The concrete types can be looked up in the list of patchers.
* **resources**: Specifies the Kubernetes resource to be patched.
* **Patcher Arguments**: (Optional) Patcher specific additional arguments.
* **kafkaConfig**: Contains the Kafka configuration.
* **bootstrapServers**: The bootstrap servers connection string.
* **topics**: List of topics to be created for each [experiment](#Experiment). Alternative theodolite offers the possibility to remove certain topics after each experiment.
* **name**: The name of the topic.
* **numPartitions**: The number of partitions of the topic.
* **replicationFactor**: The replication factor of the topic.
* **removeOnly**: determines if this topic should only be deleted after each experiement. For removeOnly topics the name can be a RegEx describing the topic.
## The Execution Object
A benchmark can be executed for different SUTs, by different users and multiple times. We call such an execution of a benchmark simply an *execution*. The *execution* object defines all conrete values of an Execution.
An exapmle for an execution object is given in [example-execution-yaml-resource](example-benchmark-yaml-resource.yaml).
```yaml
name: String
benchmark: String
load:
loadType: String
loadValues:
- UnsignedInt
...
resources:
resourceType: String
resourceValues:
- UnsignedInt
...
slos:
- sloType: String
threshold: UnsignedInt
prometheusUrl: String
externalSloUrl: String
offset: SignedInt
warmup: UnsignedInt
...
executions:
strategy: "LinearSearch" or "BinarySearch"
duration: UnsignedInt
repetition: UnsignedInt
restrictions:
- "LowerBound"
...
configurationOverrides:
- patcher:
type: String
resource: String
<Patcher Arguments> ...
...
```
The properties have the following definitions:
* **name**: The name of the *execution*
* **benchmark**: The name of the *benchmark* this *execution* is referring to.
* **load**: Specifies the load values that are benchmarked.
* **loadType**: The type of the load. It must match one of the load types specified in the referenced *benchmark*.
* **loadValues**: List of load values for the specified load type.
* **resources**: Specifies the scaling resource that is benchmarked.
* **resourceType**: The type of the resource. It must match one of the resource types specified in the referenced *benchmark*.
* **resourceValues**: List of resource values for the specified resource type.
* **slos**: List of the Service Level Objective (SLO) for this *execution*. Each SLO has the following fields:
* **sloType**: The type of the SLO. It must match 'lag trend'.
* **threshold**: The threshold the SUT should meet for a sucessful experiment.
* **prometheusUrl**: Connection string for promehteus.
* **externalSloUrl**: Connection string for a external slo analysis.
* **offset**: Hours by which the start and end timestamp will be shifted (for different timezones).
* **warmup**: Seconds of time that are ignored in the analysis.
* **executions**: Defines the overall parameter for the execution.
* **strategy**: Defines the used strategy for the execution: either 'LinearSearch' or 'BinarySearch'
* **duration**: Defines the duration of each [experiment](#Experiment) in seconds.
* **repetition**: Unused.
* **restrictions**: List of restriction strategys used to delimit the search space.
**- LowerBound**: Currently only supported *restriction strategy*.
* **configurationOverrides**: List of patchers that are used to override existing configurations.
* **patcher**: Patcher used to patch a resource. Each patcher has the following structure:
* **type**: Type of the Patcher. The concrete types can be looked up in the list of patchers.
* **resources**: Specifies the Kubernetes resource to be patched.
* **Patcher Arguments**: (Optional) Patcher specific additional arguments.
## Patchers
* **ReplicaPatcher**: Allows to modify the number of Replicas for a kubernetes deployment.
* **type**: "ReplicaPatcher"
* **resource**: "uc1-kstreams-deployment.yaml"
* **NumSensorsLoadGeneratorReplicaPatcher**: Allows to scale the nummer of load generators. Scales arcording to the following formular: (value + 15_000 - 1) / 15_000
* **type**: "NumSensorsLoadGeneratorReplicaPatcher"
* **resource**: "uc1-load-generator-deployment.yaml"
* **NumNestedGroupsLoadGeneratorReplicaPatcher**: Allows to scale the nummer of load generators. Scales arcording to the following formular: (4^(value) + 15_000 -1) /15_000
* **type**: "NumNestedGroupsLoadGeneratorReplicaPatcher"
* **resource**: "uc1-load-generator-deployment.yaml"
* **ReplicaPatcher**: Allows to modify the number of Replicas for a kubernetes deployment.
* **type**: "ReplicaPatcher"
* **resource**: "uc1-kstreams-deployment.yaml"
* **EnvVarPatcher**: Allows to modify the value of an environment variable for a container in a kubernetes deployment.
* **type**: "EnvVarPatcher"
* **resource**: "uc1-load-generator-deployment.yaml"
* **container**: "workload-generator"
* **variableName**: "NUM_SENSORS"
* **NodeSelectorPatcher**: Changes the node selection field in kubernetes resources.
* **type**: "NodeSelectorPatcher"
* **resource**: "uc1-load-generator-deployment.yaml"
* **variableName**: "env"
* **value**: "prod"
* **ResourceLimitPatcher**: Changes the resource limit for a kubernetes resource.
* **resource**: "uc1-kstreams-deployment.yaml"
* **container**: "uc-application"
* **variableName**: "cpu" or "memory"
* **value**:"1000m" or "2Gi"
* **SchedulerNamePatcher**: Changes the sheduler for kubernetes resources.
* **type**: "SchedulerNamePatcher"
* **resource**: "uc1-kstreams-deployment.yaml"
* **value**: "random-scheduler"
* **ImagePatcher**: Changes the image of a kubernetes resource. Currently not fully implemented.
* **type**: "ImagePatcher"
* **resource**: "uc1-kstreams-deployment.yaml"
* **container**: "uc-application"
* **value**: "dockerhubrepo/imagename"
## Experiment
According to [our benchmarking method](https://doi.org/10.1016/j.bdr.2021.100209), the execution of a benchmark requires performing multiple **Experiments**. I think what is actually done within/during an experiment is another level of detail. (But just for the sake of completeness: In an experiment, the benchmark implementation is deployed, load is generated according to the benchmark specification, some SLOs are monitored continuously, etc.)
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment