diff --git a/docs/creating-a-benchmark.md b/docs/creating-a-benchmark.md index ee55fb4ea0f8d194b9e3ce96c380ddecf91be9d4..673d8e070b41b8a03833f92d7ebd211ca6e4c6a8 100644 --- a/docs/creating-a-benchmark.md +++ b/docs/creating-a-benchmark.md @@ -1,8 +1,83 @@ +--- +title: Creating a Benchmark +has_children: false +nav_order: 5 +--- -# Defining a Benchmark +# Creating a Benchmark Please note that to simply run a benchmark, it is not required to define one. Theodolite comes with a set of benchmarks, which are ready to be executed. See the Benchmarks and Executions docs page to learn more about our distinction between benchmarks and executions. +A typical benchmark looks as follow. + +```yaml +apiVersion: theodolite.com/v1 +kind: benchmark +metadata: + name: uc1-kstreams +spec: + appResource: + - "custom/uc1-kstreams-deployment.yaml" + loadGenResource: + - "custom/uc1-load-generator-deployment.yaml" + - "custom/uc1-load-generator-service.yaml" + resourceTypes: + - typeName: "Instances" + patchers: + - type: "ReplicaPatcher" + resource: "custom/uc1-kstreams-deployment.yaml" + loadTypes: + - typeName: "NumSensors" + patchers: + - type: "EnvVarPatcher" + resource: "custom/uc1-load-generator-deployment.yaml" + properties: + variableName: "NUM_SENSORS" + container: "workload-generator" + - type: "NumSensorsLoadGeneratorReplicaPatcher" + resource: "custom/uc1-load-generator-deployment.yaml" + properties: + loadGenMaxRecords: "15000" + kafkaConfig: + bootstrapServer: "theodolite-cp-kafka:9092" + topics: + - name: "input" + numPartitions: 40 + replicationFactor: 1 + - name: "theodolite-.*" + removeOnly: True + +``` + +### System under Test (SUT) and Load Generator Resources + +In Thedolite, the system under test (SUT) and the load generator are described by Kubernetes resources files. +Based on these files, both the SUT and the load generator are started and stopped for each SLO experiment. + +All Kubernetes resource files listed under `appResource` and `loadGenResource` must be accessible by Theodolite. +The recommended way to achieve this is by bundling them in one or multiple ConfigMaps, which can be done by: + +```sh +kubectl create configmap <configmap-name> --from-file=<path-to-resources> +``` + +### Load and Resource Types + +Benchmarks need to specify at least one supported load and resource type for which scalability can be benchmarked. + +Load and resource types are described by a name (used for reference from an Execution) and a list of patchers. + +### Kafka Configuration + +Theodolite allows to automatically create and remove Kafka topics for each SLO experiment. +Use the `removeOnly: True` property for topics which are created automatically by the SUT. +For those topics, also wildcards are allowed in the topic name. + + +<!-- Further information: API Reference --> +<!-- Further information: How to deploy --> + +----- * Create a benchmark for operator mode * Create a benchmark for standalone mode diff --git a/docs/creating-an-execution.md b/docs/creating-an-execution.md index 9aeba6b285772585535c0f41bdf8c7fcb8f9f43e..2e96f62873c386ca0a1ec7f1991b0f3818c33fee 100644 --- a/docs/creating-an-execution.md +++ b/docs/creating-an-execution.md @@ -79,3 +79,4 @@ Instead, also Executions allow to do small reconfigurations, such as switching o This is done by defining `configOverrides` in the Execution. Each override consists of a patcher, defining which Kubernetes resource should be patched in which way, and a value the patcher is applied with. <!-- Further information: API Reference --> +<!-- Further information: How to run -->