Skip to content

Introduce the Benchmark class

This issue describes how benchmarks will be represented in the new Theodolite implementation. We start by giving an overview of what a benchmark is and how it is statically represented. Then, we follow up by presenting how such a representation will be converted to a OOP class, which can be executed.

This issue can be considered finished (i.e. closed) if everything is implemented and documented in the new docs. Hence, it might make sense to create appropriate subtickets for certain parts of this ticket.

Please note:

  • most of the examples in this issue are more or less pseudo code
  • Not every single aspect we will have to implement is considered already

Read-only benchmark description data structures

We start by giving an overview of how a benchmark can be represented statically.

Context

A BenchmarkExecution (name is t.b.d.), i.e. a single scalability experiment for a given benchmark is described similar to the following YAML.

name: experiment-xyz # describe this specific experiment
benchmark: UC1-KStreams-Kubernetes # type: Benchmark
load: # We might extend this to a list
  loadType: NumSensors
  loadValues: [100000, 200000, 300000]
resources: # We might extend this to a list
  resourceType: Instances
  resourceValues: [1, 2, 3, 4]
slos:
  - sloType: LagTrend
    threshold: 2000
  - sloType: MaxDiscardPerSec
    threshold: 10
execution:
  strategy: LinearSearch
  duration: 5m
  repetitions: 3
configOverrides:
  - # ...

Purpose of this ticket is not to discuss the BenchmarkExecution, but instead the Benchmark type. Benchmarks are decided to be declared separately from a BenchmarkExecution as they are expected to have a much longer life-cycle.

The Benchmark type (or rather KubernetesBenchmark)

Benchmarks can be designed for different execution environments, one of those will be Kubernetes. Hence, we will have a subtype of Benchmark called KubernetesBenchmark. (In the first iteration of our new implementation, we will only consider Kubernetes benchmarks. However, having potentially other execution environments in mind allows already to generalize a bit.) A KubernetesBenchmark will look similar to the following YAML.

name: UC1-KStreams
appResources:
  - "uc1-kstreams/deployment.yaml"
  - "uc1-kstreams/service.yaml"
loadGenResources:
  - "uc1-load-gen/deployment.yaml"
resourceTypes: # All the resource types supported by this benchmark
  - typeName: Instances
    patchers:
    - name: ReplicaPatcher # Refers to a class which "patches" the replicas of the following resource
      deployment: "uc1-kstreams/deployment.yaml"
    - name: EnvVarPatcher # Another "patcher" which sets a specific environment variable
      deployment: "uc1-kstreams/deployment.yaml"
      container: kstreams-app # A pod can have more than one container
      variableName: "TASK_PARALLELISM" # This is an example copied from Flink, which requires TASK_PARALLELISM to be set in addition to the number of replicas
  - typeName: Threads
    patchers:
    - name: EnvVarPatcher
      deployment: "uc1-kstreams/deployment.yaml"
      container: kstreams-app
      variableName: "NUM_THREADS"
loadTypes: # All the load dimensions supported by this benchmark
  - typeName: NumSensors
    patchers:
    - name: EnvVarPatcher
      deployment: "uc1-load-gen/deployment.yaml" # no container-> set for all
      variableName: "NUM_SENSOR"
  - typeName: MessageFrequency
    patchers:
    - name: EnvVarPatcher
      deployment: "uc1-load-gen/deployment.yaml"
      variableName: "MESSAGE_FREQUENCY"
# We might also want to configure Kafka settings here or set defaults

Deploying a benchmark

So far, we have only looked at the describing data structures for a benchmark. Now, we will a look at how Theodolite will deploy software components based on these benchmarks.

Context: BenchmarkRunner

The BenchmarkRunner (or BenchmarkExecutor) is created based on the BenchmarkExecution above. It holds exactly one reference to a Benchmark.

Benchmark

A Benchmark is an interface having (currently) one method:

interface Benchmark {
  fun buildDeployment(load, resources): BenchmarkDeployment
}

A benchmark is intended to be stateless, allowing to pass a benchmark to different BenchmarkRunners.

We currently think about one implementation of a Benchmark, which is a KubernetesBenchmark.

BenchmarkDeployment

A BenchmarkDeployment represents -- as its name suggests -- one deployment of a benchmark. It can be set up and teared down. Hence, a BenchmarkDeployment can have state, for example, to keep track of what it previously deployed. It looks like:

interface BenchmarkDeployment {
  fun setup()
  fun teardown()
}

Again, we currently think only about one implementation of this interface: A KubernetesBenchmarkDeployment. (We still have to decide whether using K8s or Kubernetes for names.)

class KubernetesBenchmarkDeployment(
  val resources: List<K8sResource>, // List of already patched resources
  val kafkaConfig: ?,
  // Maybe more
): BenchmarkDeployment {
  val kafkaController: KafkaController()
  val kubernetesManager: KubernetesManager() // Maybe per resource type

  fun setup() {
    kafkaController.createTopics(this.kafkaConfig)
    resources.forEach {
      kubernetesManager.deploy(it)
    }
  }

  fun teardown() {
    kafkaController.removeTopics(this.kafkaConfig)
    resources.forEach {
      kubernetesManager.remove(it)
    }
    // ...
  }
}

A KubernetesBenchmark contains all the data specified above. When calling kubernetesBenchmark.buildDeployment(..) (kubernetesBenchmark is KubernetesBenchmark), all resources are patched based on the current scaling and other configuration and new KubernetesBenchmarkDeployment is returned.

Patchers

A Patcher (maybe KubernetesPatcher) is a Kotlin Interface, which describes how a given benchmark deployment can be altered. They are used to describe how certain load value or resource value (or potentially some static configuration) is applied to a benchmark. The interfaces looks like (not sure about the generic type yet):

interface Patcher<T> {
  fun patch(T value)
}

Possible implementation of Patcher are, for example:

class ReplicaPatcher(
  val deployment: K8sDeployment // K8s deployment to patch
): Patcher<Int> {
  fun patch(Int value) {
    this.deployment.replicas = value // Pseudo code
  }
}

or:

class EnvVarPatcher(
  val deployment: K8sDeployment, // Might also be a statefulset, job, etc., maybe individual patchers are required in this case
  val container?: String, // or some container reference
  val variablenName: String
): Patcher<Int> {
  fun patch(Int value) {
    this.deployment.containers[this.container].envs[this.variablenName] = value // Pseudo code
  }
}

Note that although these example refer to patching Kubernetes resources, we could think about other types of Patchers such as one that alters some configuration of Kafka.

Instances of Patchers can be, for example, created from YAML files via reflection or pre-defined factories. String conversion etc. still needs to be handled somehow, but that should be possible.

Edited by Sören Henning