Merge branch 'master' into oci-volumes

analysis/README.md

@@ -9,7 +9,7 @@ benchmark execution results and plotting. The following notebooks are provided:
 For legacy reasons, we also provide the following notebooks, which, however, are not documented:
 
 * [scalability-graph.ipynb](scalability-graph.ipynb): Creates a scalability graph for a certain benchmark execution.
-* [scalability-graph-final.ipynb](scalability-graph-final.ipynb): Combines the scalability graphs of multiple benchmarks executions (e.g. for comparing different configuration).
+* [scalability-graph-plotter.ipynb](scalability-graph-plotter.ipynb): Combines the scalability graphs of multiple benchmarks executions (e.g. for comparing different configuration).
 * [lag-trend-graph.ipynb](lag-trend-graph.ipynb): Visualizes the consumer lag evaluation over time along with the computed trend.
 
 ## Usage

analysis/demand-metric-plot.ipynb

@@ -34,7 +34,7 @@
   },
   {
    "source": [
-    "We need to specify the directory, where the demand CSV files can be found, and a dictionary that maps a system description (e.g. its name) to the corresponding CSV file (prefix). "
+    "We need to specify the directory, where the demand CSV files can be found, and a dictionary that maps a system description (e.g. its name) to the corresponding CSV file (prefix). To use Unicode narrow non-breaking spaces in the description format it as `u\"1000\\u202FmCPU\"`."
    ],
    "cell_type": "markdown",
    "metadata": {}

analysis/demand-metric.ipynb

@@ -4,7 +4,7 @@
    "source": [
     "# Theodolite Analysis - Demand Metric\n",
     "\n",
-    "This notebook allows applies Theodolite's *demand* metric to describe scalability of a SUT based on Theodolite measurement data.\n",
+    "This notebook applies Theodolite's *demand* metric to describe scalability of a SUT based on Theodolite measurement data.\n",
     "\n",
     "Theodolite's *demand* metric is a function, mapping load intensities to the minimum required resources (e.g., instances) that are required to process this load. With this notebook, the *demand* metric function is approximated by a map of tested load intensities to their minimum required resources.\n",
     "\n",

docs/README.md

+---
+title: Theodolite
+nav_order: 1
+permalink: /
+---
+
+# Theodolite
+
+> A theodolite is a precision optical instrument for measuring angles between designated visible points in the horizontal and vertical planes.  -- <cite>[Wikipedia](https://en.wikipedia.org/wiki/Theodolite)</cite>
+
+Theodolite is a framework for benchmarking the horizontal and vertical scalability of stream processing engines. It consists of three modules:
+
+## Theodolite Benchmarks
+
+Theodolite contains 4 application benchmarks, which are based on typical use cases for stream processing within microservices. For each benchmark, a corresponding workload generator is provided. Currently, this repository provides benchmark implementations for Kafka Streams.
+
+
+## Theodolite Execution Framework
+
+Theodolite aims to benchmark scalability of stream processing engines for real use cases. Microservices that apply stream processing techniques are usually deployed in elastic cloud environments. Hence, Theodolite's cloud-native benchmarking framework deploys as components in a cloud environment, orchestrated by Kubernetes. More information on how to execute scalability benchmarks can be found in [Thedolite execution framework](execution).
+
+
+## Theodolite Analysis Tools
+
+Theodolite's benchmarking method create a *scalability graph* allowing to draw conclusions about the scalability of a stream processing engine or its deployment. A scalability graph shows how resource demand evolves with an increasing workload. Theodolite provides Jupyter notebooks for creating such scalability graphs based on benchmarking results from the execution framework. More information can be found in [Theodolite analysis tool](analysis).

docs/_config.yml

+title: "Theodolite"
+remote_theme: pmarsceill/just-the-docs
+#color_scheme: "dark"
+aux_links:
+    "Theodolite on GitHub":
+      - "//github.com/cau-se/theodolite"
\ No newline at end of file

docs/release-process.md

+---
+title: Release Process
+has_children: false
+nav_order: 2
+---
+
 # Release Process
 
 We assume that we are creating the release `v0.1.1`. Please make sure to adjust

execution/README.md

@@ -121,7 +121,9 @@ can be installed via Helm. We also provide a [default configuration](infrastruct
 To install it:
 
 ```sh
-helm install kafka-lag-exporter https://github.com/lightbend/kafka-lag-exporter/releases/download/v0.6.3/kafka-lag-exporter-0.6.3.tgz -f infrastructure/kafka-lag-exporter/values.yaml
+helm repo add kafka-lag-exporter https://lightbend.github.io/kafka-lag-exporter/repo/
+helm repo update
+helm install kafka-lag-exporter kafka-lag-exporter/kafka-lag-exporter -f infrastructure/kafka-lag-exporter/values.yaml
 ```
 
 ### Installing Theodolite

execution/infrastructure/kafka/values.yaml

@@ -56,6 +56,7 @@ cp-kafka:
     "auto.create.topics.enable": false
     "log.retention.ms": "10000" # 10s
     # "log.retention.ms": "86400000" # 24h
+    # "group.initial.rebalance.delay.ms": "30000" # 30s
     "metrics.sample.window.ms": "5000" #5s
 
 ## ------------------------------------------------------

execution/infrastructure/prometheus/helm-values.yaml

@@ -36,6 +36,9 @@ nodeExporter:
 
 prometheusOperator:
   enabled: true
+  namespaces:
+    releaseNamespace: true
+    additional: []
 
 prometheus:
   enabled: false

execution/run_uc.py

@@ -94,11 +94,11 @@ def load_yaml_files():
     :return: wg, app_svc, app_svc_monitor ,app_jmx, app_deploy
     """
     print('Load kubernetes yaml files')
-    wg = load_yaml('uc-workload-generator/base/workloadGenerator.yaml')
-    app_svc = load_yaml('uc-application/base/aggregation-service.yaml')
-    app_svc_monitor = load_yaml('uc-application/base/service-monitor.yaml')
-    app_jmx = load_yaml('uc-application/base/jmx-configmap.yaml')
-    app_deploy = load_yaml('uc-application/base/aggregation-deployment.yaml')
+    wg = load_yaml('uc-workload-generator/workloadGenerator.yaml')
+    app_svc = load_yaml('uc-application/aggregation-service.yaml')
+    app_svc_monitor = load_yaml('uc-application/service-monitor.yaml')
+    app_jmx = load_yaml('uc-application/jmx-configmap.yaml')
+    app_deploy = load_yaml('uc-application/aggregation-deployment.yaml')
 
     print('Kubernetes yaml files loaded')
     return wg, app_svc, app_svc_monitor, app_jmx, app_deploy

execution/run_uc1.sh

-#!/bin/bash
-
-EXP_ID=$1
-DIM_VALUE=$2
-INSTANCES=$3
-PARTITIONS=${4:-40}
-CPU_LIMIT=${5:-1000m}
-MEMORY_LIMIT=${6:-4Gi}
-KAFKA_STREAMS_COMMIT_INTERVAL_MS=${7:-100}
-EXECUTION_MINUTES=${8:-5}
-
-echo "EXP_ID: $EXP_ID"
-echo "DIM_VALUE: $DIM_VALUE"
-echo "INSTANCES: $INSTANCES"
-echo "PARTITIONS: $PARTITIONS"
-echo "CPU_LIMIT: $CPU_LIMIT"
-echo "MEMORY_LIMIT: $MEMORY_LIMIT"
-echo "KAFKA_STREAMS_COMMIT_INTERVAL_MS: $KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-echo "EXECUTION_MINUTES: $EXECUTION_MINUTES"
-
-# Create Topics
-#PARTITIONS=40
-#kubectl run temp-kafka --rm --attach --restart=Never --image=solsson/kafka --command -- bash -c "./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-PARTITIONS=$PARTITIONS
-kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-
-# Start workload generator
-NUM_SENSORS=$DIM_VALUE
-WL_MAX_RECORDS=150000
-WL_INSTANCES=$(((NUM_SENSORS + (WL_MAX_RECORDS -1 ))/ WL_MAX_RECORDS))
-
-cat <<EOF >uc-workload-generator/overlay/uc1-workload-generator/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-load-generator
-spec:
-  replicas: $WL_INSTANCES
-  template:
-    spec:
-      containers:
-      - name: workload-generator
-        env:
-        - name: NUM_SENSORS
-          value: "$NUM_SENSORS"
-        - name: INSTANCES
-          value: "$WL_INSTANCES"
-EOF
-kubectl apply -k uc-workload-generator/overlay/uc1-workload-generator
-
-# Start application
-REPLICAS=$INSTANCES
-cat <<EOF >uc-application/overlay/uc1-application/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-aggregation
-spec:
-  replicas: $REPLICAS
-  template:
-    spec:
-      containers:
-      - name: uc-application
-        env:
-        - name: COMMIT_INTERVAL_MS
-          value: "$KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-        resources:
-          limits:
-            memory: $MEMORY_LIMIT
-            cpu: $CPU_LIMIT
-EOF
-kubectl apply -k uc-application/overlay/uc1-application
-
-# Execute for certain time
-sleep $(($EXECUTION_MINUTES * 60))
-
-# Run eval script
-source ../.venv/bin/activate
-python lag_analysis.py $EXP_ID uc1 $DIM_VALUE $INSTANCES $EXECUTION_MINUTES
-deactivate
-
-# Stop workload generator and app
-kubectl delete -k uc-workload-generator/overlay/uc1-workload-generator
-kubectl delete -k uc-application/overlay/uc1-application
-
-
-# Delete topics instead of Kafka
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-# kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic '.*'
-#sleep 30s # TODO check
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p'
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p' | wc -l
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list"
-
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-echo "Finished execution, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-while test $(kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(theodolite-.*|input|output|configuration)( - marked for deletion)?$/p' | wc -l) -gt 0
-do
-    kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input|output|configuration|theodolite-.*' --if-exists"
-    echo "Wait for topic deletion"
-    sleep 5s
-    #echo "Finished waiting, print topics:"
-    #kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-    # Sometimes a second deletion seems to be required
-done
-echo "Finish topic deletion, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-
-# delete zookeeper nodes used for workload generation
-echo "Delete ZooKeeper configurations used for workload generation"
-kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 deleteall /workload-generation"
-echo "Waiting for deletion"
-while kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 get /workload-generation"
-do
-    echo "Wait for ZooKeeper state deletion."
-    sleep 5s
-done
-echo "Deletion finished"
-
-echo "Exiting script"
-
-KAFKA_LAG_EXPORTER_POD=$(kubectl get pod -l app.kubernetes.io/name=kafka-lag-exporter -o jsonpath="{.items[0].metadata.name}")
-kubectl delete pod $KAFKA_LAG_EXPORTER_POD

execution/run_uc2.sh

-#!/bin/bash
-
-EXP_ID=$1
-DIM_VALUE=$2
-INSTANCES=$3
-PARTITIONS=${4:-40}
-CPU_LIMIT=${5:-1000m}
-MEMORY_LIMIT=${6:-4Gi}
-KAFKA_STREAMS_COMMIT_INTERVAL_MS=${7:-100}
-EXECUTION_MINUTES=${8:-5}
-
-echo "EXP_ID: $EXP_ID"
-echo "DIM_VALUE: $DIM_VALUE"
-echo "INSTANCES: $INSTANCES"
-echo "PARTITIONS: $PARTITIONS"
-echo "CPU_LIMIT: $CPU_LIMIT"
-echo "MEMORY_LIMIT: $MEMORY_LIMIT"
-echo "KAFKA_STREAMS_COMMIT_INTERVAL_MS: $KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-echo "EXECUTION_MINUTES: $EXECUTION_MINUTES"
-
-# Create Topics
-#PARTITIONS=40
-#kubectl run temp-kafka --rm --attach --restart=Never --image=solsson/kafka --command -- bash -c "./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-PARTITIONS=$PARTITIONS
-kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic aggregation-feedback --partitions $PARTITIONS --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-
-# Start workload generator
-NUM_NESTED_GROUPS=$DIM_VALUE
-WL_MAX_RECORDS=150000
-APPROX_NUM_SENSORS=$((4**NUM_NESTED_GROUPS))
-WL_INSTANCES=$(((APPROX_NUM_SENSORS + (WL_MAX_RECORDS -1 ))/ WL_MAX_RECORDS))
-
-cat <<EOF >uc-workload-generator/overlay/uc2-workload-generator/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-load-generator
-spec:
-  replicas: $WL_INSTANCES
-  template:
-    spec:
-      containers:
-      - name: workload-generator
-        env:
-        - name: NUM_SENSORS
-          value: "4"
-        - name: HIERARCHY
-          value: "full"
-        - name: NUM_NESTED_GROUPS
-          value: "$NUM_NESTED_GROUPS"
-        - name: INSTANCES
-          value: "$WL_INSTANCES"
-EOF
-kubectl apply -k uc-workload-generator/overlay/uc2-workload-generator
-
-# Start application
-REPLICAS=$INSTANCES
-cat <<EOF >uc-application/overlay/uc2-application/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-aggregation
-spec:
-  replicas: $REPLICAS
-  template:
-    spec:
-      containers:
-      - name: uc-application
-        env:
-        - name: COMMIT_INTERVAL_MS
-          value: "$KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-        resources:
-          limits:
-            memory: $MEMORY_LIMIT
-            cpu: $CPU_LIMIT
-EOF
-kubectl apply -k uc-application/overlay/uc2-application
-
-# Execute for certain time
-sleep $(($EXECUTION_MINUTES * 60))
-
-# Run eval script
-source ../.venv/bin/activate
-python lag_analysis.py $EXP_ID uc2 $DIM_VALUE $INSTANCES $EXECUTION_MINUTES
-deactivate
-
-# Stop workload generator and app
-kubectl delete -k uc-workload-generator/overlay/uc2-workload-generator
-kubectl delete -k uc-application/overlay/uc2-application
-
-
-# Delete topics instead of Kafka
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-# kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic '.*'
-#sleep 30s # TODO check
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p'
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p' | wc -l
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list"
-
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-echo "Finished execution, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-while test $(kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(theodolite-.*|input|aggregation-feedback|output|configuration)( - marked for deletion)?$/p' | wc -l) -gt 0
-do
-    kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input|aggregation-feedback|output|configuration|theodolite-.*' --if-exists"
-    echo "Wait for topic deletion"
-    sleep 5s
-    #echo "Finished waiting, print topics:"
-    #kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-    # Sometimes a second deletion seems to be required
-done
-echo "Finish topic deletion, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-
-# delete zookeeper nodes used for workload generation
-echo "Delete ZooKeeper configurations used for workload generation"
-kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 deleteall /workload-generation"
-echo "Waiting for deletion"
-while kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 get /workload-generation"
-do
-    echo "Wait for ZooKeeper state deletion."
-    sleep 5s
-done
-echo "Deletion finished"
-
-echo "Exiting script"
-
-KAFKA_LAG_EXPORTER_POD=$(kubectl get pod -l app.kubernetes.io/name=kafka-lag-exporter -o jsonpath="{.items[0].metadata.name}")
-kubectl delete pod $KAFKA_LAG_EXPORTER_POD

execution/run_uc3.sh

-#!/bin/bash
-
-EXP_ID=$1
-DIM_VALUE=$2
-INSTANCES=$3
-PARTITIONS=${4:-40}
-CPU_LIMIT=${5:-1000m}
-MEMORY_LIMIT=${6:-4Gi}
-KAFKA_STREAMS_COMMIT_INTERVAL_MS=${7:-100}
-EXECUTION_MINUTES=${8:-5}
-
-echo "EXP_ID: $EXP_ID"
-echo "DIM_VALUE: $DIM_VALUE"
-echo "INSTANCES: $INSTANCES"
-echo "PARTITIONS: $PARTITIONS"
-echo "CPU_LIMIT: $CPU_LIMIT"
-echo "MEMORY_LIMIT: $MEMORY_LIMIT"
-echo "KAFKA_STREAMS_COMMIT_INTERVAL_MS: $KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-echo "EXECUTION_MINUTES: $EXECUTION_MINUTES"
-
-# Create Topics
-#PARTITIONS=40
-#kubectl run temp-kafka --rm --attach --restart=Never --image=solsson/kafka --command -- bash -c "./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-PARTITIONS=$PARTITIONS
-kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-
-# Start workload generator
-NUM_SENSORS=$DIM_VALUE
-WL_MAX_RECORDS=150000
-WL_INSTANCES=$(((NUM_SENSORS + (WL_MAX_RECORDS -1 ))/ WL_MAX_RECORDS))
-
-cat <<EOF >uc-workload-generator/overlay/uc3-workload-generator/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-load-generator
-spec:
-  replicas: $WL_INSTANCES
-  template:
-    spec:
-      containers:
-      - name: workload-generator
-        env:
-        - name: NUM_SENSORS
-          value: "$NUM_SENSORS"
-        - name: INSTANCES
-          value: "$WL_INSTANCES"
-EOF
-kubectl apply -k uc-workload-generator/overlay/uc3-workload-generator
-
-
-# Start application
-REPLICAS=$INSTANCES
-cat <<EOF >uc-application/overlay/uc3-application/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-aggregation
-spec:
-  replicas: $REPLICAS
-  template:
-    spec:
-      containers:
-      - name: uc-application
-        env:
-        - name: COMMIT_INTERVAL_MS
-          value: "$KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-        resources:
-          limits:
-            memory: $MEMORY_LIMIT
-            cpu: $CPU_LIMIT
-EOF
-kubectl apply -k uc-application/overlay/uc3-application
-kubectl scale deployment uc3-titan-ccp-aggregation --replicas=$REPLICAS
-
-# Execute for certain time
-sleep $(($EXECUTION_MINUTES * 60))
-
-# Run eval script
-source ../.venv/bin/activate
-python lag_analysis.py $EXP_ID uc3 $DIM_VALUE $INSTANCES $EXECUTION_MINUTES
-deactivate
-
-# Stop workload generator and app
-kubectl delete -k uc-workload-generator/overlay/uc3-workload-generator
-kubectl delete -k uc-application/overlay/uc3-application
-
-# Delete topics instead of Kafka
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-# kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic '.*'
-#sleep 30s # TODO check
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p'
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p' | wc -l
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list"
-
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-echo "Finished execution, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-while test $(kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(theodolite-.*|input|output|configuration)( - marked for deletion)?$/p' | wc -l) -gt 0
-do
-    kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input|output|configuration|theodolite-.*' --if-exists"
-    echo "Wait for topic deletion"
-    sleep 5s
-    #echo "Finished waiting, print topics:"
-    #kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-    # Sometimes a second deletion seems to be required
-done
-echo "Finish topic deletion, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-
-# delete zookeeper nodes used for workload generation
-echo "Delete ZooKeeper configurations used for workload generation"
-kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 deleteall /workload-generation"
-echo "Waiting for deletion"
-while kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 get /workload-generation"
-do
-    echo "Wait for ZooKeeper state deletion."
-    sleep 5s
-done
-echo "Deletion finished"
-
-echo "Exiting script"
-
-KAFKA_LAG_EXPORTER_POD=$(kubectl get pod -l app.kubernetes.io/name=kafka-lag-exporter -o jsonpath="{.items[0].metadata.name}")
-kubectl delete pod $KAFKA_LAG_EXPORTER_POD

execution/run_uc4.sh

-#!/bin/bash
-
-EXP_ID=$1
-DIM_VALUE=$2
-INSTANCES=$3
-PARTITIONS=${4:-40}
-CPU_LIMIT=${5:-1000m}
-MEMORY_LIMIT=${6:-4Gi}
-KAFKA_STREAMS_COMMIT_INTERVAL_MS=${7:-100}
-EXECUTION_MINUTES=${8:-5}
-
-echo "EXP_ID: $EXP_ID"
-echo "DIM_VALUE: $DIM_VALUE"
-echo "INSTANCES: $INSTANCES"
-echo "PARTITIONS: $PARTITIONS"
-echo "CPU_LIMIT: $CPU_LIMIT"
-echo "MEMORY_LIMIT: $MEMORY_LIMIT"
-echo "KAFKA_STREAMS_COMMIT_INTERVAL_MS: $KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-echo "EXECUTION_MINUTES: $EXECUTION_MINUTES"
-
-# Create Topics
-#PARTITIONS=40
-#kubectl run temp-kafka --rm --attach --restart=Never --image=solsson/kafka --command -- bash -c "./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; ./bin/kafka-topics.sh --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-PARTITIONS=$PARTITIONS
-kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic input --partitions $PARTITIONS --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic configuration --partitions 1 --replication-factor 1; kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --create --topic output --partitions $PARTITIONS --replication-factor 1"
-
-# Start workload generator
-NUM_SENSORS=$DIM_VALUE
-WL_MAX_RECORDS=150000
-WL_INSTANCES=$(((NUM_SENSORS + (WL_MAX_RECORDS -1 ))/ WL_MAX_RECORDS))
-
-cat <<EOF >uuc-workload-generator/overlay/uc4-workload-generator/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-load-generator
-spec:
-  replicas: $WL_INSTANCES
-  template:
-    spec:
-      containers:
-      - name: workload-generator
-        env:
-        - name: NUM_SENSORS
-          value: "$NUM_SENSORS"
-        - name: INSTANCES
-          value: "$WL_INSTANCES"
-EOF
-kubectl apply -k uc-workload-generator/overlay/uc4-workload-generator
-
-# Start application
-REPLICAS=$INSTANCES
-cat <<EOF >uc-application/overlay/uc4-application/set_paramters.yaml
-apiVersion: apps/v1
-kind: Deployment
-metadata:
-  name: titan-ccp-aggregation
-spec:
-  replicas: $REPLICAS
-  template:
-    spec:
-      containers:
-      - name: uc-application
-        env:
-        - name: COMMIT_INTERVAL_MS
-          value: "$KAFKA_STREAMS_COMMIT_INTERVAL_MS"
-        resources:
-          limits:
-            memory: $MEMORY_LIMIT
-            cpu: $CPU_LIMIT
-EOF
-kubectl apply -k uc-application/overlay/uc4-application
-kubectl scale deployment uc4-titan-ccp-aggregation --replicas=$REPLICAS
-
-# Execute for certain time
-sleep $(($EXECUTION_MINUTES * 60))
-
-# Run eval script
-source ../.venv/bin/activate
-python lag_analysis.py $EXP_ID uc4 $DIM_VALUE $INSTANCES $EXECUTION_MINUTES
-deactivate
-
-# Stop workload generator and app
-kubectl delete -k uc-workload-generator/overlay/uc4-workload-generator
-kubectl delete -k uc-application/overlay/uc4-application
-
-# Delete topics instead of Kafka
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-# kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic '.*'
-#sleep 30s # TODO check
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p'
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n '/^titan-.*/p;/^input$/p;/^output$/p;/^configuration$/p' | wc -l
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list"
-
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input,output,configuration,titan-.*'"
-echo "Finished execution, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-while test $(kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(theodolite-.*|input|output|configuration)( - marked for deletion)?$/p' | wc -l) -gt 0
-do
-    kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --delete --topic 'input|output|configuration|theodolite-.*' --if-exists"
-    echo "Wait for topic deletion"
-    sleep 5s
-    #echo "Finished waiting, print topics:"
-    #kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-    # Sometimes a second deletion seems to be required
-done
-echo "Finish topic deletion, print topics:"
-#kubectl exec kafka-client -- bash -c "kafka-topics --zookeeper my-confluent-cp-zookeeper:2181 --list" | sed -n -E '/^(titan-.*|input|output|configuration)( - marked for deletion)?$/p'
-
-# delete zookeeper nodes used for workload generation
-echo "Delete ZooKeeper configurations used for workload generation"
-kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 deleteall /workload-generation"
-echo "Waiting for deletion"
-while kubectl exec zookeeper-client -- bash -c "zookeeper-shell my-confluent-cp-zookeeper:2181 get /workload-generation"
-do
-    echo "Wait for ZooKeeper state deletion."
-    sleep 5s
-done
-echo "Deletion finished"
-
-echo "Exiting script"
-
-KAFKA_LAG_EXPORTER_POD=$(kubectl get pod -l app.kubernetes.io/name=kafka-lag-exporter -o jsonpath="{.items[0].metadata.name}")
-kubectl delete pod $KAFKA_LAG_EXPORTER_POD

execution/uc-application/overlay/uc1-application/kustomization.yaml

-apiVersion: kustomize.config.k8s.io/v1beta1
-kind: Kustomization
-
-namePrefix: uc1-
-
-images:
-  - name: uc-app
-    newName: theodolite/theodolite-uc1-kstreams-app
-    newTag: latest
-
-bases:
-- ../../base
-
-patchesStrategicMerge:
-- set_paramters.yaml # Patch setting the resource parameters