Merge branch 'master' into theodolite-kotlin

analysis/README.md

@@ -3,18 +3,24 @@
 This directory contains Jupyter notebooks for analyzing and visualizing
 benchmark execution results and plotting. The following notebooks are provided:
 
+* [demand-metric.ipynb](demand-metric.ipynb): Create CSV files describing scalability according to the Theodolite `demand` metric.
+* [demand-metric-plot.ipynb](demand-metric-plot.ipynb): Create plots based on such CSV files of the `demand` metric.
+
+For legacy reasons, we also provide the following notebooks, which, however, are not documented:
+
 * [scalability-graph.ipynb](scalability-graph.ipynb): Creates a scalability graph for a certain benchmark execution.
 * [scalability-graph-final.ipynb](scalability-graph-final.ipynb): Combines the scalability graphs of multiple benchmarks executions (e.g. for comparing different configuration).
 * [lag-trend-graph.ipynb](lag-trend-graph.ipynb): Visualizes the consumer lag evaluation over time along with the computed trend.
 
 ## Usage
 
-Basically, the Theodolite Analysis Jupyter notebooks should be runnable by any Jupyter server. To make it a bit easier,
+In general, the Theodolite Analysis Jupyter notebooks should be runnable by any Jupyter server. To make it a bit easier,
 we provide introductions for running notebooks with Docker and with Visual Studio Code. These intoduction may also be
 a good starting point for using another service.
 
-For analyzing and visualizing benchmark results, either Docker or a Jupyter installation with Python 3.7 or newer is
-required (e.g., in a virtual environment).
+For analyzing and visualizing benchmark results, either Docker or a Jupyter installation with Python 3.7 or 3.8 is
+required (e.g., in a virtual environment). **Please note that Python 3.9 seems not to be working as not all our
+dependencies are ported to Python 3.9 yet.**
 
 ### Running with Docker
 

analysis/demand-metric-plot.ipynb

+{
+ "cells": [
+  {
+   "source": [
+    "# Theodolite Analysis - Plotting the Demand Metric\n",
+    "\n",
+    "This notebook creates a plot, showing scalability as a function that maps load intensities to the resources required for processing them. It is able to combine multiple such plots in one figure, for example, to compare multiple systems or configurations.\n",
+    "\n",
+    "The notebook takes a CSV file for each plot mapping load intensities to minimum required resources, computed by the `demand-metric-plot.ipynb` notebook."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "source": [
+    "First, we need to import some libraries, which are required for creating the plots."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "import pandas as pd\n",
+    "from functools import reduce\n",
+    "import matplotlib.pyplot as plt\n",
+    "from matplotlib.ticker import FuncFormatter\n",
+    "from matplotlib.ticker import MaxNLocator"
+   ]
+  },
+  {
+   "source": [
+    "We need to specify the directory, where the demand CSV files can be found, and a dictionary that maps a system description (e.g. its name) to the corresponding CSV file (prefix). "
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "results_dir = '<path-to>/results'\n",
+    "\n",
+    "experiments = {\n",
+    "    'System XYZ': 'exp200',\n",
+    "}\n"
+   ]
+  },
+  {
+   "source": [
+    "Now, we combie all systems described in `experiments`."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "dataframes = [pd.read_csv(os.path.join(results_dir, f'{v}_demand.csv')).set_index('load').rename(columns={\"resources\": k}) for k, v in experiments.items()]\n",
+    "\n",
+    "df = reduce(lambda df1,df2: df1.join(df2,how='outer'), dataframes)"
+   ]
+  },
+  {
+   "source": [
+    "We might want to display the mappings before we plot it."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "df"
+   ]
+  },
+  {
+   "source": [
+    "The following code creates a MatPlotLib figure showing the scalability plots for all specified systems. You might want to adjust its styling etc. according to your preferences. Make sure to also set a filename."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "plt.style.use('ggplot')\n",
+    "plt.rcParams['axes.facecolor']='w'\n",
+    "plt.rcParams['axes.edgecolor']='555555'\n",
+    "#plt.rcParams['ytick.color']='black'\n",
+    "plt.rcParams['grid.color']='dddddd'\n",
+    "plt.rcParams['axes.spines.top']='false'\n",
+    "plt.rcParams['axes.spines.right']='false'\n",
+    "plt.rcParams['legend.frameon']='true'\n",
+    "plt.rcParams['legend.framealpha']='1'\n",
+    "plt.rcParams['legend.edgecolor']='1'\n",
+    "plt.rcParams['legend.borderpad']='1'\n",
+    "\n",
+    "@FuncFormatter\n",
+    "def load_formatter(x, pos):\n",
+    "    return f'{(x/1000):.0f}k'\n",
+    "\n",
+    "markers = ['s', 'D', 'o', 'v', '^', '<', '>', 'p', 'X']\n",
+    "\n",
+    "def splitSerToArr(ser):\n",
+    "    return [ser.index, ser.as_matrix()]\n",
+    "\n",
+    "plt.figure()\n",
+    "#plt.figure(figsize=(4.8, 3.6)) # For other plot sizes\n",
+    "#ax = df.plot(kind='line', marker='o')\n",
+    "for i, column in enumerate(df):\n",
+    "    plt.plot(df[column].dropna(), marker=markers[i], label=column)\n",
+    "plt.legend()\n",
+    "ax = plt.gca()\n",
+    "#ax = df.plot(kind='line',x='dim_value', legend=False, use_index=True)\n",
+    "ax.set_ylabel('number of instances')\n",
+    "ax.set_xlabel('messages/second')\n",
+    "ax.set_ylim(ymin=0)\n",
+    "#ax.set_xlim(xmin=0)\n",
+    "ax.yaxis.set_major_locator(MaxNLocator(integer=True))\n",
+    "ax.xaxis.set_major_formatter(FuncFormatter(load_formatter))\n",
+    "\n",
+    "plt.savefig('temp.pdf', bbox_inches='tight')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python",
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "version": "3.8.5-final"
+  },
+  "orig_nbformat": 2,
+  "file_extension": ".py",
+  "mimetype": "text/x-python",
+  "name": "python",
+  "npconvert_exporter": "python",
+  "pygments_lexer": "ipython3",
+  "version": 3,
+  "kernelspec": {
+   "name": "python37064bitvenvvenv6c432ee1239d4f3cb23f871068b0267d",
+   "display_name": "Python 3.7.0 64-bit ('.venv': venv)",
+   "language": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file

analysis/demand-metric.ipynb

+{
+ "cells": [
+  {
+   "source": [
+    "# Theodolite Analysis - Demand Metric\n",
+    "\n",
+    "This notebook allows applies Theodolite's *demand* metric to describe scalability of a SUT based on Theodolite measurement data.\n",
+    "\n",
+    "Theodolite's *demand* metric is a function, mapping load intensities to the minimum required resources (e.g., instances) that are required to process this load. With this notebook, the *demand* metric function is approximated by a map of tested load intensities to their minimum required resources.\n",
+    "\n",
+    "The final output when running this notebook will be a CSV file, providig this mapping. It can be used to create nice plots of a system's scalability using the `demand-metric-plot.ipynb` notebook."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "source": [
+    "In the following cell, we need to specifiy:\n",
+    "\n",
+    "* `exp_id`: The experiment id  that is to be analyzed.\n",
+    "* `warmup_sec`: The number of seconds which are to be ignored in the beginning of each experiment.\n",
+    "* `max_lag_trend_slope`: The maximum tolerable increase in queued messages per second.\n",
+    "* `measurement_dir`: The directory where the measurement data files are to be found.\n",
+    "* `results_dir`: The directory where the computed demand CSV files are to be stored."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "exp_id = 200\n",
+    "warmup_sec = 60\n",
+    "max_lag_trend_slope = 2000\n",
+    "measurement_dir = '<path-to>/measurements'\n",
+    "results_dir = '<path-to>/results'\n"
+   ]
+  },
+  {
+   "source": [
+    "With the following call, we compute our demand mapping."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from src.demand import demand\n",
+    "\n",
+    "demand = demand(exp_id, measurement_dir, max_lag_trend_slope, warmup_sec)"
+   ]
+  },
+  {
+   "source": [
+    "We might already want to plot a simple visualization here:"
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "demand.plot(kind='line',x='load',y='resources')"
+   ]
+  },
+  {
+   "source": [
+    "Finally we store the results in a CSV file."
+   ],
+   "cell_type": "markdown",
+   "metadata": {}
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import os\n",
+    "\n",
+    "demand.to_csv(os.path.join(results_dir, f'exp{exp_id}_demand.csv'), index=False)"
+   ]
+  }
+ ],
+ "metadata": {
+  "language_info": {
+   "name": "python",
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "version": "3.8.5-final"
+  },
+  "orig_nbformat": 2,
+  "file_extension": ".py",
+  "mimetype": "text/x-python",
+  "name": "python",
+  "npconvert_exporter": "python",
+  "pygments_lexer": "ipython3",
+  "version": 3,
+  "kernelspec": {
+   "name": "python37064bitvenvvenv6c432ee1239d4f3cb23f871068b0267d",
+   "display_name": "Python 3.7.0 64-bit ('.venv': venv)",
+   "language": "python"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}
\ No newline at end of file

analysis/scalability-graph.ipynb

@@ -245,7 +245,7 @@
    "metadata": {},
    "outputs": [],
    "source": [
-    "min_suitable_instances.to_csv(os.path.join(directory_out, f'../results-inst/exp{exp_id}_min-suitable-instances.csv'), index=False)"
+    "min_suitable_instances.to_csv(os.path.join(directory_out, f'exp{exp_id}_min-suitable-instances.csv'), index=False)"
    ]
   },
   {

analysis/src/demand.py

+import os
+from datetime import datetime, timedelta, timezone
+import pandas as pd
+from sklearn.linear_model import LinearRegression
+
+def demand(exp_id, directory, threshold, warmup_sec):
+    raw_runs = []
+
+    # Compute SL, i.e., lag trend, for each tested configuration
+    filenames = [filename for filename in os.listdir(directory) if filename.startswith(f"exp{exp_id}") and filename.endswith("totallag.csv")]
+    for filename in filenames:
+        #print(filename)
+        run_params = filename[:-4].split("_")
+        dim_value = run_params[2]
+        instances = run_params[3]
+
+        df = pd.read_csv(os.path.join(directory, filename))
+        #input = df.loc[df['topic'] == "input"]
+        input = df
+        #print(input)
+        input['sec_start'] = input.loc[0:, 'timestamp'] - input.iloc[0]['timestamp']
+        #print(input)
+        #print(input.iloc[0, 'timestamp'])
+        regress = input.loc[input['sec_start'] >= warmup_sec] # Warm-Up
+        #regress = input
+
+        #input.plot(kind='line',x='timestamp',y='value',color='red')
+        #plt.show()
+
+        X = regress.iloc[:, 2].values.reshape(-1, 1)  # values converts it into a numpy array
+        Y = regress.iloc[:, 3].values.reshape(-1, 1)  # -1 means that calculate the dimension of rows, but have 1 column
+        linear_regressor = LinearRegression()  # create object for the class
+        linear_regressor.fit(X, Y)  # perform linear regression
+        Y_pred = linear_regressor.predict(X)  # make predictions
+
+        trend_slope = linear_regressor.coef_[0][0]
+        #print(linear_regressor.coef_)
+
+        row = {'load': int(dim_value), 'resources': int(instances), 'trend_slope': trend_slope}
+        #print(row)
+        raw_runs.append(row)
+
+    runs = pd.DataFrame(raw_runs)
+
+    # Set suitable = True if SLOs are met, i.e., lag trend is below threshold
+    runs["suitable"] =  runs.apply(lambda row: row['trend_slope'] < threshold, axis=1)
+
+    # Sort results table (unsure if required)
+    runs.columns = runs.columns.str.strip()
+    runs.sort_values(by=["load", "resources"])
+
+    # Filter only suitable configurations
+    filtered = runs[runs.apply(lambda x: x['suitable'], axis=1)]
+
+    # Compute demand per load intensity
+    grouped = filtered.groupby(['load'])['resources'].min()
+    demand_per_load = grouped.to_frame().reset_index()
+
+    return demand_per_load

execution/README.md

@@ -153,11 +153,11 @@ declarations for different volume types.
 Using a [hostPath volume](https://kubernetes.io/docs/concepts/storage/volumes/#hostpath) is the easiest option when
 running Theodolite locally, e.g., with minikube or kind.
 
-Just modify `infrastructure/kubernetes/volumeSingle.yaml` by setting `path` to the directory on your host machine where
+Just modify `infrastructure/kubernetes/volume-hostpath.yaml` by setting `path` to the directory on your host machine where
 all benchmark results should be stored and run:
 
 ```sh
-kubectl apply -f infrastructure/kubernetes/volumeSingle.yaml
+kubectl apply -f infrastructure/kubernetes/volume-hostpath.yaml
 ```
 
 ##### *local* volume
@@ -166,12 +166,12 @@ A [local volume](https://kubernetes.io/docs/concepts/storage/volumes/#local) is
 access (e.g. via SSH) to one of your cluster nodes.
 
 You first need to create a directory on a selected node where all benchmark results should be stored. Next, modify
-`infrastructure/kubernetes/volumeCluster.yaml` by setting `<node-name>` to your selected node (this node will most
-likely also execute the job). Further, you have to set `path` to the directory on the node you just created. To deploy
+`infrastructure/kubernetes/volume-local.yaml` by setting `<node-name>` to your selected node. (This node will most
+likely also execute the [Theodolite job](#Execution).) Further, you have to set `path` to the directory on the node you just created. To deploy
 you volume run:
 
 ```sh
-kubectl apply -f infrastructure/kubernetes/volumeCluster.yaml
+kubectl apply -f infrastructure/kubernetes/volume-local.yaml
 ```
 
 ##### Other volumes
@@ -195,7 +195,7 @@ RBAC is enabled on your cluster (see installation of [Theodolite RBAC](#Theodoli
 To start the execution of a benchmark run (with `<your-theodolite-yaml>` being your job definition):
 
 ```sh
-kubectl apply -f <your-theodolite-yaml>
+kubectl create -f <your-theodolite-yaml>
 ```
 
 This will create a pod with a name such as `your-job-name-xxxxxx`. You can verifiy this via `kubectl get pods`. With

execution/infrastructure/kafka-lag-exporter/values.yaml

+image:
+  pullPolicy: IfNotPresent
+
 clusters:
   - name: "my-confluent-cp-kafka"
     bootstrapBrokers: "my-confluent-cp-kafka:9092"

execution/infrastructure/kafka/values.yaml

@@ -55,6 +55,7 @@ cp-kafka:
     # "min.insync.replicas": 2
     "auto.create.topics.enable": false
     "log.retention.ms": "10000" # 10s
+    #"log.retention.ms": "86400000" # 24h
     "metrics.sample.window.ms": "5000" #5s
 
 ## ------------------------------------------------------

execution/theodolite.yaml

@@ -11,7 +11,7 @@ spec:
           claimName: theodolite-pv-claim
       containers:
         - name: theodolite
-          image: bvonheid/theodolite:latest
+          image: ghcr.io/cau-se/theodolite:latest
           # imagePullPolicy: Never # Used to pull "own" local image
           env:
             - name: UC # mandatory