diff --git a/README.md b/README.md index f2f32ccca45cb369c0117299c035c8512b866500..26c8e8696e6823bd2a2de4e509556580b5f1fafd 100644 --- a/README.md +++ b/README.md @@ -2,28 +2,55 @@ Website: http://kieker-monitoring.net/MooBench -The MooBench micro-benchmarks can be used to quantify the performance overhead caused by monitoring framework components. +The MooBench micro-benchmarks can be used to quantify the performance overhead +caused by monitoring framework components. Currenly (fully) supported monitoring frameworks are: * Kieker (http://kieker-monitoring.net) * OpenTelemetry (https://opentelemetry.io/) * inspectIT (https://inspectit.rocks/) +## Directory Structure + +- analysis = analysis scripts +- benchmark = moobench code +- continuous-integration = obsolete +- docker = Dockerfile to be used in automated runs on our intrestructure +- frameworks = benchmark setups for the different frameworks + - Kieker + - OpenTelementry + - SPASSmeter + - inspectIT +- gradle = build system, gradle libraries +- tools = tooling used to support benchmarks and process results + - compile-results = adds new results to a result log and computes partial + views of the results for presentation in websites + - getConfidenceIntervalTable = compute the confidence interval table + - receiver = receiver for Kieker TCP probe output + ## Benchmark Execution Initially, the following steps are required: -1. Make sure, that you've installed R (http://www.r-project.org/) to generate the results , awk to install intermediate results and curl to download processing tools (Ubuntu: `sudo apt install r-base gawk curl`). +1. Make sure, that you've installed R (http://www.r-project.org/) to generate + the results , awk to install intermediate results and curl to download + processing tools (Ubuntu: `sudo apt install r-base gawk curl`). 2. Compile the application by calling `./gradlew assemble` in the main folder. -All experiments are started with the provided "External Controller" scripts. The following scripts are available for every framework ($FRAMEWORK): -* In `frameworks/$FRAMEWORK/benchmark.sh` a script is provided for regular execution (with default parameters) -* In `frameworks/$FRAMEWORK/runExponentialSizes.sh` a script is provided for execution with different call tree depth sizes (exponentially growing from 2) +All experiments are started with the provided "External Controller" scripts. +The following scripts are available for every framework ($FRAMEWORK): +* In `frameworks/$FRAMEWORK/benchmark.sh` a script is provided for regular + execution (with default parameters) +* In `frameworks/$FRAMEWORK/runExponentialSizes.sh` a script is provided for + execution with different call tree depth sizes (exponentially growing from 2) -Each scripts will start different factorial experiments (started `$NUM_OF_LOOPS` times for repeatability), which will be: +Each scripts will start different factorial experiments (started `$NUM_OF_LOOPS` +times for repeatability), which will be: - baseline execution - execution with instrumentation but without processing or serialization -- execution with serialization to hard disc (currently not available for inspectIT) -- execution with serialization to tcp receiver, which might be a simple receiver (Kieker), or Zikpin and Prometheus (OpenTelemetry and inspectIT) +- execution with serialization to hard disc (currently not available for + inspectIT) +- execution with serialization to tcp receiver, which might be a simple receiver + (Kieker), or Zikpin and Prometheus (OpenTelemetry and inspectIT) All scripts have been tested on Ubuntu and Raspbian. @@ -46,8 +73,24 @@ cd frameworks/opentelemetry/ ``` ## Data Analysis -Each benchmark execution calls an R script providing mean, standard deviation and confidence intervals for the benchmark variants. If you want to get these values again, switch to `frameworks` and call `runR.sh $FRAMEWORK`, where framework is the folder name of the framework (e.g. Kieker). -If you got data from a run with exponential growing call tree depth, unzip them first (`for file in *.zip; do unzip $file; done`), copy all `results-$framework` folder to a common folder and run `./getExponential.sh` in analysis. This will create a graph for each framework and an overview graph for external processing of the traces (zipkin for OpenTelemetry and inspectIT, TCP for Kieker). +Each benchmark execution calls an R script providing mean, standard deviation +and confidence intervals for the benchmark variants. If you want to get these +values again, switch to `frameworks` and call `runR.sh $FRAMEWORK`, where +framework is the folder name of the framework (e.g. Kieker). + +If you got data from a run with exponential growing call tree depth, unzip them +first (`for file in *.zip; do unzip $file; done`), copy all `results-$framework` +folder to a common folder and run `./getExponential.sh` in analysis. This will +create a graph for each framework and an overview graph for external processing +of the traces (zipkin for OpenTelemetry and inspectIT, TCP for Kieker). + +In the folder /bin/r are some R scripts provided to generate graphs to visualize +the results. In the top the files, one can configure the required paths and the +configuration used to analyze the data. + +## Quality Control + +We also use MooBench as a performance regression test which is run periodically +when new features are added to Kieker. -In the folder /bin/r are some R scripts provided to generate graphs to visualize the results. In the top the files, one can configure the required paths and the configuration used to analyze the data. diff --git a/build.gradle b/build.gradle index 73de2cfac6650d35d8bee3eb2464e0f9bcae0c8f..46c5c8cada7d73a73005beaa1808edb9ba21b0da 100644 --- a/build.gradle +++ b/build.gradle @@ -1,9 +1,5 @@ /* - * This file was generated by the Gradle 'init' task. - * - * This generated file contains a sample Java project to get you started. - * For more details take a look at the Java Quickstart chapter in the Gradle - * User Manual available at https://docs.gradle.org/6.5/userguide/tutorial_java_projects.html + * Main gradle build script. */ plugins { @@ -11,9 +7,6 @@ plugins { } repositories { - // Use jcenter for resolving dependencies. - // You can declare any Maven/Ivy/file repository here. - jcenter() mavenCentral() maven { url "https://oss.sonatype.org/content/repositories/snapshots/" } } @@ -29,9 +22,6 @@ dependencies { subprojects { repositories { - // Use jcenter for resolving dependencies. - // You can declare any Maven/Ivy/file repository here. - jcenter() mavenCentral() maven { url "https://oss.sonatype.org/content/repositories/snapshots/" } } diff --git a/frameworks/Kieker/common-functions b/frameworks/common-functions similarity index 100% rename from frameworks/Kieker/common-functions rename to frameworks/common-functions diff --git a/frameworks/common-functions.sh b/frameworks/common-functions.sh index 853bec16a108e98568d95e9c7792b888b00826c6..377ea99fd6cabcbc9cfe55cce4110f67a2f18b39 100755 --- a/frameworks/common-functions.sh +++ b/frameworks/common-functions.sh @@ -1,5 +1,20 @@ #!/bin/bash +# +# Common functions used in scripts. +# + +# ensure the script is sourced +if [ "${BASH_SOURCE[0]}" -ef "$0" ] +then + echo "Hey, you should source this script, not execute it!" + exit 1 +fi + +# +# functions +# + function getSum { awk '{sum += $1; square += $1^2} END {print "Average: "sum/NR" Standard Deviation: "sqrt(square / NR - (sum/NR)^2)" Count: "NR}' } @@ -141,6 +156,77 @@ function printIntermediaryResults { done } +# +# reporting +# + +export RED='\033[1;31m' +export WHITE='\033[1;37m' +export YELLOW='\033[1;33m' +export NC='\033[0m' + +if [ "$BATCH_MODE" == "yes" ] ; then + export ERROR="[error]" + export WARNING="[warning]" + export INFO="[info]" +else + export ERROR="${RED}[error]${NC}" + export WARNING="${YELLOW}[warning]${NC}" + export INFO="${WHITE}[info]${NC}" +fi + +function error() { + echo -e "${ERROR} $@" +} + +function warn() { + echo -e "${WARNING} $@" +} + +function info() { + echo -e "${INFO} $@" +} + +# $1 = NAME, $2 = EXECUTABLE +function checkExecutable() { + if [ "$2" == "" ] ; then + error "$1 variable for executable not set." + exit 1 + fi + if [ ! -x "$2" ] ; then + error "$1 not found at: $2" + exit 1 + fi +} + +# $1 = NAME, $2 = FILE +function checkFile() { + if [ "$2" == "" ] ; then + error "$1 variable for file not set." + exit 1 + fi + if [ ! -f "$2" ] ; then + error "$1 not found at: $2" + exit 1 + fi +} + +# $1 = NAME, $2 = FILE +function checkDirectory() { + if [ "$2" == "" ] ; then + error "$1 directory variable not set." + exit 1 + fi + if [ ! -d "$2" ] ; then + if [ "$3" == "create" ] ; then + information "$1: directory does not exist, creating it" + mkdir $2 + else + error "$1: directory $2 does not exist." + exit 1 + fi + fi +} FRAMEWORK_NAME=$(basename -- "${BASE_DIR}") RESULTS_DIR="${BASE_DIR}/results-${FRAMEWORK_NAME}"