Merge branch 'concepts-docs' into 'master'

Add docs for fundamental concepts

See merge request !205

docs/_includes/head_custom.html

+{% if page.usemathjax %}
+<script type="text/javascript" async
+ src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-MML-AM_CHTML">
+</script>
+{% endif %}
\ No newline at end of file

docs/_sass/custom/custom.scss

+.main-content a {
+    white-space: normal;
+}
+
 .theodolite-logo {
     height: 18em;
-}
\ No newline at end of file
+}
+
+.MJXc-display {
+    overflow-x: auto;
+}
+
+table.fixed-colums {
+    th, td {
+       width: 1%;
+    }
+}

docs/benchmarks-and-executions.md → docs/concepts/benchmarks-and-executions.md

 ---
-title: Fundamental Concepts
+title: Data and Role Model
 has_children: false
-nav_order: 2
+parent: Fundamental Concepts
+nav_order: 3
 ---
 
 # Benchmarks and Executions

docs/concepts/index.md

+---
+title: Fundamental Concepts
+has_children: true
+nav_order: 2
+---
+
+# Fundamental Concepts
+
+> A theodolite is a precision optical instrument for measuring angles between designated visible points in the horizontal and vertical planes.  -- <cite>[Wikipedia](https://en.wikipedia.org/wiki/Theodolite)</cite>
+
+Theodolite is a framework for benchmarking the horizontal and vertical scalability of cloud-native applications.
+
+Theodolite adopts established definitions of scalability in cloud computing for its benchmarking method. It quantifies
+scalability by running isolated experiments for different load intensities and provisioned resource amounts, which assess whether specified SLOs are fulfilled. [Two metrics are available](metrics): The demand metric describes how the amount of minimal required resources evolve with increasing load intensities, while the capacity metric describes how the maximal processable load evolves with increasing resources. Hence, both metrics are functions. <!--Example?-->
+
+The terms load, resources and SLOs are consciously kept abstract as Theodolite leaves it to the benchmark designer to define what type of load, resources, and SLOs should be evaluated. For example, horizontal scalability can be benchmarked by varying the amount of Kubernetes Pods, while vertical scalability can be benchmarked by varying CPU and memory constraints of Pods.
+
+To balance statistical grounding and time-efficient benchmark execution, Theodolite comes with different [heuristic for
+evaluating the search space](search-strategies) of load and resource combinations. Other configuration options include the number of repetitions, the experiment and warm-up duration, as well as the amount of different load and resource values to be evaluated.
\ No newline at end of file

docs/concepts/metrics.md

+---
+title: Scalability Metrics
+has_children: false
+parent: Fundamental Concepts
+nav_order: 1
+usemathjax: true
+---
+
+# Theodolite's Scalability Metrics
+
+Theodolite's scalability metrics are based on the following definition:
+
+> Scalability is the ability of [a] system to sustain increasing workloads by making use of additional resources. -- <cite>[Herbst et al. (2013)](https://www.usenix.org/conference/icac13/technical-sessions/presentation/herbst)</cite>
+
+Based on this definition, scalability can be characterized by the following three attributes:
+
+* **Load intensity** is the input variable to which a system is subjected. Scalability is evaluated within a range of load intensities.
+* **Service levels objectives (SLOs)** are measurable quality criteria that have to be fulfilled for every load intensity.
+* **Provisioned resources** can be increased to meet the SLOs if load intensities increase.
+
+
+## Scalability Metrics
+
+Theodolite uses these attributes to define two scalability metrics:
+
+| Resource Demand Metric | Load Capacity Metric |
+|:----|:----|
+| The resource demand metric quantifies scalability by describing how the amount of minimal required resources (i.e., all SLOs are fulfilled) evolves with increasing load intensities. | The load capacity metric quantifies scalability by describing how the maximal processable load (i.e., all SLOs are fulfilled) evolves with increasing resources. |
+| ![Example for resource demand metric](../../assets/images/demand.svg){: .d-block .mx-auto } *Example: Scalability of two stream processing engines measured with the demand metric.*{: .d-block .text-center } | ![Example for load capacity metric](../../assets/images/capacity.svg){: .d-block .mx-auto }  *Example: Scalability of two stream processing engines measured with the capacity metric.*{: .d-block .text-center } |
+{: .fixed-colums }
+
+<!--
+## Resource Demand Metric
+
+![](../../assets/images/capacity.svg){: .d-block .mx-auto }
+
+
+## Load Capacity Metric
+
+![](../../assets/images/demand.svg){: .d-block .mx-auto }
+-->
+
+
+## Formal Definition
+
+For a more formal definition of both metrics, we define the load type as the set of possible load intensities for that
+type, denoted as $$L$$.
+Similarly, we define the resource type as the set of possible resources, denoted as $$R$$.
+We also require that there exists an ordering on both sets $$L$$ and $$R$$.
+We define the set of all SLOs as $$S$$ and denote an SLO $$s \in S$$ as Boolean-valued function
+$$\text{slo}_s: L \times R \to \{\text{false},\text{true}\}$$ with $$\text{slo}_s(l,r) = \text{true}$$ if a system deployed with $$r$$ resource amounts does not violate SLO $$s$$ when processing load intensity $$l$$.
+
+We can denote our **resource demand** metric as $$\text{demand: } L \to R$$, defined as:
+
+$$
+\forall l \in L: \text{demand}(l) = \min\{r \in R \mid \forall s \in S: \text{slo}_s(l,r) = \text{true}\}
+$$
+
+And similarly denote our **resource capacity** metric as $$\text{capacity: } R \to L$$, defined as:
+
+$$
+\forall r \in R: \text{capacity}(r) = \max\{l \in L \mid \forall s \in S: \text{slo}_s(l,r) = \text{true}\}
+$$
+
+## Further Reading
+
+S. Henning and W. Hasselbring. “[How to Measure Scalability of Distributed Stream Processing Engines?](https://research.spec.org/icpe_proceedings/2021/companion/p85.pdf)” In: International Conference on Performance Engineering. 2021. DOI: [10.1145/3447545.3451190](https://doi.org/10.1145/3447545.3451190).
+<!-- TODO replace with EMSE paper one day-->

docs/concepts/search-strategies.md

+---
+title: Search Strategies
+has_children: false
+parent: Fundamental Concepts
+nav_order: 2
+---
+
+# Theodolite's Search Strategies
+
+Theodolite measures [its metrics](metrics) by performing isolated experiments for different load intensities and provisioned resource amounts.
+However, to determine a system's resource demand or load capacity or a good approximation of those, it is often not necessary to evaluate each possible combination. Instead, Theodolite provides search strategies, which decide at benchmark runtime which combinations of load and resources to evaluate.
+
+The following search strategies are available:
+
+### Full Search
+
+The full search strategy performs SLO experiments for each combination of resource configuration and load intensity. Its advantage is that it allows for extensive evaluation after the benchmark has been executed. This also includes that based on the same SLO experiments, both the demand and the capacity metric can be evaluated. However, this comes at the cost of significantly longer execution times.
+
+### Linear Search
+
+The linear search strategy reduces the overall execution time by not running SLO experiments whose results are not required by the metric.
+For the resource demand metric this means, as soon as a sufficient resource configuration for a certain load intensity is found, no further resource configurations are tested for that load.
+
+### Binary Search
+
+The binary search strategy adopts the well known binary search algorithm for sorted arrays.
+For the resource demand metric this means, the strategy starts by performing the SLO experiments for the middle resource configuration.
+Depending on whether this experiment was successful or not, it then continues searching in the lower or upper half, respectively.
+The binary search is particularly advantageous if the search space is very large.
+However it is based on the assumption that with additional resources for the same load, performance does not substantially decrease.
+
+### Lower Bound Restriction Search
+
+The lower bound restriction is a search strategy that uses the results of already performed SLO experiments to narrow the search space.
+For the resource demand, it starts searching (with another strategy) beginning from the minimal required resources of all lower load intensities.
+The lower bound restriction is based on the assumption that with increasing load intensity, the resource demand never decreases.
+
+<!-- further reading in EMSE paper -->
\ No newline at end of file

docs/creating-a-benchmark.md

@@ -6,7 +6,7 @@ nav_order: 5
 
 # Creating a Benchmark
 
-Please note that to simply run a benchmark, it is not required to define one. Theodolite comes with a [set of benchmarks](theodolite-benchmarks), which are ready to be executed. See the [fundamental concepts](benchmarks-and-executions) page to learn more about our distinction between benchmarks and executions.
+Please note that to simply run a benchmark, it is not required to define one. Theodolite comes with a [set of benchmarks](theodolite-benchmarks), which are ready to be executed. See the [fundamental concepts](concepts) page to learn more about our distinction between benchmarks and executions.
 
 A typical benchmark looks like this:
 

docs/index.md

@@ -12,7 +12,7 @@ Theodolite is a framework for benchmarking the horizontal and vertical scalabili
 {: .fs-6 .fw-300 .text-center }
 
 [Get started now](quickstart){: .btn .btn-primary .fs-5 .mb-4 .mb-md-0 .mr-4 }
-[Learn the underlying concepts](benchmarks-and-executions){: .btn .fs-5 .mb-4 .mb-md-0 }
+[Learn the underlying concepts](concepts){: .btn .fs-5 .mb-4 .mb-md-0 }
 {: .text-center }
 
 ---