Skip to content
Snippets Groups Projects
Commit 85e27391 authored by Sören Henning's avatar Sören Henning
Browse files

Revert merge of docs branch

parent 4b83719f
No related branches found
No related tags found
2 merge requests!159Re-implementation of Theodolite with Kotlin/Quarkus,!152Adjust Benchmark Definitions for Operator
Pipeline #4031 passed
......@@ -15,28 +15,6 @@ stages:
variables:
DOCKER_TLS_CERTDIR: "/certs"
# Theodolite Docs
build-docs:
stage: build
image: ruby:2.7
tags:
- exec-docker
cache:
paths:
- docs/vendor
before_script:
- cd docs
- gem install bundler
- bundle config set --local path 'vendor'
- bundle install
script: bundle exec jekyll build
artifacts:
paths:
- docs/_site
# Theodolite Helm Chart
lint-helm:
......
_site
.sass-cache
.jekyll-cache
.jekyll-metadata
vendor
source "https://rubygems.org"
# Hello! This is where you manage which Jekyll version is used to run.
# When you want to use a different version, change it below, save the
# file and run `bundle install`. Run Jekyll with `bundle exec`, like so:
#
# bundle exec jekyll serve
#
# This will help ensure the proper Jekyll version is running.
# Happy Jekylling!
#gem "jekyll", "~> 4.2.0"
# This is the default theme for new Jekyll sites. You may change this to anything you like.
gem "minima", "~> 2.5"
# If you want to use GitHub Pages, remove the "gem "jekyll"" above and
# uncomment the line below. To upgrade, run `bundle update github-pages`.
# gem "github-pages", group: :jekyll_plugins
gem "github-pages", "~> 215", group: :jekyll_plugins
# If you have any plugins, put them here!
#group :jekyll_plugins do
#gem "jekyll-feed", "~> 0.12"
#end
# Windows and JRuby does not include zoneinfo files, so bundle the tzinfo-data gem
# and associated library.
platforms :mingw, :x64_mingw, :mswin, :jruby do
gem "tzinfo", "~> 1.2"
gem "tzinfo-data"
end
# Performance-booster for watching directories on Windows
gem "wdm", "~> 0.1.1", :platforms => [:mingw, :x64_mingw, :mswin]
GEM
remote: https://rubygems.org/
specs:
activesupport (6.0.4)
concurrent-ruby (~> 1.0, >= 1.0.2)
i18n (>= 0.7, < 2)
minitest (~> 5.1)
tzinfo (~> 1.1)
zeitwerk (~> 2.2, >= 2.2.2)
addressable (2.7.0)
public_suffix (>= 2.0.2, < 5.0)
coffee-script (2.4.1)
coffee-script-source
execjs
coffee-script-source (1.11.1)
colorator (1.1.0)
commonmarker (0.17.13)
ruby-enum (~> 0.5)
concurrent-ruby (1.1.9)
dnsruby (1.61.7)
simpleidn (~> 0.1)
em-websocket (0.5.2)
eventmachine (>= 0.12.9)
http_parser.rb (~> 0.6.0)
ethon (0.14.0)
ffi (>= 1.15.0)
eventmachine (1.2.7)
execjs (2.8.1)
faraday (1.4.3)
faraday-em_http (~> 1.0)
faraday-em_synchrony (~> 1.0)
faraday-excon (~> 1.1)
faraday-net_http (~> 1.0)
faraday-net_http_persistent (~> 1.1)
multipart-post (>= 1.2, < 3)
ruby2_keywords (>= 0.0.4)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.1.0)
ffi (1.15.3)
forwardable-extended (2.6.0)
gemoji (3.0.1)
github-pages (215)
github-pages-health-check (= 1.17.2)
jekyll (= 3.9.0)
jekyll-avatar (= 0.7.0)
jekyll-coffeescript (= 1.1.1)
jekyll-commonmark-ghpages (= 0.1.6)
jekyll-default-layout (= 0.1.4)
jekyll-feed (= 0.15.1)
jekyll-gist (= 1.5.0)
jekyll-github-metadata (= 2.13.0)
jekyll-mentions (= 1.6.0)
jekyll-optional-front-matter (= 0.3.2)
jekyll-paginate (= 1.1.0)
jekyll-readme-index (= 0.3.0)
jekyll-redirect-from (= 0.16.0)
jekyll-relative-links (= 0.6.1)
jekyll-remote-theme (= 0.4.3)
jekyll-sass-converter (= 1.5.2)
jekyll-seo-tag (= 2.7.1)
jekyll-sitemap (= 1.4.0)
jekyll-swiss (= 1.0.0)
jekyll-theme-architect (= 0.1.1)
jekyll-theme-cayman (= 0.1.1)
jekyll-theme-dinky (= 0.1.1)
jekyll-theme-hacker (= 0.1.2)
jekyll-theme-leap-day (= 0.1.1)
jekyll-theme-merlot (= 0.1.1)
jekyll-theme-midnight (= 0.1.1)
jekyll-theme-minimal (= 0.1.1)
jekyll-theme-modernist (= 0.1.1)
jekyll-theme-primer (= 0.5.4)
jekyll-theme-slate (= 0.1.1)
jekyll-theme-tactile (= 0.1.1)
jekyll-theme-time-machine (= 0.1.1)
jekyll-titles-from-headings (= 0.5.3)
jemoji (= 0.12.0)
kramdown (= 2.3.1)
kramdown-parser-gfm (= 1.1.0)
liquid (= 4.0.3)
mercenary (~> 0.3)
minima (= 2.5.1)
nokogiri (>= 1.10.4, < 2.0)
rouge (= 3.26.0)
terminal-table (~> 1.4)
github-pages-health-check (1.17.2)
addressable (~> 2.3)
dnsruby (~> 1.60)
octokit (~> 4.0)
public_suffix (>= 2.0.2, < 5.0)
typhoeus (~> 1.3)
html-pipeline (2.14.0)
activesupport (>= 2)
nokogiri (>= 1.4)
http_parser.rb (0.6.0)
i18n (0.9.5)
concurrent-ruby (~> 1.0)
jekyll (3.9.0)
addressable (~> 2.4)
colorator (~> 1.0)
em-websocket (~> 0.5)
i18n (~> 0.7)
jekyll-sass-converter (~> 1.0)
jekyll-watch (~> 2.0)
kramdown (>= 1.17, < 3)
liquid (~> 4.0)
mercenary (~> 0.3.3)
pathutil (~> 0.9)
rouge (>= 1.7, < 4)
safe_yaml (~> 1.0)
jekyll-avatar (0.7.0)
jekyll (>= 3.0, < 5.0)
jekyll-coffeescript (1.1.1)
coffee-script (~> 2.2)
coffee-script-source (~> 1.11.1)
jekyll-commonmark (1.3.1)
commonmarker (~> 0.14)
jekyll (>= 3.7, < 5.0)
jekyll-commonmark-ghpages (0.1.6)
commonmarker (~> 0.17.6)
jekyll-commonmark (~> 1.2)
rouge (>= 2.0, < 4.0)
jekyll-default-layout (0.1.4)
jekyll (~> 3.0)
jekyll-feed (0.15.1)
jekyll (>= 3.7, < 5.0)
jekyll-gist (1.5.0)
octokit (~> 4.2)
jekyll-github-metadata (2.13.0)
jekyll (>= 3.4, < 5.0)
octokit (~> 4.0, != 4.4.0)
jekyll-mentions (1.6.0)
html-pipeline (~> 2.3)
jekyll (>= 3.7, < 5.0)
jekyll-optional-front-matter (0.3.2)
jekyll (>= 3.0, < 5.0)
jekyll-paginate (1.1.0)
jekyll-readme-index (0.3.0)
jekyll (>= 3.0, < 5.0)
jekyll-redirect-from (0.16.0)
jekyll (>= 3.3, < 5.0)
jekyll-relative-links (0.6.1)
jekyll (>= 3.3, < 5.0)
jekyll-remote-theme (0.4.3)
addressable (~> 2.0)
jekyll (>= 3.5, < 5.0)
jekyll-sass-converter (>= 1.0, <= 3.0.0, != 2.0.0)
rubyzip (>= 1.3.0, < 3.0)
jekyll-sass-converter (1.5.2)
sass (~> 3.4)
jekyll-seo-tag (2.7.1)
jekyll (>= 3.8, < 5.0)
jekyll-sitemap (1.4.0)
jekyll (>= 3.7, < 5.0)
jekyll-swiss (1.0.0)
jekyll-theme-architect (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-cayman (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-dinky (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-hacker (0.1.2)
jekyll (> 3.5, < 5.0)
jekyll-seo-tag (~> 2.0)
jekyll-theme-leap-day (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-merlot (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-midnight (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-minimal (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-modernist (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-primer (0.5.4)
jekyll (> 3.5, < 5.0)
jekyll-github-metadata (~> 2.9)
jekyll-seo-tag (~> 2.0)
jekyll-theme-slate (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-tactile (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-theme-time-machine (0.1.1)
jekyll (~> 3.5)
jekyll-seo-tag (~> 2.0)
jekyll-titles-from-headings (0.5.3)
jekyll (>= 3.3, < 5.0)
jekyll-watch (2.2.1)
listen (~> 3.0)
jemoji (0.12.0)
gemoji (~> 3.0)
html-pipeline (~> 2.2)
jekyll (>= 3.0, < 5.0)
kramdown (2.3.1)
rexml
kramdown-parser-gfm (1.1.0)
kramdown (~> 2.0)
liquid (4.0.3)
listen (3.5.1)
rb-fsevent (~> 0.10, >= 0.10.3)
rb-inotify (~> 0.9, >= 0.9.10)
mercenary (0.3.6)
minima (2.5.1)
jekyll (>= 3.5, < 5.0)
jekyll-feed (~> 0.9)
jekyll-seo-tag (~> 2.1)
minitest (5.14.4)
multipart-post (2.1.1)
nokogiri (1.11.7-x86_64-linux)
racc (~> 1.4)
octokit (4.21.0)
faraday (>= 0.9)
sawyer (~> 0.8.0, >= 0.5.3)
pathutil (0.16.2)
forwardable-extended (~> 2.6)
public_suffix (4.0.6)
racc (1.5.2)
rb-fsevent (0.11.0)
rb-inotify (0.10.1)
ffi (~> 1.0)
rexml (3.2.5)
rouge (3.26.0)
ruby-enum (0.9.0)
i18n
ruby2_keywords (0.0.4)
rubyzip (2.3.0)
safe_yaml (1.0.5)
sass (3.7.4)
sass-listen (~> 4.0.0)
sass-listen (4.0.0)
rb-fsevent (~> 0.9, >= 0.9.4)
rb-inotify (~> 0.9, >= 0.9.7)
sawyer (0.8.2)
addressable (>= 2.3.5)
faraday (> 0.8, < 2.0)
simpleidn (0.2.1)
unf (~> 0.1.4)
terminal-table (1.8.0)
unicode-display_width (~> 1.1, >= 1.1.1)
thread_safe (0.3.6)
typhoeus (1.4.0)
ethon (>= 0.9.0)
tzinfo (1.2.9)
thread_safe (~> 0.1)
unf (0.1.4)
unf_ext
unf_ext (0.0.7.7)
unicode-display_width (1.7.0)
zeitwerk (2.4.2)
PLATFORMS
x86_64-linux
x86_64-linux-musl
DEPENDENCIES
github-pages (~> 215)
minima (~> 2.5)
tzinfo (~> 1.2)
tzinfo-data
wdm (~> 0.1.1)
BUNDLED WITH
2.2.21
---
title: Theodolite
nav_order: 1
nav_exclude: true
permalink: /
---
......
title: "Theodolite"
description: >-
Theodolite is a framework for benchmarking the horizontal and vertical
scalability of stream processing engines.
remote_theme: pmarsceill/just-the-docs
#color_scheme: "dark"
aux_links:
"Theodolite on GitHub":
- "//github.com/cau-se/theodolite"
exclude:
- Gemfile
- Gemfile.lock
- vendor
- "//github.com/cau-se/theodolite"
\ No newline at end of file
---
title: Benchmarks and Executions
has_children: false
nav_order: 2
---
# Benchmarks and Executions
In Theodolite, we distinguish between the static description of a scalability benchmark and its execution.
## Benchmarks
Benchmarks define what should be executed in scalability experiments. They
consists of system under test (SUT) and an associated load generator, where
both SUT and load generator are represented as sets of Kubernetes resources
such Pods, Services, or ConfigMaps.
Additionally, benchmarks define one or more load and resource types, scalability
can be evaluated for.
[TODO]: # (Link to metrics)
Benchmarks are portable, that is they can be provided by standardization
organizations, researchers, etc. and installed by other researchers,
engineers, etc.
They do not have a life-cycle. Instead, to run a benchmark, an execution of a
benchmark is created and passed to Theodolite.
## Execution
An execution represents a one-time execution of a benchmark with a
specific configuration. Hence, an execution refers to a benchmark and one of
the load and resource types defined by that benchmark. Additionally, executions
allow to alter the SUT or the load generator to evaluate regarding specific
configurations.
Executions also describe details regarding the scalability measurement method,
such as for long experiment should be performed or how often they should be
repeated.
In contrast to benchmarks, execution have a life-cycle. They can be planned,
executed, or aborted. Each execution of a benchmark is represented by an
individual entity. This supports repeatability as executions can be archived
and shared.
# Defining a Benchmark
Please note that to simply run a benchmark, it is not required to define one. Theodolite comes with a set of benchmarks, which are ready to be executed. See the Benchmarks and Executions docs page to learn more about our distinction between benchmarks and executions.
* Create a benchmark for operator mode
* Create a benchmark for standalone mode
* Deploy a benchmark
## Creating a benchmark for operator mode
### App and Workload Generator Resources
### Load and Resource Types
### Kafka Configuration
### Make benchmark Kubernetes resources accessible
* A benchmark is defined by a set of Kubernetes resources, which are automatically deployed and scaled by Theodolite.
* To make these resources accessible by the Theodolite operator, we need to add them to a volume, already mounted by Theodolite. The easiest way to do this is by adding the resources to the configmap `xyz`, which is mounted by Theodilte with its default installation:
* `sh`
## Creating a benchmark for standalone mode
Creating a benchmark in standalone mode is similar to operator mode. However,
instead of defining a benchmark as `Benchmark` resource, it is defined as a
benchmark YAML file. Such a file deviates in the following points from a
`Benchmark` resource:
* The fields `apiVersion`, `kind` and `metadata` should be removed.
* The benchmark's name (`metadata.name` in `Benchmark` resources) must be defined by the top-level field `name`.
* Everything that is defined in `spec` has to be moved to the top-level.
**TODO: example**
**TODO: Create a ConfigMap containing the benchmark YAML files as well as all Kubernetes resources for that benchmark + deploy**
\ No newline at end of file
---
title: Installation
has_children: false
nav_order: 3
---
# Installing Theodolite
The easiest option to install Theodolite is using [Helm](https://helm.sh).
To install Theodolite with all its dependencies run:
```sh
helm repo add theodolite https://cau-se.github.io/theodolite
helm repo update
helm install theodolite theodolite/theodolite
```
This installs Theodolite in operator mode. Operator mode is the easiest to be used, but requires some permissions in the installation. If those cannot be granted, Theodolite can also be installed for standalone mode.
## Installation Options
As usual, the installation via Helm can be configured by passing a values YAML file:
```
helm install theodolite theodolite/theodolite --values <your-config.yaml>
```
For this purpose the [default values file]() can serve as a template for your custom configuration.
### Operator mode
The Theodolite operator allows to run and manage benchmarks through the Kubernetes API. It is installed by setting `operator.enabled` to `true`. In addition to Theodolite's dependencies, this will deploy all resources that are required for the operator as well as the CRDs, users will interact with.
**TODO:** link
### Standalone mode
For running Theodolite in standalone mode, it is sufficient to disable the operator by setting `operator.enabled` to `false`. With this setting, only Theodolite's dependencies as well as resources to get the necessary permissions are installed.
### Random Scheduler
Installation of the random scheduler can be enabled and via `randomScheduler.enabled`. Please note that the random scheduler is neither required in operator mode nor in standalone mode. However, it has to be installed if benchmark executions should use random scheduling.
**TODO:** link
### Multiple installations in the same cluster
Multiple Theodolite installations in the same namespace are currently not fully tested.
In cases, where you need to install multiple Theodolite instances, it's best to use dedicated namespaces **and** different release names.
*Note that for meaningful results, usually only one benchmark should be executed at a time.*
## Installation Hints
### Installation with a release name other than `theodolite`
When using another release name than `theodolite`, make sure to adjust the Kafka Lag Exporter configuration of you `values.yaml` accordingly:
```yaml
kafka-lag-exporter:
clusters:
- name: "<your-release-name>-cp-kafka"
bootstrapBrokers: "<your-release-name>-cp-kafka:9092"
```
This seems unfortunately to be necessary as Helm does not let us inject values into dependency charts.
---
title: Release Process
has_children: false
nav_order: 5
nav_order: 2
---
# Release Process
......@@ -35,7 +35,7 @@ again be merged into master.
7. Tag this commit `v0.3.1` (can be done via GitLab). The corresponding Docker images will be uploaded.
8. Create *releases* on GitLab and GitHub. Upload the generated Helm package to these releases via the UIs of GitLab and GitHub.
8. Create *releases* on GitLab and GitHub. Upload the generated Helm package to these releases.
9. Switch to the `master` branch.
......
---
title: Running Benchmarks
has_children: false
nav_order: 4
---
# Running Scalability Benchmarks
There are two options to run scalability benchmarks with Theodolite:
1. The [Theodolite Operator](#running-benchmarks-with-the-theodolite-operator) is a long-running service in your cluster, which accepts submitted benchmarks and executes them.
2. In [standalone mode](#running-benchmarks-in-standalone-mode), Theodolite is started as a Kubernetes Job and only runs for the duration of one benchmark execution.
While the operator is a bit more complex to install then the standalone mode,
it makes it way easier to manage benchmarks and their executions. In
particular, it allows to submit a set of experiments, which are then executed
automatically one after another.
**Therefore, running benchmarks with the operator is recommended.**
## Running Benchmarks with the Theodolite Operator
The general process for running Theodolite benchmarks in operator mode is as follows:
1. Create and deploy a new `Benchmark` resource or deploy one of the already existing ones.
2. Define your benchmark execution as an `Execution` resource and deploy it.
### 1. Creating a benchmark
Benchmarks are defined as resources of our custom resource definition
`Benchmark`. You can either create a new `Benchmark` for your custom benchmark
or system under test or deploy one of the existing benchmarks, which can be
found in [`theodolite-benchmarks/definitions`](theodolite-benchmarks/definitions). **TODO link**
Suppose your `Benchmark` resource is stored in `example-benchmark.yaml`, you
can deploy it by running:
```sh
kubectl apply -f example-benchmark.yaml
```
To see the list of all deployed benchmarks run:
```sh
kubectl get benchmarks
```
Additionally you need to make all your benchmark's Kubernetes resources available to the operator.
**TODO benchmark resources**
Once your benchmark is deployed, it is ready to be executed.
### 2. Creating an execution
To execute a benchmark, you need to describe this execution by creating an `Execution` resource.
**TODO: details execution**
Suppose your `Execution` resource is stored in `example-execution.yaml`, you
can deploy it by running:
```sh
kubectl apply -f example-execution.yaml
```
To see the list of all deployed benchmarks run:
```sh
kubectl get executions
```
The `STATUS` field will tell you whether a benchmark execution has been
started, finished or failed due to some error. The `DURATION` field tells you
for how long that execution is running (so far). Similar to a Kubernetes Job,
an `Execution` is not automatically deleted once it is finished. This makes it
easier to keep track of all the benchmark executions and to organize benchmark
results.
## Running Benchmarks in Standalone Mode
The general process for running Theodolite benchmarks in standalone mode is as follows:
1. Create a benchmark by writing a YAML file or select one of the already existing ones and create a ConfigMap from it.
2. Define your benchmark execution as a YAML file and create a ConfigMap from it. We provide a template for this.
3. Create a Theodolite Job from our template and mount your benchmark and benchmark execution with it.
### 1. Creating a benchmark
Creating a benchmark in standalone mode is similar to operator mode. However,
instead of defining a benchmark as `Benchmark` resource, it is defined as a
benchmark YAML file. Such a file deviates in the following points from a
`Benchmark` resource:
* The fields `apiVersion`, `kind` and `metadata` should be removed.
* The benchmark's name (`metadata.name` in `Benchmark` resources) must be defined by the top-level field `name`.
* Everything that is defined in `spec` has to be moved to the top-level.
**TODO: example**
**TODO: Create a ConfigMap containing the benchmark YAML files as well as all Kubernetes resources for that benchmark + deploy**
### 2. Creating an execution
**TODO: see above**
### 3. Create a Theodolite Job
**TODO example**
\ No newline at end of file
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment