Link Search Menu Expand Document Documentation Menu
Documentation

run

Whether you’re using the included SmartObserve Benchmark workloads or a custom workload, use the run command to gather data about the performance of your SmartObserve cluster according to the selected workload.

Usage

The following example runs a test using the geonames workload in test mode:

smartobserve-benchmark run --workload=geonames --test-mode

After the test runs, SmartObserve Benchmark responds with a summary of the benchmark metrics:

------------------------------------------------------
    _______             __   _____
   / ____(_)___  ____ _/ /  / ___/_________  ________
  / /_  / / __ \/ __ `/ /   \__ \/ ___/ __ \/ ___/ _ \
 / __/ / / / / / /_/ / /   ___/ / /__/ /_/ / /  /  __/
/_/   /_/_/ /_/\__,_/_/   /____/\___/\____/_/   \___/
------------------------------------------------------

|                         Metric |                 Task |     Value |   Unit |
|-------------------------------:|---------------------:|----------:|-------:|
|            Total indexing time |                      |   28.0997 |    min |
|               Total merge time |                      |   6.84378 |    min |
|             Total refresh time |                      |   3.06045 |    min |
|               Total flush time |                      |  0.106517 |    min |
|      Total merge throttle time |                      |   1.28193 |    min |
|               Median CPU usage |                      |     471.6 |      % |
|             Total Young Gen GC |                      |    16.237 |      s |
|               Total Old Gen GC |                      |     1.796 |      s |
|                     Index size |                      |   2.60124 |     GB |
|                  Total written |                      |   11.8144 |     GB |
|         Heap used for segments |                      |   14.7326 |     MB |
|       Heap used for doc values |                      |  0.115917 |     MB |
|            Heap used for terms |                      |   13.3203 |     MB |
|            Heap used for norms |                      | 0.0734253 |     MB |
|           Heap used for points |                      |    0.5793 |     MB |
|    Heap used for stored fields |                      |  0.643608 |     MB |
|                  Segment count |                      |        97 |        |
|                 Min Throughput |         index-append |   31925.2 | docs/s |
|              Median Throughput |         index-append |   39137.5 | docs/s |
|                 Max Throughput |         index-append |   39633.6 | docs/s |
|      50.0th percentile latency |         index-append |   872.513 |     ms |
|      90.0th percentile latency |         index-append |   1457.13 |     ms |
|      99.0th percentile latency |         index-append |   1874.89 |     ms |
|       100th percentile latency |         index-append |   2711.71 |     ms |
| 50.0th percentile service time |         index-append |   872.513 |     ms |
| 90.0th percentile service time |         index-append |   1457.13 |     ms |
| 99.0th percentile service time |         index-append |   1874.89 |     ms |
|  100th percentile service time |         index-append |   2711.71 |     ms |
|                           ...  |                  ... |       ... |    ... |
|                           ...  |                  ... |       ... |    ... |
|                 Min Throughput |     painless_dynamic |   2.53292 |  ops/s |
|              Median Throughput |     painless_dynamic |   2.53813 |  ops/s |
|                 Max Throughput |     painless_dynamic |   2.54401 |  ops/s |
|      50.0th percentile latency |     painless_dynamic |    172208 |     ms |
|      90.0th percentile latency |     painless_dynamic |    310401 |     ms |
|      99.0th percentile latency |     painless_dynamic |    341341 |     ms |
|      99.9th percentile latency |     painless_dynamic |    344404 |     ms |
|       100th percentile latency |     painless_dynamic |    344754 |     ms |
| 50.0th percentile service time |     painless_dynamic |    393.02 |     ms |
| 90.0th percentile service time |     painless_dynamic |   407.579 |     ms |
| 99.0th percentile service time |     painless_dynamic |   430.806 |     ms |
| 99.9th percentile service time |     painless_dynamic |   457.352 |     ms |
|  100th percentile service time |     painless_dynamic |   459.474 |     ms |

-------------------------------------
[INFO] ✅ SUCCESS (took 2634 seconds)
-------------------------------------

Options

Use the following options to customize the run command for your use case. Options in this section are categorized by their use case.

General settings

The following options shape how each test runs and how results appear:

  • --test-mode: Runs the given workload in test mode, which is useful when checking a workload for errors.
  • --user-tag: Defines user-specific key-value pairs to be used in metric record as meta information, for example, intention:baseline-ticket-12345.
  • --results-format: Defines the output format for the command line results, either markdown or csv. Default is markdown.
  • --results-number-align: Defines the column number alignment for when the compare command outputs results. Default is right.
  • --results-file: When provided a file path, writes the compare results to the file indicated in the path.
  • --show-in-results: Determines whether or not to include the comparison in the results file.

Distributions

The following options set which version of SmartObserve and the SmartObserve plugins the benchmark test uses:

  • --distribution-version: Downloads the specified SmartObserve distribution based on version number. For a list of released SmartObserve versions, see Version history.
  • --distribution-repository: Defines the repository from where the SmartObserve distribution should be downloaded. Default is release.
  • --revision: Defines the current source code revision to use for running a benchmark test. Default is current.
    • current: Uses the source tree’s current revision based on your SmartObserve distribution.
    • latest: Fetches the latest revision from the main branch of the source tree.
    • You can also use a timestamp or commit ID from the source tree. When using a timestamp, specify @ts, where “ts” is a valid ISO 8601 timestamp, for example, @2013-07-27T10:37:00Z.
  • --smartobserve-plugins: Defines which SmartObserve plugins to install. By default, no plugins are installed.
  • --plugin-params: Defines a comma-separated list of key:value pairs that are injected verbatim into all plugins as variables.
  • --runtime-jdk: The major version of JDK to use.
  • --client-options: Defines a comma-separated list of clients to use. All options are passed to the SmartObserve Python client. Default is timeout:60.

Cluster

The following option relates to the target cluster of the benchmark.

  • --target-hosts: Defines a comma-separated list of host-port pairs that should be targeted if using the pipeline benchmark-only. Default is localhost:9200.

Distributed workload generation

The following options help users who want to use multiple hosts to generate load to the benchmark cluster:

  • --worker-ips: Defines a comma-separated list of hosts that coordinate loads. Default is localhost.
  • --enable-worker-coordinator-profiling: Enables an analysis of the performance of SmartObserve Benchmark’s worker coordinator. Default is false.

Provisioning

The following options help customize how SmartObserve Benchmark provisions SmartObserve and workloads:

  • --cluster-config-repository: Defines the repository from which SmartObserve Benchmark loads cluster-configs and cluster-config-instances.
  • --cluster-config-path: Defines the path to the --cluster-config-instance and any SmartObserve plugin configurations to use.
  • --cluster-config-revision: Defines a specific Git revision in the cluster-config that SmartObserve Benchmark should use.
  • --cluster-config-instance: Defines the --cluster-config-instance to use. You can see possible configuration instances using the command smartobserve-benchmark list cluster-config-instances.
  • --cluster-config-instance-params: A comma-separated list of key-value pairs injected verbatim as variables for the cluster-config-instance.

Workload

The following options determine which workload is used to run the test:

  • --workload-repository: Defines the repository from which SmartObserve Benchmark loads workloads.
  • --workload-path: Defines the path to a downloaded or custom workload.
  • --workload-revision: Defines a specific revision from the workload source tree that SmartObserve Benchmark should use.
  • --workload: Defines the workload to use based on the workload’s name. You can find a list of preloaded workloads using smartobserve-benchmark list workloads.

Test procedures

The following options define what test procedures the test uses and which operations are contained inside the procedure:

  • --test-execution-id: Defines a unique ID for this test run. Defines the test procedures to use with each workload. You can find a list of test procedures that the workload supports by specifying the workload in the info command, for example, smartobserve-benchmark info --workload=<workload_name>. To look up information on a specific test procedure, use the command smartobserve-benchmark info --workload=<workload_name> --test-procedure=<test-procedure>.
  • --include-tasks: Defines a comma-separated list of test procedure tasks to run. By default, all tasks listed in a test procedure array are run.
  • --exclude-tasks: Defines a comma-separated list of test procedure tasks not to run.
  • --enable-assertions: Enables assertion checks for tasks. Default is false.

Pipelines

The --pipeline option selects a pipeline to run. You can find a list of pipelines supported by SmartObserve Benchmark by running smartobserve-benchmark list pipelines.

Telemetry

The following options enable telemetry devices on SmartObserve Benchmark:

  • --telemetry: Enables the provided telemetry devices when the devices are provided using a comma-separated list. You can find a list of possible telemetry devices by using smartobserve-benchmark list telemetry.
  • --telemetry-params: Defines a comma-separated list of key-value pairs that are injected verbatim into the telemetry devices as parameters.

Errors

The following options set how SmartObserve Benchmark handles errors when running tests:

  • --on-error: Controls how SmartObserve Benchmark responds to errors. Default is continue.
    • continue: Continues to run the test despite the error.
    • abort: Aborts the test when an error occurs.
  • --preserve-install: Keeps the Benchmark candidate and its index. Default is false.
  • --kill-running-processes: When set to true, stops any SmartObserve Benchmark processes currently running and allows SmartObserve Benchmark to continue to run. Default is false.