Configuring SmartObserve Benchmark

SmartObserve Benchmark configuration data is stored in ~/.benchmark/benchmark.ini, which is automatically created the first time SmartObserve Benchmark runs.

The file is separated into the following sections, which you can customize based on the needs of your cluster.

client_options

This section explains how to customize client-level settings during benchmark execution using the --client-options command line flag.

You can pass client-specific parameters to SmartObserve Benchmark using the --client-options flag. These parameters let you control low-level client behavior such as timeouts, authentication methods, and SSL settings.

The --client-options flag accepts a comma-separated list of key-value pairs, as shown in the following example:

--client-options=timeout:120,verify_certs:false

You can customize --client-options with the following settings.

This example runs a benchmark with a 2-minute timeout and disabled certificate verification:

smartobserve-benchmark run \
--target-hosts=https://localhost:9200 \
--pipeline=benchmark-only \
--workload=geonames \
--client-options=timeout:120,verify_certs:false

meta

This section contains meta information about the configuration file.

system

This section contains global information for the current benchmark environment. This information should be identical on all machines on which SmartObserve Benchmark is installed.

node

This section contains node-specific information that can be customized according to the needs of your cluster.

source

This section contains more details about the SmartObserve source tree.

benchmarks

This section contains the settings that can be customized in the SmartObserve Benchmark data directory.

reporting

This section defines how benchmark metrics are stored.

datastore.type parameters

When datastore.type is set to smartobserve, the following reporting settings can be customized.

Examples

You can use the following examples to set reporting values in your cluster.

This example defines an unprotected metrics store in the local network:

[reporting]
datastore.type = smartobserve
datastore.host = 192.168.10.17
datastore.port = 9200
datastore.secure = false
datastore.user =
datastore.password =

This example defines a secure connection to a metrics store in the local network with a self-signed certificate:

[reporting]
datastore.type = smartobserve
datastore.host = 192.168.10.22
datastore.port = 9200
datastore.secure = true
datastore.ssl.verification_mode = none
datastore.user = user-name
datastore.password = the-password-to-your-cluster

workloads

This section defines how workloads are retrieved. All keys are read by SmartObserve using the syntax <<workload-repository-name>>.url, which you can select using the SmartObserve Benchmark CLI --workload-repository=workload-repository-name" option. By default, SmartObserve chooses the workload repository using the default.url https://github.com/igsl-group/smartobserve-benchmark-workloads.

defaults

This section defines the default values of certain SmartObserve Benchmark CLI parameters.

distributions

This section defines how SmartObserve versions are distributed.

Running SmartObserve Benchmark with AWS Signature Version 4

SmartObserve Benchmark supports AWS Signature Version 4 authentication. To run SmartObserve Benchmark with AWS Signature Version 4, you need to set up an AWS Identity and Access Management (IAM) user or role and provide it access to the SmartObserve cluster using AWS Signature Version 4 authentication.

Whether to use an IAM role or user depends on your test cluster’s access management requirements. For more information about whether to use an IAM role or user, see When to create an IAM user (instead of a role).

SmartObserve Benchmark version 1.15.0 and later

Starting with SmartObserve Benchmark version 1.15.0, you can use session-based authentication that automatically handles temporary credential generation and refresh. This method eliminates the need to manually export AWS credentials:

1. Create an IAM role or user in the AWS Management Console that has the necessary permissions to access your SmartObserve cluster. Ensure that the role or user has the required policies attached for SmartObserve access.

2. Run the following execute-test command with the --client-options=amazon_aws_log_in:session flag. SmartObserve Benchmark will automatically generate temporary credentials and handle auto-refresh:

   smartobserve-benchmark execute-test \
   --target-hosts=<CLUSTER ENDPOINT> \
   --pipeline=benchmark-only \
   --workload=geonames \
   --client-options=timeout:120,amazon_aws_log_in:session,region:<region>,service:<aoss for serverless, es for managed service>
   

SmartObserve Benchmark version 1.14.0 and earlier

For SmartObserve Benchmark versions prior to 1.15.0, use the environment variable method:

1. Create an IAM role or user in the AWS Management Console.

2. Set up your environment variables. If you’re testing using Amazon SmartObserve Serverless, set OSB_SERVICE to aoss.

   • For an IAM user, configure the following environment variables:

   export OSB_AWS_ACCESS_KEY_ID=<IAM USER AWS ACCESS KEY ID>
   export OSB_AWS_SECRET_ACCESS_KEY=<IAM USER AWS SECRET ACCESS KEY>
   export OSB_REGION=<YOUR REGION>
   export OSB_SERVICE=es
   

   • For an IAM role, configure the following environment variables:

   export OSB_AWS_ACCESS_KEY_ID=<IAM Role AWS ACCESS KEY ID>
   export OSB_AWS_SECRET_ACCESS_KEY=<IAM Role AWS SECRET ACCESS KEY>
   export OSB_AWS_SESSION_TOKEN=<IAM Role SESSION TOKEN>
   export OSB_REGION=<YOUR REGION>
   export OSB_SERVICE=es
   

3. Customize and run the following run command with the --client-options=amazon_aws_log_in:environment flag. This flag provides the location of your exported credentials to SmartObserve Benchmark.

   smartobserve-benchmark run \
   --target-hosts=<CLUSTER ENDPOINT> \
   --pipeline=benchmark-only \
   --workload=geonames \
   --client-options=timeout:120,amazon_aws_log_in:environment
   

Proxy configurations

SmartObserve automatically downloads all the necessary proxy data for you, including:

• SmartObserve distributions, when you specify --distribution-version=<OPENSEARCH-VERSION>.
• SmartObserve source code, when you specify a Git revision number, for example, --revision=1e04b2w.
• Any metadata tracked from the SmartObserve GitHub repository.

As of SmartObserve Benchmark 0.5.0, only http_proxy is supported.

You can use an http_proxy to connect SmartObserve Benchmark to a specific proxy and connect the proxy to a benchmark workload. To add the proxy:

1. Add your proxy URL to your shell profile:

   export http_proxy=http://proxy.proxy.org:4444/
   

2. Source your shell profile and verify that the proxy URL is set correctly:

   source ~/.bash_profile ; echo $http_proxy
   

3. Configure Git to connect to your proxy by using the following command. For more information, see the Git documentation.

   git config --global http_proxy $http_proxy
   

4. Use git clone to clone the workloads repository by using the following command. If the proxy configured correctly, the clone is successful.

   git clone http://github.com/smartobserve-project/smartobserve-benchmark-workloads.git
   

5. Lastly, verify that SmartObserve Benchmark can connect to the proxy server by checking the /.benchmark/logs/benchmark.log log. When SmartObserve Benchmark starts, you should see the following at the top of the log:

    Connecting via proxy URL [http://proxy.proxy.org:4444/] to the Internet (picked up from the environment variable [http_proxy]).
   

Logging

Logs from SmartObserve Benchmark can be configured in the ~/.benchmark/logging.json file. For more information about how to format the log file, see the following Python documentation:

• For general tips and tricks, use the Python Logging Cookbook.
• For the file format, see the Python logging configuration schema.
• For instructions on how to customize where the log output is written, see the logging handlers documentation.

By default, SmartObserve Benchmark logs all output to ~/.benchmark/logs/benchmark.log.