Lab 2 : Performance Analyzer

When you want to monitor your Elasticsearch cluster or debug an issue, you have a number of choices. You can use the various _cat and stats APIs to pull information out of the cluster. You can monitor and profile the JVM itself. These options can be cumbersome, and they lack visual displays. While you could push _cat and stats data back into Elasticsearch and visualize with Kibana, sometimes you want a more lightweight method.

Open Distro for Elasticsearch ships with two components that combine to give you the lightweight tool you need to quickly retrieve and display core metrics. Performance Analyzer is an agent and REST API that allows you to query numerous performance metrics for your cluster, including aggregations of those metrics, independent of the Java Virtual Machine (JVM). Performance Analyzer runs on and collects metrics from the same nodes that you use to run Elasticsearch.

PerfTop

PerfTop is a lightweight, command-line tool patterned after Linux's top command. You use simple JSON to define dashboards; PerfTop displays these dashboards in your favorite terminal application.

PerfTop is a Javascript application that polls Performance Analyzer's API to pull metrics and display them in Graphs —widgets that display one or more metrics across one or more of their dimensions. You specify which metrics and graphs PerfTop displays by providing a dashboard configuration. For this lab, we will use in-built dashboards.

PerfTop CLI can be downloaded from https://opendistro.github.io/for-elasticsearch/downloads.html

Login to one of the client nodes. As part of the CloudFormation, we have already installed PerfTop CLI on the client nodes under /home/ubuntu

Run the below commands

cd /home/ubuntu
./perf-top-linux --dashboard ClusterOverview

Leave PerfTop running to monitor the cluster.

Performance Analyzer Data to Elasticsearch

We have a code sample that reads Performance Analyzer's metrics and writes them to Elasticsearch. You might wonder why Performance Analyzer doesn't do that already. Performance Analyzer is designed as a lightweight co-process for Elasticsearch that decouples Elasticsearch Monitoring from Elasticsearch failures. If your Elasticsearch cluster is in trouble, it might not be able to respond to requests, and Kibana might be down. In production, it is recommended that you send the data to a different Open Distro for Elasticsearch cluster to avoid this issue.

This code sample provides a main.py script that collects all of the metrics surfaced by Performance Analyzer, across their dimensions and aggregations. It pushes those metrics to Elasticsearch for visualization with Kibana.

Open a new terminal window and login to one of the client nodes.

Apply the below template to your cluster using the below command. This is ensure Elasticsearch will interpret the timestamp properly instead of a long integer.

curl -k -X PUT  https://127.0.0.1:9200/_template/pa_template -u admin:admin -H "Content-Type: application/json" -d '
{
    "index_patterns": ["pa-*"],
    "settings": {
        "number_of_shards": 1
    },
    "mappings": {
        "properties": {
            "@timestamp": {
                "type": "date"
            },
            "node_ip": {
                "type": "ip"
            }
        }
    }
}'

Run these commands to start streaming Performance Analyzer data to elasticsearch.

cd /home/ubuntu
git clone https://github.com/opendistro-for-elasticsearch/community.git
cd community/pa-to-es/
pip3 install pytz
python3 main.py --seven

Leave this session running.

Let's stress test

Now that we are monitoring using PerfTop and collecting metrics into Elasticsearch, let's put some load by ingesting data and running queries. We will use Rally to run some standard benchmarks.

Open another terminal and login to one of the client nodes and run the below commands. Please ensure you are not using sudo for these commands

cd /home/ubuntu
pip3 install esrally
. ./.profile
esrally configure
esrally --pipeline benchmark-only --track=metricbeat --challenge append-no-conflicts --target-host=https://localhost:9200 --client-options="use_ssl:true,basic_auth_user:'admin',basic_auth_password:'admin',verify_certs:false"

Explore Performance metrics

Login to Kibana and explore the performance metrics by creating Visualizations.

In Kibana, under Management, Click Index Patterns , and Create Index Pattern. Under index pattern, type pa-* and click Next step

Select @timestamp as Time Filter field and click "Create Index Pattern"

Under Visualizations, Click Create new Visualization and Select Line as the type. And in the next screen select pa-* as the source.

Under Metrics, Choose Max as Aggregation and CPU_Utilization as field.

Under Buckets, Click X-Axis, Choose Date Histogram as aggregation and @timestamp as field

Scroll to the bottom and click " Add sub-buckets" and select " Split Series". Select " Terms" as Aggregation and "node_ip" as the Field.

Click the blue triangle button at the top to visualize the graph.

Explore other visualization types and metrics. You can then create a Dashboard with multiple visualizations. An example screenshot below.