The situation

Lets say we have 6 servers that push events to elasticsearch, we need to know if one of them stops publishing events or if a new one is added.

The easy way out

The first idea that came to my mind when I found myself in this situation was to set up an alarm with a static treshold. This way, if the unique count of hostnames is not 6 I’d get notified.

The thing is that when we manage dozens of servers in different environments, that number might change quite often.

The solution

What if we compare the actual metric with the one from an hour…


The current state of affairs

So you got this far into the series, nice. You have your Elasticsearch cluster, your beats sending data and you got Kibana all set up with pretty dashboards. Thats al fine and dandy but its a saturday night and a bad system update brought down one of your nodes and the other two are barely handling the load.

Some background first

ODFE comes with the alerting plugin preinstalled. This allows us to set up alerts when certain conditions are met.

There are three ways to set up alerts in ODFE: Visual graphs, Extraction queries and the anomaly detector.

The first one is pretty…


If you are following along with the Getting started with the ELK Stack series, by now you should have a three Elasticsearch nodes with Filebeat and Metricbeat feeding data into your cluster.

In this write-up I will give you a brief introduction on what Kibana is and what we can use it for. Then we will install ODFE’s Kibana version and create a basic monitoring dashboard.

What is Kibana?

Kibana is a user interface that we can leverage to visualize data, it runs in a web interface and can connect to an Elasticsearch cluster to query the data. …


Introduction

In the last article we created an Elasticsearch cluster and checked its status using the cluster API. In this installment I’ll give you an overview of some alternatives for generating data so we can query it and generate dashboards and alerts.

There are many ways of ingesting data into Elasticsearch. Elastic provides lightweight shippers for logs, metrics and much more, see the “Beats site” in the Other useful information section for information on other shippers.

The most common ways of ingesting data are through Beats and Logstash. Logstash is a solution offered by Elastic to collect, parse and transform your…


In this series we will set up a basic three node Elasticsearch cluster to hold our data and use Elasticsearch’s built in redundancy and scalability features to make our cluster perform adequately and allow some failures to be tolerated.

Secondly we will install Metricbeat and Filebeat to generate some data and Kibana for visualizing our Elasticsearch data and to create dashboards and alerts.

In the third installment of this series, Ill guide you through the steps to create index templates and lifecycle policies so you can keep your cluster running smoothly.

Finally, we will set up alerting so we can…


Before we start

During this installment, we will be setting up Elasticsearch in three nodes with as little configuration as we can. There are more settings you can tune to fit your specific needs but for now I’ll try to keep it simple.

This is not a step by step guide on how to install Elasticsearch, there are detailed guides scattered around the internet for that. Our goal is to set up everything so we can build from there.

If you are just testing

The easiest way to get started is to run the whole stack in docker. Amazon has a great docker guide here. …

Franco martin

Im a solutions architect, passionate about scalable and maintainable architectures.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store