Overview of Logstash
Logstash is an open-source tool that is used for the real-time pipeline for the data injection between two different independent resources. Earlier it was used only for log collection but now it is used beyond that. Any data type can be injected through the logstash and transformed through the array of input, filter, and output plugins along with other code plugins which helps to transform the input data to required format types.
It is an open-source event processing engine that can manipulate data to destination with numerous plugins. It is an event-based tool developed by the Elasticsearch Company. It is a part of the ELK stack. Plugin receive, manipulate or ship data to the destination resource.
Logstash Configuration File Format
Pipeline = input + (filter) + Output
Logstash is not limited to processing only logs. It can handle XML, JSON, CSV, etc. alike easily. It has a very strong synergy with Elasticsearch and Kibana+ beats.
- Pipeline: Pipeline is the collection of different stages as input, output, and filter. The input data enters into the pipeline and processed as an event and at last send to the new stage which may be a filter or last output stage.
- Event: When any input data enters the pipeline, we call it an event.
- Input: The input means the data entering into the pipeline. This is the first stage of the pipeline.
- Filter: Filter is the next stage of the pipeline. In this stage, we can add more fields to the data or can change the data (manipulation stage) before sending it to the output stage.
- Output: This is the last stage of the pipeline. In this, the coming data will be formatted according to the requirements of the destination source.
Architecture of Logstach
Logstash is based on the decouple-architecture i.e. it is centralized event processing. Decouple architecture means two independently systems interact with each other without being directly connected to each other.
Logstash Design and Architecture:
In this ecosystem view, our three components are:
- Shipper: It sends all data from the source to logstash.
- Broker and Indexer: This component will receive and formulate data for indexer events.
- Search and storage: Allows to search and store events.
To install logstash, we need java installed in a given system. Even it is mentioned in JRuby, its developer given all dependencies in a tarbale so now logstash is easy to install.
$ wget https://artifacts.elastic.co/downloads/logstash/logstash-7.5.1.tar.gz
After download tarball, unzip it in your required directory.
$ cd logstash-7.5.1
Inside of that logatsh directory create a dir called conf and save all future configuration files in that directory. Configuration file sample is given below:
Running the Logstash Agent:
After creating a configuration file, to run logstash we use the following commands:
$ cd logstash-7.5.1
$ bin/logstash -f path/of/the/conf/file/dir
If you change something in the configuration file then every time you have to restart the configuration file so that it can pick up the new and updated configuration.
How to Install Plugin?
To install different kinds of plugin input/output which are available use the following command:
$ bin/logstash_plugin install /name-of-the-plugin-name/
$ bin/logstash_plugin install logstash-input-beats
One of the great things about Logstash is that it is made up of easy to fit together components: Logstash itself, Redis as a broker, Elasticsearch and the various other pluggable elements of your Logstash configuration. One of the advantages of this method is the ease with which you can scale it and those components.
- Redis: Which we’re using as a broker for incoming events.
- Elasticsearch: Handling search and storage.
- Logstash: Which is consuming and indexing the events.
This is a fairly basic introduction to scaling these components with a focus on trying to achieve some simple objectives:
- To make Logstash as redundant as possible with no single points of failure.
- To avoid messages being lost in transit from inputs and outputs.
- To make it perform in the most efficient way.
Our final scaled architecture will look like this:
Logstash Scaled Architecture
In our implementation, we’re using Redis as a broker between our Logstash agents and the Logstash central server. One of the reasons we chose Redis is that it is very simple. Thankfully, making Redis redundant is also simple. This can send events to and receive events from multiple Redis instances in a failover configuration. It’s important to note that this is a failover rather than true high availability. Events are not round robined or load-balanced between Redis instances. It will try to connect to a Redis instance and send events.
If the sending succeeds then it will continue to send events to that Redis instance. If the send fails, Logstash will pick the next Redis instance and try to send it to the next instance instead of the failed one. This does, however, provide you with some basic redundancy for your broker through the deployment of additional Redis instances but has a limited impact if your Redis instance is a performance bottleneck for your environment. If this is an issue for you then you can designate Redis instances for specific agents or groups of agents with additional Redis instances defined if you’d like redundancy.
Logstash Redis failover
Benefits of Logstash
- With the help of logstash, we can connect two different independent sources together.
- It can send data with very high speed from one source to another source.
- It can also help to parse data in different formats before ingesting to the destination.
- It also helps to filter log data.
- It also supports different kinds of databases, protocols.
- It works as a central point which makes it easy to process and collect data from different sources.
Logstash helps in transferring data from one source to another source. Logstash has fully optimized itself but using scale features, We can do more logstash control for the data pipeline.
This is a guide to What is Logstash? Here we discuss the overview and configuration file format of logstash along with architecture and its benefits. You may also look at the following articles to learn more-