Introduction to Logstash multiple outputs
Logstash multiple outputs refer to the process where the ingested data by the processing pipeline is transformed and further transferred to more than one output by the open-source pipeline of Logstash on the server-side. Along with producing and distributing the multiple outputs, it is also possible for Logstash to get the input data to be ingested from more than one source. In this article, we will have a detailed look at the Logstash multiple outputs along with studying its subtopics, including Logstash multiple outputs overviews, using Logstash multiple outputs, Installing multiple outputs, multiple outputs examples, and Conclusion about the same.
Logstash multiple outputs overviews
Logstash can process the input, whether ingested from single or multiple sources and transform it further, which can be distributed distinctly over more than one output. Certain steps need to be taken while creating Logstash multiple outputs, which are as described below –
- Take the input stream and create a copy of each of the documents in it.
- Collect all the copies and scrutinize them for filtered data that will contain only the valid fields as per requirement.
- To indicate to which type the copy of data belongs, we can create the metadata for each copy and add it to the resultant.
- To direct each of the documents to the correct output, a proper evaluation of the contained metadata should be done, and the direction of a document should be decided accordingly to the respective output.
Using Logstash multiple outputs
Logstash is the completely open-source pipeline on the server-side. The functionality of Logstash includes ingesting the data from single or multiple sources and then transforming it and sending the outputs further for storage or processing to other services. The supported services of Logstash include SQS, S3, Kafka, BigQuery of google, Cloudwatch of amazon web services, and many more. Furthermore, we can forward the filtered data of Logstash either to a single output destination or multiple outputs by filtering the inputs in a specific manner, resulting in the outputs being distributed to that particular stream for each of the inputs received.
Internally, what we do is the output is written on multiple nodes of elastic search. This lightens the burden of resource demands on each node of elastic search. One more advantage is that it provides redundant points inside the cluster from where entry can be made when a specific node in the cluster is unavailable for use.
The configurations that need to be made for multiple outputs of Logstash are done by simply changing the output section of the configuration contents present inside the config file of the pipeline.conf so that the data is written on multiple nodes of elastic search. The config looks as shown in the below template where we mention details of various hosts –
Output{
Elasticsearch{
Hosts=> [“sample Ip address: port number”, “sample Ip address 1: port number 1”, “sample Ip address 2: port number 2”, “sample Ip address 3: port number 3”]
}
}
In short, what we do here to implement multiple outputs in Logstash is make the specification of host addresses of three non-master nodes of our cluster in elasticsearch that are present in the line of host specification above. Load balancing into account and implemented by Logstash when multiple addresses are specified in the host’s parameter. We can even omit the default port of 9200, which is used for elastic search in those mentioned above, and a specified list of address ports inside the configuration.
Installing Logstash multiple outputs
We can only use Logstash multiple outputs if we have Logstash installed. Let us look at how we can install Logstash using the binary file. Use Downloaded binary file – We can go for the installation of Logstash from the downloaded binaries. This binary file can be downloaded from the mentioned link.
Step 1 – The first step will be to visit this link, and then according to the host environment of your system, which can be zip, deb, rpm, or targ.gz, you can download the binary file in the required format. You can choose your host system, and the corresponding download option button will appear in blue at the bottom, as shown below where we have chosen the windows platform.
Step 2 – Once downloaded, unzip or extract the file using any of the extractors as shown below, which will lead to unpacking of binaries –
Step 3 – Note that now the installation process is to begin, and Logstash should not be placed in a path that will have the special character of the colon “:” in it.
We can also use the package manager to install Logstash on the Linux platform.
When you go for extracting, it will install on windows, and you are free to use all the packages as they are licensed under elastic. It is an open-source functionality with free commercial features while some are paid. For the provided 30-day free trial, you can even use the paid features free of cost for 30 days, after which you will have to pay for them. We can even go for downloading the package of oss, which comes along with only those features that are available under the license of Apache 2.0.
Logstash multiple outputs examples
Let us consider a sample example of the configuration file contents of the pipeline.conf file when used for implementing multiple outputs in it –
input {
linkedIn {
consumer_key => "sampleEducbaConsumerKey"
consumer_secret => "sampleEducbaSecret"
keywords => ["business"]
oauth_token => "sampleEducbaAccessToken"
oauth_token_secret => "sampleEducbaAccessTokenSecret"
}
beats {
port => "5432"
}
}
output {
elasticsearch {
hosts => [“sample Ip address: port number”, “sample Ip address 1: port number 1”, “sample Ip address 2: port number 2”, “sample Ip address 3: port number 3”]
}
file {
path => "/educba/sample/filepath"
}
}
Now, to verify whether the configurations are set as expected, we can execute the following command –
bin/logstash -f pipeline.conf --config.test_and_exit
Execution of the above command gives the following output –
Conclusion
Logstash multiple outputs are how the Logstash pipeline generates more than one output by segregating each input stream to the correct output stream, which also teaches load balancing within the cluster.
Recommended Articles
This is a guide to Logstash Multiple Outputs. Here we discuss the Logstash multiple outputs along with studying its subtopics, including Logstash multiple outputs overviews. You may also look at the following articles to learn more –
