EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login
Home Data Science Data Science Tutorials Logstash Tutorial Logstash Multiple Outputs
Secondary Sidebar
Logstash Tutorial
  • Basic
    • What is Logstash?
    • Kibana Logstash
    • Logstash Alternatives
    • Logstash if field exists
    • Logstash Test Config
    • Logstash Date Filter
    • Logstash Version
    • Logstash Filter
    • Logstash Pipeline
    • Logstash Port
    • Logstash Filter JSON
    • Logstash JDBC
    • Logstash Timestamp
    • Logstash Multiline
    • Logstash split
    • Logstash Codec
    • Logstash HTTP input
    • Logstash Filebeat
    • Logstash if
    • Logstash File Input
    • Logstash mutate
    • Logstash Syslog
    • Logstash AWS
    • Logstash.yml
    • Logstash Install Plugin
    • Logstash Geoip
    • Logstash add field
    • Logstash Multiple Pipelines
    • Logstash TCP input
    • Logstash Multiple Outputs
    • Logstash Stdout
    • Logstash Pipeline Configuration
    • Logstash Monitoring
    • LogstashEncoder
    • Logstash CSV
    • Logstash Cluster
    • Logstash Debug

Logstash Multiple Outputs

Logstash Multiple Outputs

Introduction to Logstash multiple outputs

Logstash multiple outputs refer to the process where the ingested data by the processing pipeline is transformed and further transferred to more than one output by the open-source pipeline of Logstash on the server-side. Along with producing and distributing the multiple outputs, it is also possible for Logstash to get the input data to be ingested from more than one source. In this article, we will have a detailed look at the Logstash multiple outputs along with studying its subtopics, including Logstash multiple outputs overviews, using Logstash multiple outputs, Installing multiple outputs, multiple outputs examples, and Conclusion about the same.

Logstash multiple outputs overviews

Logstash can process the input, whether ingested from single or multiple sources and transform it further, which can be distributed distinctly over more than one output. Certain steps need to be taken while creating Logstash multiple outputs, which are as described below –

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

  • Take the input stream and create a copy of each of the documents in it.
  • Collect all the copies and scrutinize them for filtered data that will contain only the valid fields as per requirement.
  • To indicate to which type the copy of data belongs, we can create the metadata for each copy and add it to the resultant.
  • To direct each of the documents to the correct output, a proper evaluation of the contained metadata should be done, and the direction of a document should be decided accordingly to the respective output.

Using Logstash multiple outputs

Logstash is the completely open-source pipeline on the server-side. The functionality of Logstash includes ingesting the data from single or multiple sources and then transforming it and sending the outputs further for storage or processing to other services. The supported services of Logstash include SQS, S3, Kafka, BigQuery of google, Cloudwatch of amazon web services, and many more. Furthermore, we can forward the filtered data of Logstash either to a single output destination or multiple outputs by filtering the inputs in a specific manner, resulting in the outputs being distributed to that particular stream for each of the inputs received.

Internally, what we do is the output is written on multiple nodes of elastic search. This lightens the burden of resource demands on each node of elastic search. One more advantage is that it provides redundant points inside the cluster from where entry can be made when a specific node in the cluster is unavailable for use.

The configurations that need to be made for multiple outputs of Logstash are done by simply changing the output section of the configuration contents present inside the config file of the pipeline.conf so that the data is written on multiple nodes of elastic search. The config looks as shown in the below template where we mention details of various hosts –

Output{
Elasticsearch{
Hosts=> [“sample Ip address: port number”, “sample Ip address 1: port number 1”, “sample Ip address 2: port number 2”, “sample Ip address 3: port number 3”] }
}

In short, what we do here to implement multiple outputs in Logstash is make the specification of host addresses of three non-master nodes of our cluster in elasticsearch that are present in the line of host specification above. Load balancing into account and implemented by Logstash when multiple addresses are specified in the host’s parameter. We can even omit the default port of 9200, which is used for elastic search in those mentioned above, and a specified list of address ports inside the configuration.

Installing Logstash multiple outputs

We can only use Logstash multiple outputs if we have Logstash installed. Let us look at how we can install Logstash using the binary file. Use Downloaded binary file – We can go for the installation of Logstash from the downloaded binaries. This binary file can be downloaded from the mentioned link.

Step 1 – The first step will be to visit this link, and then according to the host environment of your system, which can be zip, deb, rpm, or targ.gz, you can download the binary file in the required format. You can choose your host system, and the corresponding download option button will appear in blue at the bottom, as shown below where we have chosen the windows platform.

Logstash Multiple Outputs output 1

Step 2 – Once downloaded, unzip or extract the file using any of the extractors as shown below, which will lead to unpacking of binaries –

Logstash Multiple Outputs output 2

Step 3 – Note that now the installation process is to begin, and Logstash should not be placed in a path that will have the special character of the colon “:” in it.

output 3

We can also use the package manager to install Logstash on the Linux platform.

When you go for extracting, it will install on windows, and you are free to use all the packages as they are licensed under elastic. It is an open-source functionality with free commercial features while some are paid. For the provided 30-day free trial, you can even use the paid features free of cost for 30 days, after which you will have to pay for them. We can even go for downloading the package of oss, which comes along with only those features that are available under the license of Apache 2.0.

Logstash multiple outputs examples

Let us consider a sample example of the configuration file contents of the pipeline.conf file when used for implementing multiple outputs in it –

input {
linkedIn {
consumer_key => "sampleEducbaConsumerKey"
consumer_secret => "sampleEducbaSecret"
keywords => ["business"] oauth_token => "sampleEducbaAccessToken"
oauth_token_secret => "sampleEducbaAccessTokenSecret"
}
beats {
port => "5432"
}
}
output {
elasticsearch {
hosts => [“sample Ip address: port number”, “sample Ip address 1: port number 1”, “sample Ip address 2: port number 2”, “sample Ip address 3: port number 3”] }
file {
path => "/educba/sample/filepath"
}
}

Now, to verify whether the configurations are set as expected, we can execute the following command –

bin/logstash -f pipeline.conf --config.test_and_exit

Execution of the above command gives the following output –

output 4

Conclusion

Logstash multiple outputs are how the Logstash pipeline generates more than one output by segregating each input stream to the correct output stream, which also teaches load balancing within the cluster.

Recommended Articles

This is a guide to Logstash Multiple Outputs. Here we discuss the Logstash multiple outputs along with studying its subtopics, including Logstash multiple outputs overviews. You may also look at the following articles to learn more –

  1. Logstash Alternatives
  2. Logstash Version
  3. What is Logstash?
  4. Webflow alternative
Popular Course in this category
Kibana Training (1 Course, 3 Project)
  1 Online Courses |  3 Hands-on Project |  14+ Hours |  Verifiable Certificate of Completion
4.5
Price

View Course
Primary Sidebar
Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

ISO 10004:2018 & ISO 9001:2015 Certified

© 2023 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more