EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login
Home Data Science Data Science Tutorials Logstash Tutorial Logstash Multiple Pipelines
Secondary Sidebar
Logstash Tutorial
  • Basic
    • What is Logstash?
    • Kibana Logstash
    • Logstash Alternatives
    • Logstash if field exists
    • Logstash Test Config
    • Logstash Date Filter
    • Logstash Version
    • Logstash Filter
    • Logstash Pipeline
    • Logstash Port
    • Logstash Filter JSON
    • Logstash JDBC
    • Logstash Timestamp
    • Logstash Multiline
    • Logstash split
    • Logstash Codec
    • Logstash HTTP input
    • Logstash Filebeat
    • Logstash if
    • Logstash File Input
    • Logstash mutate
    • Logstash Syslog
    • Logstash AWS
    • Logstash.yml
    • Logstash Install Plugin
    • Logstash Geoip
    • Logstash add field
    • Logstash Multiple Pipelines
    • Logstash TCP input
    • Logstash Multiple Outputs
    • Logstash Stdout
    • Logstash Pipeline Configuration
    • Logstash Monitoring
    • LogstashEncoder
    • Logstash CSV
    • Logstash Cluster
    • Logstash Debug

Logstash Multiple Pipelines

Logstash Multiple Pipelines

Introduction to Logstash Multiple Pipelines

Logstash multiple pipelines is the feature provided by Logstash with the help of the configuration file named pipelines.yml, enabling us to execute one or more pipelines inside the same process. In this article, we will study the Logstash multiple pipelines by learning its subtopics: Logstash multiple pipelines overviews, creating Logstash multiple pipelines, configuring Logstash multiple pipelines, and examples of Logstash multiple pipelines, and conclusion about the same.

Logstash multiple pipelines overviews

Multiple pipelines in Logstash are the way to execute pipelines that are more than one in number inside the same process as per our requirement, which can be achieved by using a file named pipelines.yml that should be positioned and placed in the folder named path. Settings as it is a configuration file.

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

Multiple pipelines are used when the events in the configuration don’t have the same common set of filters and inputs being shared among themselves. At the same time, conditionals and tags are used to separate all the outputs generated from them.

Suppose we have a single instance that contains multiple pipelines. In that case, events can have varied parameters for durability and performance, such as different setting values set for the persistent queues and different for pipeline workers. This means that the separation will result in the exertion of no pressure between the blocked outputs of multiple pipelines.

It is very important to handle the resource competition among various pipelines if, by default, the values are set only for a single pipeline. One of the examples that will help you understand this scenario is that we apply the measure of assigning a single worker to each of the CPU cores to reduce the pipeline workers used by the pipelines.

The namespace of the location for dead-letter queues and persistent queues are completely segregated and isolated for each of the pipelines and are specified by using the pipeline.id value for identification.

Create Logstash multiple pipelines

The first step is ensuring that the configuration file of pipelines.yml is set correctly.

The structure of the pipelines.yml file is somewhat as shown below. That contains the details of each pipeline we will use for that process.

For a single pipeline, the sample definition can be as shown below –

1

You can add as many pipelines as you want per your requirement for multiple pipelines. The sample definition can be as shown below –

2

The last step is to restart the logstash by using the command –

Sudo systemctl restart logstash

The output of the command is –

3

As we can observe above, the format followed for defining this configuration file is YAML, which has a list of dictionaries. Inside each dictionary is the pipeline description where all the key-value pairs are present. These pairs help describe the configuration’s path and their respective identifiers. The above content as a sample specifies two pipelines containing the path of configuration and their respective identifiers. If we fail to specify the setting’s value for the respective pipeline, then the default value as set in the file of Logstash.yml will be considered as the tn settings file. When no arguments are specified while starting the Logstash, the process will involve reading the file named pipelines.yml for configurations and will initialize the specified pipelines in the file. If we use options such as -f or -e, the pipelines.yml settings file is completely ignored, and a warning is logged for the same.

Configure Logstash multiple pipelines

The configuration of multiple pipelines and the description of their contents must be specified in the file named pipelines.yml, as shown above. The steps required to configure Logstash multiple pipelines using the YAML file are as specified below –

  • Our configuration file for multiple pipelines is inside the directory named etc/ search for the file named pipelines.yml. In case you don’t find it there, check where you have installed the Logstash; you will find it there. The default location where Logstash is installed and where you will find the configuration file is /etc/Logstash/pipelines.yml if you are using the Linux platform. The file looks as shown below –

Logstash Multiple Pipelines - 4

  • The next step is to open ay editor; in my case, I am using nano so that we can go for the modification of the file. We will execute the following command, which will allow us to modify the pipelines.yml file where we will place the content for now, as shown in the above sample where we created two pipelines out of which one was persisted while the other one has two workers –
    Sudo nano edit pipelines.yml

The above command’s execution will open up the pipelines.yml file where we have made the changes per our requirement. The final output file looks like this –

Logstash Multiple Pipelines - 5

  • We can click on ctrl+ O, which will save the file’s contents, and ctrl+X will make us exit the editor of nano that we opened.

Examples Logstash multiple pipelines

We can go for creating multiple pipelines with the help of a config map. Let us consider an example. We will follow certain steps to do so, which are –

  • configMap containing the input, output, and filter details should be created.

We have created the following sample configMap –

cat end.conf
input {
file {
path => "/tmp/end"
}
}
output {
stdout { }
}
cat start.conf
input {
file {
path => "/tmp/start"
}
}
output {
stdout { }
}
cat pipelines.yml
- pipeline.id: start
path.config: "https://cdn.educba.com/opt/bitnami/Logstash/config/start.conf"
- pipeline.id: end
path.config: "https://cdn.educba.com/opt/bitnami/Logstash/config/end.conf"
kubectl create cm multipleconfig --from-file=pipelines.yml --from-file=start.conf --from-file=end.conf

  • Along with the parameter of enableMultiplePipelines, try to execute the below command, with will help in deploying the helm chart.

helm install Logstash --set existingConfiguration=multipleconfig --set enableMultiplePipelines=true

We will get the following output after executing the above command –

Logstash Multiple Pipelines - 6

  • Go to the tracked file and create the dummy events inside it. Also, give a check in the output of the result generated by Logstash by using the commands –

kubectl exec -ti Logstash-0 -- bash -c 'echo good morning >> /tmp/start'
kubectl exec -ti Logstash-0 -- bash -c 'echo good night >> /tmp/end'

  • Check whether there is a proper reflection of the events in the output of the logs –

Logstash Multiple Pipelines - 7

Conclusion

Logstash multiple pipelines are very helpful in scenarios where the configurations contain the events that do not have common filters or inputs. Instead, the conditionals and tags are present, which help complete output segregation for each event flow.

Recommended Articles

This is a guide to Logstash Multiple Pipelines. Here we discuss the Introduction overviews; Create Logstash multiple pipelines Examples with code implementation. You may also have a look at the following articles to learn more –

  1. Logstash Version
  2. Logstash Alternatives
  3. Fluentd vs. Logstash
  4. Kibana Logstash
Popular Course in this category
Kibana Training (1 Course, 3 Project)
  1 Online Courses |  3 Hands-on Project |  14+ Hours |  Verifiable Certificate of Completion
4.5
Price

View Course
Primary Sidebar
Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

ISO 10004:2018 & ISO 9001:2015 Certified

© 2023 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more