EDUCBA Logo

EDUCBA

MENUMENU
  • Explore
    • EDUCBA Pro
    • PRO Bundles
    • Featured Skills
    • New & Trending
    • Fresh Entries
    • Finance
    • Data Science
    • Programming and Dev
    • Excel
    • Marketing
    • HR
    • PDP
    • VFX and Design
    • Project Management
    • Exam Prep
    • All Courses
  • Blog
  • Enterprise
  • Free Courses
  • Log in
  • Sign Up
Home Data Science Data Science Tutorials Logstash Tutorial logstash.yml
 

logstash.yml

Updated March 15, 2023

logstash.yml

 

 

Introduction to logstash.yml

logstash.yml is a settings file used to specify configurations for controlling the execution of logstash. In this article, we will throw certain light on the topic of logstash.yml by looking at the subtopics, including What is logstash.yml, logstash.yml, Configure logstash.yml, and Conclusion about the same.

Watch our Demo Courses and Videos

Valuation, Hadoop, Excel, Mobile Apps, Web Development & many more.

What is logstash.yml?

Logstash.yml is a configuration settings file that helps maintain control over the execution of logstash. First, we can try to understand the usage and purpose of the logstash.yml configuration settings file by considering a small example. Then, when we have to mention the settings of the pipeline, options related to logging, details of the location of configuration files, and other values of settings, we can use the logstash.yml file.

The settings we specify in this file often also support the usage of command-line flags. Note that if we specify any settings inside the logstash.yml configuration file and change the same setting on the command line by using the flag system, the file value gets overridden by the corresponding command-line flag value.

The configuration file of logstash.yml is written in the format language of YAML, and the location of this file changes as per the platform the user is using. The value of settings mentioned inside the file can be specified in either flat keys or hierarchical format. Let us consider a sample example of how we can specify settings in flat keys format –

Pipeline.batch.delay :65
Pipeline.batch.size: 100

While the same values in hierarchical format can be specified as –

Pipeline:
Batch:
Delay: 65
Size: 100

Interpolation of the environment variables in bash style is also supported by logstash.yml. Along with that, the support for the Keystore secrets inside the values of settings is also supported by logstash, where the specification looks somewhat as shown below –

Pipeline:
Batch:
Delay: $ {BATCH_DELAY:65}
Size: ${BATCH_SIZE}
Path:
Queue: “/c/users/educba/${QUEUE_DIR:queue}”
Node:
Name: “node_ ${LS_NAME_OF_NODE}”

The notation used above of $NAME_OF_VARIABLE: value set to be by default is supported by logstash.

We can even go for the specification of the model inside the configuration settings file of logstash.yml, where the format that is followed should be as shown below –

Modules:

-name: EDUCBA_MODEL1
Var.PLUGIN_TYPE1.SAMPLE_PLUGIN1.SAMPLE_KEY1: SAMPLE_VALUE
Var.PLUGIN_TYPE2.SAMPLE_PLUGIN2.SAMPLE_KEY2: SAMPLE_VALUE
Var.PLUGIN_TYPE3.SAMPLE_PLUGIN3.SAMPLE_KEY3: SAMPLE_VALUE
Var.PLUGIN_TYPE3.SAMPLE_PLUGIN4.SAMPLE_KEY2: SAMPLE_VALUE
Var.PLUGIN_TYPE4.SAMPLE_PLUGIN5.SAMPLE_KEY4: SAMPLE_VALUE
-name: EDUCBA_MODEL2
Var.PLUGIN_TYPE1.SAMPLE_PLUGIN1.SAMPLE_KEY1: SAMPLE_VALUE
Var.PLUGIN_TYPE2.SAMPLE_PLUGIN1.SAMPLE_KEY2: SAMPLE_VALUE

There will be ignorance of the values specified inside the logstash.yml file for defining the modules if the usage of –modules is a command line flag for modules.

logstash.yml.

There are various settings present inside the logstash.yml file that we can set. Some of them are as mentioned in the below table –

Settings name The value set by default Description
Path.data LOGSTASH_HOME/data This directory is used by the plugins and the logstash itself for all the persistent requirements.
Node.name The hostname of the machine It is used for assigning the descriptive name to the node.
Pipeline.java_execution Set to true Used to specify whether to use or not the java execution engine.
Pipeline.id Main It is the ID that is an identifier set to the pipeline.
Pipeline.batch.size 125 It specifies that the maximum amount of events will be collected by an individual worker thread before executing the output and filter. Larger the size of the batch is the efficiency but note that it also comes along with the overhead for the memory requirement.
Pipeline.workers It is set to the value cores count of CPU cores present for the host. This is the count of workers working in parallel and going through the filters and the output stage executions.
Pipeline.batch.delay The default value is 50 The number of milliseconds to wait while pipeline even batches creation for every event before the dispatch of the batch to the workers.
Pipeline.plugins_classloaders False Whether to load the plugins of java to independently running class loaders for the segregation of the dependency or not.
Pipeline.unsafe_shutdown False. Refuses to exit if any event is inflight. Whether to force the logstash to close and exit while the shutdown is performed even though some of the inflight events are present inside the memory of the system or not.
Path.config The default value is set as per the platform being used. Refer to this link for more details. For the main pipeline, the path to navigate for the configuration of logstash is set in this setting.
Config.debug False Whether to display the debug notifications for the completely compiled configurations or not.
Modules Nothing (None) value Modules are used and described in the structure format of YAML in a nested way when configured in the logstash.yml file.

Many other settings can still be configured and specified in the logstash.yml file. For a complete list, refer to this link.

Configure logstash.yml

There are two files for the configuration of logstash, which include the settings file and the pipeline configuration files used for the specification of execution and startup-related options that control logstash execution and help define the processing pipeline of logstash respectively.

Logstash.yml is one of the settings files defined in the installation of logstash and can be configured simply by specifying the values of various settings required in the file or by using command line flags.

To configure logstash, a config file needs to be created, which will contain the details about all the plugins that will be required and the details of settings regarding each of the specified plugins. We can simplify the config file by specifying the input and output inside. We can define the standard input-output of the customized ones from the elasticsearch and host value specification. Further, you can run it by executing the command of

bin/logstash -f logstash-simple.conf

where -f is for the configuration file that results in the following output –

logstash.yml output 1

Conclusion

Logstash.yml is used to specify flags of settings for the configuration of logstash, which defines the execution of logstash. We can specify and set many settings in this file discussed priorly.

Recommended Articles

This is a guide to logstash.yml. Here we discuss What logstash.yml is, logstash.yml, Configure logstash.yml, and Conclusion. You may also look at the following articles to learn more –

  1. Kibana Logstash
  2. What is Logstash?
  3. Logstash Version
  4. Kubernetes Daemonset

Primary Sidebar

Footer

Follow us!
  • EDUCBA FacebookEDUCBA TwitterEDUCBA LinkedINEDUCBA Instagram
  • EDUCBA YoutubeEDUCBA CourseraEDUCBA Udemy
APPS
EDUCBA Android AppEDUCBA iOS App
Blog
  • Blog
  • Free Tutorials
  • About us
  • Contact us
  • Log in
Courses
  • Enterprise Solutions
  • Free Courses
  • Explore Programs
  • All Courses
  • All in One Bundles
  • Sign up
Email
  • [email protected]

ISO 10004:2018 & ISO 9001:2015 Certified

© 2025 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

By continuing above step, you agree to our Terms of Use and Privacy Policy.
*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA Login

Forgot Password?

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more

🚀 Limited Time Offer! - ENROLL NOW