Introduction to Logstash Test Config
Logstash test config is the process in which we can debug and test the configurations specified in the configuration file in the easiest way possible. In this article, we are going to look at what is Logstash test config and how we can do that. To study it systematically, the subtopics that we will be focusing on include How to check Logstash test config, Logstash test config Setup, Logstash test config Input, Logstash test config Output, and Conclusion about the same. Let’s begin our journey of learning.
How to Check Logstash test config?
As we all know that Logstash is used for the processing of data using pipelines which is capable of getting the inputs from multiple sources, working and processing the same, and send the transformed data to the desired target location. Logstash has most of its use being done in Elastic search for sending the data to it which is a search engine as well as a tool used for the analytics which can further have a pictorial representation on kibana and can be seen over there. ELK stack consists of Elastic search, Logstash, and Kibana. In order to manage the incoming events, Logstash makes use of configuration files for configuring them.
In order to check the config in Logstash, what the problem with testing the Logstash config is for the specified inputs of logs that are of JSON format, we need to retrieve the output which will contain a view of the output events after the application of specified configurations on input logs.
Logstash test config Setup
We will first need to create one directory inside which we will be setting all the files and storing them. For demonstration purposes, I have created one directory C:\educba where we will need to create three files which will be initially blank which will be –
• Logstash.conf which will contain all the configurations of Logstash that we want to test.
• Output_file_for_Logstash.logs – In this file, we will be saving all the output of Logstash configurations.
• Input_file_for_Logstash.logs – This file will contain all the configurations of logsatsh which will act as the input logs that will be further consumed.
Other than this, we don’t need the ELK stack that is Elastic search or Kibana to be installed in our system as the output generated will be stored inside the local file. Having Logstash will suffice our requirements. We can do this by using binary files, though there are other ways too to install Logstash.
Use Downloaded binary file – We can go for the installation of Logstash from the downloaded binaries. This binary file can be downloaded from the mentioned link.
Step 1 – The first step will be to visit this link and then according to the host environment of your system which can be zip, deb, rpm, or targ.gz, ypu can download the binary file in the required format. You can choose your host system and the corresponding download option button will appear in blue color at the bottom as shown below where we have chosen the windows platform.
Step 2 – Once downloaded, unzip or extract the file using any of the extractors as shown below which will lead to unpacking of binaries –
Step 3 – Note that now the installation process is to begin and Logstash should not be place in a path that will have the special character of colon “:” in it.
We can also make the use of the package manager for the installation of Logstash when using Linux platform.
Remember, having java is also mandatory as this is what will be used by Logstash to run.
Logstash test config Input
Now, the file names input_file_for_Logstash.logs that you created early should be modified and all the logs of input in JSON should be saved in it so that each log is on one line as it will be further interpreted that a single line corresponds to a single individual event. We will be adding the following three logs to the file for demonstration –
{index:"79729”, id:61, message:"Educba"}
{index:"79729”, id:62, message:"Logstash Tutorial"}
{index:"79729”, id:63, message:"Testing configurations"}
Further, the configuration file of Logstash is Logstash. conf should be added the following lines of config –
input {
file{
path => "C:/educba/*.logs"
codec => json
sincedb_path => "NUL"
start_position => "beginning"
}
}
NUL setting of sincedb path has been kept so that all the pointers of Logstash are ignored and we get the whole file being read. Codec specifies that we will be making use of JSON data and the path is for our directory where all our log files reside.
Further, you can also go for the plugins specification in the filters section where all the customization can be specified. For example, we add the following filter section in our configurations –
filter {
mutate {
add_field => {"domain" => "Technology"}
}
}
Logstash test config Output
We can set the output destination to any of the files we need. For our demo, we will be setting the destination output file as the output_file_for_Logstash.logs which we created early.
Hence, the output section of our configurations will be as shown below –
Output {
File {
Path => “C:/educba/output_file_for_Logstash.logs”
}
}
After you are done with the specification of all the sections of configurations like input, filter, and output. It’s time to test Logstash configurations. We will first navigate to the folder where we installed Logstash and then run Logstash by using the below command –
logstash.bat -f C:\educba\logstash.conf
After execution of the above command, we can see that the output contains our inputs passed, and hence, the test is successful –
We can observe the newly added field by our filter named domain having value as Technology. Similarly, you can add whatever is your requirement in the input, filter, and out section and test the Logstash configurations.
Conclusion
Logstash Test configurations involve retrieving the output which will contain a view of the output events after the application of specified configurations on input logs in Logstash. We need to install Logstash in our system and then create three main files which will be for input logs, output logs that are destination target file, and the main Logstash’s configuration file.
Recommended Articles
This is a guide to Logstash Test Config. Here we discuss the Introduction, How to check Logstash test config, Examples with code implementation. You may also have a look at the following articles to learn more –
