Boosting Algorithm

What is the Boosting Algorithm?

Boosting algorithms are the special algorithms that are used to augment the existing result of the data model and help to fix the errors. They use the concept of the weak learner and strong learner conversation though the weighted average values and higher votes values for prediction. These algorithms uses decision stamp, margin maximizing classification for processing. There are three types of Algorithms available such as AdaBoost or Adaptive boosting Algorithm, Gradient, and XG Boosting algorithm. These are the machine learning algorithms that follow the process of training for predicting and fine-tuning the result.

Example:

Let’s understand this concept with the help of the following example. Let’s take an example of the email. How will you recognize your email whether it is spam or not? You can recognize it by the following conditions:

• If an email contains lots of source like that means it is spam.
• If an email contains only one file image then it is spam.
• If an email contains the message of “You Own a lottery of $xxxxx” that means it is spam.
• If an email contains some known source then it is not spam.
• If it contains the official domain like educba.com, etc that means it is not spam.

The above-mentioned rules are not that much powerful to recognize the spam or not, hence these rules are called as weak learners.

To convert weak learner to strong learner combine the prediction of the weak learner using the following methods.

1. Using average or weighted average.
2. Consider prediction has a higher vote.

Consider the above mentioned 5 rules, there are 3 votes for spam and 2 votes for not spam. Since there is high vote spam, we consider it as spam.

How Boosting Algorithms Works?

Boosting Algorithms combines each weak learner to create one strong prediction rule. To identify the weak rule, there is a base Learning algorithm (Machine Learning). Whenever Base algorithm is applied, it creates new prediction rules using the iteration process. After some iteration, it combines all weak rules to create one single prediction rule.

To choose the right distribution follows the below-mentioned steps:

Step 1: Base Learning algorithm combines each distribution and applies equal weight to each distribution.

Step 2: If any prediction occurs during the first base learning algorithm then we pay high attention to that prediction error.

Step 3: Repeat step 2 until the limit of Base Learning algorithm has been reached or high accuracy.

Step 4: Finally, it combines all the weak learner to create one strong prediction tule.

Types of Boosting Algorithm

Boosting algorithms uses different engines such as decision stamp, margin maximizing classification algorithm, etc. There are three types of Boosting Algorithms which are as follows:

1. AdaBoost (Adaptive Boosting) algorithm
2. Gradient Boosting algorithm
3. XG Boost algorithm

AdaBoost (Adaptive Boosting) Algorithm

To understand AdaBoost, please refer the below image:

Box 1: In Box 1 for each data set we assigned equal weights and to classify plus (+) and minus (-) sign we apply decision stump D1 which creates a vertical line at the left-hand side of box 1. This line incorrectly predicted three plus sign(+) as minus(-) hence we apply higher weights to these plus sign and apply another decision stump.

Box 2: In Box 2 the size of three incorrectly predicted plus signs (+) becomes bigger as compared to another. The second decision stump D2 at the right-hand side of the block predict this incorrectly predicted plus sign (+) sign as correct. But since misclassification error occurred because of the unequal weight with a minus sign (-) we assign a higher weight to a minus sign (-) and apply another decision stump.

Box 3: In Box three because of misclassification error three minus sign (-) has a high weight. here decision stump D3 is applied to predict this misclassification and correct it. This time to classify plus (+) and minus (-) sign horizontal line is created.

Box 4: In Box 4 decision stump D1, D2 and D3 are combined to create a new strong prediction.

Adaptive Boosting works are similar as mentioned above. It combines the group of weak learner base on weight age to create a strong learner. In the first iteration, it gives equal weight to each data set and the starts predicting that data set. If incorrect prediction occurs it gives high weight to that observation. Adaptive Boosting repeat this procedure in the next iteration phase and continue until the accuracy has been achieved. Then combines this to create a strong prediction.

Gradient Boosting Algorithm

Gradient boosting algorithm is a machine learning technique to define loss function and reduce it. It is used to solve problems of classification using prediction models. It involves the following steps:

1. Loss Function

The use of loss function depends on the type of problem. The advantage of gradient boosting is that there is no need for a new boosting algorithm for each loss function.

2. Weak Learner

In gradient boosting, decision trees are used as a weak learner. A regression tree is used to give true values which can be combined together to create correct predictions. Like in the AdaBoost algorithm, small trees with single split are used i.e decision stump. Larger trees are used for large levels i,e 4-8 levels.

3. Additive Model

In this model, trees are added one at a time. existing trees remains the same. During the addition of trees, gradient descent is used to minimize the loss function.

XG Boost

XG Boost is short for Extreme Gradient Boosting. XG Boost is upgraded implementation of Gradient Boosting Algorithm which is developed for high computational speed, scalability, and better performance.

XG Boost has various features which are as follows:

1. Parallel Processing: XG Boost provides Parallel Processing for tree construction which uses CPU cores while training.
2. Cross-Validation: XG Boost enables users to run cross-validation of the boosting process at each iteration, making it easy to get the exact optimum number of boosting iterations in one run.
3. Cache Optimization: It provides Cache Optimization of the algorithms for higher execution speed.
4. Distributed Computing: For training large models, XG Boost allows Distributed Computing.

Recommended Articles

In this article, we have seen what is Boosting Algorithm, Various Types of Boosting Algorithm in machine learning and their working. You can also go through our other suggested articles to learn more –

1. What is Machine Learning? | A Definition
2. Programming Languages for learning Algorithms
3. What is Blockchain Technology?
4. What is an Algorithm?