EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login

Boosting Algorithm

By Swati TawdeSwati Tawde

Home » Data Science » Data Science Tutorials » Machine Learning Tutorial » Boosting Algorithm

Boosting-Algorithm

What is the Boosting Algorithm?

Boosting algorithms are the special algorithms that are used to augment the existing result of the data model and help to fix the errors. They use the concept of the weak learner and strong learner conversation through the weighted average values and higher votes values for prediction. These algorithms uses decision stamp, margin maximizing classification for processing. There are three types of Algorithms available such as AdaBoost or Adaptive boosting Algorithm, Gradient, and XG Boosting algorithm. These are the machine learning algorithms that follow the process of training for predicting and fine-tuning the result.

Example:

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

Let’s understand this concept with the help of the following example. Let’s take the example of the email. How will you recognize your email, whether it is spam or not? You can recognize it by the following conditions:

  • If an email contains lots of source like that means it is spam.
  • If an email contains only one file image, then it is spam.
  • If an email contains the message of “You Own a lottery of $xxxxx”, that means it is spam.
  • If an email contains some known source, then it is not spam.
  • If it contains the official domain like educba.com, etc., that means it is not spam.

The above-mentioned rules are not that powerful to recognize the spam or not; hence these rules are called as weak learners.

To convert weak learner to strong learner, combine the prediction of the weak learner using the following methods.

  1. Using average or weighted average.
  2. Consider prediction has a higher vote.

Consider the above-mentioned 5 rules; there are 3 votes for spam and 2 votes for not spam. Since there is high vote spam, we consider it as spam.

How Boosting Algorithms Works?

Boosting Algorithms combines each weak learner to create one strong prediction rule. To identify the weak rule, there is a base Learning algorithm (Machine Learning). Whenever the Base algorithm is applied, it creates new prediction rules using the iteration process. After some iteration, it combines all weak rules to create one single prediction rule.

Popular Course in this category
Sale
Machine Learning Training (20 Courses, 29+ Projects)19 Online Courses | 29 Hands-on Projects | 178+ Hours | Verifiable Certificate of Completion | Lifetime Access
4.7 (14,238 ratings)
Course Price

View Course

Related Courses
Deep Learning Training (17 Courses, 24+ Projects)Artificial Intelligence Training (5 Courses, 2 Project)

To choose the right distribution follows the below-mentioned steps:

Step 1: The base Learning algorithm combines each distribution and applies equal weight to each distribution.

Step 2: If any prediction occurs during the first base learning algorithm, then we pay high attention to that prediction error.

Step 3: Repeat step 2 until the limit of the Base Learning algorithm has been reached or high accuracy.

Step 4: Finally, it combines all the weak learner to create one strong prediction tule.

Types of Boosting Algorithm

Boosting algorithms uses different engines such as decision stamp, margin maximizing classification algorithm, etc. There are three types of Boosting Algorithms which are as follows:

  1. AdaBoost (Adaptive Boosting) algorithm
  2. Gradient Boosting algorithm
  3. XG Boost algorithm

AdaBoost (Adaptive Boosting) Algorithm

To understand AdaBoost, please refer to the below image:

AdaBoost (Adaptive Boosting) algorithm

Box 1: In Box 1, for each data set, we assigned equal weights and to classify plus (+) and minus (-) sign, we apply decision stump D1, which creates a vertical line at the left-hand side of box 1. This line incorrectly predicted three-plus sign(+) as minus(-); hence we apply higher weights to these plus sign and apply another decision stump.

Box 2: In Box 2, the size of three incorrectly predicted plus signs (+) becomes bigger as compared to another. The second decision stump D2 at the right-hand side of the block predict this incorrectly predicted plus sign (+) sign as correct. But since misclassification error occurred because of the unequal weight with a minus sign (-), we assign a higher weight to a minus sign (-) and apply another decision stump.

Box 3: In Box three, because of misclassification error, three minus sign (-) has a high weight. here decision stump D3 is applied to predict this misclassification and correct it. This time to classify plus (+) and minus (-) sign horizontal line is created.

Box 4: In Box 4, decision stump D1, D2 and D3 are combined to create a new strong prediction.

Adaptive Boosting works are similar to those mentioned above. It combines the group of weak learner base on weight age to create a strong learner. In the first iteration, it gives equal weight to each data set and the starts predicting that data set. If incorrect prediction occurs, it gives high weight to that observation. Adaptive Boosting repeat this procedure in the next iteration phase and continue until the accuracy has been achieved. Then combines this to create a strong prediction.

Gradient Boosting Algorithm

Gradient boosting algorithm is a machine learning technique to define loss function and reduce it. It is used to solve problems of classification using prediction models. It involves the following steps:

1. Loss Function

The use of the loss function depends on the type of problem. The advantage of gradient boosting is that there is no need for a new boosting algorithm for each loss function.

2. Weak Learner

In gradient boosting, decision trees are used as a weak learner. A regression tree is used to give true values, which can be combined together to create correct predictions. Like in the AdaBoost algorithm, small trees with a single split are used, i.e. decision stump. Larger trees are used for large levels I,e 4-8 levels.

3. Additive Model

In this model, trees are added one at a time. existing trees remains the same. During the addition of trees, gradient descent is used to minimize the loss function.

XG Boost

XG Boost is short for Extreme Gradient Boosting. XG Boost is an upgraded implementation of the Gradient Boosting Algorithm, which is developed for high computational speed, scalability, and better performance.

XG Boost has various features, which are as follows:

  1. Parallel Processing: XG Boost provides Parallel Processing for tree construction which uses CPU cores while training.
  2. Cross-Validation: XG Boost enables users to run cross-validation of the boosting process at each iteration, making it easy to get the exact optimum number of boosting iterations in one run.
  3. Cache Optimization: It provides Cache Optimization of the algorithms for higher execution speed.
  4. Distributed Computing: For training large models, XG Boost allows Distributed Computing.

Recommended Articles

In this article, we have seen what is Boosting Algorithm is, Various Types of Boosting Algorithm in machine learning and their working. You can also go through our other suggested articles to learn more –

  1. What is Machine Learning?
  2. Learning Algorithms
  3. What is Blockchain Technology?
  4. What is an Algorithm?

Machine Learning Training (17 Courses, 27+ Projects)

19 Online Courses

29 Hands-on Projects

178+ Hours

Verifiable Certificate of Completion

Lifetime Access

Learn More

0 Shares
Share
Tweet
Share
Primary Sidebar
Machine Learning Tutorial
  • Algorithms
    • Machine Learning Algorithms
    • Apriori Algorithm in Machine Learning
    • Types of Machine Learning Algorithms
    • Bayes Theorem
    • AdaBoost Algorithm
    • Classification Algorithms
    • Clustering Algorithm
    • Gradient Boosting Algorithm
    • Mean Shift Algorithm
    • Hierarchical Clustering Algorithm
    • Hierarchical Clustering Agglomerative
    • What is a Greedy Algorithm?
    • What is Genetic Algorithm?
    • Random Forest Algorithm
    • Nearest Neighbors Algorithm
    • Weak Law of Large Numbers
    • Ray Tracing Algorithm
    • SVM Algorithm
    • Naive Bayes Algorithm
    • Neural Network Algorithms
    • Boosting Algorithm
    • XGBoost Algorithm
    • Pattern Searching
    • Loss Functions in Machine Learning
    • Decision Tree in Machine Learning
    • Hyperparameter Machine Learning
    • Unsupervised Machine Learning
    • K- Means Clustering Algorithm
    • KNN Algorithm
    • Monty Hall Problem
  • Basic
    • Introduction To Machine Learning
    • What is Machine Learning?
    • Uses of Machine Learning
    • Applications of Machine Learning
    • Naive Bayes in Machine Learning
    • Dataset Labelling
    • DataSet Example
    • Dataset ZFS
    • Careers in Machine Learning
    • What is Machine Cycle?
    • Machine Learning Feature
    • Machine Learning Programming Languages
    • What is Kernel in Machine Learning
    • Machine Learning Tools
    • Machine Learning Models
    • Machine Learning Platform
    • Machine Learning Libraries
    • Machine Learning Life Cycle
    • Machine Learning System
    • Machine Learning Datasets
    • Machine Learning Certifications
    • Machine Learning Python vs R
    • Optimization for Machine Learning
    • Types of Machine Learning
    • Machine Learning Methods
    • Machine Learning Software
    • Machine Learning Techniques
    • Machine Learning Feature Selection
    • Ensemble Methods in Machine Learning
    • Support Vector Machine in Machine Learning
    • Decision Making Techniques
    • Restricted Boltzmann Machine
    • Regularization Machine Learning
    • What is Regression?
    • What is Linear Regression?
    • Dataset for Linear Regression
    • Decision tree limitations
    • What is Decision Tree?
    • What is Random Forest
  • Supervised
    • What is Supervised Learning
    • Supervised Machine Learning
    • Supervised Machine Learning Algorithms
    • Perceptron Learning Algorithm
    • Simple Linear Regression
    • Polynomial Regression
    • Multivariate Regression
    • Regression in Machine Learning
    • Hierarchical Clustering Analysis
    • Linear Regression Analysis
    • Support Vector Regression
    • Multiple Linear Regression
    • Linear Algebra in Machine Learning
    • Statistics for Machine Learning
    • What is Regression Analysis?
    • Clustering Methods
    • Backward Elimination
    • Ensemble Techniques
    • Bagging and Boosting
    • Linear Regression Modeling
    • What is Reinforcement Learning
  • Classification
    • Kernel Methods in Machine Learning
    • Clustering in Machine Learning
    • Machine Learning Architecture
    • Automation Anywhere Architecture
    • Machine Learning C++ Library
    • Machine Learning Frameworks
    • Data Preprocessing in Machine Learning
    • Data Science Machine Learning
    • Classification of Neural Network
    • Neural Network Machine Learning
    • What is Convolutional Neural Network?
    • Single Layer Neural Network
    • Kernel Methods
    • Forward and Backward Chaining
    • Forward Chaining
    • Backward Chaining
  • Deep Learning
    • What Is Deep learning
    • Overviews Deep Learning
    • Application of Deep Learning
    • Careers in Deep Learnings
    • Deep Learning Frameworks
    • Deep Learning Model
    • Deep Learning Algorithms
    • Deep Learning Technique
    • Deep Learning Networks
    • Deep Learning Libraries
    • Deep Learning Toolbox
    • Types of Neural Networks
    • Convolutional Neural Networks
    • Create Decision Tree
    • Deep Learning for NLP
    • Caffe Deep Learning
    • Deep Learning with TensorFlow
  • RPA
    • What is RPA
    • What is Robotics?
    • Benefits of RPA
    • RPA Applications
    • Types of Robots
    • RPA Tools
    • Line Follower Robot
    • What is Blue Prism?
    • RPA vs BPM
  • PyTorch
    • PyTorch Tensors
    • What is PyTorch?
    • PyTorch MSELoss()
    • PyTorch NLLLOSS
    • PyTorch MaxPool2d
    • PyTorch Pretrained Models
    • PyTorch Squeeze
    • PyTorch Reinforcement Learning
    • PyTorch zero_grad
    • PyTorch norm
    • PyTorch VAE
    • PyTorch Early Stopping
    • PyTorch requires_grad
    • PyTorch MNIST
    • PyTorch Conv2d
    • Dataset Pytorch
    • PyTorch tanh
    • PyTorch bmm
    • PyTorch profiler
    • PyTorch unsqueeze
    • PyTorch adam
    • PyTorch backward
    • PyTorch concatenate
    • PyTorch Embedding
    • PyTorch Tensor to NumPy
    • PyTorch Normalize
    • PyTorch ReLU
    • PyTorch Autograd
    • PyTorch Transpose
    • PyTorch Object Detection
    • PyTorch Autoencoder
    • PyTorch Loss
    • PyTorch repeat
    • PyTorch gather
    • PyTorch sequential
    • PyTorch U-NET
    • PyTorch Sigmoid
    • PyTorch Neural Network
    • PyTorch Quantization
    • PyTorch Ignite
    • PyTorch Versions
    • PyTorch TensorBoard
    • PyTorch Dropout
    • PyTorch Model
    • PyTorch optimizer
    • PyTorch ResNet
    • PyTorch CNN
    • PyTorch Detach
    • Single Layer Perceptron
    • PyTorch vs Keras
    • torch.nn Module
  • UiPath
    • What is UiPath
    • UiPath Action Center
    • UiPath?Orchestrator
    • UiPath web automation
    • UiPath Orchestrator API
    • UiPath Delay
    • UiPath Careers
    • UiPath Insights
    • UiPath Split String
    • UiPath Installation
    • UiPath Filter Data Table
    • UiPath Test Suite
    • UiPath Competitors
    • UiPath Architecture
    • UiPath version
    • Uipath Reframework
    • UiPath Studio
  • Interview Questions
    • Deep Learning Interview Questions And Answer
    • Machine Learning Cheat Sheet

Related Courses

Machine Learning Training

Deep Learning Training

Artificial Intelligence Training

Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

© 2022 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more

Special Offer - Machine Learning Training (17 Courses, 27+ Projects) Learn More