EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login
Home Data Science Data Science Tutorials Machine Learning Tutorial Loss Functions in Machine Learning
Secondary Sidebar
Machine Learning Tutorial
  • Algorithms
    • Machine Learning Algorithms
    • Apriori Algorithm in Machine Learning
    • Types of Machine Learning Algorithms
    • Bayes Theorem
    • AdaBoost Algorithm
    • Classification Algorithms
    • Clustering Algorithm
    • Gradient Boosting Algorithm
    • Mean Shift Algorithm
    • Hierarchical Clustering Algorithm
    • Hierarchical Clustering Agglomerative
    • What is a Greedy Algorithm?
    • What is Genetic Algorithm?
    • Random Forest Algorithm
    • Nearest Neighbors Algorithm
    • Weak Law of Large Numbers
    • Ray Tracing Algorithm
    • SVM Algorithm
    • Naive Bayes Algorithm
    • Neural Network Algorithms
    • Boosting Algorithm
    • XGBoost Algorithm
    • Pattern Searching
    • Loss Functions in Machine Learning
    • Decision Tree in Machine Learning
    • Hyperparameter Machine Learning
    • Unsupervised Machine Learning
    • K- Means Clustering Algorithm
    • KNN Algorithm
    • Monty Hall Problem
  • Basic
    • Introduction To Machine Learning
    • What is Machine Learning?
    • Uses of Machine Learning
    • Applications of Machine Learning
    • Naive Bayes in Machine Learning
    • Dataset Labelling
    • DataSet Example
    • Deep Learning Techniques
    • Dataset ZFS
    • Careers in Machine Learning
    • What is Machine Cycle?
    • Machine Learning Feature
    • Machine Learning Programming Languages
    • What is Kernel in Machine Learning
    • Machine Learning Tools
    • Machine Learning Models
    • Machine Learning Platform
    • Machine Learning Libraries
    • Machine Learning Life Cycle
    • Machine Learning System
    • Machine Learning Datasets
    • Machine Learning Certifications
    • Machine Learning Python vs R
    • Optimization for Machine Learning
    • Types of Machine Learning
    • Machine Learning Methods
    • Machine Learning Software
    • Machine Learning Techniques
    • Machine Learning Feature Selection
    • Ensemble Methods in Machine Learning
    • Support Vector Machine in Machine Learning
    • Decision Making Techniques
    • Restricted Boltzmann Machine
    • Regularization Machine Learning
    • What is Regression?
    • What is Linear Regression?
    • Dataset for Linear Regression
    • Decision tree limitations
    • What is Decision Tree?
    • What is Random Forest
  • Supervised
    • What is Supervised Learning
    • Supervised Machine Learning
    • Supervised Machine Learning Algorithms
    • Perceptron Learning Algorithm
    • Simple Linear Regression
    • Polynomial Regression
    • Multivariate Regression
    • Regression in Machine Learning
    • Hierarchical Clustering Analysis
    • Linear Regression Analysis
    • Support Vector Regression
    • Multiple Linear Regression
    • Linear Algebra in Machine Learning
    • Statistics for Machine Learning
    • What is Regression Analysis?
    • Clustering Methods
    • Backward Elimination
    • Ensemble Techniques
    • Bagging and Boosting
    • Linear Regression Modeling
    • What is Reinforcement Learning
  • Classification
    • Kernel Methods in Machine Learning
    • Clustering in Machine Learning
    • Machine Learning Architecture
    • Automation Anywhere Architecture
    • Machine Learning C++ Library
    • Machine Learning Frameworks
    • Data Preprocessing in Machine Learning
    • Data Science Machine Learning
    • Classification of Neural Network
    • Neural Network Machine Learning
    • What is Convolutional Neural Network?
    • Single Layer Neural Network
    • Kernel Methods
    • Forward and Backward Chaining
    • Forward Chaining
    • Backward Chaining
  • Deep Learning
    • What Is Deep learning
    • Overviews Deep Learning
    • Application of Deep Learning
    • Careers in Deep Learnings
    • Deep Learning Frameworks
    • Deep Learning Model
    • Deep Learning Algorithms
    • Deep Learning Technique
    • Deep Learning Networks
    • Deep Learning Libraries
    • Deep Learning Toolbox
    • Types of Neural Networks
    • Convolutional Neural Networks
    • Create Decision Tree
    • Deep Learning for NLP
    • Caffe Deep Learning
    • Deep Learning with TensorFlow
  • RPA
    • What is RPA
    • What is Robotics?
    • Benefits of RPA
    • RPA Applications
    • Types of Robots
    • RPA Tools
    • Line Follower Robot
    • What is Blue Prism?
    • RPA vs BPM
  • Interview Questions
    • Deep Learning Interview Questions And Answer
    • Machine Learning Cheat Sheet

Related Courses

Machine Learning Training

Deep Learning Training

Artificial Intelligence Training

Loss Functions in Machine Learning

By Priya PedamkarPriya Pedamkar

loss function in machine learning

Overview of Loss Functions in Machine Learning

In Machine learning, the loss function is determined as the difference between the actual output and the predicted output from the model for the single training example while the average of the loss function for all the training examples is termed as the cost function. This computed difference from the loss functions( such as Regression Loss, Binary Classification, and Multiclass Classification loss function) is termed the error value; this error value is directly proportional to the actual and predicted value.

How does Loss Functions Work?

The word ‘Loss’ states the penalty for failing to achieve the expected output. If the deviation in the predicted value than the expected value by our model is large, then the loss function gives the higher number as output, and if the deviation is small & much closer to the expected value, it outputs a smaller number.

Here’s an example of when we are trying to predict house sales price in metro cities.

Predicted

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

Sales Price (In lakh)

Actual

Sales Price(In lakh)

Deviation (Loss)
Bangalore: 45 0 (All predictions are correct)
Pune: 35
Chennai: 40
Bangalore: 40 Bangalore: 45  5 lakh for Bangalore, 2 lakh for Chennai
Pune: 35 Pune: 35
Chennai: 38 Chennai: 40
Bangalore: 43  2 lakh for Bangalore, 5 lakh for, Pune2 lakh for Chennai,
Pune: 30
Chennai: 45

It is important to note that, amount of deviation doesn’t matter; the thing which matters here is whether the value predicted by our model is right or wrong. Loss functions are different based on your problem statement to which machine learning is being applied. The cost function is another term used interchangeably for the loss function, but it holds a slightly different meaning. A loss function is for a single training example, while a cost function is an average loss over the complete train dataset.

Types of Loss Functions in Machine Learning

Below are the different types of the loss function in machine learning which are as follows:

1. Regression loss functions

Linear regression is a fundamental concept of this function. Regression loss functions establish a linear relationship between a dependent variable (Y) and an independent variable (X); hence we try to fit the best line in space on these variables.

Y = X0 + X1 + X2 + X3 + X4….+ Xn
  • X = Independent variables
  • Y = Dependent variable
Mean Squared Error Loss

MSE(L2 error) measures the average squared difference between the actual and predicted values by the model. The output is a single number associated with a set of values. Our aim is to reduce MSE to improve the accuracy of the model.

All in One Data Science Bundle(360+ Courses, 50+ projects)
Python TutorialMachine LearningAWSArtificial Intelligence
TableauR ProgrammingPowerBIDeep Learning
Price
View Courses
360+ Online Courses | 50+ projects | 1500+ Hours | Verifiable Certificates | Lifetime Access
4.7 (86,527 ratings)

Consider the linear equation, y = mx + c, we can derive MSE as:

MSE=1/N ∑i=1 to n (y(i)−(mx(i)+b))2

Here, N is the total number of data points, 1/N ∑i=1 to n is the mean value, and y(i) is the actual value and mx(i)+b its predicted value.

Mean Squared Logarithmic Error Loss (MSLE)

MSLE measures the ratio between actual and predicted value. It introduces an asymmetry in the error curve. MSLE only cares about the percentual difference between actual and predicted values. It can be a good choice as a loss function when we want to predict house sales prices, bakery sales prices, and the data is continuous.

Here, the loss can be calculated as the mean of observed data of the squared differences between the log-transformed actual and predicted values, which can be given as:

L=1nn∑i=1(log(y(i)+1)−log(^y(i)+1))2
Mean Absolute Error (MAE)

MAE calculates the sum of absolute differences between actual and predicted variables. That means it measures the average magnitude of errors in a set of predicted values. Using the mean square error is easier to solve, but using the absolute error is more robust to outliers. Outliers are those values, which deviate extremely from other observed data points.

MAE can be calculated as:

L=1nn∑i=1∣∣y(i)−^y(i)∣∣

2. Binary Classification Loss Functions

These loss functions are made to measure the performances of the classification model. In this, data points are assigned one of the labels, i.e. either 0 or 1. Further, they can be classified as:

Binary Cross-Entropy

It’s a default loss function for binary classification problems. Cross-entropy loss calculates the performance of a classification model, which gives an output of a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability value deviate from the actual label.

Hinge loss

Hinge loss can be used as an alternative to cross-entropy, which was initially developed to use with a support vector machine algorithm. Hinge loss works best with the classification problem because target values are in the set of {-1,1}. It allows to assign more error when there is a difference in sign between actual and predicted values. Hence resulting in better performance than cross-entropy.

Squared Hinge loss

An extension of hinge loss, which simply calculates the square of the hinge loss score. It reduces the error function and makes it numerically easier to work with. It finds the classification boundary that specifies the maximum margin between the data points of various classes. Squared hinge loss fits perfect for YES OR NO kind of decision problems, where probability deviation is not the concern.

3. Multi-class Classification Loss Functions

Multi-class classification is the predictive models in which the data points are assigned to more than two classes. Each class is assigned a unique value from 0 to (Number_of_classes – 1). It is highly recommended for image or text classification problems, where a single paper can have multiple topics.

Multi-class Cross-Entropy

In this case, the target values are in the set of 0 to n i.e {0,1,2,3…n}. It calculates a score that takes an average difference between actual and predicted probability values, and the score is minimized to reach the best possible accuracy. Multi-class cross-entropy is the default loss function for text classification problems.

Sparse Multi-class Cross-Entropy

One hot encoding process makes multi-class cross-entropy difficult to handle a large number of data points. Sparse cross-entropy solves this problem by performing the calculation of error without using one-hot encoding.

Kullback Leibler Divergence Loss

KL divergence loss calculates the divergence between probability distribution and baseline distribution and finds out how much information is lost in terms of bits. The output is a non-negative value that specifies how close two probability distributions are. To describe KL divergence in terms of probabilistic view, the likelihood ratio is used.

In this article, initially, we understood how loss functions work and then we went on to explore a comprehensive list of loss functions with used case examples. However, understanding it practically is more beneficial, so try to read more and implement it. It will clarify your doubts thoroughly.

Recommended Articles

This is a guide to Loss Functions in Machine Learning. Here we discuss How does Loss Functions Work and the Types of Loss Functions in Machine Learning in detail. You may also have a look at the following articles to learn more –

  1. Machine Learning Methods
  2. Introduction To Machine Learning
  3. Big Data Technologies
  4. Learn the Catogories of Hyperparameter
  5. Machine Learning Life Cycle | Top 8 Stages
Popular Course in this category
Machine Learning Training (20 Courses, 29+ Projects)
  19 Online Courses |  29 Hands-on Projects |  178+ Hours |  Verifiable Certificate of Completion
4.7
Price

View Course

Related Courses

Deep Learning Training (18 Courses, 24+ Projects)4.9
Artificial Intelligence AI Training (5 Courses, 2 Project)4.8
0 Shares
Share
Tweet
Share
Primary Sidebar
Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Live Classes
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

ISO 10004:2018 & ISO 9001:2015 Certified

© 2022 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA
Free Data Science Course

SPSS, Data visualization with Python, Matplotlib Library, Seaborn Package

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA Login

Forgot Password?

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you

By signing up, you agree to our Terms of Use and Privacy Policy.

Let’s Get Started

By signing up, you agree to our Terms of Use and Privacy Policy.

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

Loading . . .
Quiz
Question:

Answer:

Quiz Result
Total QuestionsCorrect AnswersWrong AnswersPercentage

Explore 1000+ varieties of Mock tests View more