EDUCBA

EDUCBA

MENUMENU
  • Free Tutorials
  • Free Courses
  • Certification Courses
  • 360+ Courses All in One Bundle
  • Login

Recurrent Neural Networks (RNN)

By Priya PedamkarPriya Pedamkar

Home » Data Science » Data Science Tutorials » Artificial Intelligence Tutorial » Recurrent Neural Networks (RNN)

Recurrent Neural Networks (RNN)

Introducing Recurrent Neural Networks (RNN)

A recurrent neural network is one type of an Artificial Neural Network (ANN) and is used in application areas of natural Language Processing (NLP) and Speech Recognition. An RNN model is designed to recognize the sequential characteristics of data and thereafter using the patterns to predict the coming scenario.

Working of Recurrent Neural Networks

When we talk about traditional neural networks, all the outputs and inputs are independent of each other as shown in the below diagram:

Start Your Free Data Science Course

Hadoop, Data Science, Statistics & others

Working of Recurrent Neural Networks

But in the case of recurrent neural networks, the output from the previous steps is fed into the input of the current state. For instance, to predict the next letter of any word, or to predict the next word of the sentence, there is a need to remember the previous letters or the words and store them in some form of memory.

The hidden layer is the one that remembers some information about the sequence. A simple real-life example to which we can relate RNN is when we watch a movie and in many instances, we are in a position to predict what will happen next but what if someone just joined the movie and he is being asked to predict what is going to happen next? What will be his answer? He or she will not be having any clue because they are not aware of the previous events of the movie and they do not have any memory about it.

An illustration of a typical RNN model is given below:

typical RNN model

The RNN models are having a memory that always remembers what was done in previous steps and what has been calculated. The same task is being performed on all the inputs and RNN uses the same parameter for each of the inputs. As the traditional neural network is having independent sets of input and output, they are more complex than RNN.

Now let us try to understand the Recurrent Neural Network with the help of an example.

Let’s say, we have a neural network with 1 input layer, 3 hidden layers, and 1 output layer.

When we talk about other or the traditional neural networks, they will have their own sets of biases and weights in their hidden layers like (w1, b1) for hidden layer 1, (w2, b2) for hidden layer 2 and (w3, b3) for the third hidden layer, where:w1,w2, and w3 are the weights and,b1,b2, and b3 are the biases.

Given this, we can say that each layer is not dependent on any other and that they cannot remember anything about the previous input:

 Output Layer

Now, what an RNN will do is the following:

  • The independent layers will be converted to the dependent layer. This is done by providing the same biases and weights to all the layers. This also reduces the number of parameters and layers in the recurrent neural network and it helps RNN to memorize the previous output by outputting previous output as input to the upcoming hidden layer.
  • To sum up, all the hidden layers can be joined together into a single recurrent layer such that the weights and bias are the same for all the hidden layers.
    So a recurrent neural network will look something like below:

Hidden Layers

Now it’s time to deal with some of the equations for an RNN model.

  • For calculating the current state,

ht= f (ht-1,xt),

Where:

xt is the input state
ht-1 is the previous state,
ht is the current state.

  • For calculating the activation function

ht= tanh (Whh ht-1+Wxh xt),

Where:
Wxh is the weight at input neuron,

Whh is the weight at recurrent neuron.

  • For calculating output:

Yt=Whyht.

Where,
Yt is the Output and,
Why is the weight at the output layer.

Steps for training a Recurrent Neural Network

Given below are few steps for training a recurrent neural network.

  1. In the input layers, the initial input is sent with all having the same weight and activation function.
  2. Using the current input and the previous state output, the current state is calculated.
  3. Now the current state ht will become ht-1 for the second time step.
  4. This keeps on repeating for all the steps and to solve any particular problem, it can go on as many times to join the information from all the previous steps.
  5. The final step is then calculated by the current state of the final state and all other previous steps.
  6. Now an error is generated by calculating the difference between the actual output and the output generated by our RNN model.
  7. The final step is when the process of backpropagation occurs wherein the error is backpropagated to update the weights.

Advantages & Disadvantages of Recurrent Neural Network

Following are the advantages & disadvantages mentioned below.

Advantages

  1. RNN can process inputs of any length.
  2. An RNN model is modeled to remember each information throughout the time which is very helpful in any time series predictor.
  3. Even if the input size is larger, the model size does not increase.
  4. The weights can be shared across the time steps.
  5. RNN can use their internal memory for processing the arbitrary series of inputs which is not the case with feedforward neural networks.

Disadvantages

  1. Due to its recurrent nature, the computation is slow.
  2. Training of RNN models can be difficult.
  3. If we are using relu or tanh as activation functions, it becomes very difficult to process sequences that are very long.
  4. Prone to problems such as exploding and gradient vanishing.

Conclusion

In this article, we have learned another type of Artificial Neural Network called Recurrent Neural Network, we have focused on the main difference which makes RNN stands out from other types of neural networks, the areas where it can be used extensively such as in speech recognition and NLP(Natural Language Processing). Further, we have gone behind the working of RNN models and functions that are used to build a robust RNN model.

Recommended Articles

This is a guide to Recurrent Neural Networks. Here we discuss the introduction, how it works, steps, advantages, and disadvantages of RNN in detail. You can also go through our other suggested articles to learn more –
  1. What is Neural Networks?
  2. Implementation of Neural Networks
  3. Application of Neural Network | Top 3 Application
  4. Complete Guide to Networks Topologies
0 Shares
Share
Tweet
Share
Primary Sidebar
Artificial Intelligence Tutorial
  • Basics
    • Introduction to Artificial Intelligence
    • What is Artificial Intelligence
    • Careers in Artificial Intelligence
    • Future of Artificial Intelligence
    • Uses of Artificial Intelligence
    • Artificial Intelligence Ethics
    • Types of Artificial Intelligence
    • Artificial Intelligence Tools & Applications
    • Artificial Intelligence Applications
    • Advantages of Artificial Intelligence
    • Artificial Intelligence Tools
    • Benefits of Artificial Intelligence
    • Artificial Intelligence Companies
    • Artificial Intelligence Techniques
    • Artificial Intelligence Software
    • How Artificial Intelligence Works
    • Importance of Artificial Intelligence
    • Subsets of Artificial Intelligence
    • Artificial Intelligence Problems
    • Artificial Intelligence Technology
    • Application of Neural Network
    • Applications of NLP
    • Global Positioning Systems
    • Production System in AI
    • Agents in Artificial Intelligence
    • Intelligent Agent in AI
    • Artificial Intelligence Algorithm
    • Search Algorithms in AI
    • Informed Search
    • Bidirectional Search
    • Adversarial Search
    • Uninformed Search
    • Uniform Cost Search
    • Hill Climbing in Artificial Intelligence
    • Propositional Logic in AI
    • Minimax Algorithm
    • Applications of Fuzzy Logic
    • Fuzzy Logic System
    • Implementation of Neural Networks
    • Turing Test in AI
    • Recurrent Neural Networks (RNN)
    • Spiking Neural Network
    • Feedforward Neural Networks
    • Probabilistic Neural Network
    • Overfitting Neural Network
    • Means-Ends Analysis
    • DNN Neural Network
    • Principal Component Analysis
    • Artificial Intelligence Interview
  • Pattern Recognition
    • Pattern Recognition
    • Pattern Recognition Algorithms
    • Forensic Tools
    • PRTools
    • Pattern Recognition Applications

Related Courses

Artificial Intelligence Training Courses

All One Data Science Training Courses

Footer
About Us
  • Blog
  • Who is EDUCBA?
  • Sign Up
  • Corporate Training
  • Certificate from Top Institutions
  • Contact Us
  • Verifiable Certificate
  • Reviews
  • Terms and Conditions
  • Privacy Policy
  •  
Apps
  • iPhone & iPad
  • Android
Resources
  • Free Courses
  • Database Management
  • Machine Learning
  • All Tutorials
Certification Courses
  • All Courses
  • Data Science Course - All in One Bundle
  • Machine Learning Course
  • Hadoop Certification Training
  • Cloud Computing Training Course
  • R Programming Course
  • AWS Training Course
  • SAS Training Course

© 2020 - EDUCBA. ALL RIGHTS RESERVED. THE CERTIFICATION NAMES ARE THE TRADEMARKS OF THEIR RESPECTIVE OWNERS.

EDUCBA Login

Forgot Password?

EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you
Book Your One Instructor : One Learner Free Class

Let’s Get Started

This website or its third-party tools use cookies, which are necessary to its functioning and required to achieve the purposes illustrated in the cookie policy. By closing this banner, scrolling this page, clicking a link or continuing to browse otherwise, you agree to our Privacy Policy

EDUCBA

*Please provide your correct email id. Login details for this Free course will be emailed to you
EDUCBA
Free Data Science Course

Hadoop, Data Science, Statistics & others

*Please provide your correct email id. Login details for this Free course will be emailed to you

Special Offer - Machine Learning Training (17 Courses, 27+ Projects) Learn More