Skip to content
Concise and Practical AI/ML
  • Pages
    • Preface
    • What are AI and ML
    • Mathematics Recap
      • Calculus
      • Algebra
    • Libraries to Use
    • Models for ML
    • Methods of ML
    • Neuralnet Alphabet
    • Neuralnet
      • Neuron
        • Types of Neurons
        • Input Separations
        • Activation Functions
      • Layers in Network
      • Loss Functions
      • Gradient Descent
      • Feedforward
      • Backpropagation
      • Optimisers & Training
      • Techniques in ML
        • Normalisation
        • Regularisation
        • Concatenation
        • Boosted & Combinatory
        • Heuristic Hyperparams
      • Problems in Neuralnet
        • Overfitting
        • Explosion and Vanishing
    • Supervised Learning
      • Regression
      • Classification
    • Reinforcement Learning
      • Concepts
      • Learning Tactics
      • Policy Network
      • Bellman Equation
      • Q-table
      • Q-network
    • Unsupervised Learning
      • Some Applications
    • Incremental Learning
    • Case Studies
      • Algorithm Approximator
      • Regression
      • Classification
      • Sequence Learning
      • Pattern Learning
      • Generative
    • Notable Mentions
Concise and Practical AI/ML
/
/
Explosion and Vanishing
Share
Explore

Explosion and Vanishing

Value Flow Explosion

During feedforward or backpropagation, large values may multiply with one another and become bigger and bigger very fast even above the numeric limit of computer, eg. 64bit size. Even without such explosion. Values in the network should be limited, or they carry meanings no more.

Value Flow Vanishing

The same multiple problem as in value flow explosion; however, this time is multiplication with zeros or very small values, and the flow end values will be come all zeros.

Value Flow Explosion
Value Flow Vanishing
Gallery
 
Want to print your doc?
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.