Skip to content
[New] Concise and Practical AI/ML
  • Pages
    • Preface
    • Artificial Intelligence
      • Concepts
      • High-level Intelligence
    • Maths for ML
      • Calculus
      • Algebra
    • Machine Learning
      • History of ML
      • ML Models
        • ML Model is Better
        • How a Model Learns
        • Boosted vs Combinatory
      • Neuralnet
        • Neuron
          • Types of Neurons
        • Layers
        • Neuralnet Alphabet
        • Heuristic Hyperparams
      • Feedforward
        • Input Separation
      • Backprop
        • Activation Functions
        • Loss Functions
        • Gradient Descent
        • Optimizers
      • icon picker
        Design Techniques
        • Normalization
        • Regularization
          • Drop-out Technique
        • Concatenation
        • Overfitting & Underfitting
        • Explosion & Vanishing
      • Engineering Techniques
    • Methods of ML
      • Supervised Learning
        • Regression
        • Classification
      • Reinforcement Learning
        • Concepts
        • Bellman Equation
        • Q-table
        • Q-network
        • Learning Tactics
          • Policy Network
      • Unsupervised Learning
        • Some Applications
      • Other Methods
    • Practical Cases
    • Ref & Glossary
[New] Concise and Practical AI/ML
/
/
Design Techniques
Share
Explore

Limiters

There are 3 types of limiters in neuralnet, they are: Normalization, Regularization, and Threshold Function (activation function), with different usefulnesses.

Normalization

Normalization is for the weights and bias to avoid adapting with the infinite range of input, which is that infinity can’t be learnt.

Regularization

Regularization is to punish weight change so the weights and biases won’t comsume the whole infinite range of value, leave spare rooms for unknown cases.

Threshold Function

Threshold function (activation function) is optional in regression but kinda required in classification, it limits the outputs into cases, even integers. It limits output, not separate inputs which is done by params.

Train-Test Split

Training data should have multiple entries with similar values for each case. In such condition, similar cases can be split into training set and test set, to verify during training if the model is generalizing well.

Fine-tuning

Full Fine-tuning

Fine-tune all params.

PEFT (Param-Efficient Fine-Tuning)

Fine-tune some params.

LoRA (Low-Rank Adaption PEFT)

Fine-tune by adding related params.

Prefix PEFT

Fine-tune the first some layers.

Adaptor PEFT

Fine-tune by injecting a layer in middle.

Problems in Training

Overfitting

Explosion

Vanish

Imbalanced Data

Data set for training should be balanced, not too many of 1 output and so few of another.

⁠

Limiters
Normalization
Regularization
Threshold Function
Train-Test Split
Fine-tuning
Full Fine-tuning
PEFT (Param-Efficient Fine-Tuning)
LoRA (Low-Rank Adaption PEFT)
Prefix PEFT
Adaptor PEFT
Problems in Training
Overfitting
Explosion
Vanish
Imbalanced Data
Gallery
 
Want to print your doc?
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.