Explore

AML 3304 Lecture and Lab Notebook: F23 October 17

Learning Outcomes:

Learn how make Embeddings ← → Learning how the architecture of the AI Application works (which is what we need to do the Project)

How it all starts with the concept of the PYTHON NEURON building up to ANNs and GANs.

⁠

Lecture on Generative Adversarial Networks and Artificial Neural Networks

⁠

Outline:

Epoch 1: Foundation and Basics

⁠

Introduction to Neural Networks

⁠

Definition and Inspiration

Neural networks, in the context of machine learning, are algorithms inspired by the structure of the human brain. They are designed to recognize patterns.

Based on Bayesian Math:

https://coda.io/@peter-sigurdson/baysian-models⁠

⁠

The brain consists of billions of cells called neurons, which process and transmit information. Similarly, in an ANN artificial neural network, basic units called artificial neurons process information using mathematical functions. Python AI programs start with the Python class “neuron”. Then we build our AI Application up in layers: which yields the Layered Architecture of the AI Application.

#todo: add references to UMEP book and LI Blog Articles

Why Neural Networks?

Capability to learn and make decisions from data, based on mathematical statistical algorithms called Bayesian algorithms applied to your input Training Corpus.

⁠

Flexibility: Can adjust themselves to the data and can generalize to unseen data. We need to study the operation of JSON “Big Data” data structures to provide conversional memory.

Versatility: Used in a wide range of applications like image and voice recognition, medical diagnosis, financial forecasting, scientific and engineering (Wolfram Mathematica for Science and Engineering). We can present business processes in an XML formatted language called BPEL and BPMN. Wolfram’s Mathematica can do do AI processing on scientific and engineering formulas.

⁠

The Starting Point: Basic Neuron: The Building Block of ANNs

⁠

Anatomy of a Neuron

Inputs: Analogous to dendrites in a biological neuron. They receive the data.

Weights: These are the strength or amplitude of a connection. If a weight is large, it means that the neuron's input has a strong influence on the output.

Summation Function: All the weighted inputs are summed up in this function.

Activation Function: It determines the neuron's output based on the sum of the weighted inputs. It's used to introduce non-linearity into the output of a neuron. This is where we get probabilistic programming.

Output: Analogous to axon in a biological neuron. This is the data that the neuron sends out to the next layer after processing.

The Perceptron Model

The perceptron is the simplest type of artificial neuron. It's a binary classification algorithm that makes its decisions based on a linear predictor function.

A perceptron takes several binary inputs, multiplies them by their weights, sums them up, and then produces a single binary output using a step function. (In future classes, we will experiment with simple R programming to see visualizations and to develop an intuition for this).

⁠

Structure of Artificial Neural Networks (ANNs)

⁠

Layers in an ANN

Input Layer: The layer that receives input from the dataset. Typically, nodes in this layer represent features or attributes of the dataset.

⁠

Hidden Layer(s): Layers that are neither input nor output. They perform computations and transfer information from the input nodes to the output nodes. Deep Neural Networks have multiple hidden layers: These to the intermediate processing steps.

Output Layer: This layer produces the result for given inputs: This produces the “next token generation” stream which is sent back to user who is chatting with our AI Language Model.

Nodes or Neurons

Image NODES in a Weighted Graph.

#todo insert photo of whiteboard

Each node in a layer represents a specific output. For instance, in a neural network designed for multiclass classification, each node in the output layer represents a specific class.

Connections and Weights

Every node in a layer is connected to every node in the previous and next layers. These connections represent weights, which get adjusted during learning.

⁠

Training an ANN: Feedforward and Backpropagation

⁠

Feedforward Process: Layered Architecture

The process where the input is passed through the neural network, layer by layer, until it reaches the output layer.

Each neuron processes the input, and this processed data is passed as input to the next layer.

Loss Function

Once we have the predicted output, we compare it to the actual output. This comparison is quantified using a loss function (e.g., Mean Squared Error for regression tasks or Cross-Entropy for classification tasks). [We will use R programming to get a visualization on what these mean.]

The result from the loss function, termed as 'loss', indicates how well the neural network's prediction matched the actual output. If you did good Bayesian Training, your loss will be very small.

Backpropagation Algorithm

The heart of the neural network training process.

It's an optimization algorithm used for minimizing the error in the predictions.

The main principle behind backpropagation is the chain rule from calculus. It calculates the gradient of the loss function with respect to each weight by propagating the gradient backward in the network.

Once the gradient is calculated, weights are adjusted using optimization techniques like Gradient Descent.

Gradient Descent and Learning Rate

Gradient Descent is an optimization algorithm used to minimize the loss function by adjusting weights in the direction of the steepest decrease in the loss.

The size of the steps taken to reach the minimum is determined by the learning rate. If the learning rate is too high, the algorithm might overshoot the optimal solution.

If it's too low, the algorithm might get stuck and take too long.

⁠

At the end of this epoch, the students should have a clear understanding of the fundamental concepts and workings of Artificial Neural Networks, at least to the point of getting an intuition on our code labs.

They should be able to visualize the structure of ANNs, understand the significance of each component, and appreciate the intricacies of the training process.

Epoch 2: Diving Deeper with GANs

Introduction to Generative Adversarial Networks (GANs)

Components of GANs: Generator and Discriminator: The Generator develops solutions. The Discriminator looks for problems with those solutions.

Training Process of GANs

Types of GANs:

Vanilla GAN,

DCGAN

CGAN

StyleGAN

Introduction to Generative Adversarial Networks (GANs)

Generative Adversarial Networks, abbreviated as GANs, have made a significant splash in the world of deep learning.

As their name suggests, GANs are generative models that can produce or "generate" new, previously unseen data.

How AIs generate new information / Very analogous to how doing SQL query on a Database can generate new information:

⁠

https://www.linkedin.com/pulse/computational-philosophy-building-implicate-order-new-sigurdson/⁠

⁠

They're particularly known for creating realistic images, but their capabilities stretch far beyond that.

Relatable Example: Think of GANs like an artist (generator) trying to create counterfeit money and a detective (discriminator) trying to distinguish between real and counterfeit money. Over multiple epochs of interaction, the artist becomes so skilled that the detective can't tell the difference between the counterfeit and the real thing. (Loss function is nearing Zero).

⁠

Components of GANs: Generator and Discriminator

The Generator: It takes random noise as an input and produces data (like images). Its primary aim is to make its generated data indistinguishable from real data.

Relatable Example: Consider the generator as a student trying to cheat on an exam by writing an answer in their words. Initially, the answers might not make sense, but with practice, they get better.

The Discriminator: It's a binary classifier that tries to distinguish between real data and fake data produced by the generator. If it gets fooled by the generator, it provides feedback, helping the generator improve.

Relatable Example: Now think of the discriminator as the teacher checking the student's answer. The teacher (discriminator) knows the correct answer and can easily spot when a student is trying to cheat. However, as the student becomes better at paraphrasing, it gets tougher for the teacher to catch the cheating.

⁠

Training Process of GANs

The training process of GANs is like a two-player game:

Generator's Turn: The generator creates a batch of fake data.

Discriminator's Turn: It evaluates both real data and the fake data produced by the generator.

Feedback Loop: The discriminator then provides feedback to the generator about how convincing its fake data was.

Adjustments: Based on this feedback, the generator tries to produce even more convincing data.

This process is repeated through multiple EPOCHS until the generator produces data that the discriminator can't distinguish from real data or until a set number of iterations is reached. [The loss function is so close to Zero that it is effectively Zero).

Relatable Example: It's like training a dog. When the dog (generator) does something right, it gets a treat. If not, it gets corrected and tries again. Over time, with repetition, the dog learns to perform tricks correctly. AIs get trained (Human Feedback Training) by being told that their performance is good enough or not, and modify their perform to optimize their reward. We let the AI algorthm self-discover the path to minimize gradient descent loss.

⁠

Types of GANs

Vanilla GAN: The simplest form of GAN, and often the starting point for those new to the topic. It consists of a basic generator and discriminator architecture.

Relatable Example: Think of Vanilla GAN as the basic model car in a series. It has essential features and gets the job done, but there might be more advanced models with additional features.

DCGAN (Deep Convolutional GAN): Uses deep convolutional networks, making it especially effective for image generation tasks. DCGANs can capture complex image patterns and structures.

Relatable Example: Imagine trying to replicate a famous painting. While a basic approach (Vanilla GAN) might capture the colors and basic shapes, DCGAN would capture the intricate brush strokes and details.

CGAN (Conditional GAN): Adds a twist to GANs by introducing conditional parameters, allowing users to guide the data generation process based on certain conditions or labels.

Relatable Example: Suppose you're cooking a meal (data generation). With CGAN, it's like following a specific recipe or adding ingredients based on dietary restrictions.

StyleGAN: Particularly renowned for producing high-quality, photorealistic images. It can adjust the style of the generated image at different levels of granularity.

Relatable Example: Think of StyleGAN as a fashion designer creating clothes. While the basic structure of a shirt remains the same, the designer can change its style - from the type of collar, the print, the fabric texture, and so on.

⁠

Conclusion

GANs, with their adversarial approach, have revolutionized the way we think about data generation.

By pitting two AI Agents against each other, they harness a competitive spirit to produce astonishingly realistic results. As we've seen through relatable examples, the dynamics between the generator and discriminator are both competitive and collaborative, leading to continuous improvement and refinement.

⁠

Lab Activity: Implementing a Basic Vanilla GAN

Note: We will be using Juypter Notebook within Anaconda Python for our Activities, Assignment and Project hand-in. It is great for combining Code Execution and Documentation with Markdown.

Objective: To give students a hands-on experience in setting up and training a basic GAN to understand the interaction between the generator and the discriminator.

Steps:

Setup: Install required libraries and tools.

Data Preparation: Use a simple dataset (e.g., MNIST) for ease of understanding.

Building the Generator and Discriminator: Using TensorFlow/Keras.

Training the GAN: Observe how the generator improves over time.

Evaluating Results: Visualizing generated data after various epochs.

Discussion: Analyzing the challenges faced during training and ways to overcome them.

⁠

By the end of the lecture and lab activity, students will have a comprehensive understanding of GANs' intricacies, from their foundational principles to their diverse variations.

⁠

image.jpeg failed to upload

⁠

Generate a python program to illustrate this operation

Let's implement a basic Vanilla GAN using the TensorFlow/Keras library. This will provide a hands-on illustration of how GANs operate. For simplicity, we'll use the MNIST dataset of handwritten digits.

import numpy as np

import tensorflow as tf

from tensorflow.keras.datasets import mnist

from tensorflow.keras.layers import Dense, Flatten, Reshape

from tensorflow.keras.models import Sequential

# Load and preprocess the MNIST dataset

(x_train, _), (_, _) = mnist.load_data()

x_train = x_train / 255.0 # normalize to [0, 1]

x_train = np.expand_dims(x_train, axis=-1)

x_train = x_train.astype(np.float32)

# Define the generator model

def build_generator(latent_dim):

model = Sequential([

Dense(128, activation='relu', input_dim=latent_dim),

Dense(784, activation='sigmoid'),

Reshape((28, 28))

])

return model

# Define the discriminator model

def build_discriminator():

model = Sequential([

Flatten(input_shape=(28, 28)),

Dense(128, activation='relu'),

Dense(1, activation='sigmoid')

])

return model

# Compile the discriminator model

discriminator = build_discriminator()

discriminator.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])

# Build and compile the GAN model with discriminator set as non-trainable

discriminator.trainable = False

generator = build_generator(latent_dim=100)

gan_input = tf.keras.Input(shape=(100,))

gan_output = discriminator(generator(gan_input))

gan = tf.keras.Model(gan_input, gan_output)

gan.compile(optimizer='adam', loss='binary_crossentropy')

# Training the GAN

epochs = 10000

batch_size = 128

for epoch in range(epochs):

# Train discriminator

idx = np.random.randint(0, x_train.shape[0], batch_size)

real_imgs = x_train[idx]

noise = np.random.normal(0, 1, (batch_size, 100))

generated_imgs = generator.predict(noise)

real_labels = np.ones((batch_size, 1))

fake_labels = np.zeros((batch_size, 1))

d_loss_real = discriminator.train_on_batch(real_imgs, real_labels)

d_loss_fake = discriminator.train_on_batch(generated_imgs, fake_labels)

d_loss = 0.5 * np.add(d_loss_real, d_loss_fake)

# Train generator

noise = np.random.normal(0, 1, (batch_size, 100))

valid_labels = np.ones((batch_size, 1))

g_loss = gan.train_on_batch(noise, valid_labels)

# Print progress

print(f"{epoch}/{epochs} [D loss: {d_loss[0]} | D accuracy: {d_loss[1]}] [G loss: {g_loss}]")

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.