Skip to content
TRELLO ( look at my Trello as an Example)TEXT Cells of your Collab Collab Notebook.
You will choose a TEACHER (and in your documentation: why WHY that model)Provision some input training data (by example, by walking over a Text File and glob all the text contents in a variable, and pass that variable into your Training Method.Once you have created the embedding, that is good for now.
Local premises: Your own laptop : collaborate on code via GITHUB repo pushes and pulls. If you do this, provide in your TRELLO Board you GITHUB REPO URL.Google Collab Workbook : Provide in your TRELLO Board your URL: Make a collaborator on your Google Collab Workbook. Hugging Face Spaces: make me a member of your TEAM ROOM and post the URL to your TRELLO Board.
Documentation which you will present as Documentation Cells in your Google Collab Notebook.Code presented in your Google Collab notebook.
IntroductionRecap of Neural Networks and ANNs.Introduction to Embeddings: Definition and importance in Machine Learning.Deep Dive into the NeuronRevisit the concept of a neuron: weighted inputs, activation function, and output.Importance of the neuron as the basic computational unit.Lecture: From Neuron to Layer
A neuron takes a set of inputs, applies weights, sums them up, and then passes the sum through an activation function to produce an output.Inputs: Data fed into the neuron (e.g., feature values).Weights: Parameters that determine the importance or influence of a given input.Bias: An additional parameter that allows for flexibility in fitting the model.Activation Function: A function (e.g., sigmoid, ReLU) that introduces non-linearity and determines the neuron's output.
�y is the output,�w are the weights,�x are the inputs,�b is the bias, and�f is the activation function.
Real-world data is multi-dimensional. A single neuron can't capture the complexity of such data.Multiple neurons can learn different features or patterns from the input data.A layer consists of multiple neurons arranged in parallel.Each neuron in a layer receives the same input but has its weights, biases, and potentially different activation functions.In dense layers, each neuron in the previous layer is connected to every neuron in the next layer.This ensures that features learned by one neuron are available for subsequent neurons.
Input Layer: Receives raw data. The number of neurons typically corresponds to the number of features.Hidden Layer(s): Layers between the input and output. They process and transform data, extracting and refining features. Most of the "learning" occurs here.Output Layer: Produces the final prediction or classification. The number of neurons corresponds to the number of output classes or regression outputs.As data progresses through layers, it undergoes transformations. Initial layers might learn basic patterns (e.g., edges in images or specific sounds in audio). Subsequent layers combine these to detect more complex features (shapes, textures, or semantic meaning).While deeper networks can model more complex relationships, they can also introduce challenges like vanishing and exploding gradients.Careful selection of activation functions (e.g., ReLU for hidden layers) can help mitigate these issues.The architecture of layers (number of layers, number of neurons per layer) directly impacts the network's performance and its ability to generalize.Design decisions should be informed by the nature of the data and the problem being addressed.
Setup: Install and import necessary libraries (TensorFlow, Keras, and others).Single Neuron Implementation: Build a neuron that takes a set of inputs, applies weights and bias, and produces an output using an activation function.Scaling to a Layer: Expand the single neuron to a layer with multiple neurons. Observe how each neuron produces different outputs for the same input.Visualization: Use libraries like Matplotlib or Seaborn to visualize the outputs of each neuron in the layer for different inputs.Discussion: Analyze how different neurons in a layer can capture different patterns or features from the same input.
Introduction to EmbeddingsDefinition and significance of embeddings in representing categorical data.Differences between one-hot encoding and embeddings.Building an Embedding LayerArchitecture of an embedding layer.How embeddings reduce dimensionality and capture semantic relationships.Embeddings in PracticePractical applications: Word embeddings in NLP (e.g., Word2Vec, GloVe).Importance in collaborative filtering for recommendation systems.ConclusionRecap of the journey from a single neuron to creating embeddings.Future scope and advanced topics related to embeddings.
Setting Up the EnvironmentInstalling necessary libraries and tools.Overview of the dataset: Using a sample dataset with categorical features.Building a Basic NeuronConstructing a single neuron using Python.Applying a sample input and observing the output after an activation function.Constructing a Neural Network LayerExpanding the single neuron to a basic neural network layer.Forward pass with sample data.One-hot Encoding vs. EmbeddingsImplementing one-hot encoding on a sample categorical feature.Observing the sparsity and dimensionality issues.Building an Embedding LayerUsing TensorFlow/Keras to build an embedding layer.Initializing and inspecting embedding weights.Training a Neural Network with an Embedding LayerConstructing a simple neural network model with an embedding layer for categorical data representation.Training the model and adjusting embedding weights.Visualizing EmbeddingsUsing tools like TensorBoard to visualize the embeddings.Observing how similar categories cluster together in the embedding space.Conclusion and CleanupSummarizing the lab activities.Resources for further exploration and learning.
Share
Explore
F23 AML3304 Assignment 1 Embeddings
Clarification Questions:
For DOCUMENTATION:
The central task of Assignment 1 is to create an Embedding (which is a Python Data Structure).
You will create this Embedding using the procedures in this Lab Workbook:
Later on: You will use this embedding in your AI model to do Next Token Generation for project.
The preferred way to ask questions is the Slack Channel:
Resources:
3 Ways to use compute to do your assignment and project:
{Tell me in your TRELLO board what kind of compute you are using}
Collab - Huggingface, and TRELLO Board LINK: - After the course is over: Post this on LINKED IN to showcase to employers.
What to do for Assignment 1:
Make an Embedding!
Questions to answer: Answer these in your TRELLO Board.
The purpose is “meta-cognition”: (Thinking about what you are thinking about will soak the insights and intuitions about the subject into your Understanding Space faster).
Describe the MODEL that you are using in your code as the Model for your Embedding.
Research and Discuss WHY you choose that model.
How is it of particular value to your Project Business Domain.
*** Additional Requirement: Make a TEAM Trello Board for your Assignment and Project.
IMPORTANT: Add as AN EDITOR of your TRELLO BOARD.
The EDIT Access URL for your TRELLO Board: Put it into a TEXT file :
Name the text file as teamname.txt and upload to
Course Outline: High Performance compute is what we need to make real world AI Language Models work, at work, with realistically large sets of input training data sets, and with many users making many requests everyday.
The preferred way to communicate about course materials, assignments and project is Slack:
Due November 24 Friday
Format of how to hand this in:
You can work in Teams of up to 4 members.
Hand in format Options: {Your choice, any way you can do it is fine}
Can we simply submit a Google Colab notebook with markdown notes?
Google collab notebook must contain::
- all team members name/id
- answers to the assignment questions.
Local Premises : VSC: Put your team’s GITHUB url into a TEXT file and Dropbox submit.
Team’s Hugging Face Spaces: test, prototype, experment with code
Google Collab Workbook: Hand in by making an editting member of your Collab workbook
Advantages of Collab: Use cells to put code and documentation in MD format.
Easy for team members to collaborate.
Deliverables for your Assignment: Discuss these Questions in your TRELLO Board
Research and select a MODEL for your Embedding (and therefore later your Project), and support and defend your reasoning and decision making as to why you choose that MODEL for your Use Cases and Business Domain:
*—- If you were doing this at work: What licensing and pricing considerations for using the APIs would factor into account?
Prerequisite understanding for doing this:
The concept of the PYTHON NEURON, and the concepts of the ANN and the GAN: In this Lecture Notebook:
Using a Cloud Graphical Processor to run your programs:
Resources:
Click to access →
For Assignment 1, you will create a Word Embedding.
How you will deliver this:
You can word in Teams of up to 4 people.
Your deliverables will include;
In addition to your code, I would like to see documentation explaining your thinking and design process in putting together your Embedding.
Learning Outcomes: Understand what an Embedding is.
Lecture and Lab Outline: Building an Embedding Starting with a Neuron
Lecture Outline:
Introduction
Understanding neural networks begins with grasping the concept of the fundamental unit that forms them: the neuron. Just as neurons in the brain work in harmony to enable complex thought and action, artificial neurons, or perceptrons, come together to form layers, enabling computational prowess in artificial neural networks (ANNs).
1. The Basic Neuron: A Quick Recap
a. Definition of a Neuron (Perceptron)
b. Components
c. Mathematical Representation
Where:
2. Transition from a Single Neuron to Multiple Neurons Forming a Layer
a. The Need for Multiple Neurons
b. Structure of a Layer
c. Concept of Dense or Fully Connected Layers
3. How Layers Work in Tandem in ANNs
a. Layered Architecture
b. Data Transformation Across Layers
c. Activation Functions in Layers
d. Importance of Layer Design
Conclusion
Moving from the concept of a singular neuron to the intricate web of layers in an ANN helps us appreciate the layered structure's power and sophistication. Each layer, built upon a foundation of individual neurons, processes data, learns patterns, and collaboratively contributes to the network's final decision or prediction. As we further delve into neural network architectures, understanding this transition from neuron to layer remains fundamental.
Lab Activity: Building and Visualizing a Layer in Python
Objective: Understand the transition from a single neuron to a layer by constructing and visualizing a basic neural network layer using Python and TensorFlow/Keras.
Steps:
By the end of this lecture and lab activity, students will have a concrete understanding of how individual neurons come together to form layers in a neural network, setting the foundation for more advanced topics in deep learning.
Lab Outline:
Summation:
Embeddings play a critical role in representing and processing categorical data in neural networks, especially in areas like natural language processing. Starting with the foundational concept of a neuron and building up to embeddings allows students to grasp the evolution and intricacies of data representation in deep learning. This lecture and lab sequence provides a structured approach to understand and implement embeddings, bridging the gap between theory and practical application.
Note that for the Project I will be demonstrating this using the standard teaching corpus of the Guttenburg Corpus:
This Github also has alot of good sample Python Code you can study and use for your Project.
Let's create an embedding for textual data using TensorFlow's Keras API. This example will show you how to convert text into dense vectors (embeddings) that capture the semantic meaning of the words.
Embedding Creation using Word2Vec
1. Preparing Data
Let's use a small corpus for simplicity:
pythonCopy code
corpus = [
'I love machine learning',
'I love deep learning',
'Deep learning is a subfield of machine learning',
'AI is fascinating',
'Machine learning is fascinating'
]
2. Text Preprocessing
We'll tokenize the sentences and convert them to integers:
pythonCopy code
from tensorflow.keras.preprocessing.text import Tokenizer
from tensorflow.keras.preprocessing.sequence import pad_sequences
# Tokenization
tokenizer = Tokenizer()
tokenizer.fit_on_texts(corpus)
total_words = len(tokenizer.word_index) + 1
# Convert text to sequence of integers
Want to print your doc?
This is not the way.
This is not the way.
Try clicking the ··· in the right corner or using a keyboard shortcut (
CtrlP
) instead.