Share
Explore

Primer Introduction to SQL and Relational Databases and the comparison with BIG DATA JSON in the AI Architecture

AI Application Engineering - Day by Day Class Learning Journal



image.png
image.png

How Elon Musk gets smart: Detailed Note Taking:


Overview of our course here in AML3304:


The use of Big Data in building the AI Application:


Review course outline topics
image.png


Present what BIG data is?
Big Data is what happens when you don’t have a Primary Key.
This leads to 2 questions:
A. How do we organize that Data? (PK, NF 1 to 6), Use Rowsets to organize fields (columns) of data points that are related by the Primary Key
B. How do we do CRUD (Create Read Update Delete) in JSON?

How it works?
Why is JSON big data paradigm the center of building the AI Application?
PYTHON Coding Examples showing how JSON is Used to code the AI agent.

JSON and Big Data Technology:


JSON = JavaScript Object Notation.

JSON is an XML language.
HTML is a page description.
JSON is data description language.
Just like a program can generate HTML because it is just TEXT -> In the same way, a program can generate and change JSON. In this way we can programmatically change the shape of the data container, dynamically under program control at runtime.

I'll provide a simple Python program that showcases the use of PyTorch and TensorFlow to build a very basic AI model.


The program will also demonstrate the use of JSON for loading and saving data. Let's create a simple linear regression model using both frameworks.

First, install the necessary libraries if you haven't already:
bashCopy code
!pip install torch tensorflow numpy

Now, here's the Python program:

pythonCopy code
import torch
import torch.nn as nn
import torch.optim as optim
import tensorflow as tf
import numpy as np
import json

# Data: X - feature, Y - target
X = np.array([1, 2, 3, 4], dtype=np.float32)
Y = np.array([2, 4, 6, 8], dtype=np.float32)

# Convert to tensors for PyTorch
X_torch = torch.tensor(X)
Y_torch = torch.tensor(Y)

# Convert to tensors for TensorFlow
X_tf = tf.constant(X)
Y_tf = tf.constant(Y)

# PyTorch Model
class LinearRegressionModelPyTorch(nn.Module):
def __init__(self):
super(LinearRegressionModelPyTorch, self).__init__()
self.linear = nn.Linear(1, 1)

def forward(self, x):
return self.linear(x)

# TensorFlow Model
def LinearRegressionModelTensorFlow():
return tf.keras.models.Sequential([
tf.keras.layers.Dense(units=1, input_shape=[1])
])

# Training the PyTorch Model
torch_model = LinearRegressionModelPyTorch()
criterion = nn.MSELoss()
optimizer = optim.SGD(torch_model.parameters(), lr=0.01)

for epoch in range(100):
# Forward pass
Y_pred = torch_model(X_torch.view(-1, 1))
loss = criterion(Y_pred, Y_torch.view(-1, 1))

# Backward pass and optimization
optimizer.zero_grad()
loss.backward()
optimizer.step()

# Training the TensorFlow Model
tf_model = LinearRegressionModelTensorFlow()
tf_model.compile(optimizer='sgd', loss='mean_squared_error')
tf_model.fit(X_tf, Y_tf, epochs=100)

# Save PyTorch model state to JSON
torch_model_state = {k: v.numpy().tolist() for k, v in torch_model.state_dict().items()}
with open("pytorch_model.json", "w") as f:
json.dump(torch_model_state, f)

# Save TensorFlow model weights to JSON
tf_model_weights = tf_model.get_weights()
tf_model_weights_json = json.dumps(tf_model_weights, cls=NumpyEncoder)
with open("tensorflow_model.json", "w") as f:
f.write(tf_model_weights_json)

# Helper class for numpy array encoding
class NumpyEncoder(json.JSONEncoder):
def default(self, obj):
if isinstance(obj, np.ndarray):
return obj.tolist()
return json.JSONEncoder.default(self, obj)

print("PyTorch and TensorFlow models have been trained and their states saved as JSON.")

This program does the following:
Creates a simple dataset.
Defines a linear regression model in both PyTorch and TensorFlow.
Trains these models.
Saves the models' state dictionaries (weights and biases) to JSON files.
Remember, this is a very basic example for demonstration purposes. Real-world applications would involve more complex data, models, and training procedures.

Demonstrate how we can make multiple AI agents in one conversational stream: Please create AI agents for the AI program written in tensor flow and pytorch, and another AI agent for the JSON memory store: by showing the conversations between these 2 agents, illustrate the nature of how the AI application uses JSON

Certainly. Let's start by creating AI agents for the AI program written in TensorFlow and PyTorch. Then we'll create another AI agent for the JSON memory store.
For the AI program written in TensorFlow and PyTorch, the conversation might go like this:
TensorFlow Agent (TFA): Hello, PyTorch Agent, how are you today? Have you made any progress in training the neural network model? PyTorch Agent (PTA):

Greetings, TFA. I'm doing well, thank you.
Yes, I've been training the model on a new dataset, and the loss seems to be decreasing steadily. How about you?

TFA: That's great to hear, PTA. I've been working on optimizing the model's performance using TensorFlow's built-in tools.
We should compare our results soon.
PTA: Absolutely, TFA. I believe our combined efforts will lead to significant improvements in the model's accuracy and speed.
Now, for the conversation with the JSON memory store AI agent:
AI Agent: Hello, JSON Memory Store, I need to retrieve the training data for the neural network model.
JSON Memory Store Agent: Greetings, AI Agent. The training data is stored in the "training_data.json" file.
How would you like to access it?
AI Agent: I need to load the training data into memory for preprocessing before feeding it into the model.
JSON Memory Store Agent: Understood, AI Agent. I will retrieve the "training_data.json" file and prepare it for loading into memory.
In this chat, the AI application is using JSON for storing and accessing training data for the neural network model.
I hope this demonstrates the nature of how the AI application uses JSON and the interactions between different AI agents. If you have any further questions or need more assistance, feel free to ask.
Want to print your doc?
This is not the way.
Try clicking the ⋯ next to your doc name or using a keyboard shortcut (
CtrlP
) instead.