Gallery

Autogen Tut

RIP to the milkman, elevator operator, and the guy who walked around waking people up before alarm clocks.

Final Project Work

A2 DLNN

CS559 Notes

Explore

A2 DLNN

Project Work

Assignment 2 Neural Machine Translation with RNN’s

⁠

...

Implemtation

⁠

max_len = max([len(sent) for sent in sents]) # or max_len = max(map(len, sents))

sents_padded = [sent + [pad_token] * (max_len - len(sent)) for sent in sents]

⁠

(a)

⁠

self.source = nn.Embedding(len(vocab.src), embed_size, padding_idx=src_pad_token_idx)

self.target = nn.Embedding(len(vocab.tgt), embed_size, padding_idx=tgt_pad_token_idx)

⁠

(b)

self.encoder = nn.LSTM(input_size=embed_size, hidden_size=self.hidden_size, bias=True, bidirectional=True)

self.decoder = nn.LSTMCell(input_size=embed_size + self.hidden_size, hidden_size=self.hidden_size, bias=True)

self.h_projection = nn.Linear(self.hidden_size*2, self.hidden_size, bias=False) # W_{h}

self.c_projection = nn.Linear(self.hidden_size*2, self.hidden_size, bias=False) # W_{c}

self.att_projection = nn.Linear(self.hidden_size*2, self.hidden_size, bias=False) # W_{attProj}

self.combined_output_projection = nn.Linear(self.hidden_size*3, self.hidden_size , bias=False) # W_{u}

self.target_vocab_projection = nn.Linear(self.hidden_size, len(self.vocab.tgt), bias=False) # W_vocab

self.dropout = nn.Dropout(p=self.dropout_rate)

⁠

(c)

⁠

X = self.model_embeddings.source(source_padded) # X.shape (src_len, b, e)

X_packed = pack_padded_sequence(X, source_lengths) # X-packed.shape (src_len, b, e)

enc_hiddens, (last_hidden, last_cell) = self.encoder(X_packed) # enc.hiddens.shape (src_len, b, h*2)

enc_hiddens, len_pad = pad_packed_sequence(enc_hiddens, batch_first=True) # shape (b, src_len, h*2)

init_decoder_hidden = self.h_projection(torch.cat((last_hidden[0], last_hidden[1]), 1)) # shape (b, 2*h)

init_decoder_cell = self.c_projection(torch.cat((last_cell[0], last_cell[1]), 1)) # shape (b, 2*h)

dec_init_state = (init_decoder_hidden, init_decoder_cell)

⁠

(d)

⁠

enc_hiddens_proj = self.att_projection(enc_hiddens) # shape (b, src_len, h)

Y = self.model_embeddings.target(target_padded) # shape shape (tgt_len, b, e)

for i in torch.split(Y, 1, dim=0): # torch.split(tensor, split_size_or_section, dim=0), Y_t.shape (1, b, e)

Y_t = i.squeeze(0) # Y_t.shape (b, e), o_prev.shape (b, h), squeeze explicitly at location

Ybar_t = torch.cat((Y_t, o_prev), dim=1) # Ybar_t.shape(b, e+h), torch.cat(tensors, dim=0, out=None)

dec_state, o_t, e_t = self.step(Ybar_t, dec_state, enc_hiddens, enc_hiddens_proj, enc_masks) # dec_state, combined_output, e_t

combined_outputs.append(o_t) # shape (b, h)

o_prev = o_t

combined_outputs = torch.stack(combined_outputs, dim=0) # (tgt_len, b, h)

⁠

(e)

⁠

(dec_hidden, dec_cell) = self.decoder(Ybar_t, dec_state)

dec_state = (dec_hidden, dec_cell)

e_t = torch.bmm(enc_hiddens_proj, dec_hidden.unsqueeze(2)).squeeze(2)

⁠

(f)

⁠

....

Deliverables

Submit all files within the src/submission subdirectory. This includes:

src/submission/__init__.py

src/submission/model_embeddings.py

src/submission/nmt_model.py

src/submission/utils.py

src/submission/test_outputs.txt

Want to print your doc?
This is not the way.

Try clicking the ⋯ next to your doc name or using a keyboard shortcut (

CtrlP

) instead.