Natural Language Processing

Word Embedding

1) Hot Embedding: V is the vocabulary, the representation of the i-th word will have a 1 in the i-the position and 0 in the remaining |V | − 1 positions

Problems with Hot Embedding

V tends to be very large
These representations do not capture similarity
Euclidean distance = v2 and cosine similarity of 0

Distributed Representations of words

A co-occurrence matrix is a terms x terms matrix which captures the number of times a term appears in the context of another term

The context is defined as a window of k words around the terms

Some fixable problems:

Stop words are frequent
Solution 1: Ignore very frequent words
Solution 2: Use a threshold t
Solution 3: Instead of count(w, c) using PMI(w,c)

PMI(w,c) = logp(c|w)/p(c) = log count(w,c) * N / count(c) * count(w)

Very high dimensional
Very sparse
Solution: Use dimensionality reduction (SVD)

Continuous bag of words models

Predict n-th word given previous n-1 words

Feedforward neural network

Input: One hot representation of the context word

Output: There are V words possible and we get the probability distribution

k→Embedding dimension

Pasted image 20260415225837.png

Hot vector representation basically selects the ith column in Wcontext, giving the word representation of the context word

Update rule: vw = vw + n * u (1 - y)

vw - word
n - learning rate
u - ith column vector of Wcontext

Goal - Maximise the cosine similarity between uc and vc

Evaluating word representations

Semantic Relatedness: Ask humans to judge the relatedness between two pair of words, compute the cosine corresponding cosine similarity word vectors learnt by the model.
Synonym Detection: Given a term, find the candidate synonyms with largest cosine similarity

Smodel(cat,dog)=vcatTvdog/∥vcat∥∥vdog∥ → Cosine similarity (how close the vectors are in a higher dimensional space)

Semantic Analogy: brother:sister::grandson:? → vsister - vbrother + vgrandson (find the nearest neighbour in the word space)