Skip to content

Commit 3347bdd

Browse files
authored
Update Attention Maths.md
1 parent 9f1a1f0 commit 3347bdd

File tree

1 file changed

+12
-7
lines changed
  • Chapter-wise code/Code - PyTorch/7. Attention Models/2. Neural Text Summarization/1. Transformer Models

1 file changed

+12
-7
lines changed

Chapter-wise code/Code - PyTorch/7. Attention Models/2. Neural Text Summarization/1. Transformer Models/Attention Maths.md

Lines changed: 12 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Attention Maths
22

3+
## Basic Variables
4+
35
```buildoutcfg
46
Q: Queries (embeddings of German words)
57
K: Keys (embeddings of English words)
@@ -8,16 +10,19 @@ D: Dimensionality of word embeddings
810
Lq: no. of. queries
911
Lk: no. of keys
1012
11-
```
13+
dim[Q] = [Lq, D];
14+
dim[K] = [Lk, D];
15+
dim[V] = [Lk, D];
1216
13-
1. Input to attention: Q, K, V Often Vs are same as Ks.
17+
Input to attention: Q, K, V.
18+
Often Vs are same as Ks.
19+
20+
```
1421

15-
2. dim[Q] = [Lq, D];
16-
3. dim[K] = [Lk, D];
17-
4. dim[V] = [Lk, D]
22+
## Delinating Maths behind Attention
1823

19-
<br>
20-
<img src="../images/15. step - 1.png" width="50%"></img> <br><br>
24+
1. Input to attention are Queries, Keys and Values. <br>
25+
<img src="../images/15. step - 1.png" width="40%"></img> <br><br>
2126

2227
4. A query Q, will assign each key K, a probability that key K is a match for Q. Similarity is measured by taking dot
2328
product of vectors. So Q and K are similar iff `Q dot K` is large. <br>

0 commit comments

Comments
 (0)