Rotary Positional Embedding (RoPE) Visualization

Interactive exploration of how RoPE encodes position information in transformer models

Configuration

1,000 10,000 50,000
0 0 1024
-32 1 32

Dot Product Analysis

q · k (original): 0.00
R(q,p) · R(k,p+k): 0.00
Difference: 0.00

3D Vector Rotation

Visualization of how RoPE rotates vector components in 3D space. Each pair of vector dimensions is rotated by an angle determined by the position and frequency.

Frequency Spectrum

Shows the geometric progression of frequencies across dimensions. Lower dimensions have higher frequencies (shorter wavelengths) while higher dimensions have lower frequencies.

Vector Component Rotation

Dimension Pair Original Values Rotated Values Rotation Angle Frequency

How RoPE Works

Rotary Positional Embedding (RoPE) encodes position information by rotating pairs of vector components:

  • Each pair of dimensions (2i, 2i+1) is treated as a 2D vector
  • The vector is rotated by an angle θ = p * ωᵢ where p is position and ωᵢ is frequency
  • Frequencies decrease geometrically with dimension: ωᵢ = 1/(θ^(2i/d))
  • This creates relative position encoding in the attention dot product: q·k depends only on m-n
  • The norm (length) of vectors remains unchanged, preserving semantic information

The rotation matrix for each pair is:

Mᵢ = [ cos(pωᵢ) -sin(pωᵢ) ]
[ sin(pωᵢ) cos(pωᵢ) ]