Update README.md

README.md

@@ -20,41 +20,70 @@ tags:
 - new
 ---
 
+### NLP/ASR model with acoustic variable radii relative position embedding (vRoPE) that maps pitch to token. 
+----
 
-## Echo - NLP/ASR model with acoustic variable radii relative position embedding (vRoPE) that maps pitch to token.  And some other stuff...
+Pitch-Aware Processing: Integrates F0/pitch information throughout the processing pipeline, making the model sensitive to prosodic features of speech.
 
-https://github.com/sine2pi/asr_model_echo
+To highlight the relationship between pitch and rotary embeddings echo implements two complementary pitch-based enhancements:
 
-freqs = (theta / 220.0) * 700 * (torch.pow(10, torch.linspace(0, 2595 * torch.log10(torch.tensor(1 + 8000/700)), dim // 2, device=device, dtype=dtype) / 2595) - 1) / 1000
+1. The first uses pitch to modify theta (rotary frequency)*
+2. The second adds direct similarity bias to attention
+3. Variable radii added in place of unit circle radius(1.0) associated with torch.polar. The frequencies (f0) are time aligned with tokens creating acoustically-weighted positional encodings where the "loudness" of each position in the embedding space reflects the acoustic prominence in the original speech.
 
-The Magic of Domain-Specific Knowledge in ML..
+By modulating the RoPE frequencies based on pitch (F0), we are essentially telling the model to pay attention to the acoustic features relate to sequence position in a way that's proportional to the voice characteristics.  This approach creates a more speech-aware positional representation that helps the model better understand the relationship between acoustic features and text.
 
-Experimental - research model. Some of the modules and functions in the code are not part of the active model just yet. 
+![rotpatterns](https://github.com/user-attachments/assets/165a3f18-659a-4e2e-a154-a3456b667bae)
 
-Pitch-Aware Processing: Integrates F0/pitch information throughout the processing pipeline, making the model sensitive to prosodic features of speech.
 
+These visualizations show how F0 (fundamental frequency/pitch) information affects the model's rotary position embeddings (RoPE)
 
-To highlight the relationship between pitch and rotary embeddings echo implements two complementary pitch-based enhancements:
 
-1. The first uses pitch to modify theta (rotary frequency)*
-2. The second adds direct similarity bias to attention
-3. Variable radii added in place of unit circle radius(1.0) associated with torch.polar. The frequencies (f0) are time aligned with tokens creating acoustically-weighted positional encodings where the "loudness" of each position in the embedding space reflects the acoustic prominence in the original speech.
+Each figure shows 4 subplots (one for each of the first 4 dimensions of your embeddings in the test run). These visualizations show how pitch information modifies position encoding patterns in the model.
 
-1000 steps no f0:
+In each subplot:
 
-<img width="470" alt="123" src="https://github.com/user-attachments/assets/1b3ca1e8-0b7d-47dd-802b-5eda9537ae13" />
+- **Thick solid lines**: Standard RoPE rotations for even dimensions (no F0 adaptation)
+- **Thick dashed lines**: Standard RoPE rotations for odd dimensions (no F0 adaptation)
+- **Thin solid lines**: F0-adapted RoPE rotations for even dimensions
+- **Thin dashed lines**: F0-adapted RoPE rotations for odd dimensions
 
-1000 steps with f0 / theta substitutions:
 
-<img width="470" alt="rhg" src="https://github.com/user-attachments/assets/ddfad0c5-21b5-4f1d-879f-ae41411444a8" />
+1. **Differences between thick and thin lines**: This shows how much the F0 information is modifying the standard position encodings. Larger differences indicate stronger F0 adaptation.
 
+2. **Pattern changes**: The standard RoPE (thick lines) show regular sinusoidal patterns, while the F0-adapted RoPE (thin lines) show variations that correspond to the audio's pitch contour.
 
+3. **Dimension-specific effects**: Compared across four subplots to see if F0 affects different dimensions differently.
 
+4. **Position-specific variations**: In standard RoPE, frequency decreases with dimension index, but F0 adaptation modify this pattern.
 
 
-By modulating the RoPE frequencies based on pitch (F0), we are essentially telling the model to pay attention to the acoustic features relate to sequence position in a way that's proportional to the voice characteristics.  This approach creates a more speech-aware positional representation that helps the model better understand the relationship between acoustic features and text.
+These visualizations help confirm that:
+- F0 information is being properly integrated into the model
+- The adaptation creates meaningful variations in the position encodings
+- The signal is strong enough to potentially help the model understand pitch-sensitive aspects of speech
+  
+----
+#### Domain-Specific ASR model. 
 
+#### freqs = (theta / 220.0) * 700 * (torch.pow(10, torch.linspace(0, 2595 * torch.log10(torch.tensor(1 + 8000/700)), dim // 2, device=device, dtype=dtype) / 2595) - 1) / 1000
 
+#### Static frequency's are perfectly fine for text models but not for NLP.
+-----
+
+1000 steps no f0:
+
+<img width="470" alt="123" src="https://github.com/user-attachments/assets/1b3ca1e8-0b7d-47dd-802b-5eda9537ae13" />
+
+1000 steps with f0 / theta substitutions:
+
+<img width="470" alt="rhg" src="https://github.com/user-attachments/assets/ddfad0c5-21b5-4f1d-879f-ae41411444a8" />
+
+https://huggingface.co/Sin2pi/Echo17/tensorboard?params=scalars#frame
+
+<img width="470" alt="4345" src="https://github.com/user-attachments/assets/b918fa73-fde0-40ca-9f09-bfc4e97e1ddf" />
+
+----
 
 The patterns below show how positions "see" each other in relation to theta and f0. 
 
@@ -65,10 +94,13 @@ Narrow bands: More focus on nearby positions (good for local patterns)
 <img width="470" alt="cc" src="https://github.com/user-attachments/assets/28d00fc5-2676-41ed-a971-e4d857af43f8"  />
 <img width="470" alt="cc2" src="https://github.com/user-attachments/assets/9089e806-966b-41aa-8793-bee03a6e6be1"  />
 
+
+Echos rotary implementation maps the perceptual properties of audio to the mathematical properties of the rotary embeddings, creating a more adaptive and context-aware representation system. Pitch is optionally extracted from audio in the data processing pipeline and can be used for an additional feature along with spectrograms and or used to inform the rotary and or pitch bias.
+
 Pitch bias
 
-The pitch bias implementation creates an attention bias matrix.
-This makes tokens with similar pitch attend to each other more.
+The pitch bias implementation creates an attention bias matrix:
+This makes tokens with similar pitch attend to each other more, which helps:
 
 - Track speaker consistency
 - Maintain coherent pitch patterns
@@ -79,4 +111,17 @@ The theoretical foundation:
 - Speech has inherent rhythmic and tonal patterns that correlate with semantic content
 - Varying the rotation frequency based on pitch creates a more speech-aware positional encoding
 
+--- 
+
+<img width="470" alt="cc2" src="https://github.com/user-attachments/assets/d52a48b1-8717-4d29-9452-cfdf43c92fe8"  />
+
+## The F0-Conditioned Rotation Mechanism
+
+The high gate usage validates the fundamental frequency conditioning approach:
+
+- Pitch-adaptive rotary embeddings are providing meaningful signal that the gates are actively utilizing
+- The decoder is learning to selectively attend to pitch-relevant patterns
+- The gates are functioning as a kind of "pitch-aware filter" that determines which information should flow through the network
+
+
 ---