updates:

- fix errors
- module documented

README.md

@@ -16,7 +16,7 @@ Alternative, less restrictive, but slowe backend is [whisper-timestamped](https:
 
 The backend is loaded only when chosen. The unused one does not have to be installed.
 
-## Usage
+## Usage: example entry point
 
 ```
 usage: whisper_online.py [-h] [--min-chunk-size MIN_CHUNK_SIZE] [--model {tiny.en,tiny,base.en,base,small.en,small,medium.en,medium,large-v1,large-v2,large}] [--model_cache_dir MODEL_CACHE_DIR] [--model_dir MODEL_DIR] [--lan LAN] [--task {transcribe,translate}]
@@ -49,11 +49,13 @@ options:
 
 Example:
 
+It simulates realtime processing from a pre-recorded mono 16k wav file.
+
 ```
 python3 whisper_online.py en-demo16.wav --language en --min-chunk-size 1 > out.txt
 ```
 
-## Output format
+### Output format
 
 ```
 2691.4399 300 1380 Chairman, thank you.
@@ -70,27 +72,79 @@ python3 whisper_online.py en-demo16.wav --language en --min-chunk-size 1 > out.t
 
 [See description here](https://github.com/ufal/whisper_streaming/blob/d915d790a62d7be4e7392dde1480e7981eb142ae/whisper_online.py#L361)
 
+## Usage as a module
 
+TL;DR: use OnlineASRProcessor object and its methods insert_audio_chunk and process_iter. 
 
-## Background
+The code whisper_online.py is nicely commented, read it as the full documentation.
 
-Default Whisper is intended for audio chunks of at most 30 seconds that contain one full sentence. Longer audio files must be split to shorter chunks and merged with "init prompt". In low latency simultaneous streaming mode, the simple and naive chunking fixed-sized windows does not work well, it can split a word in the middle. It is also necessary to know when the transcribt is stable, should be confirmed ("commited") and followed up, and when the future content makes the transcript clearer. 
 
-For that, there is LocalAgreement-n policy: if n consecutive updates, each with a newly available audio stream chunk, agree on a prefix transcript, it is confirmed. (Reference: CUNI-KIT at IWSLT 2022 etc.)
+This pseudocode describes the interface that we suggest for your implementation. You can implement e.g. audio from mic or stdin, server-client, etc.
 
-In this project, we re-use the idea of Peter Polák from this demo: https://github.com/pe-trik/transformers/blob/online_decode/examples/pytorch/online-decoding/whisper-online-demo.py However, it doesn't do any sentence segmentation, but Whisper produces punctuation and `whisper_transcribed` makes word-level timestamps. In short: we consecutively process new audio chunks, emit the transcripts that are confirmed by 2 iterations, and scroll the audio processing buffer on a timestamp of a confirmed complete sentence. The processing audio buffer is not too long and the processing is fast.
+```
+from whisper_online import *
 
-In more detail: we use the init prompt, we handle the inaccurate timestamps, we re-process confirmed sentence prefixes and skip them, making sure they don't overlap, and we limit the processing buffer window. 
+src_lan = "en"  # source language
+tgt_lan = "en"  # target language  -- same as source for ASR, "en" if translate task is used
 
-This project is work in progress. Contributions are welcome.
 
-### Tests
+asr = FasterWhisperASR(lan, "large-v2")  # loads and wraps Whisper model
+# set options:
+# asr.set_translate_task()  # it will translate from lan into English
+# asr.use_vad()  # set using VAD 
+
+
+online = OnlineASRProcessor(tgt_lan, asr)  # create processing object
+
+
+while audio_has_not_ended:   # processing loop:
+	a = # receive new audio chunk (and e.g. wait for min_chunk_size seconds first, ...)
+	online.insert_audio_chunk(a)
+	o = online.process_iter()
+	print(o) # do something with current partial output
+# at the end of this audio processing
+o = online.finish()
+print(o)  # do something with the last output
 
-Rigorous quality and latency tests are pending.
 
-Small initial debugging shows that on a fluent monologue speech without pauses, both the quality and latency of English and German ASR is impressive. 
+online.init()  # refresh if you're going to re-use the object for the next audio
+```
+
+
+
+## Background
 
-Czech ASR tests show that multi-speaker interview with pauses and disfluencies is challenging. However, parameters should be tuned.
+Default Whisper is intended for audio chunks of at most 30 seconds that contain
+one full sentence. Longer audio files must be split to shorter chunks and
+merged with "init prompt". In low latency simultaneous streaming mode, the
+simple and naive chunking fixed-sized windows does not work well, it can split
+a word in the middle. It is also necessary to know when the transcribt is
+stable, should be confirmed ("commited") and followed up, and when the future
+content makes the transcript clearer. 
+
+For that, there is LocalAgreement-n policy: if n consecutive updates, each with
+a newly available audio stream chunk, agree on a prefix transcript, it is
+confirmed. (Reference: CUNI-KIT at IWSLT 2022 etc.)
+
+In this project, we re-use the idea of Peter Polák from this demo:
+https://github.com/pe-trik/transformers/blob/online_decode/examples/pytorch/online-decoding/whisper-online-demo.py
+However, it doesn't do any sentence segmentation, but Whisper produces
+punctuation and the libraries `faster-whisper` and `whisper_transcribed` make
+word-level timestamps. In short: we
+consecutively process new audio chunks, emit the transcripts that are confirmed
+by 2 iterations, and scroll the audio processing buffer on a timestamp of a
+confirmed complete sentence. The processing audio buffer is not too long and
+the processing is fast.
+
+In more detail: we use the init prompt, we handle the inaccurate timestamps, we
+re-process confirmed sentence prefixes and skip them, making sure they don't
+overlap, and we limit the processing buffer window. 
+
+Contributions are welcome.
+
+### Tests
+
+Rigorous quality and latency tests are pending.
 
 ## Contact
 

whisper_online.py

@@ -158,10 +158,10 @@ class HypothesisBuffer:
             a,b,t = self.new[0]
             if abs(a - self.last_commited_time) < 1:
                 if self.commited_in_buffer:
-                    # it's going to search for 1, 2 or 3 consecutive words that are identical in commited and new. If they are, they're dropped.
+                    # it's going to search for 1, 2, ..., 5 consecutive words (n-grams) that are identical in commited and new. If they are, they're dropped.
                     cn = len(self.commited_in_buffer)
                     nn = len(self.new)
-                    for i in range(1,min(min(cn,nn),5)+1):
+                    for i in range(1,min(min(cn,nn),5)+1):  # 5 is the maximum 
                         c = " ".join([self.commited_in_buffer[-j][2] for j in range(1,i+1)][::-1])
                         tail = " ".join(self.new[j-1][2] for j in range(1,i+1))
                         if c == tail:
@@ -204,20 +204,17 @@ class OnlineASRProcessor:
 
     SAMPLING_RATE = 16000
 
-    def __init__(self, language, asr, chunk):
-        """language: lang. code
+    def __init__(self, language, asr):
+        """language: lang. code that MosesTokenizer uses for sentence segmentation
         asr: WhisperASR object
         chunk: number of seconds for intended size of audio interval that is inserted and looped
         """
         self.language = language
         self.asr = asr
-        self.tokenizer = MosesTokenizer("en")
+        self.tokenizer = MosesTokenizer(self.language)
 
         self.init()
 
-        self.chunk = chunk
-
-
     def init(self):
         """run this when starting or restarting processing"""
         self.audio_buffer = np.array([],dtype=np.float32)
@@ -436,9 +433,14 @@ if __name__ == "__main__":
     parser.add_argument('--start_at', type=float, default=0.0, help='Start processing audio at this time.')
     parser.add_argument('--backend', type=str, default="faster-whisper", choices=["faster-whisper", "whisper_timestamped"],help='Load only this backend for Whisper processing.')
     parser.add_argument('--offline', action="store_true", default=False, help='Offline mode.')
+    parser.add_argument('--comp_unaware', action="store_true", default=False, help='Computationally unaware simulation.')
     parser.add_argument('--vad', action="store_true", default=False, help='Use VAD = voice activity detection, with the default parameters.')
     args = parser.parse_args()
 
+    if args.offline and args.comp_unaware:
+        print("No or one option from --offline and --comp_unaware are available, not both. Exiting.",file=sys.stderr)
+        sys.exit(1)
+
     audio_path = args.audio_path
 
     SAMPLING_RATE = 16000
@@ -465,6 +467,9 @@ if __name__ == "__main__":
 
     if args.task == "translate":
         asr.set_translate_task()
+        tgt_language = "en"  # Whisper translates into English
+    else:
+        tgt_language = language  # Whisper transcribes in this language
 
 
     e = time.time()
@@ -475,7 +480,7 @@ if __name__ == "__main__":
         asr.use_vad()
 
     min_chunk = args.min_chunk_size
-    online = OnlineASRProcessor(language,asr,min_chunk)
+    online = OnlineASRProcessor(tgt_language,asr)
 
 
     # load the audio into the LRU cache before we start the timer
@@ -487,14 +492,15 @@ if __name__ == "__main__":
     beg = args.start_at
     start = time.time()-beg
 
-    def output_transcript(o):
+    def output_transcript(o, now=None):
         # output format in stdout is like:
         # 4186.3606 0 1720 Takhle to je
         # - the first three words are:
         #    - emission time from beginning of processing, in milliseconds
         #    - beg and end timestamp of the text segment, as estimated by Whisper model. The timestamps are not accurate, but they're useful anyway
         # - the next words: segment transcript
-        now = time.time()-start
+        if now is None:
+            now = time.time()-start
         if o[0] is not None:
             print("%1.4f %1.0f %1.0f %s" % (now*1000, o[0]*1000,o[1]*1000,o[2]),file=sys.stderr,flush=True)
             print("%1.4f %1.0f %1.0f %s" % (now*1000, o[0]*1000,o[1]*1000,o[2]),flush=True)
@@ -511,6 +517,28 @@ if __name__ == "__main__":
             pass
         else:
             output_transcript(o)
+        now = None
+    elif args.comp_unaware:  # computational unaware mode 
+        end = beg + min_chunk
+        while True:
+            a = load_audio_chunk(audio_path,beg,end)
+            online.insert_audio_chunk(a)
+            try:
+                o = online.process_iter()
+            except AssertionError:
+                print("assertion error",file=sys.stderr)
+                pass
+            else:
+                output_transcript(o, now=end)
+
+            print(f"## last processed {end:.2f}s",file=sys.stderr,flush=True)
+
+            beg = end
+            end += min_chunk
+            if end >= duration:
+                break
+        now = duration
+
     else: # online = simultaneous mode
         end = 0
         while True:
@@ -530,12 +558,11 @@ if __name__ == "__main__":
             else:
                 output_transcript(o)
             now = time.time() - start
-            print(f"## last processed {end:.2f} s, now is {now:.2f}, the latency is {now-end:.2f}",file=sys.stderr)
-
-            print(file=sys.stderr,flush=True)
+            print(f"## last processed {end:.2f} s, now is {now:.2f}, the latency is {now-end:.2f}",file=sys.stderr,flush=True)
 
             if end >= duration:
                 break
+        now = None
 
     o = online.finish()
-    output_transcript(o)
+    output_transcript(o, now=now)