Sync all skills and memories 2026-04-14 07:27

2026-04-14 07:27:20 +09:00
parent 516bb44fe6
commit 1eba2bca95
386 changed files with 167655 additions and 0 deletions
--- a/skills/mlops/models/whisper/SKILL.md
+++ b/skills/mlops/models/whisper/SKILL.md
@@ -0,0 +1,320 @@
+---
+name: whisper
+description: OpenAI's general-purpose speech recognition model. Supports 99 languages, transcription, translation to English, and language identification. Six model sizes from tiny (39M params) to large (1550M params). Use for speech-to-text, podcast transcription, or multilingual audio processing. Best for robust, multilingual ASR.
+version: 1.0.0
+author: Orchestra Research
+license: MIT
+dependencies: [openai-whisper, transformers, torch]
+metadata:
+  hermes:
+    tags: [Whisper, Speech Recognition, ASR, Multimodal, Multilingual, OpenAI, Speech-To-Text, Transcription, Translation, Audio Processing]
+
+---
+
+# Whisper - Robust Speech Recognition
+
+OpenAI's multilingual speech recognition model.
+
+## When to use Whisper
+
+**Use when:**
+- Speech-to-text transcription (99 languages)
+- Podcast/video transcription
+- Meeting notes automation
+- Translation to English
+- Noisy audio transcription
+- Multilingual audio processing
+
+**Metrics**:
+- **72,900+ GitHub stars**
+- 99 languages supported
+- Trained on 680,000 hours of audio
+- MIT License
+
+**Use alternatives instead**:
+- **AssemblyAI**: Managed API, speaker diarization
+- **Deepgram**: Real-time streaming ASR
+- **Google Speech-to-Text**: Cloud-based
+
+## Quick start
+
+### Installation
+
+```bash
+# Requires Python 3.8-3.11
+pip install -U openai-whisper
+
+# Requires ffmpeg
+# macOS: brew install ffmpeg
+# Ubuntu: sudo apt install ffmpeg
+# Windows: choco install ffmpeg
+```
+
+### Basic transcription
+
+```python
+import whisper
+
+# Load model
+model = whisper.load_model("base")
+
+# Transcribe
+result = model.transcribe("audio.mp3")
+
+# Print text
+print(result["text"])
+
+# Access segments
+for segment in result["segments"]:
+    print(f"[{segment['start']:.2f}s - {segment['end']:.2f}s] {segment['text']}")
+```
+
+## Model sizes
+
+```python
+# Available models
+models = ["tiny", "base", "small", "medium", "large", "turbo"]
+
+# Load specific model
+model = whisper.load_model("turbo")  # Fastest, good quality
+```
+
+| Model | Parameters | English-only | Multilingual | Speed | VRAM |
+|-------|------------|--------------|--------------|-------|------|
+| tiny | 39M | ✓ | ✓ | ~32x | ~1 GB |
+| base | 74M | ✓ | ✓ | ~16x | ~1 GB |
+| small | 244M | ✓ | ✓ | ~6x | ~2 GB |
+| medium | 769M | ✓ | ✓ | ~2x | ~5 GB |
+| large | 1550M | ✗ | ✓ | 1x | ~10 GB |
+| turbo | 809M | ✗ | ✓ | ~8x | ~6 GB |
+
+**Recommendation**: Use `turbo` for best speed/quality, `base` for prototyping
+
+## Transcription options
+
+### Language specification
+
+```python
+# Auto-detect language
+result = model.transcribe("audio.mp3")
+
+# Specify language (faster)
+result = model.transcribe("audio.mp3", language="en")
+
+# Supported: en, es, fr, de, it, pt, ru, ja, ko, zh, and 89 more
+```
+
+### Task selection
+
+```python
+# Transcription (default)
+result = model.transcribe("audio.mp3", task="transcribe")
+
+# Translation to English
+result = model.transcribe("spanish.mp3", task="translate")
+# Input: Spanish audio → Output: English text
+```
+
+### Initial prompt
+
+```python
+# Improve accuracy with context
+result = model.transcribe(
+    "audio.mp3",
+    initial_prompt="This is a technical podcast about machine learning and AI."
+)
+
+# Helps with:
+# - Technical terms
+# - Proper nouns
+# - Domain-specific vocabulary
+```
+
+### Timestamps
+
+```python
+# Word-level timestamps
+result = model.transcribe("audio.mp3", word_timestamps=True)
+
+for segment in result["segments"]:
+    for word in segment["words"]:
+        print(f"{word['word']} ({word['start']:.2f}s - {word['end']:.2f}s)")
+```
+
+### Temperature fallback
+
+```python
+# Retry with different temperatures if confidence low
+result = model.transcribe(
+    "audio.mp3",
+    temperature=(0.0, 0.2, 0.4, 0.6, 0.8, 1.0)
+)
+```
+
+## Command line usage
+
+```bash
+# Basic transcription
+whisper audio.mp3
+
+# Specify model
+whisper audio.mp3 --model turbo
+
+# Output formats
+whisper audio.mp3 --output_format txt     # Plain text
+whisper audio.mp3 --output_format srt     # Subtitles
+whisper audio.mp3 --output_format vtt     # WebVTT
+whisper audio.mp3 --output_format json    # JSON with timestamps
+
+# Language
+whisper audio.mp3 --language Spanish
+
+# Translation
+whisper spanish.mp3 --task translate
+```
+
+## Batch processing
+
+```python
+import os
+
+audio_files = ["file1.mp3", "file2.mp3", "file3.mp3"]
+
+for audio_file in audio_files:
+    print(f"Transcribing {audio_file}...")
+    result = model.transcribe(audio_file)
+
+    # Save to file
+    output_file = audio_file.replace(".mp3", ".txt")
+    with open(output_file, "w") as f:
+        f.write(result["text"])
+```
+
+## Real-time transcription
+
+```python
+# For streaming audio, use faster-whisper
+# pip install faster-whisper
+
+from faster_whisper import WhisperModel
+
+model = WhisperModel("base", device="cuda", compute_type="float16")
+
+# Transcribe with streaming
+segments, info = model.transcribe("audio.mp3", beam_size=5)
+
+for segment in segments:
+    print(f"[{segment.start:.2f}s -> {segment.end:.2f}s] {segment.text}")
+```
+
+## GPU acceleration
+
+```python
+import whisper
+
+# Automatically uses GPU if available
+model = whisper.load_model("turbo")
+
+# Force CPU
+model = whisper.load_model("turbo", device="cpu")
+
+# Force GPU
+model = whisper.load_model("turbo", device="cuda")
+
+# 10-20× faster on GPU
+```
+
+## Integration with other tools
+
+### Subtitle generation
+
+```bash
+# Generate SRT subtitles
+whisper video.mp4 --output_format srt --language English
+
+# Output: video.srt
+```
+
+### With LangChain
+
+```python
+from langchain.document_loaders import WhisperTranscriptionLoader
+
+loader = WhisperTranscriptionLoader(file_path="audio.mp3")
+docs = loader.load()
+
+# Use transcription in RAG
+from langchain_chroma import Chroma
+from langchain_openai import OpenAIEmbeddings
+
+vectorstore = Chroma.from_documents(docs, OpenAIEmbeddings())
+```
+
+### Extract audio from video
+
+```bash
+# Use ffmpeg to extract audio
+ffmpeg -i video.mp4 -vn -acodec pcm_s16le audio.wav
+
+# Then transcribe
+whisper audio.wav
+```
+
+## Best practices
+
+1. **Use turbo model** - Best speed/quality for English
+2. **Specify language** - Faster than auto-detect
+3. **Add initial prompt** - Improves technical terms
+4. **Use GPU** - 10-20× faster
+5. **Batch process** - More efficient
+6. **Convert to WAV** - Better compatibility
+7. **Split long audio** - <30 min chunks
+8. **Check language support** - Quality varies by language
+9. **Use faster-whisper** - 4× faster than openai-whisper
+10. **Monitor VRAM** - Scale model size to hardware
+
+## Performance
+
+| Model | Real-time factor (CPU) | Real-time factor (GPU) |
+|-------|------------------------|------------------------|
+| tiny | ~0.32 | ~0.01 |
+| base | ~0.16 | ~0.01 |
+| turbo | ~0.08 | ~0.01 |
+| large | ~1.0 | ~0.05 |
+
+*Real-time factor: 0.1 = 10× faster than real-time*
+
+## Language support
+
+Top-supported languages:
+- English (en)
+- Spanish (es)
+- French (fr)
+- German (de)
+- Italian (it)
+- Portuguese (pt)
+- Russian (ru)
+- Japanese (ja)
+- Korean (ko)
+- Chinese (zh)
+
+Full list: 99 languages total
+
+## Limitations
+
+1. **Hallucinations** - May repeat or invent text
+2. **Long-form accuracy** - Degrades on >30 min audio
+3. **Speaker identification** - No diarization
+4. **Accents** - Quality varies
+5. **Background noise** - Can affect accuracy
+6. **Real-time latency** - Not suitable for live captioning
+
+## Resources
+
+- **GitHub**: https://github.com/openai/whisper ⭐ 72,900+
+- **Paper**: https://arxiv.org/abs/2212.04356
+- **Model Card**: https://github.com/openai/whisper/blob/main/model-card.md
+- **Colab**: Available in repo
+- **License**: MIT
+
+