Top 10 data gaps
Language and task pairs where at least two watched labs shipped models in the last 12 months, ranked by demand relative to commercially licensed supply. Languages driven by exactly the same releases are grouped into one entry.
The full grid
The 30 languages with the most lab demand. Darker means a bigger gap. Hover a cell for the numbers behind it.
Lab briefs
One card per watched lab: what they've released, how fast, in which languages, and which gaps from the map their releases point to.
This week
New lab models
New speech datasets
Popular datasets updated this week
Method & limits
Demand
Every model from the watchlist with an audio pipeline tag, created in the last 12 months. Each model adds 1 + log10(1 + downloads) to each language it's tagged with, divided by the square root of its language count so a 100-language checkpoint doesn't swamp targeted releases. Speech recognition demand includes automatic-speech-recognition and audio-text-to-text (audio LLMs); synthesis demand is text-to-speech. Downloads are Hugging Face's trailing 30-day count.
Supply
Every dataset tagged task_categories:automatic-speech-recognition or text-to-speech with a commercially usable licence (CC0, CC-BY, CC-BY-SA, Apache-2.0, MIT, ODC-BY, PDDL, BSD), weighted by its size category from 1 (under 1K rows) to 7 (100M–1B rows). Sizes above 1B rows are nearly always wrong tags, so they count as 1. Non-commercial, no-derivatives, research-only, "other" and missing licences are excluded. Datasets with fewer than 10 downloads and no likes are treated as placeholders and skipped.
Gap score
100 × demand ÷ (1 + supply). The top 10 only includes pairs with releases from at least two watched labs.
Watchlist
Limits
- Open releases only. Closed labs show up only through what they publish openly. OpenAI’s last open audio model on Hugging Face is Whisper large-v3-turbo (Oct 2024), and closed-only providers such as ElevenLabs don’t appear at all.
- Licence and size tags are self-reported by uploaders and are sometimes wrong. A CC-BY tag on a dataset of scraped audio doesn't make it clean to train on.
- Language tags are optional. Many Meta and ESPnet releases carry none, so they count in the lab briefs but not the grid.
- Row counts aren't hours. Size categories count rows, not audio duration or speaker diversity.
Pipeline
A GitHub Action runs fetch.py every day, writes data.json and a dated snapshot, and redeploys this page. The weekly diff compares today against the snapshot closest to seven days ago. Source on GitHub.