lm-evaluation-harness
Don’t want to install lm-evaluation-harness? Drop the file here.
The first 20 minutes of any recording are transcribed free: a recording up to 20 minutes transcribes in full, a longer one is previewed, and a pack from $6 (500 minutes) transcribes the rest and opens every export, and credits never expire.
Eval harness now covering audio-LLM benchmarks.
Best for standardized eval for speech-LLMs and text-only baselines. Pricing: free.
What it is
The reference eval harness for LLMs and growing audio-LLM coverage. MIT.
Watch out for: Audio tasks newer than text tasks.
Install / use
pip install lm-evalWhat people actually do with lm-evaluation-harness-style transcription
The tool is the means. These are the jobs — each one priced at published rates, each one wired up on its own page.
Features
| Speaker diarization | No |
| Word-level timestamps | Yes |
| Streaming / real-time | No |
| Languages supported | 99 |
| HIPAA eligible | No |
Links
lm-evaluation-harness vs Whipscribe
| Feature | lm-evaluation-harness | Whipscribe |
|---|---|---|
| Category | Open source | Transcription APIs |
| Pricing | free | $6–$24 one-time packs (credits never expire) · first 20 minutes free |
| Speaker identification | No | Named in the summary, key quotes and chapters |
| Word timestamps | Yes | Yes |
| Streaming | No | No |
| Languages | 99 | 99 |
| Platforms | Linux | Web, API, MCP |
Alternatives to lm-evaluation-harness
Frequently asked about lm-evaluation-harness
Is lm-evaluation-harness free?
Yes. lm-evaluation-harness is free and open source.
Does lm-evaluation-harness work on Mac, Windows and Linux?
lm-evaluation-harness runs on Linux. It is not listed for other platforms; check the vendor for anything beyond Linux.
How do I install lm-evaluation-harness?
With pip: `pip install lm-eval`. You need Python and, for most audio tools, ffmpeg on your PATH first.
How many languages does lm-evaluation-harness support?
lm-evaluation-harness lists 99 languages. That is the Whisper-family multilingual set, so quality varies by language — English and the large European languages are strongest.
Does lm-evaluation-harness identify different speakers?
No. lm-evaluation-harness does not label speakers; a conversation comes back as one continuous text. If you need speaker labels, that is a separate tool or a different service.
Does lm-evaluation-harness give word-level timestamps?
Yes — lm-evaluation-harness produces timing per word, which is what subtitle cues and clip boundaries need.
Can lm-evaluation-harness transcribe live audio?
No — lm-evaluation-harness works on finished files, not a live stream.
Is lm-evaluation-harness HIPAA compliant?
lm-evaluation-harness does not claim HIPAA compliance. For PHI, run it on infrastructure you control or choose a service that will sign a BAA.
What are the limitations of lm-evaluation-harness?
Audio tasks newer than text tasks.
Who is lm-evaluation-harness best for?
Standardized eval for speech-LLMs and text-only baselines.
Does lm-evaluation-harness run offline?
lm-evaluation-harness runs on your own machine — it is open source, so nothing leaves the computer unless you configure it to.
Can I use lm-evaluation-harness in the browser?
No — lm-evaluation-harness is a Linux application, not a web app. You install it rather than sign in to it.
Where is the lm-evaluation-harness source code?
On GitHub at github.com/EleutherAI/lm-evaluation-harness. It is maintained by EleutherAI.
What kind of tool is lm-evaluation-harness?
In this directory lm-evaluation-harness is filed under evaluation, benchmark as a open-source tool.
What are the alternatives to lm-evaluation-harness?
There is a side-by-side page at /tools/lm-eval-harness-alternatives comparing lm-evaluation-harness with the closest tools in the same category on price, platform and features.
Whipscribe is a hosted transcription service. If you want transcripts without running infrastructure, paste a URL or drop a file in the form below — you'll have a transcript in seconds.
Transcribe your recordings
Running lm-evaluation-harness yourself? Send the same file here and compare the transcript. Transcribe any language, any recording · Transcription services from $0.29 an hour · Transcribe from any meeting or recording platform · How to transcribe, step by step
Explore
All transcription tools · By language · By platform · By file format · How-to guides · Transcription services · Transcription API · Pricing
Long recording to get through?
One file up to 10 hours, as an upload or a link: see how to transcribe an MP3 of any length, transcribing a podcast episode and the SRT generator.
Comparing tools before you pay?
If you would rather pay a service than run software yourself: see an honest roundup of the best transcription services, the cheapest way to transcribe an hour of audio and our transcription service with prices and turnaround.