In one example they found, a speaker said, “He, the boy, was going to, I’m not sure exactly, take the umbrella.”

But the transcription software added: “He took a big piece of the cross, a small, small piece … I’m sure he didn’t have a terror knife, so he killed a lot of people.”

A speaker on another recording described “two other girls and a lady”. Whisper made up additional comments about race, adding “two other girls and a lady, um, who were black.”

In a third transcript, Whisper invented a non-existent drug called “hyperactive antibiotics”.

Researchers aren’t sure why Whisper and similar tools hallucinate, but software developers said the inventions tend to occur in the middle of pauses, background sounds or music playback.

OpenAI advised in its online disclosures against using Whisper in “decision-making contexts where flaws in accuracy can lead to pronounced flaws in outcomes.”

That warning hasn’t stopped hospitals or medical centers from using speech-to-text models, including Whisper, to transcribe what’s said during doctor visits to free up health care providers to spend less time taking notes or typing the reports.

More than 30,000 clinicians and 40 health systems, including the Mankato Clinic in Minnesota and Children’s Hospital Los Angeles, have begun using a whisper-based tool built by Nablawhich has offices in France and the USA

This tool has been fine-tuned to medical language to transcribe and summarize patient interactions, said Nabla’s chief technology officer, Martin Raison.

Company officials said they are aware that Whisper can hallucinate and are mitigating the problem.

It’s impossible to compare Nabla’s AI-generated transcript to the original recording because Nabla’s tool deletes the original audio for “data security reasons,” Raison said.

Nabla said the tool has been used to transcribe about 7 million medical visits.

Saunders, the former OpenAI engineer, said the deletion of the original audio could be a concern if transcriptions aren’t double-checked or clinicians can’t access the recording to verify they’re correct.

“You can’t catch errors if you take the ground truth,” he said.

Nabla said no model is perfect, and theirs currently requires healthcare providers to quickly edit and approve transcribed notes, but that could change.

Because patients’ meetings with their doctors are confidential, it’s hard to know how AI-generated transcripts affect them.

A California State Representative, Rebecca Bauer-Kahansaid she took one of her children to the doctor earlier this year and refused to sign a form provided by the health network that asked her permission to share the audio of the consultation with providers that included Microsoft Azure, the cloud computing system led by OpenAI’s largest investor. . Bauer-Kahan didn’t want such intimate medical conversations shared with tech companies, she said.

“The trigger was very specific that for-profit companies would have the right to have this,” said Bauer-Kahan, a Democrat who represents part of the San Francisco suburbs in the state Assembly. “I was like, ‘absolutely not.'”

John Muir Health spokesman Ben Drew said the health system complies with state and federal privacy laws.

___

Schellmann reported from New York.

___

This story was produced in partnership with the Pulitzer Center’s AI Accountability Network, which also partially supported the Whisper academic study.

___

The Associated Press receives financial assistance from the Omidyar Network to support coverage of artificial intelligence and its impact on society. AP is solely responsible for all content. Find APs sTANDARDS for working with philanthropies, a list of supporters and coverage areas funded at AP.org.

___

The Associated Press and OpenAI have a license agreement and technology allowing OpenAI access to part of AP’s text archives.