Audio Workflows with Microsoft Foundry

Two ways to reach speech models

foundryR can use audio models in two places:

• Azure OpenAI deployments on your main resource – whisper for transcription and translation, and a text-to-speech model such as gpt-4o-mini-tts for synthesis. These reuse your main endpoint and key and are what the examples below use. Classic whisper is exposed only on the deployment path, so pass api = "deployment".
• A dedicated Speech (LLM Speech) resource for MAI-Transcribe models. This chunk is illustrative and is not run:

# Only needed for MAI-Transcribe on a dedicated Speech resource:
foundry_set_speech_endpoint(Sys.getenv("AZURE_FOUNDRY_SPEECH_ENDPOINT"))
foundry_set_speech_key("your-speech-key")

A real, public-domain sample

The examples below use a short excerpt from John F. Kennedy’s 1961 inaugural address (“And so, my fellow Americans…”). This clip ships with the package and is the de facto “hello, world” of open-source speech recognition, so the transcript is easy to check against a recording everyone knows.

sample_audio <- system.file("extdata/samples/jfk.wav", package = "foundryR")
basename(sample_audio)
#> [1] "jfk.wav"

Transcribe an audio file

foundry_transcribe() returns one row per file. The text column holds the transcript and the phrases list-column holds segment-level timing. We use the whisper deployment on the main resource; because classic whisper lives on the deployment path we pass api = "deployment", and response_format = "verbose_json" asks the service for per-segment timing.

transcript <- foundry_transcribe(
  sample_audio,
  service = "openai",
  model = "whisper",
  api = "deployment",
  response_format = "verbose_json"
)

transcript$text
#> [1] "And so my fellow Americans, ask not what your country can do for you, ask what you can do for your country."

The segment timing lives in the phrases list-column, one row per recognized segment. A short clip like this one is a single segment; longer recordings return many:

head(transcript$phrases[[1]])
#> # A tibble: 1 × 7
#>   text                    locale offset_ms duration_ms confidence speaker words 
#>   <chr>                   <chr>      <int>       <int>      <dbl> <chr>   <list>
#> 1 " And so my fellow Ame… <NA>           0       11000         NA <NA>    <list>

Synthesize speech

foundry_speak() writes binary audio to disk and returns the file path and byte count – handy for experiment stimuli, accessibility assets, and demos. Use your text-to-speech deployment name for model. The examples use temporary files and remove them after use; choose an explicit path in your own workflow for audio you want to keep.

speech_path <- tempfile(fileext = ".mp3")
speech <- foundry_speak(
  "Hello, world.",
  model = "gpt-4o-mini-tts",
  voice = "alloy",
  path = speech_path
)

speech[, c("bytes", "model", "voice", "format")]
#> # A tibble: 1 × 4
#>   bytes model           voice format
#>   <int> <chr>           <chr> <chr> 
#> 1 25728 gpt-4o-mini-tts alloy mp3

Those bytes are the real audio the model returned. Play them here when the suggested base64enc package is installed:

Translate multilingual recordings

Use foundry_translate_audio() when you want an analysis corpus in a common language. To keep the example fully reproducible we first synthesize a short Spanish clip, then translate it to English with whisper – both are real API calls.

spanish_path <- tempfile(fileext = ".mp3")
spanish_clip <- foundry_speak(
  "La reunion fue muy util.",
  model = "gpt-4o-mini-tts",
  voice = "alloy",
  path = spanish_path
)

Listen to the synthesized Spanish input:

Now translate it to English with the whisper deployment:

translation <- foundry_translate_audio(
  spanish_clip$path,
  service = "openai",
  model = "whisper",
  api = "deployment"
)

translation$text
#> [1] "The meeting was very useful."

Notes for researchers

• Inspect head(transcript$phrases[[1]]) before processing long recordings so you know the segment structure your coding scheme has to handle.
• Keep raw audio out of your project repository; store transcripts and IDs.
• Request response_format = "verbose_json" to get segment-level timing from whisper; the default format returns the transcript text only.