Add audio rendering support

[danielweinmann]

Summary

This PR adds audio rendering support to toucan. Audio tracks in OTIO timelines are currently ignored — this adds a complete audio pipeline that runs parallel to the existing video pipeline.

I'm new to the project and to C++ in general, so I'd really appreciate any guidance on how to better follow the existing patterns, naming conventions, or architectural decisions. Happy to rework anything that doesn't fit the project's direction.

What it does

• Decodes audio from source media files using FFmpeg (libavcodec + libswresample), supporting all common formats (AAC, MP3, PCM, FLAC, etc.)
• Walks OTIO timeline audio tracks — handles clips, gaps, transitions (crossfade), LinearTimeWarp, and multi-track mixing
• Extracts audio from video clips — when a video clip references a media file that contains audio (the common case), the audio is rendered alongside the video
• Muxes audio+video in output containers (MP4, MOV) using the existing FFmpegWrite pipeline
• Writes separate audio files via a new -afile option (useful when piping video to stdout)

New CLI options

Option	Description	Default
-acodec <CODEC>	Audio codec (pcm_s16le, pcm_s24le, pcm_f32le, flac, aac)	pcm_s16le
-arate <RATE>	Audio sample rate	48000
-achannels <N>	Audio channel count	2
-afile <PATH>	Write audio to a separate file	—
-no_audio	Disable audio rendering entirely	—

Architecture

The audio pipeline mirrors the video pipeline:

Video	Audio (new)
ImageGraph	AudioGraph
ffmpeg::Read	ffmpeg::AudioRead
ffmpeg::Write (video only)	ffmpeg::Write (video+audio) + ffmpeg::AudioWrite
IImageNode / ImageBuf	AudioBuffer (interleaved float32)

AudioGraph follows the same OTIO timeline walking pattern as ImageGraph — same _track(), _item(), _timeWarps() structure. Audio is processed in frame-aligned chunks (one video frame's worth of samples per call).

Known limitations

• LinearTimeWarp changes playback position correctly but doesn't pitch-shift — the audio for a 0.5x slow-mo section will sound choppy rather than pitch-shifted. Proper time-stretching would need a dedicated algorithm.
• Audio effects are limited to volume adjustment via OTIO Effect metadata. No EQ, reverb, etc.

Files changed

New files (7):

• lib/toucanRender/AudioBuffer.h — simple audio buffer struct
• lib/toucanRender/FFmpegAudioRead.h/cpp — FFmpeg audio decoder + libswresample
• lib/toucanRender/AudioGraph.h/cpp — OTIO timeline walker for audio
• lib/toucanRender/FFmpegAudioWrite.h/cpp — standalone audio file writer

Modified files (10):

• lib/toucanRender/FFmpeg.h/cpp — AudioCodec enum
• lib/toucanRender/FFmpegWrite.h/cpp — audio stream in combined muxer
• bin/toucan-render/App.h/cpp — CLI options + render loop
• lib/toucanRender/TimelineAlgo.h/cpp — getAudioClips() helper
• lib/toucanRender/CMakeLists.txt — new sources + libswresample
• cmake/SuperBuild/BuildFFmpeg.cmake — AAC in minimal build

Test plan

• All 19 existing tests pass (backward compatible)
• Video-only .otio files render with no audio stream (unchanged behavior)
• MP4 with audio renders to MOV with both video+audio streams
• Audio levels match source (peak within 1 dB)
• No periodic silence blocks or clicking artifacts
• -no_audio flag correctly suppresses audio
• -afile writes separate audio file alongside muxed output
• Test on Linux and Windows (only tested on macOS arm64 so far)

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: danielweinmann / name: Daniel Weinmann (3b9bf3c)

[darbyjohnston]

Hi and thanks for the contribution! You mentioned you are relatively new to C++ but this looks like a fairly substantial change. I took a quick look and overall it looks good, I should have more time to review it next week. I can also try testing it out on Windows.

I was curious about this line in App.cpp:

const int samplesPerFrame = static_cast<int>(
            std::round(static_cast<double>(audioSampleRate) / timeRange.duration().rate()));

What happens when the frame rate is not a multiple of the sample rate (like 29.97)?

[darbyjohnston]

Hi, and sorry for the long delay! I think this PR still needs some work for the frame rate issue I mentioned, are you still interested in working on it?

[danielweinmann]

Hi, and sorry for the long delay! I think this PR still needs some work for the frame rate issue I mentioned, are you still interested in working on it?

Hey, @darbyjohnston! Sorry, I had to move on to other projects and won't have time to work on this in the near future. Feel free to take it over or close it. Thank you for your time, and sorry for not being able to take this one across the finish line.

[darbyjohnston]

No problem, I will add a couple of fixes and merge the work keeping your attribution intact. Thanks for creating the PR.

[darbyjohnston]

Replaced with: #65