Include codecs needed by Jami VoIP/IM application
Context
When trying to build Jami using either of the provided FFMPEG platforms. I ended up with a build that was unable to do any audio calls. The reason being that FFMPEG lacks some protocols and possibly codecs even in the “ffmpeg-full” version of the library.
By looking through the build options used by upstream with the ones used in this image, I've found quite a few features enabled in their build of FFMPEG not used in the Freedesktop platform image or its “ffmpeg-full” extension:
General flags: --enable-zlib --enable-swscale --enable-bsfs
Audio/video codecs and muxers for calls
At least rtp
, g722
, webm
and mjpeg
are essential I believe. The more formats are enabled the better interop will work with different platforms and other SIP software (think embedded SIP handheld devices, etc). The H.26[3-5] bundle can likely be done without, if it is not acceptable at all for license reasons, but will break video acceleration in many cases.
Extra muxers (free): rtp,g722,webm,pcm_s16be,pcm_s16le
Extra demuxers (free): mjpeg,rtp,image_jpeg_pipe,image_png_pipe,image_webp_pipe,g722,pcm_mulaw,pcm_alaw,pcm_s16be,pcm_s16le
Extra parsers (free): vp9
Extra encoders (free): adpcm_g722,libspeex,libopus,mjpeg
Extra decoders (free): adpcm_g722,libspeex,libopus,mjpeg,mjpegb
Extra muxers (non-free): h263,h264,hevc
Extra demuxers (non-free): mjpeg_2000,h263,hevc
Extra parsers (non-free): h263
Extra encoders (non-free): libx264,mpeg4,h263p
Extra decoders (non-free): mpeg4
Additional flags (non-free): --enable-libx264
Audio decoders for ringtones
Can be done without, the selection of possible ringtone files will just be reduced for users. I have no idea what all of those PCM variants are for however.
Extra decoders (free): pcm_u24be,pcm_u24le,pcm_u32be,pcm_u32le,pcm_u8,pcm_f16le,pcm_f24le,pcm_f32be,pcm_f32le,pcm_f64be,pcm_f64le,pcm_s16be,pcm_s16be_planar,pcm_s16le,pcm_s16le_planar,pcm_s24be,pcm_s24le,pcm_s24le_planar,pcm_s32be,pcm_s32le,pcm_s32le_planar,pcm_s64be,pcm_s64le,pcm_s8,pcm_s8_planar,pcm_u16be,pcm_u16le
Extra decoders (non-free): aac
Image codecs for avatars and messages
Lossless JPEG and BMP should be pretty uncontroversial. Not including JPGE-2000 (tricky) would be acceptable.
Extra encoders (free): jpegls,bmp
Extra decoders (free): jpegls,bmp
Extra encoders (non-free): ljpeg
Extra decoders (non-free): jpeg2000
Filters
Extra filters: overlay,amix,amerge,aresample,format,aformat,fps,transpose,pad
Protocols
Of these rtp
and srtp
are definitely required, I am not sure about the others.
Extra protocols: crypto,rtp,srtp,tcp,udp,unix,pipe
Input device support
Video support is currently broken even with these enabled, but it without this video won't work for a while. (maybe PipeWire will get us there at some point, we'll see…)
Extra flags: --enable-indev=v4l2,xcbgrab
Hardware acceleration for 4k streaming
Allowing for high-quality video calls, having support for both hardware decoding and encoding is essential to provide a seamless experience. I am not sure how important support for the NVCodec
API is in practice and if we could do without on NVidia drivers.
Extra flags (generic): --enable-vdpau --enable-vaapi
Extra hardware accelerators (generic): mpeg4_vdpau,mpeg4_vaapi,h263_vaapi,vp8_vaapi,mjpeg_vaapi
Extra encoders (generic): h264_vaapi,vp8_vaapi,mjpeg_vaapi
Extra flags (NVCodec, x86 only): --enable-cuvid --enable-ffnvcodec --enable-nvdec --enable-nvenc
Extra hardware accelerators (NVCodec, x86 only): h264_nvdec,hevc_nvdec,vp8_nvdec,mjpeg_nvdev
Extra encoders (NVCodec, x86 only): h264_nvenc,hevc_nvenc
Description
-
Get upstream to comment on the above and point out any mistakes I may have made. -
Discuss which of these items would be eligible (both legally and practially) for inclusion into the “ffmpeg-full” runtime extension or the base runtime. (Needs input from SDK maintainers!) -
Prepare a MR with the items that have been agreed on.
Acceptance Criteria
The Jami app can do audio and (if we find why else it's broken right now) video calls using the “ffmpeg-full” extension provided by the Freedesktop SDK, rather then compiling its own.
Bonus points if it also works with the base Freedesktop Runtime image, even it it means that codec selection and video acceleration is restricted.