GStreamer Compositing for Streaming H.264 Video

Gateworks recently featured a blog in which 8 video cameras were connected to a Gateworks Ventana SBC and then displayed on a HDMI monitor. This is useful for localized applications. For remote applications there is another solution.

Remote applications require streaming the multiple video streams over the network (Ethernet or WiFi). For bandwidth efficiency, all camera inputs can be joined together into a single frame and then transmitted across the network.


To join all the streams into a single frame, a software element of GStreamerĀ called a compositor is used. Older versions of the compositor relied on the CPU and caused choppy video. Gateworks recently started using gstreamer-imx which contains a hardware accelerated compositor which is far superior. With this compositor, each stream can be positioned on the frame and then linked to a RTSP stream in the H.264 format.

An example is shown with two Gateworks Ventana SBCs that are on the same network.

Start the following pipeline on the SBC with the cameras connected:

gst-variable-rtsp-server -u \
 "imxv4l2videosrc device=/dev/video2 queue-size=55 ! queue2 ! c.sink_0 \
 imxv4l2videosrc device=/dev/video3 queue-size=55 ! queue2 ! c.sink_1 \
 imxv4l2videosrc device=/dev/video4 queue-size=55 ! queue2 ! c.sink_2 \
 imxv4l2videosrc device=/dev/video5 queue-size=55 ! queue2 ! c.sink_3 \
 imxv4l2videosrc device=/dev/video6 queue-size=55 ! queue2 ! c.sink_4 \
 imxv4l2videosrc device=/dev/video7 queue-size=55 ! queue2 ! c.sink_5 \
 imxv4l2videosrc device=/dev/video8 queue-size=55 ! queue2 ! c.sink_6 \
 imxv4l2videosrc device=/dev/video9 queue-size=55 ! queue2 ! c.sink_7 \
 imxg2dcompositor name=c background-color=0xffffff \
 sink_0::xpos=0 sink_0::ypos=0 sink_0::width=320 sink_0::height=360 sink_0::fill_color=0x00000000 \
 sink_1::xpos=320 sink_1::ypos=0 sink_1::width=320 sink_1::height=360 sink_1::fill_color=0x00000000 \
 sink_2::xpos=640 sink_2::ypos=0 sink_2::width=320 sink_2::height=360 sink_2::fill_color=0x00000000 \
 sink_3::xpos=960 sink_3::ypos=0 sink_3::width=320 sink_3::height=360 sink_3::fill_color=0x00000000 \
 sink_4::xpos=0 sink_4::ypos=360 sink_4::width=320 sink_4::height=360 sink_4::fill_color=0x00000000 \
 sink_5::xpos=320 sink_5::ypos=360 sink_5::width=320 sink_5::height=360 sink_5::fill_color=0x00000000 \
 sink_6::xpos=640 sink_6::ypos=360 sink_6::width=320 sink_6::height=360 sink_6::fill_color=0x00000000 \
 sink_7::xpos=960 sink_7::ypos=360 sink_7::width=320 sink_7::height=360 sink_7::fill_color=0x00000000 \
 ! queue2 ! video/x-raw, width=1280, height=720 ! imxipuvideotransform \
 ! imxvpuenc_h264 bitrate=20000 ! rtph264pay name=pay0 pt=96"

Then, on the receiving board that is connected to an HDMI display, start the following pipeline with the actual IP address (example IP below) of the board with the cameras:

gst-launch-1.0 rtspsrc location=rtsp:// latency=100 ! \
queue2 ! decodebin ! autovideosink

For more information andĀ code examples, please visit the related Gateworks Software Wiki links below: