GStreamer Compositing for Streaming H.264 Video

Gateworks recently featured a blog in which 8 video cameras were connected to a Gateworks Ventana SBC and then displayed on a HDMI monitor. This is useful for localized applications. For remote applications there is another solution.

Remote applications require streaming the multiple video streams over the network (Ethernet or WiFi). For bandwidth efficiency, all camera inputs can be joined together into a single frame and then transmitted across the network.

streamingdiagram3

To join all the streams into a single frame, a software element of GStreamer called a compositor is used. Older versions of the compositor relied on the CPU and caused choppy video. Gateworks recently started using gstreamer-imx which contains a hardware accelerated compositor which is far superior. With this compositor, each stream can be positioned on the frame and then linked to a RTSP stream in the H.264 format.

An example is shown with two Gateworks Ventana SBCs that are on the same network.

Start the following pipeline on the SBC with the cameras connected:

gst-variable-rtsp-server -u \
 "imxv4l2videosrc device=/dev/video2 queue-size=55 ! queue2 ! c.sink_0 \
 imxv4l2videosrc device=/dev/video3 queue-size=55 ! queue2 ! c.sink_1 \
 imxv4l2videosrc device=/dev/video4 queue-size=55 ! queue2 ! c.sink_2 \
 imxv4l2videosrc device=/dev/video5 queue-size=55 ! queue2 ! c.sink_3 \
 imxv4l2videosrc device=/dev/video6 queue-size=55 ! queue2 ! c.sink_4 \
 imxv4l2videosrc device=/dev/video7 queue-size=55 ! queue2 ! c.sink_5 \
 imxv4l2videosrc device=/dev/video8 queue-size=55 ! queue2 ! c.sink_6 \
 imxv4l2videosrc device=/dev/video9 queue-size=55 ! queue2 ! c.sink_7 \
 imxg2dcompositor name=c background-color=0xffffff \
 sink_0::xpos=0 sink_0::ypos=0 sink_0::width=320 sink_0::height=360 sink_0::fill_color=0x00000000 \
 sink_1::xpos=320 sink_1::ypos=0 sink_1::width=320 sink_1::height=360 sink_1::fill_color=0x00000000 \
 sink_2::xpos=640 sink_2::ypos=0 sink_2::width=320 sink_2::height=360 sink_2::fill_color=0x00000000 \
 sink_3::xpos=960 sink_3::ypos=0 sink_3::width=320 sink_3::height=360 sink_3::fill_color=0x00000000 \
 sink_4::xpos=0 sink_4::ypos=360 sink_4::width=320 sink_4::height=360 sink_4::fill_color=0x00000000 \
 sink_5::xpos=320 sink_5::ypos=360 sink_5::width=320 sink_5::height=360 sink_5::fill_color=0x00000000 \
 sink_6::xpos=640 sink_6::ypos=360 sink_6::width=320 sink_6::height=360 sink_6::fill_color=0x00000000 \
 sink_7::xpos=960 sink_7::ypos=360 sink_7::width=320 sink_7::height=360 sink_7::fill_color=0x00000000 \
 ! queue2 ! video/x-raw, width=1280, height=720 ! imxipuvideotransform \
 ! imxvpuenc_h264 bitrate=20000 ! rtph264pay name=pay0 pt=96"

Then, on the receiving board that is connected to an HDMI display, start the following pipeline with the actual IP address (example IP below) of the board with the cameras:

gst-launch-1.0 rtspsrc location=rtsp://172.24.10.210:9099/stream latency=100 ! \
queue2 ! decodebin ! autovideosink

For more information and code examples, please visit the related Gateworks Software Wiki links below:

 

Capturing 8 Video Inputs on Gateworks Ventana SBCs

Gateworks would like to introduce software support for the AVC8000nano Mini-PCIe card on the Ventana Single Board Computers.

final_screen_camerasFigure 1: Screen capture of 8 analog cameras displayed on a monitor using the Gateworks Ventana SBC

Many applications, such as surveillance, require multiple analog video inputs from cameras for monitoring. These cameras can then be displayed on an HDMI monitor or streamed over the network.

final_camerasonboardFigure 2: Eight analog video cameras mounted in a circular fashion for a panoramic capture

Gateworks has added driver support for the AVC8000nano in it’s Yocto Linux board support package. This driver support will reveal 8 video interfaces in Linux, such as /dev/video0, /dev/video1, etc. These video interfaces can then be accessed using GStreamer.

final_pictureofcardFigure 3: The AVC8000nano installed on a Gateworks Ventana SBC with 8 analog cameras connected 

The OpenCV software library could be used to stitch the different inputs together to create a seamless panorama.

For more information and example GStreamer pipelines on how to use this card, please visit our Software Wiki Page:

AVC8000nano Software Wiki Page

Future blog posts will cover streaming multiple camera inputs as well as adaptive bitrate streaming! Please subscribe to the Gateworks blog to the right!

 

 

i.MX6 GStreamer-imx Plugins – Tutorial & Example Pipelines

Gateworks would like to announce the support of the GStreamer-imx plugins starting with Yocto 1.8 on the Ventana family of Single Board Computers.

gstreamerpipeline

Gateworks, the leading supplier of Powerful ARM based Single Board Computer solutions using the Freescale i.MX6, has invested countless engineering hours researching and mastering GStreamer for the i.MX series of processors. Gateworks would like to share this GStreamer research with the rest of the i.MX community of developers!

There are two main versions of GStreamer used on the i.MX6 processor:0.10 and 1.0. Version 1.0 is now the latest standard.

The i.MX6 processor has hardware blocks such as the IPU (image processing unit), VPU (video processing unit), and GPU (graphical processing unit). The main advantage of using these hardware blocks is that there is no CPU cost for decoding/encoding a stream because another hardware block in the i.MX6 takes care of it. This leaves the CPU free to deal with other programs etc.

The GStreamer app works with ‘plugins’. A plugin comprises of elements that can do work on a media stream. For example, the imxvpudec is a VPU based decoder plugin.

This post is specifically about the plugins. There are different versions and sets of plugins available.

Gateworks has chosen to use the GStreamer-imx plugins for the following reasons:

  • Open Source Development model: The project is on github and is very active
  • The main developer has been a GStreamer contributer for some time now and is very active in the GStreamer community
  • The source is very well documented and easy to follow
  • Things are done in a very standard GStreamer way

Plugin List

For a thorough description of each plugin, why and how to use it, please visit the Gateworks Software Wiki GStreamer Page

The following is a list of plugins provided by the latest version of gstreamer-imx (0.11.0)

Type Plugin(s) Element(s) Comments
Audio Decoder imxaudio imxuniaudiodec Uses i.MX uniaudio codecs for decoding
Audio Encoder imxaudio imxmp3audioenc Uses i.MX for MP3 audio encoding
Device Sources imxv4l2video imxv4l2videosrc Get camera source via v4l2
Video Decoder imxvpu imxvpudec VPU Based decoder
Video Encoder imxvpu imxvpuenc_mjpeg; imxvpuenc_mpeg4; imxvpuenc_h264; imxvpuenc_h263 VPU Based encoders
Video Render (sink) imxg2d; imxpxp; imxeglvivsink; imxipu imxg2dvideosink; imxpxpvideosink; imxeglvivsink; imxipuvideosink g2d1, ipu1, pxp2, and egl (overlay) video sinks
Video Converter imxg2d; imxpxp; imxipu imxg2dvideotransform; imxpxpvideotransform; imxipuvideotransform g2d, pxp, egl and ipu video filter/converter/scalars3
Video Compositing imxg2d; imxipu imxg2dcompositor, imxipucompositor gpu/ipu accelerated compositing

1. The g2d sink is very flexible in the types of input video it can take, but doesn’t have the ability to convert to as many formats as the IPU can. On the other hand, the IPU is very picky with it’s input (e.g. requiring a 1px offset) and the kernel driver is very undocumented, but as stated before, it can convert between many colorspace formats.
2. Note that the PXP sinks are only applicable to the i.mx6solo and i.mx6dl processors.
3. Please see note 1 above.

Plugin Example Pipeline

For example, to encode a video from a camera on /dev/video2 into h.264 and save it to a file:

#Take camera input /dev/video2, encode it to h264 at a bitrate of 10mbit/s (CBR) and save to a file.
gst-launch-1.0 imxv4l2videosrc device=/dev/video2 ! imxvpuenc_h264 bitrate=10000 ! filesink location=/tmp/file.mp4

Many more pipeline examples are described and listed on the Gateworks Software Wiki GStreamer Pipelines page

Summary

Using GStreamer 1.0 with the GStreamer-imx plugins is a powerful way to access and apply the multimedia capabilities of the Freescale i.MX6 processors on the Gateworks SBCs.

For more information, visit the Gateworks Software Wiki GStreamer Pipelines page

If there are other examples you would like to see, please add to the discussion!

Yocto 1.8 Linux BSP – Gateworks i.MX6 SBCs

yoctoproject_logo_cmyk

Gateworks would like to announce the release & support of Yocto 1.8 on the Ventana Family of Single Board Computers that feature the Freescale i.MX6 processor. Yocto 1.8 is under the code name Fido. Yocto is a Linux operating system that Gateworks uses for video input and output, networking, GUI, IoT and more. Gateworks recommends all customers and engineers update from Yocto 1.7 to Yocto 1.8.

Yocto 1.8 Updates and Highlights:

  • Updated Linux Kernel. The kernel has been updated to the Gateworks downstream 3.14 vendor kernel.
  • Updated the gstreamer video framework to gstreamer-imx and gstreamer-1.0
    • Updated to the gstreamer-imx community based plugins for utilizing i.MX6 hardware acceleration to provide increased flexibility over what is provided from Freescale in gst-fsl-plugins
    • Included RTSP server application, gst-variable-rtsp-server
    • More gstreamer-imx information is available here
  • AVC8000nano Video Capture Mini-PCIe card for up to 8x D1 inputs
    • The driver is now loaded by default in Yocto 1.8 and tested with gstreamer-imx
    • This is very useful for surveillance and compositing many video sources into one screen
    • Find more information here
  • Updated uboot-envtools
  • Updated gsc-update tool
  • Support for the GW16113 IO Expansion module via the gwsoc software tool. More information available here.

To get started, Gateworks has posted Pre-Built images on the Yocto Wiki page. This includes tarballs and ubis for multimedia and gui images. The wiki page also documents building Yocto 1.8 from source.

Yocto Software Wiki Page

Please update to Yocto 1.8 and contact Gateworks with any questions. Their support team is happy and available to work with you!

Speeding Up Boot Times with Falcon Mode

Everyone desires computers that boot very fast! The software that is shipped by default on Gateworks boards is tuned to be very ‘developer friendly’ for getting started in the development cycle. Common questions like: ‘How can I make my software boot quicker?’ are asked frequently and Gateworks has devoted time into documenting various methods of increasing boot speed or boot speed perception (see http://trac.gateworks.com/wiki/boot_speed). Gateworks has additionally improved boot speeds by implementing U-Boot’s Falcon mode in the Ventana bootloader. Gateworks Ventana is the first IMX6 based board in mainline U-Boot to get this support.

What is U-Boot Falcon mode? Falcon mode is named after the Peregin Falcon, the fastest member of the animal kingdom. U-Boot Falcon mode speeds up boot time by bypassing the bootloader completely which can save 3 to 10 seconds depending on the system configuration.

Continuing with the technical details, Falcon mode is a feature of the Secondary Program Loader (SPL) which is built from the same U-Boot source that builds U-Boot proper. The job of the SPL is to configure the memory controller and load the next stage – which is typically the full-blown, feature-rich U-Boot bootloader, which in turn is responsible for loading the OS. The U-Boot bootloader can easily add 3 to 10 seconds to the boot-up of a system depending on its configuration, the boot medium (Flash vs micro-SD for example) and the filesystem on that boot medium. To bypass the loading of U-Boot, the Ventana SPL now supports reading an OS kernel and boot parameters (or ‘args’) directly from NAND flash blocks or raw micro-SD sectors (sans filesystems). This typically takes 1 to 2 seconds, depending on the size of your OS kernel and results in being able to have a fancy graphical splash-screen show up within the first couple of seconds of power. What the OS does next is very dependent on the application and OS configuration. If several seconds are shaved off in getting to the OS with Falcon mode then the system boot time has been sped up in some cases significantly.

Keep in mind that using Falcon mode is something that should be done at the end of the development cycle as the U-Boot bootloader features that are bypassed can come in very handy during development.

For reference and more information, please see:

Android KitKat Software Release for Ventana SBCs

androidkitkat

Gateworks is proud to announce the release of Android KitKat for it’s family of Ventana SBCs.  The source and pre-built images for Android 4.4.3 (KitKat) have been released.

To find out what is new in KitKat 4.4 compared to the prior Android JellyBean release, please see the Android documentation: http://www.android.com/versions/kit-kat-4-4/

Notable changes include:

  • Update to the Android Open Source Project’s 4.4.3 code base
  • Added NAND flash support (for Ventana boards that have 2GB NAND)
  • Added support for new products such as the GW551x and GW552x
  • Freescale has added many IMX6 performance and stability improvements.
  • Newer 3.10.53 based Linux kernel which has many improvements.

Freescale has verified that their Kitkat 4.4.3 BSP (which Gateworks BSP is sourced from), passes Android Compatibility Test Suite (CTS 4.4_r3).

We recommend everyone switch to this new branch for all future Android support.

To get started, follow the building instructions on Gateworks Software Wiki site here: http://trac.gateworks.com/wiki/Android/Building#BuildingAndroidforVentanafromsource

H.264 Video Streaming over Network

The Gateworks Ventana Family of Single Board Computers is well suited for multimedia applications. The Freescale i.MX6 processor has hardware encoding and decoding making for very little processor overhead. To take advantage of this hardware decoding it is important to always use the ‘vpuenc’ and ‘vpudec’ gstreamer pipeline elements.

  • vpuenc – VPU (Video Processing Unit) encoder
  • vpudec – VPU (Video Processing Unit) decoder

It is also important to use a high compression video format such as H.264 which uses significantly less network bandwidth while retaining good quality. To use this, we encode by using the following pipeline element:

  • vpuenc codec=avc

When streaming over the network, TCP is a reliable protocol but often too stringent for video applications. UDP is often preferred so that even if packets are lost, the stream will continue without waiting for the lost packet. To use UDP, we use the udpsrc and udpsink pipeline elements.

Here is an example pipeline to stream live video over the network using two Ventana Single Board Computers. Be sure the two single board computers are on the same network and can ping each other. Both should also be running OpenEmbedded Yocto operating system. An analog video camera is connected to the video input on a Ventana board. Another Ventana board is connected to a HDMI monitor.

Start the server board pipeline first: (board with HDMI monitor connected)

root@ventana:~# gst-launch udpsrc port=9001 ! h264parse ! queue max-size-time=0 max-size-buffers=0 ! vpudec low-latency=true frame-plus=1 framedrop=false ! mfw_v4lsink sync=false async=false

Then start the client pipeline (board with analog video camera connected)

root@ventana:~# gst-launch tvsrc device=/dev/video0 ! vpuenc codec=avc ! udpsink host=192.168.1.2 port=9001 sync=false async=false

For more information, please view our GStreamer Software Wiki here: http://trac.gateworks.com/wiki/Yocto/gstreamer

Ventana Family Model Numbers & Ordering Options

The Ventana Family of Single Board Computers has been a very popular offering with the HDMI HD video input and HDMI output, the CAN Bus port and high performance i.MX6 processor along with many more great features.  To provide the widest range of options, variations of each family member was created to make ordering more simple. These variations along with their part numbers are listed on our Ventana ordering options Wiki page: http://trac.gateworks.com/wiki/ventana/orderoptions

Ventana JTAG Software Release

Gateworks is proud to announce JTAG programming support for the Ventana family of Single Board Computers. This support allows precompiled binaries that include the kernel, bootloader, and root filesystem to be flashed via Gateworks JTAG software onto any Ventana board. This is valuable as it is the most simple way to load new software and updates onto the board. The same JTAG software is also backwards compatible with all previous Gateworks boards. This release is limited to Linux and provides programming support and does not include upload or verify support. Ventana is unique in that it is the first family to use NAND flash which requires additional routines to handle bad blocks.

The new JTAG Linux executables, can be found on the Gateworks Software Wiki:

http://trac.gateworks.com/wiki/jtag_instructions#JTAGExecutables

To find the most current and up to date prebuild binary images of OpenWrt for Ventana:

http://trac.gateworks.com/wiki/ventana/openwrt#Step2.InstallingFirmware

Gateworks Support Team

Android App Development on Gateworks SBCs

Gateworks supports Android on the Ventana family of Single Board Computers. Once loading Android onto the SBC, the next logical step is to create an app. Some of these apps may want to communicate with a GPIO and other hardware.  Gateworks has documented some examples and tutorials on creating a simple Android app and correctly integrating it into the system.

Please find all the information on the Gateworks Software Wiki link: http://trac.gateworks.com/wiki/Android/app_development