Monitoring a Panorama app

Real-time smart video analytics application development and edge device deployment is a tricks task. In the last few years, the industry players built platforms to support this activity. Notable examples include NVIDIA Metropolis, Intel’s OpenVINO, and AWS Panorama. While these solutions make some aspects of the video-analytics application development more straightforward, there are still many issues to deal with before deploying a video analytics application in production. This post introduces SkyLine, the first in a series of open-source tools to make developing AWS Panorama application simpler.
AWS Panorama is a machine learning appliance and software framework that allows you to deploy video analytics applications on edge. For a thorough introduction and a step-by-step tutorial on deploying a Panorama application, refer to Deploy an Object-Detector Model at the Edge on AWS Panorama.
AWS Panorama framework eases many aspects of video analytics application development, including camera connection management, video decoding, frame extraction, model optimization and loading, display output management, over-the-air deployment of your application, and other features. Nevertheless, some tasks are still challenging, including diagnostic tasks when a model does not work as expected.
Introducing SkyLine
The only available way to get visual feedback on the correct functionality of a Panorama application is to physically connect a display to the HDMI port of the appliance. The display will show the output video stream of a single application deployed on the device. However, physically accessing the appliance is not always feasible. SkyLine allows to re-stream the output video of any Panorama application to an external service, for example, to AWS Kinesis Video Streams. This feature can be very convenient for monitoring an application remotely.
How does SkyLine work?
SkyLine instantiates a GStreamer pipeline with an [appsrc](https://gstreamer.freedesktop.org/documentation/app/appsrc.html)
element at the head. An OpenCV VideoWriter
is configured to write to the appsrc
: instead of saving the consecutive frames to a video file, it streams to the output sink. When opening the VideoWriter
instance, the user should specify the frame width and height and the frame rate of the output stream. You can manually set these parameters or let SkyLine infer them from the input dimensions and the frequency you feed new frames to it. If using this auto-configuration feature, some frames (by default 100) will be discarded at the beginning of the streaming because SkyLine will use them to calculate statistics of the frame rate and measure the frame dimensions. This is the "warmup" of SkyLine. If you send frames with different sizes, SkyLine will resize the input, but this comes with a performance penalty. You are also expected to send new frames to SkyLine with the frequency specified in the frame-per-second parameter. If frames are sent at a different frequency, the video fragments get out of sync in Kinesis Video Streams, and you won’t be able to replay the video smoothly.

SkyLine
is an abstract base class that handles the state machine of the GStreamer pipeline. The concrete implementation KVSSkyLine
sends frames to Amazon Kinesis Video Streams service. It is easy to extend SkyLine to support other services, especially if there is already a GStreamer plugin for that.
Integrating SkyLine into a Panorama application
Configuring the Docker container
SkyLine depends on custom compiled external libraries. All these libraries must be compiled and configured correctly in your application’s docker container to make SkyLine
work. These libraries include:
- GStreamer 1.0 installed with standard plugins pack, libav, tools, and development libraries;
- OpenCV 4.2.0, compiled with GStreamer support and Python bindings;
- numpy (it is typically installed by the base docker image of your Panorama application).
In the case you want to use KVSSkyLine
, the SkyLine
implementation that streams the video to Amazon Kinesis Video Streams, it will need also the following libraries:
- Amazon Kinesis Video Streams (KVS) Producer SDK compiled with GStreamer plugin support;
- Environment variable
GST_PLUGIN_PATH
configured to point to the directory where the compiled binaries of KVS Producer SDK GStreamer plugin is placed; - Environment variable
LD_LIBRARY_PATH
including the third-party open-source dependencies compiled by KVS Producer SDK; boto3
(it is typically installed by the base docker image of your Panorama application).
In the source code repository of SkyLine, a sample Dockerfile is provided in the examples folder that shows how to install these libraries and SkyLine in any container correctly. In most cases, it is required just to copy the relevant sections from the sample to your application’s Dockerfile. Please note that the first time you compile the docker container, it might take up to one hour to correctly compile all libraries.
Setting up AWS IAM privileges for KVSSkyLine
KVSSkyLine
uses the Amazon Kinesis Video Streams (KVS) Producer library, wrapped in a GStreamer sink element, to send the processed frames to the KVS service. KVS Producer needs AWS credentials. It can use the credentials associated with the Panorama Application Role, but you must explicitly configure it. KVSSkyLine
needs privileges to execute the following actions:
kinesisvideo:DescribeStream
kinesisvideo:GetStreamingEndpoint
kinesisvideo:PutMedia
If KVSSkyLine
needs to automatically create the Kinesis Video Stream the first time it is used, you should also include the kinesisvideo:CreateStream
action. An example policy allowing KVSSkyLine
to write data to Kinesis Video Streams could look like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"kinesisvideo:DescribeStream",
"kinesisvideo:CreateStream",
"kinesisvideo:GetDataEndpoint",
"kinesisvideo:PutMedia"
],
"Resource": [
"*"
]
}
]
}
Setting up credentials
There are two types of AWS security credentials: static and temporary. The former never expire and in case of a leak, you should invalidate them manually and reconfigure the applications. For this reason, their use is strongly discouraged in a production environment. Examples of static AWS credentials include the IAM user’s Access Key Id and Secret Access Key pair.

Temporary credentials expire after a predefined period. In case of a leak, they can be used only within their expiration time, typically in the order of several hours. Temporary credentials can be renewed before they expire so their lifespan can be extended, or they can be exchanged for new ones with a later expiration time. This process needs additional coordination from the application using this type of credentials. Examples of temporary credentials include the AWS Access Key Id, the Secret Access Key, and the Session Token.
We offer different options to provide credentials withKVSCredentialsHandler
subclasses provided in the kvs
module. If you want to use static credentials for testing purposes, create an IAM user in your AWS account and attach a policy like the one above to the user. You should configure this user to have programmatic access to AWS resources and get the user’s AWS Access Key and Secret Key pair: create a KVSInlineCredentialsHandler
or KVSEnvironmentCredentialsHandler
instance in the application’s code to pass these credentials to KVS Producer Plugin directly in the GStreamer pipeline definition or as environment variables. However, as these credentials do not expire, using this configuration in a production environment is not recommended. Even in a development and testing environment, you should take the appropriate security measures to protect these credentials: never hardcode them in the source code. Instead, use AWS Secret Manager or a similar service to provide these parameters to your application.
KVSSkyLine
can also use the Panorama Application Role to pass the application’s credentials to KVS Producer. These credentials are temporary, meaning that they expire within a couple of hours, and the user should renew them before expiration. The Producer library expects temporary credentials in a text file. KVSFileCredentialsHandler
manages the renewal of the credentials and periodically updates the text file with the new credentials. If you want to use this method, attach a policy similar to the example above to your Application Role. Remember to always test your Panorama application – KVS integration if it still works after the KVSFileCredentialsHandler
refreshed the credentials. Let your application run for several hours and periodically check if it continues to stream the video to KVS. You will also find diagnostic information in the CloudWatch logs of your application when the credentials were renewed.
Environment variables
KVSSkyLine
needs two environment variables to have GStreamer find the KVS Producer plugin. The name of these variables is GST_PLUGIN_PATH
and LD_LIBRARY_PATH
. They point to the folder of KVS Producer GStreamer plugin and its 3rd party dependencies. In the sample Dockerfile provided, the correct values of these variables are written to a small configuration file named /panorama/.env
in the container. You should pass the path of this file to KVSSkyLine
or otherwise ensure that these environment variables contain the correct value.
Usage example
When initializing a KVSSkyLine
instance, you should pass the AWS region name where your stream is created, the name of the stream, and a credentials handler instance. If you want to configure the frame rate and the dimensions of the frames manually, you should also set them here. When both parameters are specified, SkyLine skips the warmup period and sends the first frame directly to KVS. When you are ready to send frames, call the start_streaming
method which opens the GStreamer pipeline. After this method is called, you are expected to send new frames to the stream calling the put
method periodically, with the frequency specified in the frame rate or inferred by KVSSkyLine
. You can stop and restart streaming on the same KVSSkyLine
instance.
The following example uses the temporary credentials of the IAM Application Role assumed by your Panorama application:
If everything works well, you can watch the restreamed video on the Kinesis Video Streams page of the AWS console. Certainly, you can modify the image before sending it to SkyLine: draw annotations on it based on the inference result of the deep-learning model.
Annotations
Annotations and annotation drivers provide a unified way to draw annotations on different rendering backends. Currently, two annotation drivers are implemented:
PanoramaMediaAnnotationDriver
allows you to draw onpanoramasdk.media
object, andOpenCVImageAnnotationDriver
allows you to draw on an OpenCV image (numpy array) object.

The library can draw two types of annotations: labels and boxes.
Using annotations
Depending on the available backends, you can create one or more annotation driver instances at the beginning of the video frame processing loop. During the process of a single frame, you are expected to collect all annotations to be drawn on the frame in a python collection (for example, in a list
). When the processing is finished, you call the render
method on any number of drivers, passing the same collection of annotations. All coordinates used in annotation are normalized to the [0; 1)
range.
Example usage:
Postface
Even if SkyLine can be a helpful tool, its usage might raise two concerns that you should consider carefully. We discourage using SkyLine in a production environment: it is a development aid or and a Debugging tool.

The first concern is of technical nature. Currently, the application code in a Panorama app does not have direct access to the onboard GPU; thus, all video encoding done by SkyLine run on the device’s CPU. This behavior could slow down the appliance: streaming a single output stream with SkyLine could require anything between 10–30% of the CPU capacity of the device.
The second concern is related to data protection. The Panorama appliance is designed to protect the video streams being processed. It has two ethernet interfaces to separate the network of the video cameras (typically a closed-circuit local area network) from the device’s Internet access. Using SkyLine, you relay the video stream from the protected, closed-circuit camera network to the public Internet. You should carefully examine the data protection requirements of your application and the camera network before using SkyLine. Moreover, you are responsible for keeping private all AWS credentials used by SkyLine, as required by AWS Shared Responsibility Model.
Where to go from here?
SkyLine
is part of Backpack
, a broader set of tools that aims to help software development on AWS Panorama. Other components will be presented in a future post, so keep tuned and follow us to get updated.
Backpack
is open source and available on GitHub. You can build and run the example docker container on an ARM64 based system. For example, suppose you’ve already set up the Panorama Test Utility on a t4g type EC2 instance. In this case, you can build and launch a shell in this container with the following commands on the EC2 instance:
$ git clone https://github.com/Neosperience/backpack.git
$ cd backpack/examples
$ docker build . -t my_backpack_example:1.0
$ docker run -it my_backpack_example:1.0 bash
The container can be built and executed also on a MacBook Pro with the new M1 ARM-based chip.
The backpack library is extensively documented with docstrings. You can read the detailed API docs online or using the python3
interpreter in the container shell:
root@my_backpack_example:/# python3
>>> import backpack.skyline, backpack.kvs, backpack.annotation
>>> help(backpack.skyline)
>>> help(backpack.kvs)
>>> help(backpack.annotation)
You can also try other SkyLine
implementations found in Backpack
. For example, RTSPSkyLine
allows you to host an RTSP server right in the application container, and connect directly to your Panorama appliance with an RTSP client to monitor your application.
Let me know how you use SkyLine
, which new features you would like to see, and stay tuned for future posts about the other components of Backpack
.
Update as of 20/06/2022: The remote viewing module of the backpack
library was renamed to skyline
.
About the author
Janos Tolgyesi is an AWS Community Builder working as Machine Learning Solution Architect at Neosperience. He has worked with ML technologies for five years and with AWS infrastructure for eight years. He loves building things, let it be a video analytics application on the edge or a user profiler based on clickstream events. You can find me on Twitter, Medium, and LinkedIn.
The open-source project of Backpack was supported by Neosperience.
I would like to express my special thanks to Luca Bianchi for proofreading this article.