SafeZone2D with AI (Advanced Motion Detection)

Description

SafeZoneEdge 2D from Digital Barriers: this plugin performs an advanced plug-and-play motion detection on a video stream; it does not need any configuration or calibration.
It is typically used to detect human activity, filtering efficiently artifacts coming from weather conditions (e.g. wind, rain or snow).

The output data are a list of alarms (PAnalytics::Apply() method returns a PList of PEvent which represent alarms and detected objects).

The analytics engine of this plugin comes from SmartVis SafeZone: see presentation https://www.youtube.com/watch?v=LPYTJ09lrVU

  • Plugin: PPluginAnalyticsSafeZoneEdge2d
  • Product name: SafeZoneEdge2d
  • Version: 2.0

Overview

Technical requirements to get good results using SafeZoneEdge2D

  • Use in a sterile zone ONLY – it does not work in a busy area and will stop detecting.
  • An object must be visible for at least 2 seconds in the scene.
  • Works best with fixed cameras (small vibrations are tolerated).
  • Using SafeZone with a PTZ camera is possible, but has limitations. Once the camera stops moving it can take a few seconds for the analytics to learn the new background, and any object already in the scene will likely be treated as background.
  • There must be suitable contrast between background and objects to detect.
  • To detect an object, it must be:
    • Taller than 15 pixels
    • Not taller than 60% of the height of the image
    • Not bigger than 35% of the area of the image
  • Images should be provide to a framerate as close as possible to 8 fps. Different framerates will lead to degraded performances both in terms of detection accuracy and false alarm rate.
  • To avoid interference from insects, we recommend not using the built-in IR lighting of the camera (especially cameras with sun shield).
  • Raindrops on camera lens may strongly reduce detection and false alarm mitigation performances. Avoid fixed dome for outdoor use. Install a shield on the top of the camera when necessary to prevent from raindrops on camera lens.

CPU requirements

CPU Mark on windows (figures on Linux can vary slightly):

  • QCIF analysis, 4x3 image: 150 CPU Mark, 50 MB RAM
  • QCIF analysis, 16x9 image: 200 CPU Mark, 50 MB RAM

Input requirements

Image resolution : QCIF like Most common resolution depending on the source aspect ratio:

Aspect Ratio Resolution
16/9 256 x 144
16/10 224 x 140
8/5 224 x 140
4/3 192 x 144
5/4 180 x 144
11/9 176 x 144

For optimal performance, it is HIGHLY recommended to use a QCIF like resolution (e.g. 176x144 pixels) at 8 frames per second.
Basically, it means you have to control the framerate by calling PAnalytics::Apply() method about 8 times per second on a live stream. If you want to process a video file, be sure the video has been registered at 6 to 10 FPS. If less than 6 FPS, the detection could fail. If more than 10 FPS, you should skip some frames to approximate 8 FPS (e.g. if 24 FPS than skip 2 frames over 3).

Good design consists in using a separate thread to perform SafeZone2D analytics so that SafeZone2D has enough CPU power to process images at the right framerate.

You do not have to resize input images, it will be done automatically by the plugin, using the specified resolution (see parameters below).

Please contact us at suppo.nosp@m.rt-s.nosp@m.dk@di.nosp@m.gita.nosp@m.lbarr.nosp@m.iers.nosp@m..com if you have any trouble (e.g. poor performance) while trying to use this plugin with different video streams.

How to create a PAnalytics SafeZoneEdge2d object?

The easiest way to configure the SafeZone engine is to use the helper class PSafeZoneEdge2dParameters. This class allows you to easily set the most common configuration options for the SafeZone engine.

// assumes: using namespace papillon;
PAnalytics safeZoneEdge2d;
PResult ret = PAnalytics::Create("SafeZoneEdge2d", safeZoneEdge2dParameters, safeZoneEdge2d);
if (ret.Failed()) ...

See PAnalytics to learn more.

Parameters

For simpler usage, use PSafeZoneEdge2dParameters to setup video analytics parameters. This class has a set of Set/Get methods to easily configure the algorithm without knowing raw parameter names described below.

In this section, ROI stands for "Region Of Interest". It is the rectangle area where detection is performed.

Important note: all parameters must be passed as PString.

The following parameters are mandatory:

Parameter Type Description
camera-type string type of sensor: can be "dayNight" (colour sensor), "BW" (black-and-white) or "thermal" (thermal sensor)
image-width, image-height string resolution in pixels used to perform video analytics. Typical (optimal) parameters are QCIF resolution, i.e. image-width="176" and image-height="144". Note: actual process resolution might be slightly different because of aspect ratio of input images
sensitivity string a value in "0".."1" - default is "0.5" - which specify detection sensitivity. Higher sensitivity (towards 1) means more detections but more false alarms
filter-headlights string either "true" indicating that there is headlight illumination in the scene (Head light filtering will be activated if the camera type is not thermal) or "false" if not

Other optional parameters:

Parameter Type Description
min-object-width string filter: minimum object width (width of the bounding box), in pixels
max-object-width string filter: maximum object width (width of the bounding box), in pixels
min-object-height string filter: minimum object height (height of the bounding box), in pixels
max-object-height string filter: maximum object height (height of the bounding box), in pixels
post-alarm-time-ms string an integer value in milliseconds, "0" by default - amount of time during which alarm will be kept after disappearance of a target (used to smooth alarms)
use-roi string either "true" (use a Region Of Interest) or "false" (analyse the full image); "false" by default: the entire image will be analyzed
detection-roi string a string representing a PRectanglef (can be used to set "roi-x", "roi-y", "roi-w", "roi-h" values at once). Overwrite "roi-?" parameters if both are set. No detection-roi by default. Note: ROI will be automatically clamped to be inside analysed images.
roi-x string a string representing a normalised float value (in "0".."1") = x-coordinate of the upper-left corner of the ROI (Region Of Interest)
roi-y string a string representing a normalised float value (in 0..1) = y-coordinate of the upper-left corner of the ROI (Region Of Interest)
roi-w string a string representing a normalised float value (in 0..1) = width of the ROI (Region Of Interest): "1" corresponds to the original image width
roi-h string a string representing a normalised float value (in 0..1) = height of the ROI (Region Of Interest): "1" corresponds to the original image height

Small objects filtering note: small objects are filtered if their width is smaller than min-object-width AND their height smaller than min-object-height. Large objects filtering note: large objects are filtered if their width is larger than max-object-width AND their height larger than max-object-height.

SafeZone AI

SafeZone AI is a special mode of SafeZone 2D in which either a DNN classifier or DNN detector can be enabled. This will significantly decrease the number of false-positives, however it will come at the cost of increased computation time.

  • The DNN classifier is run on all objects detected by SafeZone2D and is good at filtering out false-positive detections.
  • The DNN detector is run on every frame to perform the initial detection. This can be more accurate, however it comes at significant computational cost.

The code below shows an example of how to enable the DNN classifier. Code for the DNN detector is very similar.

// assumes: using namespace papillon;
PAnalytics safeZoneEdge2d;
safeZoneEdge2dParameters.EnableDNNClassifier(true);
PResult ret = PAnalytics::Create("SafeZoneEdge2d", safeZoneEdge2dParameters, safeZoneEdge2d);
if (ret.Failed()) ...

By default the DNN classifier will try an load into to Intel GPU and use the GPU to perform the processing. If your computer does not have a GPU it will default back to the CPU. You can also request that the DNN classifier uses the CPU by default.

safeZoneEdge2dParameters.SetDNNClassifierEngine("cpu");

The example program gives a complete example of how to use both the classifier or detector.

Output Events

Each time an object (e.g. Person) is detected, an alarm (PEvent) will be generated.

The output of this plugin is a list (PList) containing only PEvent objects with the following type:

  • Type = "metadata"
  • Annotation = a XML document which is the metadata associated with the current frame (use GetAnnotation() to get this document).

Example 1: XML document when no detection performed. Note: OSD_SIZE can be slightly different from the original resolution because SafeZone2D try to optimise resolution while preserving aspect ratio.

<NODE NAME="ivs_results" REFERENTIAL="0" SITE="" OSD_SIZE="256x144" VERSION="5.0.0">
<TIMESTAMP> 2016-08-22T12-08-21-072-000000005_XXXXXXXXXX </TIMESTAMP>
<FRAME_NUMBER> 5 </FRAME_NUMBER>
<FRAMERATE> -1 </FRAMERATE>
<STATUS> LEARNING </STATUS>
<ACTOR_LIST>
</ACTOR_LIST>
<ALERT_LIST>
</ALERT_LIST>
<ALERT_ZONE_LIST>
<ZONE NAME="zone-1">
<POINT NUMBER="0" X2D="252" Y2D="142" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
<POINT NUMBER="1" X2D="0" Y2D="142" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
<POINT NUMBER="2" X2D="0" Y2D="0" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
<POINT NUMBER="3" X2D="252" Y2D="0" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
</ZONE>
</ALERT_ZONE_LIST>
</NODE>

Example 2: XML document when an actor (a PERSON) is detected. The bounding box of the person is given in pixels by the _2D_BBOX element.

<NODE NAME="ivs_results" REFERENTIAL="0" SITE="" OSD_SIZE="256x144" VERSION="5.0.0">
<TIMESTAMP> 2016-08-22T12-08-45-713-000000202_XXXXXXXXXX </TIMESTAMP>
<FRAME_NUMBER> 202 </FRAME_NUMBER>
<FRAMERATE> 7.9721 </FRAMERATE>
<STATUS> RUNNING </STATUS>
<ACTOR_LIST>
<ACTOR ID="5" TYPE="PERSON" TIMESTAMP="2016-08-22T12-08-45-713-000000202_XXXXXXXXXX" IS_MAINTAINED="false">
<_2D_BBOX XMIN="122" XMAX="131" YMIN="75" YMAX="103"/>
<_3D_POSITION X="122" Y="75" Z="0"/>
<_3D_SIZE WIDTH="0" HEIGHT="0"/>
</ACTOR>
</ACTOR_LIST>
<ALERT_LIST>
<ALERT ID="0" TYPE="intrusion" ZONE_NAMES="zone-1" ACTOR_ID="5" SCENARIO_NAME="intrusion-1" BEGIN-TIMESTAMP="2016-08-22T12-08-41-338-000000167_XXXXXXXXXX" END-TIMESTAMP="2016-08-22T12-08-45-713-000000202_XXXXXXXXXX">
</ALERT>
</ALERT_LIST>
<ALERT_ZONE_LIST>
<ZONE NAME="zone-1">
<POINT NUMBER="0" X2D="252" Y2D="142" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
<POINT NUMBER="1" X2D="0" Y2D="142" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
<POINT NUMBER="2" X2D="0" Y2D="0" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
<POINT NUMBER="3" X2D="252" Y2D="0" X3D="0.0" Y3D="0.0" Z3D="0.0"/>
</ZONE>
</ALERT_ZONE_LIST>
</NODE>

See PEvent to learn more.

Example

/*
* Copyright (C) 2015-2017 Digital Barriers plc. All rights reserved.
* Contact: http://www.digitalbarriers.com/
*
* This file is part of the Papillon SDK.
*
* You can't use, modify or distribute any part of this file without
* the explicit written agreements of Digital Barriers.
*/
#include <PapillonCore.h>
USING_NAMESPACE_PAPILLON
const PString SAMPLE_DIR = PPath::Join(PUtils::GetEnv("PAPILLON_INSTALL_DIR"), "Data", "Samples"); // path to find sample data: $PAPILLON_INSTALL_DIR/Data/Samples
static PResult RunDemo(int argc, char** argv)
{
POption opt(argc, argv);
opt.AddStandardOptions(); // set-up logging
PLog::SetConsoleLevel(papillon::PLog::E_LEVEL_INFO);
PLog::SetConsoleFormat("date (sev) [sf:line]: msg");
PString inputVideoFile = opt.String("inputVideo,iv", PPath::Join(SAMPLE_DIR, "vegetation.avi"), "Input video file");
bool classifier = opt.Option("classifier", "Run the DNN Classifier");
bool detector = opt.Option("detector", "Run the DNN Detector ");
int32 delay = opt.Int("delay", 0, "Delay between each frame");
bool cpu = opt.Option("cpu", "By default will use GPU unless this option specified.");
bool display = !opt.Option("noDisplay,nd", "Display video as we go");
bool verbose = opt.Option("verbose,v", "Be verbose");
bool metadata = opt.Option("metadata,m", "Display frame metadata when being verbose");
ReturnIfFailed(opt.Check());
if (opt.Has("h") || opt.Has("help")) {
P_LOG_INFO << opt.ToStringHelp();
return PResult::C_OK;
}
// ************************************************************************
// 1. Open Video Stream
// ************************************************************************
PInputVideoStream::Open(inputVideoFile, ivs).OrDie();
// ************************************************************************
// 2. Create SafeZoneEdge2d video analytics engine
// See @ref plugin_analyticsSafeZoneEdge2D to learn more
// ************************************************************************
// Have we been asked to use classifier/detector and with CPU
safeZoneEdge2dParameters.EnableDNNClassifier(classifier);
safeZoneEdge2dParameters.EnableDNNDetector(detector);
if(cpu) {
safeZoneEdge2dParameters.SetDNNClassifierEngine("cpu");
safeZoneEdge2dParameters.SetDNNDetectorEngine("cpu");
}
PAnalytics safeZoneEdge2d;
PAnalytics::Create("SafeZoneEdge2d", safeZoneEdge2dParameters, safeZoneEdge2d).OrDie();
// ************************************************************************
// 3. Now, analyse the video stream and retrieve alarms/events detected
// by SafeZoneEdge2d
// ************************************************************************
int framesProcessed = 0;
PTimer timer(25);
double totalTime = 0.;
int numberOfAlarms = 0;
PFrame frame;
while (true)
{
// Get the frame to be analysed
if (ivs.GetFrame(frame).Failed())
break;
// Apply video analytics: a PEvent object is
// generated each time an object (e.g. a person) is detected
PList events;
if (safeZoneEdge2d.Apply(frame, events).Failed())
continue;
// Has SafeZone2D detected something in this frame?
if (!events.IsEmpty()) {
PEvent event;
events.Get(0, event);
if(event.GetType() == "alarm") {
// We have this type for start and stop alarm events
numberOfAlarms++;
if (verbose) P_LOG_INFO << event.GetAnnotation();
} else if (event.GetType() == "metadata") {
if (verbose && metadata) P_LOG_INFO << event.GetAnnotation();
}
}
// Display video stream with OSD (On-Screen-Display) drawn on top of it
totalTime += timer.ElapsedSec();
timer.Stop();
if(display) {
PImage displayImage = frame.GetImageShared();
safeZoneEdge2d.DrawOSD(displayImage);
displayImage.DrawLabel(PString("FPS: %1").Arg(1./timer.Average(), 4, 'f', 2), PPoint2Di(80, 20));
frame.Display("Papillon SDK - SafeZoneEdge2d example", delay);
}
timer.Start();
framesProcessed++;
}
// Display some stats
P_LOG_INFO << "Frames Processed: " << framesProcessed << \
" Time: " << totalTime << "s" \
" Average Frame Rate: " << double(framesProcessed) / totalTime << " fps" \
" Number of Alarms: " << numberOfAlarms/2;
return PResult::C_OK;
}
int main(int argc, char** argv)
{
RunDemo(argc, argv).LogIfError();
return 0;
}