FaceLog4 (DEPRECATED, favour FaceLog6)

Description

FaceLog4 is an advanced analytic that tracks unique faces in a video stream, classifies and optionally identifies them against a supplied watchlist. This plugin offers multithread support and is able to process high resolution video streams on a multicores CPU (typically Intel i7 with 4 cores or more).

  • Product name: FaceLog4
  • Version: 1.0

How to create a PAnalytics FaceLog4 object from this plugin?

#include <PFaceLog4Parameters.h>
PAnalytics faceLog4;
PProperties parameters = PFaceLog4Parameters::BuildDefaultParameters();
// parameters.Set("MaxFaceDetectorFR", PDouble(-1.0)); // uncomment this line to process a video file (to avoid skipping frames), otherwise keep default value (8) for a live stream
PAnalytics::Create("FaceLog4", parameters, faceLog4).OrDie();

As with all PAnalytics, it works on PFrame's pulled from a video stream and releases PEvent's when they are ready. FaceLog4 releases three main kind of events; "Face", "Track" and "Sighting", see sections below to learn more.

Region-Of-Interest (ROI)

It is possible to restrict video analytics area using a region-of-interest, see parameters table below. It can be used to select a specific area in images and/or to speed-up processing.

PProperties faceLog4Options;
faceLog4Options.Set("ROI", PRectanglei(20, 110, 640, 480));
PAnalytics::Create("FaceLog4", faceLog4Options, faceLog4).OrDie();
// or
faceLog4.Set("ROI", PRectanglei(20, 110, 640, 480));

Warning: coordinates of detected faces, tracks and sighting are given in the region-of-interest coordinates system; they will be shifted by the origin of the area (e.g. (20,110) in the previous example). You can use PRectanglef::Translated(), PPoint2Df::Translated(), PFeatureMap::Translate() to relocate detections in the original full frame. Origin of the region-of-intereset is also stored in PDetection: see PDetection::GetOrigin(). See also PUtils::DrawLabel() which handles this offset and will correctly draw detections on original full frame.

Input Data

Input data is a single PFrame object, see PAnalytics::Apply(). For optimal performance, recommanded pixel format of the underlying PImage is PImage::E_BGR8U.

Detections

A detection represents a single detection of a face in an image of a frame. If there are more than one face in the image then there will be multiple detection events released. Detection events are released per frame, there is no delay (other than the processing of the frame).

Tracks

A track is a collection of detections, in successive frames, that belong to the same subject. Track events are released as soon as the track has ended, i.e. when a track has not been added to in the next frame.

plugin_analyticsFaceLog4_Sightings

A sighting is a collection of tracks of the same subject. The tracks do not need to be in successive frames, there can be a break (lag) between them.

For example, often a track of a person walking through a video can be lost; For example, if they turn their head, or become obscured by someone else walking in front of them. In FaceLog4 we attempt to join similar tracks back together using both spatial information and using face-recognition. This collection of tracks is called a sighting.

After a configurable amount of time (measured in frames), when a track has not been added to a sighting, the sighting is released as an event. So a sighting will always be released a few seconds after the subject has left the scene.

Face Recognition

If a PWatchlist is provided to FaceLog4 as a parameter, then face recognition is performed on each track and sighting. Inside a watchlist each subject is assigned a unique id (subject-id) when a description. As each track and sighting is raised, face-recognition is performed against the watchlist. If a match is found (above a user set threshold), the track/sighting is labeled with the subject-id from the watchlist. A confidence of match is also assigned.

It should be noted, the most reliable identity is from the sighting as this is aggregated over the tracks.

Face Recognition Performance

The face recognition algorithm in Papillon are based on a powerful new technique called Deep Learning. They are vastly superior to what was available a few years ago. However, they still have their limitations.

You will get best performance when the faces in the video are within +/-30 degrees from frontal, the faces are diffusely illuminated (with no strong shadows) and there is a minimum of 50 pixels between the eyes. The algorithms will still function outside these parameters, however the performance will be degraded. Ideally, face-recognition should be used as an aid to a human operator performing their task, helping them reduce the amount of data they have to search or ordering the priority of data they have to review.

Face MetaData Classifier

A face-meta classifier is applied to the faces in each track and sighting. This provides extra information about the faces in the track and sighting. Currently this is limited to Gender information.

Parameters (how to configure FaceLog4)

Parameters are set through a PProperties class which are supplied to the PAnalytics::Create() factory constructor.

PFaceLog4Parameters is a class which can be used to help configure FaceLog4.

We aim to have a sensible set of default parameters that work effectively in many scenarios, however given the range of different use-cases for this analytic this is not always possible.

Parameter Type Required Default Description
Watchlist PWatchlist No None The watchlist of identities. If not present no face-rec performed
WatchlistOptions PWatchlistOptions No Its constructor The watchlist options
MinDetectionSize int32 No 80 The minimum detection size, in pixels. This is the minimum face it will detect (minimum is 40)
MaxDetectionSize int32 No 1000 The maximum detection size, in pixels
Threshold double No 0.0 Threshold for the face-detector
MinCertainty double No 0.0 Threshold for a track
MinDetectionsPerTrack int32 No 3 The minimum number of detections to consider it is a track
MaxDetectionsPerTrack int32 No 100 The maximum number of detections before releasing a track (see also FrameLag); this parameter should be > FrameLag
SightingSimilarity double No 0.5 The recognition threshold to use when considering to join two tracks together
FrameLag int32 No 50 How many frames to elapse before releasing a sighting (see also MaxDetectionsPerTrack)
SpatialFilter bool No true Apply spatial constraints to tracks
ROI PRectanglei No (0,0,0,0) Region-Of-Interest (ROI): restrict video analytics to the specified area
MaxFaceDetectorFR double No 8.0 Limit on number of frames per second processed by face detector. Should be set to -1 for a video file (to avoid skipping frames when running an offline processing). Typically set to 8 for a live stream (processing at 8 frames per second).

The code below shows a small snippet in how to create a FaceLog4 analytic, attach a watchlist and run it on frames in a video:

#include <PapillonCore.h>
// Create and use faceLog4 analytic
void doFaceLog4(const PWatchlist& watchlist, PInputVideoStream& inputVideoStream)
{
// Set face-log options
PProperties faceLogOptions;
faceLogOptions.Set("Watchlist", watchlist);
PAnalytics faceLog;
PAnalytics::Create("FaceLog4", faceLogOptions, faceLog).OrDie();
// Loop through the video and get face-log events
PFrame frame;
PList events;
while(inputVideoStream.GetFrame(frame).Ok())
{
faceLog.Apply(frame, events);
// you can now process events (see Events section below)
}
// Mop up any left over events
faceLog.Finish(events);
// process final events...
}

Events

This analytic provides three public types of events; "Face", "Track" and "Sighting". The type of event can be checked through PEvent::GetType().

Face Events

If PEvent::GetType() returns "Face", then the event is a face event. A face event is a detection of a face in an image.

The payload of the face event includes the following information (use PEvent::GetPayload() to access these data):

Parameter Type Description
Detection PDetection The detection

Track Events

If PEvent::GetType() returns "Track", then the event is a track event. A track event is released when a track has finished. It contains the normal PEvent information. The payload consists of the following properties (use PEvent::GetPayload() to access these data):

Parameter Type Description
StartFrame int64 The start frame of the track
EndFrame int64 The end frame of the track
NumberOfFrames int32 The number of frames in the track
StartTime PDateTime The start time of the track
EndTime PDateTime The end time of the track
Thumbnail PImage A thumbnail that represents the track
FaceRectangles PList Bounding rectangles (elements are PRectanglef)
Description PDescription A description of the track
SubjectId PGuid The subject id of the track
SubjectConfidence double The confidence of the subject

If no PWatchlist was supplied to the FaceLog4 analytic then the subject information will contain the PGuid::Null() and the confidence will be set to -1.

The description will contain information about the subject in the track that can be stored in a watchlist, and later searched.

The description will also contain meta-data about the face in track, e.g. its gender.

Sighting Events

If PEvent::GetType() returns "Sighting", then the event is a sighting event.

The payload of a sighting event contains the following information (use PEvent::GetPayload() to access these data):

Parameter Type Description
Thumbnail PImage A thumbnail that represents the sighting
StartFrame int64 The start frame index
EndFrame int64 The end frame index
NumberOfFrames int32 The number of frames
StartTime PDateTime The start time of the sighting
EndTime PDateTime The end time of the track
DetectionList PDetectionList A list of all the detections in the sighting
TrackList PList A list of all the track ids (PGuid objects) in the sighting
SubjectId PGuid The global id of the subject in the sighting, PGuid::Null() if not known
SubjectConfidence double The confidence of the identity, range between 0 and 1
Description PDescription The description of the subject (you can store this in a watchlist, see PWatchlist::Add())

The following snippet shows how to process events generated by FaceLog4:

void ProcessEvents(const PList& events)
{
for (int32 i=0; i<events.Size(); ++i)
{
PEvent event;
if (events.Get(i, event).Failed()) continue;
PProperties payload = event.GetPayload();
if (event.GetType() == "Sighting"))
{
// For example we can get subjectId from sighting
PGuid subjectId;
if (sighting.GetPayload().Get("SubjectId", subjectId).Ok())
{
// do something...
}
}
}
}

Example

/*
* Copyright (C) 2015-2018 Digital Barriers plc. All rights reserved.
* Contact: http://www.digitalbarriers.com/
*
* This file is part of the Papillon SDK.
*
* You can't use, modify or distribute any part of this file without
* the explicit written agreements of Digital Barriers.
*/
// Papillon
#include <PapillonCore.h>
#include "PFaceLog4Parameters.h"
// STL
#include <algorithm>
#include <map>
#include <queue>
#include <sstream>
#include <unordered_map>
USING_NAMESPACE_PAPILLON
const PString SAMPLE_DIR = PPath::Join(PUtils::GetEnv("PAPILLON_INSTALL_DIR"),
"Data",
"Samples"); // path to find sample data: $PAPILLON_INSTALL_DIR/Data/Samples
PDetector CreateFaceDetector(const PString& detectorName, const PString& localiserName, const PString& parametersStr) {
P_LOG_DEBUG << "parametersStr:" << parametersStr;
PProperties parameters = PProperties::CreateFromKeyValueString(parametersStr);
P_LOG_INFO << "Creating face detector:" << detectorName;
if(!localiserName.IsEmpty()) {
PDetector localiser;
PProperties localiserProperties =
PProperties::CreateFromKeyValueString("maxNumDetectors=0;enableLocaliser=true");
if(!PDetector::Create(localiserName, localiserProperties, localiser).LogIfError()) {
P_LOG_ERROR << "Failed to create face localiser.";
throw std::runtime_error("Failed to create face localiser");
}
parameters.Set(PDetector::C_PARAM_DETECTOR_LOCALISER, localiser);
}
parameters.Set("enableLocaliser", true);
PDetector detector;
P_LOG_DEBUG << "Parameters" << parameters.ToStringLog();
PResult res = PDetector::Create(detectorName, parameters, detector);
if(res.Failed()) {
P_LOG_ERROR << "Failed to create face detector:" << res;
throw std::runtime_error("Failed to create face detector");
}
return detector;
}
void CreateTemplateDescriber(PDescriber& describer,
const PString& params,
int batchSize,
int gpuId,
bool sharedMode,
bool doThumbnails) {
P_LOG_INFO << "Creating face recognition describer...";
P_LOG_INFO << "describerBatchSize: " << batchSize;
PString gpuString = PString("gpuId=") + PString(gpuId);
P_LOG_INFO << "GPU:" << gpuString;
PProperties dnnParameters = PProperties::CreateFromKeyValueString(params + ";type=Face;" + gpuString +
PString(";batchSize=%1").Arg(batchSize));
if(doThumbnails) {
dnnParameters.Set("descriptorThumbnail", PBool(true));
}
if(sharedMode) {
dnnParameters.Set("sharedDnn", true);
}
PDescriber::Create("DescriberDnn", dnnParameters, describer).OrDie();
}
PResult HandleSighting(const PEvent& sighting, const PWatchlist& watchlist, bool writeVideo, bool display = false) {
const PGuid& sightingId = sighting.GetSource();
const PString shortSightingId = sightingId.ToString().Substring(0, 3);
const PString videoFile = PString("sighting_%0.avi?encode_with=opencv&fourcc=XVID&fps=10").Arg(shortSightingId);
const PString thumbnailFile = PString("thumbnail_%0.jpg").Arg(shortSightingId);
POutputVideoStream ovs;
if(writeVideo) {
if(POutputVideoStream::Open(videoFile, ovs).Failed()) {
P_LOG_ERROR << "Failed to open " << videoFile << " for writing";
writeVideo = false;
}
}
// Each sighting has a thumbnail
PImage thumbnail;
sighting.GetPayload().Get(papillon::C_PROPERTY_THUMBNAIL, thumbnail);
if(display)
thumbnail.Display("Sighting", 40);
// We can get all the detections from sighting
PDetectionList detectionList;
sighting.GetPayload().Get("DetectionList", detectionList);
// We can get subjectId from sighting
PGuid subjectId;
sighting.GetPayload().Get("SubjectId", subjectId);
// We can get confidence we are of subject
double subjectConfidence = -1.0;
sighting.GetPayload().Get("SubjectIdConfidence", subjectConfidence);
// P_LOG_INFO << "payload:" << sighting.GetPayload();
// We can also get the description
PDescription description;
sighting.GetPayload().Get("Description", description);
// We can also get start and end frames
int32 startFrame, endFrame;
sighting.GetPayload().Get("StartFrame", startFrame);
sighting.GetPayload().Get("EndFrame", endFrame);
// In the description we can get the face meta-data
PFaceMetaData faceMetaData = PFaceMetaData::FromDescription(description);
// We go to the watch list to get the identity
PString name = "Unknown";
PDescription wd;
if(watchlist.GetDescription(subjectId, wd))
name = PString("%0: %1").Arg(wd.GetName()).Arg(subjectConfidence, 0, 'g', 2);
// Draw label, but not bounding box
PUtils::ELabelColourScheme colourScheme = PUtils::E_ORANGE;
auto metaClassifications = faceMetaData.GetClassificationNames();
PString classLabel;
if(metaClassifications.Size() == 1) {
const PString& classification = metaClassifications.Get(0);
int32 classIndex{-1};
float classConfidence{0.f};
if(faceMetaData.GetClassIndex(classification, classIndex, classConfidence).Ok()) {
faceMetaData.GetClassLabel(classification, classLabel);
if(classIndex == 0)
colourScheme = PUtils::E_PINK;
else if(classIndex == 1)
colourScheme = PUtils::E_BLUE;
}
}
// Now we want to go through the actual detections in the sighting
for(int32 j = 0; j < detectionList.Size(); ++j) {
PDetection detection = detectionList.Get(j);
PImage image = detection.GetFrame().GetImage().Clone();
// Lets draw a label at the top of the image specifying the frame number and timestamp
int32 centreWidth = image.GetWidth() / 2;
image.DrawLabel(PString("%0 Frame #: %1")
.Arg(detection.GetFrame().GetTimestampUTC().ToStringISO())
.Arg(detection.GetFrame().GetFrameNumber()),
PPoint2Di(centreWidth, 20));
// Lets draw a label at the bottom with the sighting id and meta-data
PUtils::DrawLabel(image, detection, PString("Id: %0 : %1").Arg(shortSightingId).Arg(classLabel),
PUtils::E_BOTTOM_CENTRE, colourScheme);
// Draw label at the top with the identity
PUtils::ELabelColourScheme colourScheme2 = PUtils::E_BLACK;
if(subjectConfidence > 0.0)
colourScheme2 = PUtils::E_GREEN;
PUtils::DrawLabel(image, detection, name, PUtils::E_TOP_CENTRE, colourScheme2, 1.0, 0);
// Display the image
if(display)
image.Display("Sighting", 40);
// Write video if asked
if(writeVideo)
ovs.PutFrame(image);
}
P_LOG_INFO << PString("Sighting: %0 Id: %1 Frames: (%4,%5) Name: '%2' Confidence: %3")
.Arg(sightingId.ToString())
.Arg(subjectId.ToString())
.Arg(wd.GetName())
.Arg(subjectConfidence)
.Arg(startFrame)
.Arg(endFrame);
// P_LOG_DEBUG << "Event:" << sighting;
return PResult::C_OK;
}
PResult ProcessEvents(const PList& events, const PWatchlist& watchlist, bool writeVideo, bool display) {
// Got through each event
for(int32 i = 0, ni = events.Size(); i < ni; ++i) {
PEvent event;
events.Get(i, event);
P_LOG_TRACE << "Event: " << event.GetType();
if(event.GetType() == "Sighting")
HandleSighting(event, watchlist, writeVideo, display);
PDescription description;
if(event.GetPayload().Get("Description", description).Ok()) {
PFaceMetaData faceMetaData = PFaceMetaData::FromDescription(description);
P_LOG_TRACE << event.GetType() << " (" << i << "): " << faceMetaData;
}
}
return PResult::C_OK;
}
class CTrack {
public:
CTrack(int startFrame, const PPoint2Df& point)
: m_startFrame{startFrame}
, m_lastFrame{startFrame} {
m_track.push_back(point);
static int colourIdx = 0;
m_clolour = PColour3i::C_COLOUR_TABLE[colourIdx];
colourIdx = (colourIdx + 1) % (sizeof(PColour3i::C_COLOUR_TABLE) / sizeof(PColour3i));
}
void AddPoint(int curFrame, PPoint2Df point) {
for(int i = m_lastFrame; i < curFrame; i++) {
m_track.push_back(point);
}
m_lastFrame = curFrame;
}
std::vector<PPoint2Df> m_track;
int m_startFrame;
int m_lastFrame;
PColour3i m_clolour;
};
class CShowEvents {
public:
explicit CShowEvents(bool skipPlotting, bool showPoints, bool drawTracks)
: m_skipImage(false)
, m_framesTotal(0)
, m_framesSkipped(0)
, m_totalFaces(0)
, m_skipPlotting(skipPlotting)
, m_showPoints(showPoints)
, m_drawTracks(drawTracks) {}
void SetWatchList(const PWatchlist& watchlist) { m_watchlist = watchlist; }
void SetRoi(const PRectanglei& roi) { m_roi = roi; }
void StartTimer() {
m_timer = PTimer();
m_framesTotal = m_framesSkipped = m_totalFaces = 0;
}
void QueueEvents(const PList& events) {
for(int32 i = 0, ni = events.Size(); i < ni; i++) {
PEvent event;
events.Get(i, event);
m_eventQueue.push(event);
}
}
bool ShowEvents() {
bool imageReady = false;
while(!m_eventQueue.empty()) {
PEvent event = m_eventQueue.front();
m_eventQueue.pop();
const PString& eventType = event.GetType();
P_LOG_TRACE << "Processing event:" << eventType;
if(eventType == "Face") {
m_totalFaces++;
if(!m_skipPlotting) {
const PProperties& eventProperties = event.GetPayload();
PDetection detection;
if(!eventProperties.Get("Detection", detection)) {
P_LOG_ERROR << "Failed to get detection";
continue;
}
// show detection
// P_LOG_INFO << "detection:" << detection.GetFaceRectangle().GetRectangle();
// get meta data
PGuid subjectId;
if(!eventProperties.Get("SubjectId", subjectId))
P_LOG_ERROR << "Failed to get subjectId";
double subjectIdConfidence = 0;
if(!eventProperties.Get("SubjectIdConfidence", subjectIdConfidence))
P_LOG_ERROR << "Failed to get subjectIdConfidence";
P_LOG_TRACE << "Subject:" << subjectId << " confidence:" << subjectIdConfidence;
PDescription description;
if(!eventProperties.Get("Description", description))
P_LOG_ERROR << "Failed to get description";
PDescription subjectDescription;
bool subjectKnown = false;
if(m_watchlist.GetDescription(subjectId, subjectDescription).Ok())
subjectKnown = true;
PFaceMetaData faceMetaData = PFaceMetaData::FromDescription(description);
const PString classificationForChoosingColours{"mask"};
auto detectionColour{PColour3i::Red()};
{
int32 classIndex{-1};
float classConfidence{0.f};
if(faceMetaData.GetClassIndex(classificationForChoosingColours, classIndex, classConfidence)) {
detectionColour = (classIndex == 1)
? PColour3i::Blue()
: ((classIndex == 0) ? PColour3i::Pink() : PColour3i::White());
}
}
// Draw bounding box
PRectanglei detRect =
detection.GetFeatureMap().GetBoundingBox().Translated(detection.GetOrigin()).ToPRectanglei();
if(!m_workImage.DrawRectangle(detRect, detectionColour, 1)) {
P_LOG_ERROR << "Failed to draw detection rectangle";
}
//PUtils::DrawDetection(m_workImage, detection, detectionColour, false, !m_showPoints);
// Draw meta data labels
//P_LOG_INFO << "faceInfo:" << faceMetaData;
//P_LOG_INFO << "classes: " << faceMetaData.GetClassificationNames();
{
PPoint2Di at{int32(detRect.GetCentre().GetX()), detRect.GetY() + detRect.GetHeight()};
auto metaClassifications = faceMetaData.GetClassificationNames();
for(int i = 0, n = metaClassifications.Size(); i < n; i++) {
const PString& classification = metaClassifications.Get(i);
int32 classIndex{-1};
float classConfidence{0.f};
if(faceMetaData.GetClassIndex(classification, classIndex, classConfidence).Ok()) {
PString classLabel;
faceMetaData.GetClassLabel(classification, classLabel);
if(!m_workImage.DrawLabel(classLabel, at, detectionColour, PColour3i::White())) {
P_LOG_ERROR << "Failed to draw classification label";
}
at = at.Translated(0, -21);
}
}
}
// Draw name label
{
PColour3i fontColour(0, 0, 0);
PString tagColour = "#95a5a6";
PString message;
if(subjectKnown) {
PString tag;
PString name =
PString("%0: %1").Arg(subjectDescription.GetName()).Arg(subjectIdConfidence, 0, 'g', 2);
if(subjectDescription.GetProperties().Get("Tag", tag).Ok()) {
subjectDescription.GetProperties().Get("TagColour", tagColour);
message = PString("%0 @ %1").Arg(name).Arg(tag);
} else {
message = name;
}
} else {
message = "Unknown";
}
PPoint2Di at{int32(detRect.GetCentre().GetX()), detRect.GetY()};
if(!m_workImage.DrawLabel(message, at, PColour3i(tagColour), fontColour)) {
P_LOG_ERROR << "Failed to draw name label";
}
}
// add face to tracks
PGuid sightingId;
if(!eventProperties.Get("SightingId", sightingId)) {
P_LOG_ERROR << "Failed to get sightingId";
continue;
}
auto track = m_tracks.find(sightingId);
if(track == m_tracks.end()) {
m_tracks.emplace(sightingId, CTrack(detection.GetFrame().GetFrameNumber(),
detection.GetFaceRectangle().GetRectangle().GetCentre()));
} else {
track->second.AddPoint(detection.GetFrame().GetFrameNumber(),
detection.GetFaceRectangle().GetRectangle().GetCentre());
}
// show thumbnail for the track
//{
// PImage thumbnail;
// event.GetPayload().Get(papillon::C_PROPERTY_THUMBNAIL, thumbnail);
// thumbnail.Display(sightingId.ToString());
//}
}
} else if(eventType == "FrameStart") {
if(!m_skipPlotting) {
const PProperties& eventProperties = event.GetPayload();
PFrame frame;
if(!eventProperties.Get("Frame", frame)) {
P_LOG_ERROR << "Failed to get frame";
continue;
}
P_LOG_TRACE << "Went on new frame:" << frame.GetFrameNumber();
static int lastFrameNo = 0;
if(frame.GetFrameNumber() - lastFrameNo > 1) {
P_LOG_ERROR << "Missed frames between: " << lastFrameNo << " - " << frame.GetFrameNumber();
}
lastFrameNo = frame.GetFrameNumber();
m_workImage = frame.GetImage().Clone();
// show roi
if(m_roi.IsValid()) {
m_workImage.DrawRectangle(m_roi, PColour3i(255, 0, 0), 1);
}
}
m_skipImage = false;
} else if(eventType == "FrameEnd") {
// draw tracks
const PProperties& eventProperties = event.GetPayload();
PFrame frame;
if(!eventProperties.Get("Frame", frame)) {
P_LOG_ERROR << "Failed to get frame";
continue;
}
int startFrame = frame.GetFrameNumber() - 50;
if(m_drawTracks) {
for(auto track : m_tracks) {
int startIdx = std::max(0, startFrame - track.second.m_startFrame);
if(startIdx >= track.second.m_track.size()) {
continue;
}
PPoint2Di prevPoint = track.second.m_track[startIdx].ToPPoint2Di();
for(int i = startIdx + 1; i < track.second.m_track.size(); i++) {
PPoint2Di curPoint = track.second.m_track[i].ToPPoint2Di();
m_workImage.DrawLine(prevPoint.GetX(), prevPoint.GetY(), curPoint.GetX(), curPoint.GetY(),
track.second.m_clolour, 2);
prevPoint = curPoint;
}
}
}
// clean old tracks
int frameThreshold = frame.GetFrameNumber() - 50;
auto it = m_tracks.cbegin();
while(it != m_tracks.cend()) {
if(it->second.m_lastFrame < frameThreshold) {
it = m_tracks.erase(it);
} else {
++it;
}
}
++m_framesTotal;
if(!m_skipImage) {
m_readyImage = m_workImage;
imageReady = true;
break;
}
} else if(eventType == "FrameSkipped") {
++m_framesSkipped;
m_skipImage = true;
}
}
return imageReady;
}
const PImage& GetImage() const { return m_readyImage; }
const PString GetFpsString() const {
return PString("fps %0 (%1) faces per sec %3 faces per frame %4")
.Arg(double(m_framesTotal - m_framesSkipped) / m_timer.ElapsedSec(), 4, 'f', 2)
.Arg(double(m_framesTotal) / m_timer.ElapsedSec(), 4, 'f', 1)
.Arg(double(m_totalFaces) / m_timer.ElapsedSec(), 6, 'f', 1)
.Arg(double(m_totalFaces) / double(m_framesTotal - m_framesSkipped), 4, 'f', 1);
}
private:
PImage m_workImage;
bool m_skipImage;
PImage m_readyImage;
PWatchlist m_watchlist;
PTimer m_timer;
int64 m_framesTotal;
int64 m_framesSkipped;
int64 m_totalFaces;
PRectanglei m_roi;
bool m_skipPlotting;
bool m_showPoints;
bool m_drawTracks{false};
std::map<PGuid, CTrack> m_tracks;
std::queue<PEvent> m_eventQueue;
};
int CountProcessedFrames(const PList& events) {
int result = 0;
for(int32 i = 0, ni = events.Size(); i < ni; ++i) {
PEvent event;
events.Get(i, event);
const PString& eventType = event.GetType();
if(eventType == "FrameEnd")
++result;
else if(eventType == "FrameSkipped")
--result;
}
return result;
}
PResult ConvertWatchlist(PWatchlist& watchlist, const PDetector& detector, const PDescriber& describer) {
if(watchlist.Size() < 1) {
return PResult::Error("Empty watchlist");
}
// find out most popular describerId
PGuid oldDescribeId;
{
std::unordered_map<PGuid, int> counter;
PList wlItems = watchlist.GetAllDescriptions();
for(int wlIndex = 0; wlIndex < std::min(wlItems.Size(), 20); wlIndex++) {
PDescription description;
if(!wlItems.Get(wlIndex, description)) {
P_LOG_ERROR << "Failed to read description from the list";
continue;
}
PList descriptors = description.GetDescriptors();
for(int i = 0; i < descriptors.Size(); ++i) {
PDescriptor descriptor;
if(descriptors.Get(i, descriptor).Ok()) {
if(descriptor.GetObjectType() == PObjectType::C_FACE) {
++counter[descriptor.GetDescribeId()];
}
}
}
}
int popularity{0};
for(const auto& p : counter) {
P_LOG_INFO << "describeId: " << p.first << " popularity: " << p.second;
if(popularity < p.second) {
popularity = p.second;
oldDescribeId = p.first;
}
}
if(popularity < 1) {
return PResult::Error("Can't find describerId to replace");
}
}
P_LOG_INFO << "Old describe id:" << oldDescribeId;
#if 0
auto res = watchlist.RecomputeDescriptors(oldDescribeId, detector, describer);
P_LOG_INFO << "Number of descriptions converted:" << res;
if(res < 1) {
return PResult::Error("Failed to convert watchlist");
}
#else
PList wlItems = watchlist.GetAllDescriptions();
for(int wlIndex = 0; wlIndex < wlItems.Size(); wlIndex++) {
PDescription description;
if(!wlItems.Get(wlIndex, description)) {
P_LOG_ERROR << "Failed to read description from the list";
continue;
}
// create procedural "feature map" once and for all
// assumes size of the thumbnail is 240 x 320 pixels and eyes coordinates are Right=(80,144) and Left=(150,144)
PGuid manualDetectorId = PGuid("{cccccccc-f8f9-0000-0000-000000000000}"); // fake detector
float confidence = 1.0;
PFeatureRectangle boundingBoxDetection(PRectanglef(0.0f, 0.0f, 240.0f, 320.0f), "FaceRectangle", confidence);
PFeatureMap proceduralFeatureMap;
proceduralFeatureMap.Add(boundingBoxDetection);
PFeaturePoint leftEye(PPoint2Df(150.0f, 144.0f), "LeftEyeCentre", confidence);
PFeaturePoint rightEye(PPoint2Df(80.0f, 144.0f), "RightEyeCentre", confidence);
PFeaturePoint mouthCentre(PPoint2Df(120.0f, 204.0f), "MouthCentre", confidence);
proceduralFeatureMap.Add(leftEye);
proceduralFeatureMap.Add(rightEye);
proceduralFeatureMap.Add(mouthCentre);
PList descriptorsFR08 = description.GetDescriptors(oldDescribeId);
for(int i = 0; i < descriptorsFR08.Size(); ++i) {
PDescriptor descriptorFR08;
if(descriptorsFR08.Get(i, descriptorFR08).LogIfError().Failed())
continue;
PImage descriptorImage = descriptorFR08.GetDataSourceAsImage();
if(descriptorImage.IsEmpty()) {
P_LOG_ERROR << "FaceLog Thread: import watchlist: failed to convert descriptor "
<< descriptorFR08.GetDescriptorId() << ": no image attached";
continue;
}
PFrame frameForThumbnail(descriptorImage);
// 2. create a procedural PDetection (to avoid running any FaceDetector which could fail)
// from the thumbnail
PDetection detection(PObjectType::C_FACE, manualDetectorId, frameForThumbnail, proceduralFeatureMap,
confidence);
// 3. compute descriptor for this PDetection with FR09 describe engine
PDescription descriptionFR09;
describer.Describe(detection, descriptionFR09);
if(descriptionFR09.GetNumberOfDescriptors() != 1) {
P_LOG_ERROR << "FaceLog Thread: import watchlist: failed to convert descriptor "
<< descriptorFR08.GetDescriptorId() << ": expected 1 descriptor only";
continue;
}
PDescriptor descriptorFR09;
descriptionFR09.GetFirstDescriptor(descriptorFR09);
// replace FR08 descriptor by FR09 one
PByteArray dataFR09;
if(descriptorFR09.GetDescriptor(dataFR09).LogIfError().Failed())
continue;
descriptorFR08.SetDescriptor(dataFR09);
descriptorFR08.SetDescribeId(describer.GetDescribeId());
P_LOG_INFO << "FaceLog Thread: import watchlist: descriptor " << descriptorFR08.GetDescriptorId()
<< " has been converted from model FR08 to FR09";
}
}
#endif
return PResult::C_OK;
}
PResult RunDemo(int argc, char** argv) {
POption opt(argc, argv);
opt.AddStandardOptions(); // set-up logging
PLog::SetConsoleFormat("date (sev) [sf:line]: msg");
// clang-format off
PString inputVideoFile = opt.String ("inputVideo,iv", PPath::Join(SAMPLE_DIR, "busy_office.avi"), "Input video file");
PString outputVideoFile = opt.String ("outputVideo,ov", "", "Output video file");
int32 maxFrames = opt.Int (",f", 0, "Maximum number of frames to process");
int32 skip = opt.Int ("skip,skip", 0, "Skip this number of frames at the beginning");
bool createWatchlist = opt.Bool ("createWatchlist,cw", false, "Create a watchlist from the watchlistConfigFile");
PString watchlistConfigFile = opt.String ("watchlistConfig,wlc", PPath::Join(SAMPLE_DIR, "watchlist.config"), "Use a config file to build a watchlist");
PString watchlistFile = opt.String ("watchlist,wl", "", "Load watchlist from file");
PString faceLogName = opt.String ("facelog,fl", "FaceLog4", "FaceLog plugin name (FaceLog4 or FaceLog6)");
PString faceLogParsString = opt.String ("facelogPars,flp", "", "FaceLog plugin parameters (ex par1=v1;par2=v2)");
double watchlistThreshold = opt.Double ("watchlistThreshold,wt", 0.75, "The threshold to use for identification");
bool display = !opt.Option ("noDisplay,nd", "Display video as we go");
bool displayPoints = opt.Option ("displayPoints", "Display face points");
bool displayTracks = opt.Option ("displayTracks", "Display tracks");
bool displayEvents = !opt.Option ("noDisplayEvents,nde", "Display events as we go");
//bool motion = opt.Bool ("motion,m", false, "Use motion detection to speed up initial face-detection");
bool writeSightings = opt.Bool ("saveSightings,ss", false, "Write labeled video for each sighting");
int32 delay = opt.Int ("delay,delay", 0, "Delay in ms between frames");
PString roi = opt.String ("roi,roi", "", "Region of interest (ex. 100,150,600,400)");
int fdGpuId = opt.Int ("fdGpuId,fdGpuId", -1, "Face detector GpuId");
int dGpuId = opt.Int ("dGpuId,dGpuId", -1, "Describer GpuId");
bool doThumbnails = opt.Bool ("thumbnails,th", false, "Create thumbnails in describer");
PString detectorName = opt.String ("detector,d", "FaceDetector2", "Face detector name");
PString detectorParsString = opt.String ("detectorPars,fdp", "", "Detector parameters (ex par1=v1;par2=v2)");
PString describerParsString = opt.String ("describerPars,dp", "", "Describer parameters (ex par1=v1;par2=v2)");
PString localiserName = opt.String ("localiser,loc", "FaceDetector2", "Face localiser name");
PString metaDescriberPars = opt.String ("mdPars,mdPars", "", "Meta describer parameters (if empty create gender describer)");
PString metaDescriberPars2 = opt.String ("mdPars2,mdPars2", "", "Meta describer parameters for second describer");
// clang-format on
ReturnIfFailed(opt.Check());
if(opt.Has("h") || opt.Has("help")) {
P_LOG_INFO << opt.ToStringHelp();
return PResult::C_OK;
}
POutputVideoStream ovs;
P_LOG_DEBUG << "output video ------'" << outputVideoFile << "'-------------";
if(!outputVideoFile.IsEmpty()) {
if(!outputVideoFile.Contains("?")) {
outputVideoFile += "?encode_with=opencv&fourcc=XVID&fps=25";
}
P_LOG_INFO << "Will write result video to: " << outputVideoFile;
if(POutputVideoStream::Open(outputVideoFile, ovs).Failed()) {
P_LOG_ERROR << "Failed to open " << outputVideoFile << " for writing";
return PResult::Error("Failed to open output video file");
}
}
// Open video stream
PInputVideoStream ivs;
ReturnIfFailed(PInputVideoStream::Open(PString("%0").Arg(inputVideoFile), ivs));
PFaceLog4Parameters faceLogParameters = PFaceLog4Parameters::BuildDefaultParameters();
if(!faceLogParsString.IsEmpty()) {
PProperties newProps = PProperties::CreateFromKeyValueString(faceLogParsString);
PList keys = newProps.GetKeyList();
for(int32 i = 0; i < keys.Size(); ++i) {
PString key;
if(keys.Get(i, key).Ok()) {
faceLogParameters.Set(key, newProps.GetAsString(key));
}
}
}
// create detector
detectorParsString += PString(";gpuId=") + PString(fdGpuId);
PDetector detector = CreateFaceDetector(detectorName, localiserName, detectorParsString);
faceLogParameters.Set("FaceDetector", detector);
// create face recognition describer
PDescriber faceDescriber;
CreateTemplateDescriber(faceDescriber, describerParsString, 4, dGpuId, true, doThumbnails);
faceLogParameters.Set("FaceRecognitionDescriber", faceDescriber);
{
PList metaDescribers;
if(metaDescriberPars.IsEmpty()) {
metaDescriberPars = PString("gpuId=") + PString(dGpuId) + PString(";type=Gender");
}
PDescriber metaDescriber;
P_LOG_INFO << "Creating meta data describer...";
PProperties parameters = PProperties::CreateFromKeyValueString(metaDescriberPars);
PDescriber::Create("DescriberDnn", parameters, metaDescriber).OrDie();
metaDescribers.Add(metaDescriber);
if(!metaDescriberPars2.IsEmpty()) {
P_LOG_INFO << "Creating meta data describer 2...";
parameters = PProperties::CreateFromKeyValueString(metaDescriberPars2);
PDescriber::Create("DescriberDnn", parameters, metaDescriber).OrDie();
metaDescribers.Add(metaDescriber);
}
faceLogParameters.Set("MetaDescribers", metaDescribers);
}
// If a watch list is provided then an identity check will be performed
// for each sighting. For this you need to specify a watchlist to use.
// You can either provide a watchlist from file or provide a config
// file and a watchlist will be created for you. If you do not provide a
// watchlist no identification will be done, all the sightings will be unknown.
PWatchlist watchlist;
if(createWatchlist) {
P_LOG_INFO << "Creating watchlist using configuration file " << watchlistConfigFile;
PEnrollment enrollment;
LogAndReturnIfFailed(enrollment.CreateWatchlist(watchlistConfigFile, watchlist));
} else if(!watchlistFile.IsEmpty()) {
LogAndReturnIfFailed(PFileIO::ReadFromFile(watchlistFile, watchlist));
P_LOG_INFO << "Loaded watchlist " << watchlistFile << " with " << watchlist.Size() << " subjects";
if(!watchlist.IsCompatibleWith(faceDescriber.GetDescribeId())) {
P_LOG_INFO << "Watchlist is not compatible with describer. Will try to convert it.";
if(!ConvertWatchlist(watchlist, detector, faceDescriber).LogIfError()) {
return PResult::Error("Failed to convert watchlist");
}
}
} else {
P_LOG_INFO << "Not using a watchlist, anonymous sightings will be reported.";
}
// Set up the watchlist and the options for facelog
PWatchlistOptions watchlistOptions;
watchlistOptions.SetNormalise(false);
watchlistOptions.SetTopN(1);
watchlistOptions.SetThreshold(static_cast<float>(watchlistThreshold));
faceLogParameters.SetWatchlist(watchlist);
faceLogParameters.SetWatchlistOptions(watchlistOptions);
{
P_LOG_DEBUG << "FaceLog parameters:";
PList keys = faceLogParameters.GetKeyList();
for(int32 i = 0; i < keys.Size(); ++i) {
PString key;
if(keys.Get(i, key).Ok()) {
if(key == "Watchlist") {
P_LOG_DEBUG << "Watchlist:";
PList descriptions = faceLogParameters.GetWatchlist().GetAllDescriptions();
for(int32 j = 0; j < descriptions.Size(); ++j) {
PDescription d;
if(descriptions.Get(j, d).LogIfError().Ok()) {
P_LOG_DEBUG << j << " : " << d.GetIdentityId() << " : " << d.GetName();
}
}
} else {
P_LOG_DEBUG << key << " => " << faceLogParameters.GetAsString(key);
}
}
}
}
// Finally create the face-logger
PAnalytics faceLog;
ReturnIfFailed(PAnalytics::Create(faceLogName, faceLogParameters, faceLog));
// Specify a Region-Of-Interest (optional)
PRectanglei roiRect;
if(!roi.IsEmpty()) {
std::stringstream ss(std::string(roi.c_str()));
std::vector<int> v;
int i;
while(ss >> i) {
v.push_back(i);
if(ss.peek() == ',')
ss.ignore();
}
if(v.size() != 4) {
return PResult::Error("Wrong region of interest");
}
roiRect = PRectanglei(v[0], v[1], v[2], v[3]);
}
PList events;
int frameNumber = 0;
CShowEvents eventShow(!display && !ovs.IsOpened(), displayPoints, displayTracks);
eventShow.SetWatchList(watchlist);
if(roiRect.IsValid()) {
faceLog.Set("roi", roiRect);
eventShow.SetRoi(roiRect);
}
while(true) {
PFrame frame;
if(!ivs.GetFrame(frame).LogIfError().Ok())
break;
// draw frame number
// frame.GetImageShared().DrawLabel(PString("%0 Frame #: %1").Arg(frame.GetTimestampUTC().ToStringISO())
// .Arg(frame.GetFrameNumber()),
// PPoint2Di(200, 20));
if(display) {
PImage displayImage = frame.GetImage().Clone();
displayImage.Display("Source", 1);
}
if(faceLog.Apply(frame, events).LogIfError().Failed())
break;
frameNumber++;
if(frameNumber < skip)
continue;
eventShow.QueueEvents(events);
if(eventShow.ShowEvents()) {
const PImage& img = eventShow.GetImage();
if(!img.IsEmpty() && ovs.IsOpened()) {
if(!ovs.PutFrame(img).LogIfError()) {
P_LOG_ERROR << "Error writing video. Stopping...";
break;
}
}
if(display) {
img.Display("Face Log").LogIfError();
}
}
P_LOG_DEBUG << eventShow.GetFpsString();
// Let process the events
ProcessEvents(events, watchlist, writeSightings, displayEvents).LogIfError();
// Check if we are exiting early
if(frameNumber == maxFrames) {
P_LOG_INFO << "Maximum number of frames reached. Stop processing...";
break;
}
// delay
if(delay > 0) {
PConcurrentThread::Sleep(delay);
}
}
// OK we have finished video, lets check we have no events hanging around
faceLog.Finish(events).LogIfError();
eventShow.QueueEvents(events);
while(eventShow.ShowEvents()) {
const PImage& img = eventShow.GetImage();
if(!img.IsEmpty() && ovs.IsOpened()) {
ovs.PutFrame(img).LogIfError();
}
if(display) {
img.Display("Face Log").LogIfError();
}
}
P_LOG_DEBUG << eventShow.GetFpsString();
ProcessEvents(events, watchlist, writeSightings, display).LogIfError();
return PResult::C_OK;
}
int main(int argc, char** argv) {
PapillonSDK::Initialise().OrDie();
try {
RunDemo(argc, argv).LogIfError();
} catch(const std::exception& exc) {
P_LOG_ERROR << "Exception:" << exc.what();
return -1;
} catch(...) {
P_LOG_ERROR << "Unknown exception.";
return -1;
}
PapillonSDK::Finalise();
return 0;
}