FaceLog6 (Real-time face identification)

Description

FaceLog6 is an advanced analytic that tracks unique faces in a video stream, classifies and optionally identifies them against a supplied watchlist. This plugin offers multithread support and is able to process several high resolution video streams on a multicores CPU (typically Intel i7 with 4 cores or more).

  • Product name: FaceLog6
  • Version: 1.0

How to create a PAnalytics FaceLog6 object from this plugin?

#include <PFaceLog6Parameters.h>
PAnalytics faceLog6;
PProperties parameters = PFaceLog6Parameters::BuildDefaultParameters();
// parameters.Set("MaxFaceDetectorFR", PDouble(-1.0)); // uncomment this line to process a video file without skipping frames
// keep default value (8.0) for a live stream
PAnalytics::Create("FaceLog6", parameters, faceLog6).OrDie();

As with all PAnalytics, it works on PFrame's pulled from a video stream and releases PEvent's when they are ready (See Events section below).

Input Data

Input data is a single PFrame object, see PAnalytics::Apply(). For optimal performance, recommanded pixel format of the underlying PImage is PImage::E_BGR8U.

Face Recognition

If a PWatchlist is provided to FaceLog6 as a parameter (see Watchlist parameter below), then face recognition is performed on each detection. Inside a watchlist each subject is assigned a unique id (subject-id) when a description. If a match is found (above a user set threshold), the face/sighting is labeled with the subject-id from the watchlist. A confidence of match is also assigned.

It should be noted, the most reliable identity is from the Sighting event as it is aggregated over all detections in the sighting.

Face Recognition Performance

The face recognition algorithm in Papillon are based on a powerful new technique called Deep Learning. They are vastly superior to what was available a few years ago. However, they still have their limitations.

You will get best performance when the faces in the video are within +/-30 degrees from frontal, the faces are diffusely illuminated (with no strong shadows) and there is a minimum of 50 pixels between the eyes. The algorithms will still function outside these parameters, however the performance will be degraded. Ideally, face-recognition should be used as an aid to a human operator performing their task, helping them reduce the amount of data they have to search or ordering the priority of data they have to review.

Read Face Recognition to learn more.

Face MetaData Classifier

A face-meta classifier is applied to the faces in each sighting. This provides extra information about the faces in the track and sighting. Currently this is limited to Gender information.

Example: how to extract gender information from a "Face" event?

#include <PapillonCore.h>
PString ExtractGender(const PEvent& event)
{
if (event.GetType() != "Face")
return PString::Empty();
const PProperties& eventPayload = event.GetPayload();
PDescription description;
if (eventPayload.Get("Description", description).Failed())
return PString::Empty();
PFaceMetaData faceMetaData = PFaceMetaData::FromDescription(description);
PString gender = faceMetaData.GetGenderAsString();
return gender;
}

Parameters (how to configure FaceLog6?)

Parameters are set through a PProperties class which are supplied to the PAnalytics::Create() factory constructor.

PFaceLog6Parameters is an helper class which can be used to help configure FaceLog6. Because it is an additional class, it is not contain in PapillonCore.h and you will have to add # include <PFaceLog6Parameters.h> in your code to use it.

We aim to have a sensible set of default parameters that work effectively in many scenarios, however given the range of different use-cases for this analytic this is not always possible.

Running several instances of FaceLog6 on the same machine and Slowdown issues: If you encounter slowdowns while running several instances of FaceLog6, check CPU and GPU usage: if 100% is reached, you should adjust parameters to reduce resource usage and get better load balancing between Analytics engines. For example, you can try to limit video stream resolution and/or limit the number of frames processed by second (see parameter MaxFaceDetectorFR below). Another option is to update your hardware and use a CPU with more cores and/or a bigger GPU.

Parameter Type Required Default Description
Watchlist PWatchlist No None The watchlist of identities. If not present no face identification will be performed
WatchlistOptions PWatchlistOptions No Its constructor The watchlist options; this object contains threshold for Face Recognition
Detector PString No FaceDetector2 Name of the face detector (name of the plugin to use)
MinDetectionSize int32 No 80 The minimum detection size, in pixels. This is the minimum face it will detect (minimum is 40)
MaxDetectionSize int32 No 1000 The maximum detection size, in pixels
Threshold double No 0.0 Threshold for the face-detector
ROI PRectanglei No (0,0,0,0) Region of Interest (default to whole image)
MaxFaceDetectorFR double No 8.0 Limit on number of frames per second processed by face detector. Should be set to -1 for a video file (to avoid skipping frames when running an offline processing). Typically set to 8 for a live stream (processing at 8 frames per second).
GPU bool No false Use GPU or not
MaxGap int32 No 10 Number of frames to wait before releasing Sighting event
MinLength int32 No 3 Minimum number of frames required in track before releasing Sighting event
MaxLength int32 No 100 Maximum number of frames before releasing Sighting event

The code below shows a small snippet in how to create a FaceLog6 analytic, attach a watchlist and run it on frames in a video:

#include <PapillonCore.h>
// Create and use faceLog6 analytic
void DoFaceLog6(const PWatchlist& watchlist, PInputVideoStream& inputVideoStream)
{
// Set face-log options
PProperties faceLogOptions;
faceLogOptions.Set("Watchlist", watchlist);
PAnalytics faceLog;
PAnalytics::Create("FaceLog6", faceLogOptions, faceLog).OrDie();
// Loop through the video and get face-log events
PFrame frame;
PList events;
while (inputVideoStream.GetFrame(frame).Ok())
{
// run FaceLog on current image
faceLog.Apply(frame, events);
// process events (see Events section below)
for (int32 i = 0, ni = events.Size(); i < ni; i++)
{
PEvent event;
if (events.Get(i, event).Failed()) continue;
const PProperties& eventProperties = event.GetPayload();
const PString& eventType = event.GetType();
if (eventType == "Sighting")
{
PGuid subjectId;
eventProperties.Get("SubjectId", subjectId);
PDescription watchlistDescription;
bool subjectKnown = false;
if (m_watchlist.GetDescription(subjectId, watchlistDescription).Ok())
subjectKnown = true;
PString detectionName = "Unknown";
if (subjectKnown)
detectionName = subjectDescription.GetName();
// ...
}
else if (eventType == "Face")
{
PDetection detection;
if (eventProperties.Get("Detection", detection).Failed()) continue;
PRectanglef detectionBbox = detection.GetFaceRectangle().GetRectangle();
// ...
}
}
}
// Mop up any left over events
faceLog.Finish(events);
// process final events...
}

Events

This analytic provides following events:

  • FrameStart: start to process a frame
  • FrameSkipped: skip a frame (see MaxFaceDetectorFR parameter below)
  • FrameEnd: the analysis of a frame is finished
  • Face: a face has been detected in a frame
  • Sighting: a person has been seen in a video stream (and is now no more visible)

The type of event can be checked through PEvent::GetType().

Each of these events have identifiers (PGuid) to be able to link them together (e.g. to know that a face is the same person between consecutive frames). See documentation below (section Face Events) to learn more.

Because of asynchronous processing, several frame events can sometimes be released at the the same time. In the opposite, list of events can be empty for some calls to PAnalytics::Apply().

What is a sighting?

A sighting represents a person seen in a video stream. It is a collection of "faces" of the same subject. The sightings do not need to be in successive frames, there can be a break (lag) between them.

For example, often a sighting of a person walking through a video can be lost; for example, if they turn their head, or become obscured by someone else walking in front of them. In FaceLog6 we attempt to track of a person using face-recognition.

A sighting event is generated when a person has left the scene for more than 10 frames (value by default, see MaxGap parameter below).

Frame Events

These events introduced to put "milestones" in the events stream:

  • "FrameStart" event signals that next events were generated from processing of this frame.
  • "FrameEnd" event marks that there will be no more events generated from this frame.
  • "FrameSkipped" shows that the frame was skipped from processing. This normally happens when the limit on number of frames to be processed is set in order to reduce CPU/GPU load.

The payload of the frame events includes the following information (use PEvent::GetPayload() to access these data):

Parameter Type Description
Frame PFrame The frame

Example: how to get frame number and timestamp for "FrameStart" event?

if (event.GetType() == "FrameStart")
{
PFrame frame;
if (event.GetPayload().Get("Frame", frame).Ok())
{
int frameNumber = frame.GetFrameNumber();
PDateTime timestamp = frame.GetTimestampUTC();
// ...
}
}

Face Events

If PEvent::GetType() returns "Face", then the event is a face event. A face event is a detection of a face in an image.

The payload of the face event includes the following information (use PEvent::GetPayload() to access these data):

Parameter Type Description
Thumbnail PImage Thumbnail of the face
Detection PDetection The detection
SubjectId PGuid Id of the subject if identified (PGuid::Null() id if unknown)
SubjectIdConfidence PFloat Confidence of identification (see below)
SightingId PGuid Id of the sighting (this id is the link between several faces of the same person)
Description PDescription Description of this detection together with the sighting

If no PWatchlist was supplied to the FaceLog6 analytic then the subject information will contain the PGuid::Null() and the confidence will be set to -1.

About SubjectIdConficence: This is a float number representing the identification score.

  • For Unknown people, this score will be -1 for the first two frames, then it will become 1 from the third frame, indicating that an "Unknown" person has been detected and this detection is confirmed. The -1 value should be interpreted as "probably a face"; according to your needs, you can choose to display bounding boxes on early detection or prefer to wait for "confirmed" faces.
  • For Known people, this score is typically in the range ]-1, 1] but can sometimes exceed 1, depending on the underlying algorithm.

About Description:

  • The description will contain information about the subject in the sighting that can be stored in a watchlist, and later searched.
  • The description will also contain meta-data about the face in sighting, e.g. its gender.

Sighting Events

If PEvent::GetType() returns "Sighting", then the event is a sighting event.

The payload of a sighting event contains the following information (use PEvent::GetPayload() to access these data):

Parameter Type Description
Thumbnail PImage A thumbnail that represents the sighting
StartFrame PInt64 The start frame index
EndFrame PInt64 The end frame index
NumberOfFrames PInt64 The number of frames
StartTime PDateTime The start time of the track
EndTime PDateTime The end time of the track
SubjectId PGuid The global id of the subject in the sighting, PGuid::Null() if not known
SubjectIdConfidence PFloat The confidence of the identity (see below)
SightingId PGuid Id of the sighting (this id is the link between several faces of the same person)
Description PDescription The description of the subject (you can store this in a watchlist, see PWatchlist::Add())

About SubjectIdConfidence: This is a float number representing the identification score.

  • For Unknown people, this score will be -1 if the person has been seen less than 3 frames (it means the sighting is not confirmed), 1 otherwise. In this case, a value of 1 indicates a confirmed sighting.
  • For Known people, this score represents the confidence of identification (a typical score for a good confidence is 0.5 to 0.8; a value > 0.8 represents an extremely high confidence).

Example

/*
* Copyright (C) 2015-2018 Digital Barriers plc. All rights reserved.
* Contact: http://www.digitalbarriers.com/
*
* This file is part of the Papillon SDK.
*
* You can't use, modify or distribute any part of this file without
* the explicit written agreements of Digital Barriers.
*/
#include <unordered_map>
#include <vector>
#include <PapillonCore.h>
USING_NAMESPACE_PAPILLON
void CreateFaceDetector(PDetector &detector, const PProperties &parameters) {
P_LOG_INFO << "Creating face detector...";
PString detectorName = "FaceDetector2";
int32 numDetectors = 8; // number of detector worker threads
int32 numLocalisers = 2; // number of localiser worker threads
int32 minDetectionSize = 80;
int32 maxDetectionSize = 1000;
int32 maxDetections = 0;
double threshold = 0.0;
PRectanglei roi; // region of interest (empty means whole frame)
parameters.Get("Detector", detectorName);
parameters.Get("numDetectors", numDetectors);
parameters.Get("numLocalisers", numLocalisers);
parameters.Get("MinDetectionSize", minDetectionSize);
parameters.Get("MaxDetectionSize", maxDetectionSize);
parameters.Get("MaxDetections", maxDetections);
parameters.Get("Threshold", threshold);
parameters.Get("ROI", roi);
P_LOG_INFO << "Detector : " << detectorName;
P_LOG_INFO << "MinDetectionSize : " << minDetectionSize;
P_LOG_INFO << "MaxDetectionSize : " << maxDetectionSize;
P_LOG_INFO << "numDetectors : " << numDetectors;
P_LOG_INFO << "numLocalisers : " << numLocalisers;
P_LOG_INFO << "MaxDetections : " << maxDetections;
P_LOG_INFO << "Threshold : " << threshold;
P_LOG_INFO << "ROI : " << roi;
// Set up detector
PProperties detectortProps;
detectortProps.Set("numDetectors", numDetectors);
detectortProps.Get("numLocalisers", numLocalisers);
PResult res = PDetector::Create(detectorName, detectortProps, detector);
if (res.Failed()) {
P_LOG_ERROR << "Failed to create face detector:" << res;
return;
}
detector.SetMinDetectionSize(PSizei(minDetectionSize, minDetectionSize));
detector.SetMaxDetectionSize(maxDetectionSize);
detector.SetThreshold(static_cast<float>(threshold));
detector.SetMaxDetections(maxDetections);
if (roi.IsValid())
}
void CreateTemplateDescriber(PDescriber &describer, int batchSize, bool useGPU, bool sharedMode) {
P_LOG_INFO << "Creating face recognition describer...";
P_LOG_INFO << "describerBatchSize: " << batchSize;
PString gpuString = useGPU ? "gpuMode=true" : "gpuMode=false";
P_LOG_INFO << "GPU:" << gpuString;
PProperties dnnParameters =
PProperties::CreateFromKeyValueString("type=Face;" + gpuString + PString(";batchSize=%1").Arg(batchSize));
if (sharedMode) {
dnnParameters.Set("sharedDnn", true);
}
PDescriber::Create("DescriberDnn", dnnParameters, describer).OrDie();
}
void CreateMetaDescriber(PDescriber &describer, bool useGPU) {
P_LOG_INFO << "Creating gender recognition describer...";
PString gpuString = useGPU ? "gpuMode=true" : "gpuMode=false";
P_LOG_INFO << "GPU:" << gpuString;
PProperties genderDnnParameters = PProperties::CreateFromKeyValueString("type=Gender;" + gpuString);
PDescriber::Create("DescriberDnn", genderDnnParameters, describer).OrDie();
}
void ProcessEvents(PList &events, std::unordered_map<std::string, int> &counter) {
for (int i = 0; i < events.Size(); i++) {
PEvent event;
events.Get(i, event);
// P_LOG_INFO << event.GetType();
counter[event.GetType().c_str()]++;
}
}
class ProcessThreadC : public PRunnable {
public:
ProcessThreadC() {}
~ProcessThreadC() { /*std::cerr << "thread destructor " << (void*)this << " vf:" << m_videoFile.c_str() << "\n";*/
}
PResult Init(const PProperties &parameters, const PString &videoFile) {
// open video stream
P_LOG_INFO << "Trying to open'" << videoFile << "'";
PResult resVal = PInputVideoStream::Open(videoFile, m_ivs).LogIfError();
if (resVal.Failed()) {
P_LOG_ERROR << "Error in opening video stream:" << videoFile;
return resVal;
}
P_LOG_INFO << "Video stream is opened";
// create FaceLog
P_LOG_INFO << "Creating FaceLog6 instance";
resVal = PAnalytics::Create("FaceLog6", parameters, m_faceLog).LogIfError();
if (resVal.Failed()) {
P_LOG_ERROR << "Failed to create: FaceLog6 with parameters:" << parameters;
return resVal;
}
P_LOG_INFO << "Created FaceLog6 instance";
return PResult::C_OK;
}
virtual void Run() {
std::unordered_map<std::string, int> counter;
counter.insert({"FrameStart", 0});
counter.insert({"FrameEnd", 0});
counter.insert({"Track-In-Progress", 0});
counter.insert({"Face", 0});
counter.insert({"Sighting", 0});
// processing loop
P_LOG_INFO << "Starting main processing loop";
PFrame frame;
PList events;
while (m_ivs.GetFrame(frame).Ok()) {
if (!m_faceLog.Apply(frame, events).LogIfError().Ok()) {
P_LOG_ERROR << "Error in analytics";
break;
}
ProcessEvents(events, counter);
}
if (!m_faceLog.Finish(events).LogIfError().Ok()) {
P_LOG_ERROR << "Error in analytics";
}
ProcessEvents(events, counter);
P_LOG_INFO << "Finished processing";
}
private:
PAnalytics m_faceLog;
};
const PString SAMPLE_DIR = PPath::Join(PUtils::GetEnv("PAPILLON_INSTALL_DIR"),
"Data",
"Samples"); // path to find sample data: $PAPILLON_INSTALL_DIR/Data/Samples
void RunDemo() {
const bool useGPU = true; // use gpu
const bool useShared = true; // shared mode for describers (describers will be shared between FaceLogs)
// create list of sources
std::vector<std::string> sources;
sources.push_back(PPath::Join(SAMPLE_DIR, "busy_office.avi").c_str());
sources.push_back(PPath::Join(SAMPLE_DIR, "officeEntry.avi").c_str());
sources.push_back(PPath::Join(SAMPLE_DIR, "busy_office.avi").c_str());
sources.push_back(PPath::Join(SAMPLE_DIR, "officeEntry.avi").c_str());
PProperties parameters;
parameters.Set("GPU", useGPU);
parameters.Set("MaxFaceDetectorFR", -1.); // limit frame rate of face detector to 8 fps (to no overwhelm CPU)(if
// processing video files better disable this with -1.)
// create face detector
if (useShared) {
PDetector faceDetector;
CreateFaceDetector(faceDetector, parameters);
parameters.Set("FaceDetector", faceDetector);
}
// create face recognition describer
if (useShared) {
PDescriber faceDescriber;
CreateTemplateDescriber(faceDescriber, 4, useGPU, true);
parameters.Set("FaceRecognitionDescriber", faceDescriber);
}
// create gender describer
if (useShared) {
PDescriber genderDescriber;
CreateMetaDescriber(genderDescriber, useGPU);
parameters.Set("GenderDescriber", genderDescriber);
}
std::vector<ProcessThreadC> tasks; // important! we need to keep tasks until threads finish
std::vector<PConcurrentThread> threads;
// start processing threads
P_LOG_INFO << "-------------------- Initialising tasks";
for (auto source : sources) {
tasks.push_back(ProcessThreadC());
if (tasks.back().Init(parameters, source).LogIfError().Failed()) {
P_LOG_ERROR << "Failed to init processing thread.";
return;
}
}
P_LOG_INFO << "-------------------- Lunching processing threads";
for (auto &task : tasks) {
threads.push_back(PConcurrentThread(task));
}
// wait for threads to finish
P_LOG_INFO << "-------------------- Waiting for threads";
for (auto &it : threads) {
it.Join();
}
}
int main() {
RunDemo();
return 0;
}