Predicting external events from digital video content

• US 10,007,849 B2
• Filed: 05/27/2016
• Issued: 06/26/2018
• Est. Priority Date: 05/29/2015
• Status: Active Grant

• Current Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Original Assignee: Accenture Global Solutions Limited (Accenture PLC)
• Inventors: Bataller, Cyrille, Astrom, Anders
• Primary Examiner(s): Azarian, Sayed
• Application Number:
  US15/167,701
• Publication Number:
  US 20160350597A1
• Time in Patent Office: 760 Days
• Field of Search: 382100, 382103, 382106-107, 382155, 382162, 382168, 382173, 382181, 382190, 382209, 382224, 382232, 382238, 382254, 382274, 382276, 382286-295, 382305, 382312, 382297, 707706, 600301, 702 2, 705 3, 340540
• CPC Class Codes: G06F 16/51 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 18/2178 based on feedback of a supervisor
• CPC Class Codes: G06T 15/205 Image-based rendering
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20061 Hough transform
• CPC Class Codes: G06T 2207/30168 Image quality inspection
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 2207/30242 Counting objects in image
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06T 7/13 Edge detection
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 20/53 : Recognition of crowd images, e.g. recognition of crowd congestion
• CPC Class Codes: G06V 40/166 : using acquisition arrangements
• CPC Class Codes: G06V 40/173 : face re-identification, e.g. recognising unknown faces across different face tracks
• CPC Class Codes: G08B 13/19602 : Image analysis to detect motion of the intruder, e.g. by frame subtraction
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources

Abstract

Computerized methods and systems, including computer programs encoded on a computer storage medium, may detect event shown within digital video content captured by one or more video cameras, and correlate these detected events to real-world conditions that may not be captured within the digital video data. For example, a computing system may detect events shown within digital video content captured by one or more video cameras, and may obtain data that identifies at least one external event. The computer system may establish a predictive model that correlates values of event parameters that characterize the detected and external events during a first time period, and may apply the predictive model to an event parameter that characterizes an additional event detected during a second time period. Based on an outcome of the predictive model, the computing system may determine an expected value of the external event parameter during the second time period.

Claims

1. A system, comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

detecting events shown within image frames of digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period;

obtaining first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content;

establishing a predictive model that correlates the values of the external event parameters to values of the event parameters that characterize a portion of the detected events during the first time period, including establishing an existence of a time-varying pattern among the event parameters of the detected events during the first time period by applying a machine learning algorithm to data identifying the events and the values of the event parameters;

detecting an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time;

in response to the detection of the additional event, applying the predictive model to a value of the additional event parameter;

determining an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter;

based on the time-varying pattern, generating data identifying expected occurrences of one or more of the events during a second time period that includes the second detection time, the second time period occurring after the first time period; and

transmitting, over a data communications network, data identifying the expected value of the external event parameter to a communications device having a display, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface of the display.

2. The system of claim 1, wherein:

the one or more computers further perform the operation of receiving a portion of the video content from the one or more video cameras; and

the one or more video cameras comprise at least one of a pan-zoom-tilt video camera or a camera having a fixed field-of-view.

3. The system of claim 2, wherein detecting the additional event comprises:

receiving an additional video content from the one or more video cameras; and

detecting the additional event within the additional video content captured by the one or more video cameras.

4. The system of claim 1, wherein the one or more computers further perform the operation of applying one or more video analytics to portions of the digital video content, the applied video analytics comprising at least one of an image acquisition process, an object detection process, an object classification process, an object recognition process, an event detection process, or an object tracking process.

5. The system of claim 4, wherein:

detecting the event comprises, based on an outcome of applying the video analytics to the portions of the digital video content, detecting at least one of the events and establishing the value of at least one of the event parameters; and

detecting the additional event comprises, based on an outcome of applying the video analytics to the portions of the digital video content, detecting the additional event and establishing the value of the additional event parameter.

6. The system of claim 1, wherein the at least one external event is not shown within the digital video content captured by the one or more cameras.

7. The system of claim 1, wherein the establishing the predictive model comprises:

applying a second machine learning algorithm to data identifying the values of the external event parameters and the values of the event parameters that characterize the portion of the detected events during the first time period; and

establishing a correlation between the values of the external event parameters and the event parameters, based on an outcome of the second machine learning algorithm.

8. The system of claim 1, wherein the one or more computers further perform the operations of:

obtaining second data specifying an observed value of the external event parameter at the second detection time;

determining an existence of a variance between the actual and expected values of the external event parameters; and

modifying the predictive model in accordance with the determined variance.

9. The system of claim 1, wherein:

the external event comprises at least one of a number of customer inquiries or orders, a total number of individuals within a restricted area, or a total number of vehicles disposed within a restricted area;

the detected events comprise at least one of queued individuals, individuals entering a restricted area through a corresponding entrance, or vehicles entering the restricted area through a corresponding entrance; and

the event parameters comprise at least one of a number of the queued individuals, a number of the individuals that enter the restricted area through the corresponding entrance, or a number of the vehicles that enter the restricted area through the corresponding entrance.

10. The system of claim 1, wherein, in response to the transmitted data, the communications device is further configured to allocate one or more resources in accordance with the expected value of the expected event parameter.

11. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

detecting events shown within digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period;

obtaining first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content;

establishing a predictive model that correlates the values of the external event parameters to values of the event parameters that characterize a portion of the detected events during the first time period;

detecting an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time;

in response to the detection of the additional event, applying the predictive model to a value of the additional event parameter;

determining an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter;

transmitting data identifying the expected value of external event parameter to a communications device, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface;

establishing an existence of a time-varying pattern among the event parameters of the events detected during the first time period;

based on the time-varying pattern, generating data identifying expected occurrences of one or more of the events during a second time period that includes the second detection time, the second time period occurring after the first time period;

establishing a plurality of predictive models that correlate the values of the external event parameters to values of corresponding ones of the event parameters;

applying each of the predictive models to the value of the additional event parameter;

determining a plurality of expected values of the external event parameter at the second detection time, based on an outcome of corresponding ones of the predictive models;

obtaining second data specifying an observed value of the external event parameter at the second detection time;

determining variances between the observed value and corresponding ones of the expected values; and

selecting one or more of the event parameters as predictors of the observed value of the external event parameter based on the determined variances.

12. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

detecting events shown within digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period;

obtaining first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content;

establishing a predictive model that correlates the values of the external event parameters to values of the event parameters that characterize a portion of the detected events during the first time period;

detecting an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time;

in response to the detection of the additional event, applying the predictive model to a value of the additional event parameter;

determining an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter;

transmitting data identifying the expected value of external event parameter to a communications device, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface;

establishing an existence of a time-varying pattern among the event parameters of the events detected during the first time period;

based on the time-varying pattern, generating data identifying expected occurrences of one or more of the events during a second time period that includes the second detection time, the second time period occurring after the first time period;

determining that the additional event represents a deviation from the expected occurrences during the second time period; and

in response to the determination, transmitting data identifying the deviation to a communications device, the communications device being configured to present, to the user within a corresponding interface, a notification that includes at least a portion of the transmitted data.

13. A system comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

detecting events shown within digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period;

obtaining first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content;

establishing a predictive model that correlates the values of the external event parameters to values of the event parameters that characterize a portion of the detected events during the first time period;

detecting an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time, wherein the detected additional event corresponds to at least one of a detected presence of an unexpected object within a portion of the digital video content, a detected removal of an expected object within a portion of the digital video content, or a detection of smoke or flames within a portion of the digital video content;

in response to the detection of the additional event, applying the predictive model to a value of the additional event parameter;

determining an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter;

transmitting data identifying the expected value of external event parameter to a communications device, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface;

establishing an existence of a time-varying pattern among the event parameters of the events detected during the first time period; and

based on the time-varying pattern, generating data identifying expected occurrences of one or more of the events during a second time period that includes the second detection time, the second time period occurring after the first time period.

14. A computer-implemented method, comprising:

detecting, by at least one processor, events shown within image frames of digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period;

obtaining, by the at least one processor, first data that identifies at least one external event, the obtained data comprising observed values of an external event parameter that characterize the external event during the first time period, wherein the first data is not derived from the digital video content;

establishing, by the at least one processor, a predictive model that correlates the values of the external event parameters to values of the event parameters that characterize a portion of the detected events during the first time period, including establishing an existence of a time-varying pattern among the event parameters of the detected events during the first time period by applying a machine learning algorithm to data identifying the events and the values of the event parameters;

detecting, by the at least one processor, an additional event shown within the digital video content, the additional event being associated with a corresponding additional event parameter and a second detection time;

in response to the detection of the additional event, applying, by the at least one processor, the predictive model to a value of the additional event parameter;

determining, by the at least one processor, an expected value of the external event parameter at the second detection time, based on an outcome of applying the predictive model to the value of the additional event parameter;

based on the time-varying pattern, generating, by the at least one processor, data identifying expected occurrences of one or more of the events during a second time period that includes the second detection time, the second time period occurring after the first time period; and

transmitting, by the at least one processor and over a data communications network, data identifying the expected value of external event parameter to a communications device having a display, the communications device configured to present a representation of the expected value of the external event parameter to a user through a corresponding interface of the display.

15. A system, comprising:

one or more computers; and

one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising;

detecting events shown within image frames of digital video content captured by one or more video cameras, the detected events being associated with corresponding event parameters and detection times within a first time period;

applying a predictive model to values of the event parameters that characterize a portion of the detected events during the first time period;

based on an outcome of applying the predictive model to the values of the event parameters, generating data that identifies expected occurrences of one or more of the events during a second time period, the second time period occurring after the first time period, the generating including;

based on the outcome of the predictive model, establishing an existence of a time-varying pattern among the values of the event parameters of the events detected during the first time period by;

applying at least one of a machine learning algorithm or a data mining algorithm to data identifying the values of the event parameters and the detected events; and

establishing the existence of the time-varying pattern based on an outcome of the at least one machine learning algorithm or data mining algorithm; and

generating the expected occurrences of one or more of the events during the second time period in accordance with the time-varying pattern;

detecting an additional event within the captured video content, the additional event being associated with a second detection time that occurs within the second time period;

determining that the additional event represents a deviation from the expected occurrences during the second time period; and

in response to the determination, transmitting, over a data communications network, data that identifies the deviation to a communications device having a display, the communications device being configured to present, to the user within a corresponding interface of the display, a notification that includes a representation of the deviation.

16. The system of claim 15, wherein the detected additional event corresponds to at least one of a detected presence of an unexpected object within a portion of the digital video content, a detected removal of an expected object within a portion of the digital video content, or a detection of smoke or flames within a portion of the digital video content.