Systems and methods for analyzing a video stream

• US 9,501,915 B1
• Filed: 06/12/2015
• Issued: 11/22/2016
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Reddy, Prateek, Bapat, Akshay R., Laska, Jason N., Hua, Wei, Neal, Lawrence W.
• Primary Examiner(s): Werner, David N
• Application Number:
  US14/737,963
• Time in Patent Office: 529 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/196 : using television cameras
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 : Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19663 : Surveillance related processing done local to the camera
• CPC Class Codes: G08B 13/19669 : Event triggers storage or change of storage policy
• CPC Class Codes: G08B 13/19671 : Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19682 : Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19691 : Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: G11B 27/005 : Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/028 : with computer assistance
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/105 : of operating discs
• CPC Class Codes: G11B 27/28 : by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/30 : on the same track as the main recording
• CPC Class Codes: G11B 27/34 : Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04L 12/2803 : Home automation networks
• CPC Class Codes: H04N 21/23418 : involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/42204 : User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4314 : for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/6581 : Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 : Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/9201 : involving the multiplexing of an additional signal and the video signal
• CPC Class Codes: H04N 5/93 : Regeneration of the television signal or of selected parts thereof
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/52 : Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

The various embodiments described herein include methods, devices, and systems for analyzing video streams. In one aspect, a method includes, while receiving a video stream: obtaining motion start information indicating that a portion of the video stream includes a motion event candidate; and segmenting the portion of the video stream into a plurality of segments including an initial segment. The method also includes obtaining a first categorization for the motion event candidate based on the initial segment; and, in accordance with the obtained first categorization, generating a log entry for the motion event candidate including the first categorization. The method further includes: in response to obtaining motion end information, obtaining a second categorization for the motion event based on the plurality of segments; and updating the log entry for the motion event candidate based on the obtained second categorization.

Claims

1. A method, comprising:

at a server system having one or more processors and memory;

while receiving video information from one or more cameras, the video information including a video stream;

obtaining motion start information corresponding to a first time in the video stream, the motion start information indicating that a portion of the video stream subsequent to the first time includes a motion event candidate;

while receiving the portion of the video stream that includes the motion event candidate, segmenting the portion of the video stream into a plurality of temporal segments, the plurality of temporal segments including an initial segment;

obtaining a first categorization of a plurality of categorizations for the motion event candidate based on the initial segment;

in accordance with the obtained first categorization, generating an alert indicative of the first categorization for the motion event candidate;

obtaining motion end information corresponding to a second time in the video stream, the motion end information indicating that a portion of the video stream subsequent to the second time does not include the motion event candidate;

in response to obtaining the motion end information, obtaining a second categorization of the plurality of categorizations for the motion event based on the plurality of temporal segments;

determining whether the second categorization is the same as the first categorization; and

in accordance with a determination that the second categorization is not the same as the first categorization, generating an updated alert indicative of the second categorization for the motion event category.

2. The method of claim 1, wherein the video information is associated with a user;

and

the method further comprises sending the alert to the user.

3. The method of claim 1, further comprising obtaining a confidence score corresponding to the obtained first categorization;

and

wherein generating the alert for the motion event candidate includes generating the alert in accordance with a determination that the confidence score meets predefined criteria.

4. The method of claim 1, further comprising:

obtaining a confidence score corresponding to the obtained first categorization; and

storing the confidence score to a log entry for the motion event candidate.

5. The method of claim 1, further comprising:

in accordance with a determination that the second categorization is not the same as the first categorization, removing the first categorization from a log entry for the motion event candidate.

6. The method of claim 1, wherein the second categorization is more descriptive than the first categorization.

7. The method of claim 1, further comprising:

obtaining a third categorization for the motion event candidate based on at least one segment of the plurality of temporal segments; and

prior to obtaining the second categorization, updating a log entry for the motion event candidate to include the obtained third categorization.

8. The method of claim 1, wherein segmenting the video stream includes:

identifying a third time in the video stream;

in accordance with a determination that a predefined amount of time has lapsed, identifying a fourth time in the video stream; and

generating a segment corresponding to the portion of the video stream between the third time and the fourth time.

9. The method of claim 1, wherein each segment of the plurality of temporal segments has a same duration.

10. The method of claim 1, further comprising:

after obtaining the motion start information, assigning the segmented video stream to a first categorizer; and

storing each segment of the plurality of temporal segments to a particular memory portion, the particular memory portion associated with the first categorizer;

wherein obtaining the first categorization for the motion event candidate comprises;

retrieving, by the first categorizer, the initial segment from the particular memory portion; and

processing, by the first categorizer, the initial segment of the video stream to obtain the first categorization.

11. The method of claim 10, wherein the particular memory portion corresponds to a queue assigned to the first categorizer.

12. The method of claim 10, wherein the server system includes a plurality of categorizers;

and

wherein assigning the segmented video stream to the first categorizer comprises assigning the segmented video stream to the first categorizer in accordance with a load balancing of the plurality of categorizer.

13. The method of claim 10, further comprising, checking, by the first categorizer, for additional temporal segments of the video stream until a motion end event occurs.

14. The method of claim 13, wherein the motion end event includes:

processing, by the first categorizer, a segment denoted as a final segment;

or

a time-out event.

15. The method of claim 10, further comprising:

retrieving a second segment of the plurality of temporal segments from the particular memory portion;

obtaining, by the first categorizer, segment information corresponding to the initial segment; and

processing, by the first categorizer, the second segment of the video stream to obtain a third categorization, wherein the processing of the second segment is based on the segment information corresponding to the initial segment.

16. The method of claim 1, wherein the first categorization is based on a determination of an amount of motion within the initial segment of the video stream.

17. The method of claim 1, further comprising:

while receiving the video information;

obtaining second motion start information corresponding to a third time in the video stream, the second motion start information indicating that the video stream subsequent to the third time includes a second motion event candidate; and

while receiving the video stream that includes the second motion event candidate, segmenting the video stream into a second plurality of temporal segments, the second plurality of temporal segments including an initial segment;

obtaining a first categorization for the second motion event candidate based on the initial segment of the second plurality of temporal segments, the first categorization for the second motion event candidate indicating that the second motion event candidate is an unimportant event;

in accordance with the obtained first categorization for the second motion event candidate, forgoing generation of an alert for the second motion event candidate;

after obtaining the first categorization for the second motion event candidate, obtaining a second categorization for the second motion event candidate based on one or more segments of the second plurality of temporal segments, the second categorization for the second motion event candidate indicating that the second motion event candidate is an important event; and

in accordance with the obtained second categorization for the second motion event candidate, generating an alert indicative of the second categorization for the second motion event candidate.

18. A server system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

while receiving video information from one or more cameras, the video information including a video stream;

obtaining motion start information corresponding to a first time in the video stream, the motion start information indicating that a portion of the video stream subsequent to the first time includes a motion event candidate;

while receiving the portion of the video stream that includes the motion event candidate, segmenting the portion of the video stream into a plurality of temporal segments, the plurality of temporal segments including an initial segment;

obtaining a first categorization of a plurality of categorizations for the motion event candidate based on the initial segment;

in accordance with the obtained first categorization, generating an alert indicative of the first categorization for the motion event candidate;

obtaining motion end information corresponding to a second time in the video stream, the motion end information indicating that a portion of the video stream subsequent to the second time does not include the motion event candidate;

in response to obtaining the motion end information, obtaining a second categorization of the plurality of categorizations for the motion event based on the plurality of temporal segments;

determining whether the second categorization is the same as the first categorization; and

in accordance with a determination that the second categorization is not the same as the first categorization, generating an updated alert indicative of the second categorization for the motion event category.

19. The server system of claim 18, wherein the video information is associated with a user, and the one or more programs further include instructions for sending the alert to the user.

20. The server system of claim 18, wherein the one or more programs further include instructions for obtaining a confidence score corresponding to the obtained first categorization;

wherein generating the alert for the motion event candidate includes generating the alert in accordance with a determination that the confidence score meets predefined criteria.

21. The server system of claim 18, wherein segmenting the video stream includes:

identifying a third time in the video stream;

in accordance with a determination that a predefined amount of time has lapsed, identifying a fourth time in the video stream; and

generating a segment corresponding to the portion of the video stream between the third time and the fourth time.

22. The server system of claim 18, wherein the one or more programs further include instructions for:

after obtaining the motion start information, assigning the segmented video stream to a first categorizer; and

storing each segment of the plurality of temporal segments to a particular memory portion, the particular memory portion associated with the first categorizer;

wherein obtaining the first categorization for the motion event candidate comprises;

retrieving, by the first categorizer, the initial segment from the particular memory portion; and

processing, by the first categorizer, the initial segment of the video stream to obtain the first categorization.

23. The server system of claim 18, wherein the one or more program further include instructions for, while receiving the video information:

obtaining second motion start information corresponding to a third time in the video stream, the second motion start information indicating that the video stream subsequent to the third time includes a second motion event candidate; and

while receiving the video stream that includes the second motion event candidate, segmenting the video stream into a second plurality of temporal segments, the second plurality of temporal segments including an initial segment;

obtaining a first categorization for the second motion event candidate based on the initial segment of the second plurality of temporal segments, the first categorization for the second motion event candidate indicating that the second motion event candidate is an unimportant event;

in accordance with the obtained first categorization for the second motion event candidate, forgoing generation of an alert for the second motion event candidate;

after obtaining the first categorization for the second motion event candidate, obtaining a second categorization for the second motion event candidate based on one or more segments of the second plurality of temporal segments, the second categorization for the second motion event candidate indicating that the second motion event candidate is an important event; and

in accordance with the obtained second categorization for the second motion event candidate, generating an alert indicative of the second categorization for the second motion event candidate.

24. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

while receiving video information from one or more cameras, the video information including a video stream;

obtain motion start information corresponding to a first time in the video stream, the motion start information indicating that a portion of the video stream subsequent to the first time includes a motion event candidate;

while receiving the portion of the video stream that includes the motion event candidate, segment the portion of the video stream into a plurality of temporal segments, the plurality of temporal segments including an initial segment;

obtain a first categorization of a plurality of categorizations for the motion event candidate based on the initial segment;

in accordance with the obtained first categorization, generate an alert indicative of the first categorization for the motion event candidate;

obtain motion end information corresponding to a second time in the video stream, the motion end information indicating that a portion of the video stream subsequent to the second time does not include the motion event candidate;

in response to obtaining the motion end information, obtain a second categorization of the plurality of categorizations for the motion event based on the plurality of temporal segments;

determine whether the second categorization is the same as the first categorization; and

in accordance with a determination that the second categorization is not the same as the first categorization, generate an updated alert indicative of the second categorization for the motion event category.

25. The non-transitory computer-readable storage medium of claim 24, wherein the video information is associated with a user, and the one or more programs further comprise instructions, which when executed by the computing system, cause the system to send the alert to the user.

26. The non-transitory computer-readable storage medium of claim 24, the one or more programs further comprise instructions, which when executed by the computing system, cause the system to obtain a confidence score corresponding to the obtained first categorization;

wherein generating the alert for the motion event candidate includes generating the alert in accordance with a determination that the confidence score meets predefined criteria.

27. The non-transitory computer-readable storage medium of claim 24, herein segmenting the video stream includes:

identifying a third time in the video stream;

in accordance with a determination that a predefined amount of time has lapsed, identifying a fourth time in the video stream; and

generating a segment corresponding to the portion of the video stream between the third time and the fourth time.

28. The non-transitory computer-readable storage medium of claim 24, the one or more programs further comprise instructions, which when executed by the computing system, cause the system to:

after obtaining the motion start information, assign the segmented video stream to a first categorizer; and

store each segment of the plurality of temporal segments to a particular memory portion, the particular memory portion associated with the first categorizer;

wherein obtaining the first categorization for the motion event candidate comprises;

retrieving, by the first categorizer, the initial segment from the particular memory portion; and

processing, by the first categorizer, the initial segment of the video stream to obtain the first categorization.

29. The non-transitory computer-readable storage medium of claim 24, the one or more programs further comprise instructions, which when executed by the computing system, cause the system to:

while receiving the video information;

obtain second motion start information corresponding to a third time in the video stream, the second motion start information indicating that the video stream subsequent to the third time includes a second motion event candidate; and

while receiving the video stream that includes the second motion event candidate, segment the video stream into a second plurality of temporal segments, the second plurality of temporal segments including an initial segment;

obtain a first categorization for the second motion event candidate based on the initial segment of the second plurality of temporal segments, the first categorization for the second motion event candidate indicating that the second motion event candidate is an unimportant event;

in accordance with the obtained first categorization for the second motion event candidate, forgo generation of an alert for the second motion event candidate;

after obtaining the first categorization for the second motion event candidate, obtain a second categorization for the second motion event candidate based on one or more segments of the second plurality of temporal segments, the second categorization for the second motion event candidate indicating that the second motion event candidate is an important event; and

in accordance with the obtained second categorization for the second motion event candidate, generate an alert indicative of the second categorization for the second motion event candidate.