Systems and methods for categorizing motion event candidates

• US 9,449,229 B1
• Filed: 06/12/2015
• Issued: 09/20/2016
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Chaudhry, Rizwan Ahmed, Varadharajan, Srivatsan, Laska, Jason N., Hua, Wei, Heitz, George Alban III
• Primary Examiner(s): Werner, David N
• Application Number:
  US14/738,034
• Time in Patent Office: 466 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/19604 involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19606 Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan
• CPC Class Codes: G08B 13/19608 Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19669 Event triggers storage or change of storage policy
• CPC Class Codes: G08B 13/19684 Portable terminal, e.g. mobile phone, used for viewing video remotely
• CPC Class Codes: G11B 27/3027 used signal is digitally coded
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/4335 Housekeeping operations, e.g. prioritizing content for deletion because of storage space restrictions storage management, e.g. defragmentation G06F3/0604; unloading stored programs G06F9/445; storage management in file systems G06F16/10; buffering arrangements in a network node or in an end terminal in packet networks H04L49/90
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/8133 : specifically related to the content, e.g. biography of the actors in a movie, detailed information about an article seen in a video program
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/77 : between a recording apparatus and a television camera
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 9/8205 : involving the multiplexing of an additional signal and the colour video signal
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

The various embodiments described herein include methods, devices, and systems for categorizing motion event candidates. In one aspect, a method includes receiving and processing video frames that include a motion event candidate. The processing includes: (a) obtaining background factors corresponding to a background in at least a subset of the video frames; (b) utilizing the background factors to identify one or more motion entities; (c) for each motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity; (d) identifying one or more features in at least a subset of the video frames; and (e) aggregating the background factors, the representative motion vectors, and the features to generate motion features. The method further includes sending the motion features to an event categorizer, where the event categorizer assigns a motion event category to the motion event candidate based on the received motion features.

Claims

1. A method comprising:

at a computing system having one or more processors and memory;

receiving a plurality of video frames, the plurality of video frames including a motion event candidate;

processing the plurality of video frames, the processing comprising;

obtaining one or more background factors corresponding to a background in at least a subset of the plurality of video frames;

utilizing the obtained background factors to identify one or more motion entities in at least a subset of the plurality of video frames;

for each identified motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity;

identifying one or more scene features in at least a subset of the plurality of video frames; and

aggregating the obtained background factors, the obtained representative motion vectors, and the identified scene features to generate a plurality of motion features; and

sending the plurality of motion features to an event categorizer;

wherein the event categorizer assigns a motion event category to the motion event candidate based on the received motion features; and

wherein the motion event category assigned to the motion event candidate is selected from a group consisting of;

one or more known event types;

one or more unknown event types; and

a non-event type.

2. The method of claim 1, further comprising:

performing object recognition on each identified motion entity; and

classifying each of at least a subset of the one or more motion entities in accordance with the performed object recognition; and

wherein the motion event category is further based on the classified objects.

3. The method of claim 1, wherein the plurality of video frames correspond to a scene;

the method further comprises obtaining distance information for the scene;

wherein the aggregating includes aggregating the obtained distance information.

4. The method of claim 1, further comprising:

training the event categorizer, the training comprising;

obtaining a plurality of video clips, each video clip in the plurality of video clips including a respective motion event candidate;

designating a motion event category for each respective motion event candidate;

assigning, via the event categorizer, a motion event category to each respective motion event candidate; and

adjusting the event categorizer based on differences between the assigned motion event categories and the designated motion event categories.

5. The method of claim 1, wherein the plurality of video frames are associated with a first user;

the method further comprises obtaining user information corresponding to the first user; and

wherein the motion event category is further based on the obtained user information.

6. The method of claim 5, wherein the user information comprises user feedback corresponding to one or more prior motion event candidates.

7. The method of claim 1, wherein the plurality of video frames correspond to a scene;

the method further comprises obtaining environmental information corresponding to the scene; and

wherein the motion event category is further based on the obtained environmental information.

8. The method of claim 7, wherein the environmental information comprises information regarding whether the scene is within a structure.

9. The method of claim 1, wherein the plurality of video frames correspond to one or more cameras;

the method further comprises obtaining camera information corresponding to the one or more cameras; and

wherein the motion event category is further based on the obtained camera information.

10. The method of claim 9, wherein the camera information comprises information regarding a relationship between each camera in the one or more cameras and the ground.

11. The method of claim 9, wherein the camera information comprises camera mode information corresponding to each camera in the one or more cameras.

12. The method of claim 11, wherein the camera mode information comprises information regarding whether a respective camera is in a low-light mode.

13. The method of claim 1, further comprising:

receiving a second plurality of video frames, the second plurality of video frames including a second motion event candidate;

processing the second plurality of video frames to generate a second plurality of motion features; and

sending the second plurality of motion features to the event categorizer, wherein the event categorizer assigns a second motion event category to the second motion event candidate based on the second plurality of motion features.

14. The method of claim 13, wherein processing the second plurality of video frames comprises utilizing information corresponding to the processing of the plurality of video frames.

15. The method of claim 13, further comprising:

prior to receiving the second plurality of video frames, creating a log entry corresponding to the motion event candidate; and

updating the log entry based on the second motion event category.

16. The method of claim 13, further comprising utilizing both the plurality of motion features and the second plurality of motion features to assign a motion event category to a third motion event candidate, the third motion event candidate corresponding to a combination of the first motion event candidate and the second motion event candidate.

17. The method of claim 1, further comprising generating a confidence score for the motion event candidate.

18. A server system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

receiving a plurality of video frames, the plurality of video frames including a motion event candidate;

processing the plurality of video frames, the processing comprising;

obtaining one or more background factors corresponding to a background in at least a subset of the plurality of video frames;

utilizing the obtained background factors to identify one or more motion entities in at least a subset of the plurality of video frames;

for each identified motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity;

identifying one or more scene features in at least a subset of the plurality of video frames; and

aggregating the obtained background factors, the obtained representative motion vectors, and the identified scene features to generate a plurality of motion features; and

sending the plurality of motion features to an event categorizer;

wherein the event categorizer assigns a motion event category to the motion event candidate based on the received motion features; and

wherein the motion event category assigned to the motion event candidate is selected from a group consisting of;

one or more known event types;

one or more unknown event types; and

a non-event type.

19. The server system of claim 18, wherein the plurality of video frames correspond to a scene;

wherein the one or more programs further include instructions for obtaining distance information for the scene; and

wherein the aggregating includes aggregating the obtained distance information.

20. The server system of claim 18, wherein the plurality of video frames are associated with a first user;

wherein the one or more programs further include instructions for obtaining user information corresponding to the first user; and

wherein the motion event category is further based on the obtained user information.

21. The server system of claim 18, wherein the plurality of video frames correspond to a scene;

wherein the one or more programs further include instructions for obtaining environmental information corresponding to the scene; and

wherein the motion event category is further based on the obtained environmental information.

22. The server system of claim 18, wherein the plurality of video frames correspond to one or more cameras;

wherein the one or more programs further include instructions for obtaining camera information corresponding to the one or more cameras; and

wherein the motion event category is further based on the obtained camera information.

23. The server system of claim 18, wherein the one or more programs further include instructions for:

receiving a second plurality of video frames, the second plurality of video frames including a second motion event candidate;

processing the second plurality of video frames to generate a second plurality of motion features; and

sending the second plurality of motion features to the event categorizer, wherein the event categorizer assigns a second motion event category to the second motion event candidate based on the second plurality of motion features.

24. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

receive a plurality of video frames, the plurality of video frames including a motion event candidate;

process the plurality of video frames, the processing comprising;

obtaining one or more background factors corresponding to a background in at least a subset of the plurality of video frames;

utilizing the obtained background factors to identify one or more motion entities in at least a subset of the plurality of video frames;

for each identified motion entity, obtaining one or more representative motion vectors based on a motion track of the respective motion entity;

identifying one or more scene features in at least a subset of the plurality of video frames; and

aggregating the obtained background factors, the obtained representative motion vectors, and the identified scene features to generate a plurality of motion features; and

send the plurality of motion features to an event categorizer;

wherein the event categorizer assigns a motion event category to the motion event candidate based on the received motion features; and

wherein the motion event category assigned to the motion event candidate is selected from a group consisting of;

one or more known event types;

one or more unknown event types; and

a non-event type.

25. The non-transitory computer-readable storage medium of claim 24, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to:

perform object recognition on each identified motion entity; and

classify each of at least a subset of the one or more motion entities in accordance with the performed object recognition; and

wherein the motion event category is further based on the classified objects.

26. The non-transitory computer-readable storage medium of claim 24, wherein the plurality of video frames correspond to a scene;

wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to obtain distance information for the scene; and

wherein the aggregating includes aggregating the obtained distance information.

27. The non-transitory computer-readable storage medium of claim 24, wherein the plurality of video frames are associated with a first user;

wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to obtain user information corresponding to the first user; and

wherein the motion event category is further based on the obtained user information.

28. The non-transitory computer-readable storage medium of claim 24, wherein the plurality of video frames correspond to a scene;

wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to obtain environmental information corresponding to the scene; and

wherein the motion event category is further based on the obtained environmental information.

29. The non-transitory computer-readable storage medium of claim 24, wherein the plurality of video frames correspond to one or more cameras;

wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to obtain camera information corresponding to the one or more cameras; and

wherein the motion event category is further based on the obtained camera information.

30. The non-transitory computer-readable storage medium of claim 24, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to:

receive a second plurality of video frames, the second plurality of video frames including a second motion event candidate;

process the second plurality of video frames to generate a second plurality of motion features; and

send the second plurality of motion features to the event categorizer, wherein the event categorizer assigns a second motion event category to the second motion event candidate based on the second plurality of motion features.

31. The non-transitory computer-readable storage medium of claim 24, wherein the one or more programs further comprise instructions, which when executed by the computing system, cause the system to:

perform object recognition on each identified motion entity; and

classify each of at least a subset of the one or more motion entities in accordance with the performed object recognition; and

wherein the motion event category is further based on the classified objects.