Systems and methods for categorizing motion events

• US 9,672,427 B2
• Filed: 10/25/2016
• Issued: 06/06/2017
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Hua, Wei, Reddy, Prateek, Bapat, Akshay R., Neal, Lawrence W., Laska, Jason N.
• Primary Examiner(s): Werner, David N
• Application Number:
  US15/334,172
• Publication Number:
  US 20170046574A1
• Time in Patent Office: 224 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2321 using statistics or function optimisation, e.g. modelling of probability density functions
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 10/763 Non-hierarchical techniques, e.g. based on statistics of modelling distributions
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/52 : Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/172 : Classification, e.g. identification
• CPC Class Codes: G06V 40/20 : Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: G08B 13/196 : using television cameras
• CPC Class Codes: G08B 13/19608 : Tracking movement of a target, e.g. by detecting an object predefined as a target, using target direction and or velocity to predict its new position
• CPC Class Codes: G08B 13/19613 : Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19663 : Surveillance related processing done local to the camera
• CPC Class Codes: G08B 13/19669 : Event triggers storage or change of storage policy
• CPC Class Codes: G08B 13/19671 : Addition of non-video data, i.e. metadata, to video stream
• CPC Class Codes: G08B 13/19682 : Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19691 : Signalling events for better perception by user, e.g. indicating alarms by making display brighter, adding text, creating a sound
• CPC Class Codes: G11B 27/005 : Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/028 : with computer assistance
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/105 : of operating discs
• CPC Class Codes: G11B 27/28 : by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: G11B 27/30 : on the same track as the main recording
• CPC Class Codes: G11B 27/34 : Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04L 12/2803 : Home automation networks
• CPC Class Codes: H04N 21/23418 : involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/42204 : User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4314 : for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/6581 : Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application
• CPC Class Codes: H04N 21/8456 : by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/145 : Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/9201 : involving the multiplexing of an additional signal and the video signal
• CPC Class Codes: H04N 5/93 : Regeneration of the television signal or of selected parts thereof
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/52 : Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

The various embodiments described herein include methods, devices, and systems for categorizing motion events. In one aspect, a method includes: (1) obtaining a plurality of video frames, the plurality of video frames corresponding to a scene and a motion event candidate; (2) identifying one or more visual characteristics of the scene; (3) obtaining one or more background factors for the scene; (4) utilizing the obtained background factors to identify one or more motion entities; (5) for each identified motion entity: (a) classifying the motion entity by performing object recognition; and (b) obtaining one or more representative motion vectors based on a motion track of the motion entity; and (6) assigning a motion event category to the motion event candidate based on the identified visual characteristics, the obtained background factors, the classified motion entities, and the obtained representative motion vectors.

Claims

1. A method comprising:

at a computing system having one or more processors and memory;

obtaining a plurality of video frames, the plurality of video frames corresponding to a scene and a motion event candidate;

identifying one or more visual characteristics of the scene;

obtaining one or more background factors for the scene;

utilizing the obtained background factors to identify one or more motion entities;

for each identified motion entity;

classifying the motion entity by performing object recognition on the motion entity; and

obtaining one or more representative motion vectors based on a motion track of the motion entity; and

assigning a motion event category of a plurality of motion event categories to the motion event candidate based on the identified one or more visual characteristics, the obtained background factors, the classified motion entities, and the obtained representative motion vectors;

wherein the motion event category assigned to the motion event candidate is selected from a group consisting of;

one or more known event types;

one or more unknown event types; and

a non-event type.

2. The method of claim 1, wherein the motion event candidate comprises a false positive;

and

wherein assigning the motion event category to the motion event candidate comprises assigning the non-event type motion event category to the motion event candidate.

3. The method of claim 1, further comprising obtaining distance information for the scene;

and

wherein assigning the motion event category to the motion event candidate comprises assigning the motion event category to the motion event candidate based at least in part on the obtained distance information.

4. The method of claim 1, wherein assigning the motion event category to the motion event candidate comprises utilizing an event categorizer to assign the motion event category to the motion event candidate;

and the method further comprises;

training the event categorizer, the training comprising;

obtaining a plurality of video clips, each video clip in the plurality of video clips including one or more motion event candidates;

assigning, via the event categorizer, a motion event category to each motion event candidate;

designating a motion event category for each motion event candidate; and

adjusting the event categorizer based on differences between the assigned motion event categories and the designated motion event categories.

5. The method of claim 1, wherein the plurality of video frames are obtained from a camera;

and

the method further comprises obtaining user information corresponding to a user of the camera;

wherein assigning the motion event category to the motion event candidate comprises assigning the motion event category to the motion event candidate based at least in part on the obtained user information.

6. The method of claim 5, wherein the user information comprises user feedback from the user corresponding to one or more prior motion event candidates.

7. A server system comprising:

one or more processors; and

memory coupled to the one or more processors, the memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for;

obtaining a plurality of video frames, the plurality of video frames corresponding to a scene and a motion event candidate;

identifying one or more visual characteristics of the scene;

obtaining one or more background factors for the scene;

utilizing the obtained background factors to identify one or more motion entities;

for each identified motion entity;

classifying the motion entity by performing object recognition on the motion entity; and

obtaining one or more representative motion vectors based on a motion track of the motion entity; and

assigning a motion event category of a plurality of motion event categories to the motion event candidate based on the identified one or more visual characteristics, the obtained background factors, the classified motion entities, and the obtained representative motion vectors;

wherein the motion event category assigned to the motion event candidate is selected from a group consisting of;

one or more known event types;

one or more unknown event types; and

a non-event type.

8. The server system of claim 7, wherein the one or more programs further include instructions for obtaining environmental information corresponding to the scene;

and

wherein assigning the motion event category to the motion event candidate comprises assigning the motion event category to the motion event candidate based at least in part on the obtained environmental information.

9. The server system of claim 8, wherein the environmental information comprises information regarding whether the scene is within a structure.

10. The server system of claim 7, wherein the plurality of video frames correspond to one or more cameras;

wherein the one or more programs further include instructions for obtaining camera information corresponding to the one or more cameras; and

wherein assigning the motion event category to the motion event candidate comprises assigning the motion event category to the motion event candidate based at least in part on the obtained camera information.

11. The server system of claim 10, wherein the camera information comprises information regarding relative positioning of each camera in the one or more cameras and the ground.

12. The server system of claim 10, wherein the camera information comprises camera mode information indicating an operational mode of each camera in the one or more cameras.

13. The server system of claim 12, wherein the camera mode information comprises information regarding whether each camera in the one or more cameras is operating in a low-light mode.

14. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by a computing system, cause the system to:

obtain a plurality of video frames, the plurality of video frames corresponding to a scene and a motion event candidate;

identify one or more visual characteristics of the scene;

obtain one or more background factors for the scene;

utilize the obtained background factors to identify one or more motion entities;

for each identified motion entity;

classify the motion entity by performing object recognition on the motion entity; and

obtain one or more representative motion vectors based on a motion track of the motion entity; and

assign a motion event category of a plurality of motion event categories to the motion event candidate based on the identified one or more visual characteristics, the obtained background factors, the classified motion entities, and the obtained representative motion vectors;

wherein the motion event category assigned to the motion event candidate is selected from a group consisting of;

one or more known event types;

one or more unknown event types; and

a non-event type.

15. The non-transitory computer-readable storage medium of claim 14, wherein the plurality of video frames comprise a first plurality of video frames, and the motion event candidate comprises a first motion event candidate;

and

wherein the one or more programs further comprise instructions to;

receive a second plurality of video frames, the second plurality of video frames corresponding to the scene and including a second motion event candidate;

utilize the obtained background factors to identify one or more second motion entities;

for each identified second motion entity;

classify the second motion entity by performing object recognition on the second motion entity; and

obtain one or more representative second motion vectors based on a motion track of the second motion entity; and

assign a second motion event category of the plurality of motion event categories to the second motion event candidate based on the obtained background factors, the classified second motion entities, and the obtained representative second motion vectors.

16. The non-transitory computer-readable storage medium of claim 15, wherein assigning the second motion event category to the second motion event candidate comprises utilizing information corresponding to processing of the first plurality of video frames.

17. The non-transitory computer-readable storage medium of claim 15, wherein the one or more programs further comprise instructions to:

after assigning the second motion event category to the second motion event candidate, assign a third motion event category of the plurality of motion event categories to the first motion event candidate based at least in part on the assignment of the second motion event category to the second motion event candidate.

18. The non-transitory computer-readable storage medium of claim 15, wherein the one or more programs further comprise instructions to:

prior to receiving the second plurality of video frames, create a log entry for the first motion event candidate; and

update the log entry for the first motion event candidate based on the assignment of the second motion event category to the second motion event candidate.

19. The non-transitory computer-readable storage medium of claim 15, wherein the one or more programs further comprise instructions to assign a third motion event category to a third motion event candidate, the third motion event candidate corresponding to a combination of the first motion event candidate and the second motion event candidate;

and

wherein assigning the third motion event category to the third motion event candidate comprises assigning the third motion event category to the third motion event candidate based at least in part on the assignment of the motion event category to the first motion event candidate and the assignment of the second motion event category to the second motion event candidate.

20. The non-transitory computer-readable storage medium of claim 14, wherein the one or more programs further comprise instructions to generate a confidence score for the assignment of the motion event category to the motion event candidate.