Method and system for categorizing detected motion events

• US 9,420,331 B2
• Filed: 10/08/2014
• Issued: 08/16/2016
• Est. Priority Date: 07/07/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Laska, Jason N., Nelson, Greg R., Duffy, Greg
• Primary Examiner(s): Werner, David N
• Application Number:
  US14/510,042
• Publication Number:
  US 20160093338A1
• Time in Patent Office: 678 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/447 Temporal browsing, e.g. timeline
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30232 Surveillance
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49
• CPC Class Codes: G06V 20/44 Event detection
• CPC Class Codes: G06V 20/49 Segmenting video sequences, i.e. computational techniques such as parsing or cutting the sequence, low-level clustering or determining units such as shots or scenes
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G08B 13/194 using image scanning and comparing systems
• CPC Class Codes: G08B 13/196 : using television cameras
• CPC Class Codes: G08B 13/19604 : involving reference image or background adaptation with time to compensate for changing conditions, e.g. reference image update on detection of light level change
• CPC Class Codes: G08B 13/19606 : Discriminating between target movement or movement in an area of interest and other non-signicative movements, e.g. target movements induced by camera shake or movements of pets, falling leaves, rotating fan
• CPC Class Codes: G08B 13/19613 : Recognition of a predetermined image pattern or behaviour pattern indicating theft or intrusion
• CPC Class Codes: G08B 13/19615 : wherein said pattern is defined by the user
• CPC Class Codes: G08B 13/19676 : Temporary storage, e.g. cyclic memory, buffer storage on pre-alarm
• CPC Class Codes: G08B 13/19682 : Graphic User Interface [GUI] presenting system data to the user, e.g. information on a screen helping a user interacting with an alarm system
• CPC Class Codes: G08B 13/19684 : Portable terminal, e.g. mobile phone, used for viewing video remotely
• CPC Class Codes: G11B 27/005 : Reproducing at a different information rate from the information rate of recording for television signals H04N5/783
• CPC Class Codes: G11B 27/028 : with computer assistance
• CPC Class Codes: G11B 27/031 : Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/105 : of operating discs
• CPC Class Codes: G11B 27/30 : on the same track as the main recording
• CPC Class Codes: G11B 27/34 : Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04L 2209/80 : Wireless network architectures or network communication protocols for wireless network security H04W12/00
• CPC Class Codes: H04L 2463/062 : applying encryption of the keys cryptographic mechanisms or cryptographic arrangements for key distribution using key encryption key H04L9/0822
• CPC Class Codes: H04L 63/0428 : wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/083 : using passwords cryptographic mechanisms or cryptographic arrangements for entity authentication using a predetermined code H04L9/3226
• CPC Class Codes: H04L 67/10 : in which an application is distributed across nodes in the network software deployment G06F8/60; multiprogramming arrangements G06F9/46
• CPC Class Codes: H04L 9/0822 : using key encryption key
• CPC Class Codes: H04L 9/085 : Secret sharing or secret splitting, e.g. threshold schemes
• CPC Class Codes: H04L 9/0869 : involving random numbers or seeds
• CPC Class Codes: H04N 21/2187 : Live feed
• CPC Class Codes: H04N 21/2347 : involving video stream encryption
• CPC Class Codes: H04N 21/239 : Interfacing the upstream path of the transmission network, e.g. prioritizing client content requests hybrid fiber coaxial [HFC] networks for upstream channel allocation for video distribution H04L12/2801; flow control in data networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2393 : involving handling client requests scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/2541 : Rights Management protecting software against unauthorised usage in a vending or licensing environment G06F21/10; security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/2743 : Video hosting of uploaded data from client
• CPC Class Codes: H04N 21/42204 : User interfaces specially adapted for controlling a client device through a remote control device; Remote control devices therefor constructive details of casings for the remote control device H01H9/0235; user interfaces for controlling a tuning device of a television receiver through a remote control H03J9/00; remote control of peripheral devices connected to a television receiver through the remote control device of the television receiver H04B1/205
• CPC Class Codes: H04N 21/4222 : Remote control device emulator integrated into a non-television apparatus, e.g. a PDA, media center or smart toy remote control device for a television receiver integrated into a mobile phone H04M1/72415
• CPC Class Codes: H04N 21/431 : Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 : involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4314 : for fitting data in a restricted space on the screen, e.g. EPG data in a rectangular grid
• CPC Class Codes: H04N 21/4316 : for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4334 : Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/4335 : Housekeeping operations, e.g. prioritizing content for deletion because of storage space restrictions storage management, e.g. defragmentation G06F3/0604; unloading stored programs G06F9/445; storage management in file systems G06F16/10; buffering arrangements in a network node or in an end terminal in packet networks H04L49/90
• CPC Class Codes: H04N 21/4408 : involving video stream encryption, e.g. re-encrypting a decrypted video stream for redistribution in a home network
• CPC Class Codes: H04N 21/4438 : Window management, e.g. event handling following interaction with the user interface
• CPC Class Codes: H04N 21/4622 : Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4627 : Rights management associated to the content security in data switching network management H04L41/28; security management or policies for network security H04L63/20; access security in wireless networks H04W12/08
• CPC Class Codes: H04N 21/4753 : for user identification, e.g. by entering a PIN or password cryptographic authentication protocols H04L9/32; networks authentication protocols H04L63/08
• CPC Class Codes: H04N 23/65 : Control of camera operation in relation to power supply
• CPC Class Codes: H04N 23/651 : for reducing power consumption by affecting camera operations, e.g. sleep mode, hibernation mode or power off of selective parts of the camera
• CPC Class Codes: H04N 23/6811 : based on the image signal
• CPC Class Codes: H04N 23/6815 : by distinguishing pan or tilt from motion
• CPC Class Codes: H04N 5/144 : Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/93 : Regeneration of the television signal or of selected parts thereof
• CPC Class Codes: H04N 7/18 : Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04N 7/185 : from a mobile camera, e.g. for remote control
• CPC Class Codes: H04N 7/186 : Video door telephones
• CPC Class Codes: H04N 9/87 : Regeneration of colour television signals H04N9/80 takes precedence
• CPC Class Codes: H04W 12/02 : Protecting privacy or anonymity, e.g. protecting personally identifiable information [PII]
• CPC Class Codes: H04W 12/033 : of the user plane, e.g. user's traffic
• CPC Class Codes: H04W 12/04 : Key management, e.g. using generic bootstrapping architecture [GBA]
• CPC Class Codes: H04W 12/06 : Authentication
• CPC Class Codes: H04W 12/08 : Access security
• CPC Class Codes: H04W 12/50 : Secure pairing of devices
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication
• CPC Class Codes: H04W 84/12 : WLAN [Wireless Local Area Networks]

Abstract

A computing system device with processor(s) and memory displays a video monitoring user interface on the display, the video monitoring user interface including a video feed from a camera located remotely from the client device in a first region of the video monitoring user interface and an event timeline in a second region of the video monitoring user interface, and the event timeline includes indicators corresponding to motion events previously detected by the camera. The computing system obtains an indication of a detected motion event and associates the detected motion event with a respective category. The computing system displays an indicator for the detected motion event on the event timeline with a display characteristic corresponding to the respective category.

Claims

1. A method of categorizing and providing user feedback regarding a detected event, the method comprising:

receiving a selection of a plurality of event zones within a field of view of a single video camera;

displaying a video monitoring user interface on a display of a client device located remotely from the video camera, the video monitoring user interface including a first region for displaying a live video feed and/or a recorded video feed from the video camera and a second region for displaying a single event timeline, the single event timeline including a plurality of equally spaced time indicators each indicating a specific time and a current video feed indicator indicating the temporal position of the video feed displayed in the first region, the temporal position including a past time corresponding to the previously recorded video feed from the video camera and a current time corresponding to the live video feed from the video camera, wherein the current video feed indicator is movable relative to the equally spaced time indicators to facilitate a change in the temporal position of the video feed displayed in the first region;

displaying the single event timeline on the second region of the video monitoring user interface according to a user-selected scale;

displaying on the single event timeline a plurality of event indicators corresponding to one or more events previously detected from the live video feed during a period of time associated with the user-selected scale of the single event timeline, wherein;

each of the one or more events is assigned to at least one of the plurality of event zones;

each event is represented by a respective event indicator having a graphic display characteristic associated with a corresponding event zone to which the event is assigned; and

the graphic display characteristic is a visually unique color fill for each event zone; and

displaying, for at least one of the plurality of event indicators, a corresponding static image from the live video feed, the static image being representative of the event associated with the event indicator; and

in response to a detection of a new event in one of the event zones for the live video feed, displaying a new event indicator for the detected new event on the single event timeline, the new event indicator being displayed with the graphic display characteristic corresponding to the event zone in which the new motion event was detected.

2. The method of claim 1, further comprising:

in accordance with a determination that one or more characteristics for the new event satisfy one or more criteria for the event zone, displaying a notification indicating that the detected new event has been detected in the event zone.

3. The method of claim 2, wherein the notification pops-up from the new event indicator for the detected new event.

4. The method of claim 2, wherein the notification is overlaid on the video in the first region of the video monitoring user interface.

5. The method of claim 2, wherein the notification is a banner notification displayed in a location corresponding to the top of the video monitoring user interface.

6. The method of claim 2, wherein the notification includes one or more affordances for providing feedback as to whether the detected new event is properly detected in the event zone.

7. The method of claim 1, wherein the event zone corresponds to a recognized event zone.

8. The method of claim 1, wherein the event zone corresponds to a previously created event.

9. The method of claim 8, further comprising:

training a categorization model based on a training data set containing motion vectors corresponding to the event category, wherein common characteristics of the event category are extracted through the training to set forth the one or more categorization criteria and distinguish events associated with the event category from events associated with other event categories.

10. The method of claim 1, wherein the one or more events previously detected by the camera include a subset of events, further comprising:

identifying a first event category associated with the subset of events; and

retroactively modifying on the video monitoring user interface a display characteristic of a subset of event indicators corresponding to the subset of the events.

11. The method of claim 1, further comprising:

detecting selection of a first event indicator of the one or more event indicators, the first event indicator corresponding to a first event of the one or more events; and

displaying a dialog box for the first event corresponding to the first event indicator, wherein the dialog box includes at least one of a time the first event was detected and a preview of the first event.

12. The method of claim 1, further comprising:

displaying one or more affordances in response to detection of a contact on the video monitoring user interface; and

in accordance with an activation of the one or more affordances, enabling control of the camera including at least one of a zoom command and a microphone enabling/disabling command.

13. A computing system, comprising:

one or more processors; and

memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for;

receiving a selection of a plurality of event zones within a field of view of a single video camera;

displaying a video monitoring user interface on a display of a client device located remotely from the video camera, the video monitoring user interface including a first region for displaying a live video feed and/or a recorded video feed from the video camera and a second region for displaying a single event timeline, the single event timeline including a plurality of equally spaced time indicators each indicating a specific time and a current video feed indicator indicating the temporal position of the video feed displayed in the first region, the temporal position including a past time corresponding to the previously recorded video feed from the video camera and a current time corresponding to the live video feed from the video camera, wherein the current video feed indicator is movable relative to the equally spaced time indicators to facilitate a change in the temporal position of the video feed displayed in the first region;

displaying the single event timeline on the second region of the video monitoring user interface according to a user-selected scale;

displaying on the single event timeline a plurality of event indicators corresponding to one or more events previously detected from the live video feed during a period of time associated with the user-selected scale of the single event timeline, wherein;

each of the one or more events is assigned to at least one of the plurality of event zones;

each event is represented by a respective event indicator having a graphic display characteristic associated with a corresponding event zone to which the event is assigned; and

the graphic display characteristic is a visually unique color fill for each event zone; and

displaying, for at least one of the plurality of event indicators, a corresponding static image from the live video feed, the static image being representative of the event associated with the event indicator; and

in response to a detection of a new event in one of the event zones for the live video feed, displaying a new event indicator for the detected new event on the single event timeline, the new event indicator being displayed with the graphic display characteristic corresponding to the event zone in which the new motion event was detected.

14. The computing system of claim 13, wherein the one or more programs further comprise instructions for:

displaying a notification indicating that the detected new event has been assigned to the respective event category.

15. The computing system of claim 14, wherein the notification pops-up from the indicator for the detected new event.

16. The computing system of claim 14, wherein the notification includes one or more affordances for providing feedback as to whether the detected new event is properly assigned to the respective event category.

17. The computing system of claim 13, wherein the event zone corresponds to a recognized event category.

18. The computing system of claim 13, wherein the event zone corresponds to a previously created zone of interest.

19. The computing system of claim 13, wherein the one or more events previously detected by the camera include a subset of events, wherein the one or more programs further comprise instructions for:

identifying a first event category associated with the subset of events; and

retroactively modifying on the video monitoring user interface a display characteristic of a subset of event indicators corresponding to the subset of the events.

20. The computing system of claim 13, wherein the one or more programs further comprise instructions for:

detecting selection of a first event indicator of the one or more event indicators, the first event indicator corresponding to a first event of the one or more events; and

displaying a dialog box for the first event corresponding to the first event indicator, wherein the dialog box includes at least one of a time the first event was detected and a preview of the first event.

21. The computing system of claim 13, wherein the one or more programs further comprise instructions for:

displaying one or more affordances in response to detection of a contact on the video monitoring user interface; and

in accordance with an activation of the one or more affordances, enabling control of the camera including at least one of a zoom command and a microphone enabling/disabling command.

22. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by a computing system with one or more processors, cause the computing system to perform operations comprising:

receiving a selection of a plurality of event zones within a field of view of a single video camera;

displaying a video monitoring user interface on a display of a client device located remotely from the video camera, the video monitoring user interface including a first region for displaying a live video feed and/or a recorded video feed from the video camera and a second region for displaying a single event timeline, the single event timeline including a plurality of equally spaced time indicators each indicating a specific time and a current video feed indicator indicating the temporal position of the video feed displayed in the first region, the temporal position including a past time corresponding to the previously recorded video feed from the video camera and a current time corresponding to the live video feed from the video camera, wherein the current video feed indicator is movable relative to the equally spaced time indicators to facilitate a change in the temporal position of the video feed displayed in the first region;

displaying the single event timeline on the second region of the video monitoring user interface according to a user-selected scale;

displaying on the single event timeline a plurality of event indicators corresponding to one or more events previously detected from the live video feed during a period of time associated with the user-selected scale of the single event timeline, wherein;

each of the one or more events is assigned to at least one of the plurality of event zones;

each event is represented by a respective event indicator having a graphic display characteristic associated with a corresponding event zone to which the event is assigned; and

the graphic display characteristic is a visually unique color fill for each event zone; and

displaying, for at least one of the plurality of event indicators, a corresponding static image from the live video feed, the static image being representative of the event associated with the event indicator; and

in response to a detection of a new event in one of the event zones for the live video feed, displaying a new event indicator for the detected new event on the single event timeline, the new event indicator being displayed with the graphic display characteristic corresponding to the event zone in which the new motion event was detected.

23. The non-transitory computer readable storage medium of claim 22, wherein the instructions cause the computing system to perform operations further comprising:

displaying a notification indicating that the detected new event has been detected in the event zone.

24. The non-transitory computer readable storage medium of claim 23, wherein the notification pops-up from the new event indicator for the detected new event.

25. The non-transitory computer readable storage medium of claim 23, wherein the notification includes one or more affordances for providing feedback as to whether the detected new event is properly detected in the event zone.

26. The non-transitory computer readable storage medium of claim 22, wherein the event zone corresponds to a recognized event zone.

27. The non-transitory computer readable storage medium of claim 22, wherein the event zone corresponds to a previously created event category.

28. The non-transitory computer readable storage medium of claim 22, wherein the one or more events previously detected by the camera include a subset of events, and wherein the instructions cause the computing system to perform operations further comprising:

identifying a first event category associated with the subset of events; and

retroactively modifying on the video monitoring user interface a display characteristic of a subset of event indicators corresponding to the subset of the events.

29. The non-transitory computer readable storage medium of claim 22, wherein the instructions cause the computing system to perform operations further comprising:

detecting selection of a first event indicator of the one or more event indicators, the first event indicator corresponding to a first event of the one or more events; and

displaying a dialog box for the first event corresponding to the first event indicator, wherein the dialog box includes at least one of a time the first event was detected and a preview of the first event.

30. The non-transitory computer readable storage medium of claim 22, wherein the instructions cause the computing system to perform operations further comprising:

displaying one or more affordances in response to detection of a contact on the video monitoring user interface; and

in accordance with an activation of the one or more affordances, enabling control of the camera including at least one of a zoom command and a microphone enabling/disabling command.