Timeline-video relationship presentation for alert events

• US 10,386,999 B2
• Filed: 10/26/2016
• Issued: 08/20/2019
• Est. Priority Date: 10/26/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Burns, Christopher Charles, Heitz, III, George Alban, Stewart, James Edward, Hill, Cameron, Yang, Seungho, Venters, Joe Delone, Finlayson, William Alex, Hinz, Carsten, Psiaki, Timothy Samuel, Klee, Nathan Scott, Nelson, Gregory Rourk, Avery, Kevin James, Neal, Lawrence W., Davidsson, Martin
• Primary Examiner(s): Tran, Quoc A
• Application Number:
  US15/335,399
• Publication Number:
  US 20180113577A1
• Time in Patent Office: 1,028 Days
• CPC Class Codes: G05B 15/02 electric
• CPC Class Codes: G06F 16/739 in form of a video summary, e.g. the video summary being a video sequence, a composite still image or having synthesized frames
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G11B 27/3081 used signal is a video-frame or a video-field (P.I.P)
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/61 for supporting one-way streaming services, e.g. Internet radio
• CPC Class Codes: H04N 21/2187 Live feed
• CPC Class Codes: H04N 21/2387 Stream processing in response to a playback request from an end-user, e.g. for trick-play
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/431 Generation of visual interfaces for content selection or interaction; Content or additional data rendering
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4334 Recording operations recording of a television signal H04N5/76; arrangements for recording or accumulating broadcast information or broadcast-related information H04H60/27
• CPC Class Codes: H04N 21/47217 for controlling playback functions for recorded or on-demand content, e.g. using progress bars, mode or play-point indicators or bookmarks specific graphical features in visual interfaces H04N21/4312
• CPC Class Codes: H04N 21/488 Data services, e.g. news ticker systems specially adapted for using meteorological information in broadcast systems H04H60/71
• CPC Class Codes: H04N 21/632 using a connection between clients on a wide area network, e.g. setting up a peer-to-peer communication via Internet for retrieving video segments from the hard-disk of other client devices broadcast-related systems characterised by transmission among terminal devices H04H60/80; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739
• CPC Class Codes: H04N 21/858 : Linking data to content, e.g. by linking an URL to a video object, by creating a hotspot
• CPC Class Codes: H04N 7/186 : Video door telephones

Abstract

A method at an electronic device with a display includes: displaying a user interface having a first region and a second region; receiving, and displaying in the first region of the user interface, a live video stream of a physical environment captured by a remote video camera, where at least some of the live video stream is recorded at a remote server; displaying, in the second region, a timeline corresponding to a timespan for a first portion of a duration during which the live video stream may have been recorded; in response to receiving a user interaction to move the timespan to a second portion of the duration, transitioning the displayed timeline to a new timeline that corresponds to the timespan for the second portion, and while transitioning, displaying, in the first region, a subset of video frames representing the first and/or second portion of the duration.

Claims

1. A method, comprising:

at an electronic device having a display, one or more processors, and memory storing one or more programs for execution by the one or more processors;

displaying, via the display of the electronic device, a user interface having a first region and a second region;

receiving, and displaying in the first region of the user interface, a portion of a video stream of a physical environment captured by a video camera located remotely from the electronic device, wherein at least some of the video stream is recorded at a remote server;

displaying, in the second region of the user interface;

a first portion of a continuous timeline; and

a position marker on the timeline;

in response to receiving a user scroll command indicating a desire to display a second portion of the timeline;

scrolling the timeline to display the second portion, wherein the timeline scrolls with respect to the position marker; and

while scrolling the timeline to the second portion, displaying, in the first region of the user interface, successive video frames corresponding to updated positions of the position marker, wherein each successive video frame graphically replaces preceding video frames and is representative of a set of video frames proximate to the position marker.

2. The method of claim 1, further comprising:

receiving information indicating one or more alert events within the timespan; and

in accordance with the alert events information, displaying on the timeline an alert event indicator for each of the alert events within the timespan.

3. The method of claim 2, wherein the alert events include alert events of one or more alert event types, the types including one or more of the group consisting of:

motion events in a defined zone, motion events not in a defined zone, hazard events, sound events, thermostat events, and door events.

4. The method of claim 3, wherein an alert event indicator for an alert event of a first type of the alert event types differs from an alert event indicator for an alert event of a second type of the alert event types in at least one visual characteristic.

5. The method of claim 4, wherein the differing visual characteristic is color.

6. The method of claim 3, wherein:

alert event indicators for alert events of a first type of the alert event types are arranged along a first linear path, parallel to the timeline, in the second region;

alert event indicators for alert events of a second type of the alert event types are arranged along a second linear path, parallel to the first linear path and to the timeline, in the second region; and

the first linear path and the second linear path are parallel to each other and to the timeline.

7. The method of claim 3, further comprising:

displaying in the second region one or more event type icons, each event type icon associated with an alert event indicator and indicating an event type of an alert event to which the alert event indicator corresponds.

8. An electronic device, comprising:

a display;

one or more processors; and

memory storing one or more programs to be executed by the one or more processors, the one or more programs comprising instructions for;

displaying, via the display of the electronic device, a user interface having a first region and a second region;

receiving, and displaying in the first region of the user interface, a portion of a video stream of a physical environment captured by a video camera located remotely from the electronic device, wherein at least some of the video stream is recorded at a remote server;

displaying, in the second region of the user interface;

a first portion of a continuous timeline; and

a position marker on the timeline;

in response to receiving a user scroll command indicating a desire to display a second portion of the timeline;

scrolling the timeline to display the second portion, wherein the timeline scrolls with respect to the position marker; and

while scrolling the timeline to the second portion, displaying, in the first region of the user interface, successive video frames corresponding to updated positions of the position marker, wherein each successive video frame graphically replaces preceding video frames and is representative of a set of video frames proximate to the position marker.

9. The electronic device of claim 8, further comprising instructions for:

receiving information indicating one or more alert events within the timespan; and

in accordance with the alert events information, displaying on the timeline an alert event indicator for each of the alert events within the timespan.

10. The electronic device of claim 9, wherein:

the successive video frames comprise a first set of video frames and a second set of video frames distinct from the first set;

the first set of video frames is associated with an alert event of the alert events;

the second set of video frames is not associated with an alert event;

the first set has a first frame rate; and

the second set has a second frame rate lower than the first frame rate.

11. The electronic device of claim 9, wherein the alert event indicator for an alert event comprises a graphical object, the graphical object having a length parallel to the timeline, the length representing a duration of the alert event.

12. The electronic device of claim 9, further comprising instructions for:

displaying a plurality of consecutive time increments, the plurality of consecutive time increments corresponding to a cumulative time duration and not associated with any alert event; and

while displaying the plurality of consecutive time increments, displaying in the second region a first affordance to collapse the plurality of consecutive time increments into a cumulative time increment in the timeline, the cumulative time increment corresponding to the cumulative time duration;

detecting a user interaction activating the first affordance; and

in response to detecting the user interaction activating the first affordance, replacing display of the plurality of consecutive time increments with display of the cumulative time increment.

13. The electronic device of claim 12, further comprising instructions for:

while displaying the cumulative time increment, displaying in the second region a second affordance to expand the cumulative time increment into the plurality of consecutive time increments;

detecting a user interaction activating the second affordance; and

in response to detecting the user interaction activating the second affordance, replacing display of the cumulative time increment with display of the plurality of consecutive time increments.

14. The electronic device of claim 9, further comprising instructions for:

displaying in the second region a thumbnail associated with a first alert event indicator of the alert event indicators;

wherein;

the thumbnail is displayed proximate to the first alert event indicator; and

the thumbnail comprises a frame from a recorded portion of the live video stream proximate in time to an alert event to which the first alert event indicator corresponds.

15. The electronic device of claim 14, further comprising instructions for:

displaying in the second region, for a predefined time increment unit on the timeline, up to a predefined number of thumbnails associated with respective alert event indicators.

16. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device with a display and one or more processors, cause the electronic device to perform operations comprising:

displaying, via the display of the electronic device, a user interface having a first region and a second region;

receiving, and displaying in the first region of the user interface, a portion of a video stream of a physical environment captured by a video camera located remotely from the electronic device, wherein at least some of the video stream is recorded at a remote server;

displaying, in the second region of the user interface;

a first portion of a continuous timeline; and

a position marker on the timeline;

in response to receiving a user scroll command indicating a desire to display a second portion of the timeline;

scrolling the timeline to display the second portion, wherein the timeline scrolls with respect to the position marker; and

while scrolling the timeline to the second portion, displaying, in the first region of the user interface, successive video frames corresponding to updated positions of the position marker, wherein each successive video frame graphically replaces preceding video frames and is representative of a set of video frames proximate to the position marker.

17. The computer readable storage medium of claim 16, wherein the user scroll command comprises a swipe gesture that activates a scrolling effect on the timeline.

18. The computer readable storage medium of claim 16, further comprising instructions, which, when executed by the electronic device, cause the electronic device to perform operations comprising:

obtaining the successive video frames separately from receiving the video stream.

19. The computer readable storage medium of claim 16, further comprising instructions, which, when executed by the electronic device, cause the electronic device to perform operations comprising:

after the scrolling;

determining whether a location of the position marker on the timeline corresponds to the current time; and

in accordance with a determination that the location of the position marker on the timeline does not correspond to the current time;

obtaining a recorded portion of the video stream corresponding to a location of the position marker on the timeline; and

replacing display of the successive video frames in the first region with display of the recorded portion of the video stream.

20. The computer readable storage medium of claim 19, further comprising:

in accordance with a determination that the location of the position marker on the timeline corresponds to the current time, replacing display of the successive video frames in the first region with display of a live video stream from the video camera.