Video integration with home assistant

• US 10,200,746 B1
• Filed: 07/19/2017
• Issued: 02/05/2019
• Est. Priority Date: 07/19/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Yuan, Jessica, Stewart, James, Nongpiur, Rajeev, Lister, Patrick, Ng, Chi Yeung Jonathan
• Primary Examiner(s): McFadden, Susan
• Application Number:
  US15/654,491
• Publication Number:
  US 20190028759A1
• Time in Patent Office: 566 Days
• Field of Search: 704275
• CPC Class Codes: G06Q 10/109 Time management, e.g. calendars, reminders, meetings or time accounting
• CPC Class Codes: G10L 17/22 Interactive procedures; Man-machine interfaces
• CPC Class Codes: H04L 12/2803 Home automation networks
• CPC Class Codes: H04L 12/281 indicating a format for calling an appliance service function in a home automation network for remote control or remote monitoring of applications H04L67/025
• CPC Class Codes: H04L 12/282 based on user interaction within the home receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204
• CPC Class Codes: H04L 67/62 Establishing a time schedule for servicing the requests
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/25875 involving end-user authentication restricting access to computer systems by authenticating users using a predetermined code G06F21/33; arrangements for secret or secure communication including means for verifying the identity or authority of a user of the system H04L9/32; networks authentication protocols H04L63/08; authentication in wireless network security H04W12/06
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/43615 Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals home Audio Video Interoperability [HAVI] data switching networks H04L12/2805

Abstract

Various arrangements are detailed herein related to managing scheduled events using a home assistant and an integrated video service. Using a network-enabled video camera, a video stream of a location may be captured. The video stream captured may be used to determine an identity of a user present at the location. In response to identifying the identity of the user, access to a cloud-based calendar account for the user may be determined to be authorized. The cloud-based calendar account linked with the user may be accessed. A next calendar event mapped to the cloud-based calendar account that is scheduled to occur within a threshold time duration may be determined. A description linked with the next calendar event may be accessed. A network-enabled smart home assistant device may be used to synthesize speech to announce details related to the next event.

Claims

1. A method for managing scheduled events using a home assistant and an integrated video service, the method comprising:

capturing, using a network-enabled video camera, a video stream of a location at which a user is present;

analyzing the video stream captured using the network-enabled video camera to determine an identity of the user;

in response to identifying the identity of the user, determining that access to a cloud-based calendar account for the user is authorized;

accessing the cloud-based calendar account linked with the user;

determining that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;

in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, accessing a description linked with the next calendar event; and

synthesizing speech, using a network-enabled smart home assistant device, to announce a scheduled start time and read a description linked with the next calendar event while the user remains present within the video stream.

2. The method for managing scheduled events using the home assistant of claim 1, further comprising:

determining that an event location is linked with the next calendar event;

accessing a traffic service to determine a travel time from a home location linked with the network-enabled smart home assistant device or the network-enabled video camera to the event location; and

synthesizing speech, using the network-enabled smart home assistant device, that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time.

3. The method for managing scheduled events using the home assistant of claim 1, further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests a video clip be recorded; and

storing, to a cloud-based video storage platform, the video clip from the video stream, wherein the video clip is mapped to a user account of the user.

4. The method for managing scheduled events using the home assistant of claim 3, further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests the video clip be recorded at a time in the future;

scheduling, by the network-enabled smart home assistant device, a recording event based on the spoken command and the time; and

storing, to the cloud-based video storage platform, the video clip from the video stream starting at the time, wherein the video clip is mapped to a user account of the user.

5. The method for managing scheduled events using the home assistant of claim 1, further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests an image be captured;

synthesizing speech, using the network-enabled smart home assistant device, indicating an amount of time until the image is captured;

capturing the image using the network-enabled video camera; and

storing, to a cloud-based video storage platform, the image, wherein the image is mapped to a user account of the user.

6. The method for managing scheduled events using the home assistant of claim 1, further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests a time-lapse video clip be recorded, the spoken command specifying a start time and an end time;

capturing, using the network-enabled video camera, a plurality of images between the start time and the end time, wherein a frequency of images captured is dependent on a duration of time between the start time and the end time; and

creating the time-lapse video clip using the plurality of images.

7. The method for managing scheduled events using the home assistant of claim 1, further comprising:

receiving, by the network-enabled smart home assistant device, a spoken command that requests a history of one or more events at the location;

analyzing a stored video stream for at least a predefined time period;

identifying a plurality of events linked with an audio level above a defined threshold; and

creating a video highlight reel that includes video from the stored video stream recorded during the plurality of events.

8. The method for managing scheduled events using the home assistant of claim 1, further comprising:

receiving a first spoken query from an identified user;

determining the user is authorized to access the cloud-based calendar account linked with the network-enabled smart home assistant device; and

synthesizing speech, by the network-enabled smart home assistant device, to announce a response to the first spoken query that incorporates information from the cloud-based calendar account linked with the network-enabled smart home assistant device.

9. The method for managing scheduled events using the home assistant of claim 8, further comprising:

analyzing the video stream captured using the network-enabled video camera to determine an unidentified user is present;

receiving a second spoken query from the unidentified user;

determining the unidentified user is not authorized to access any calendar linked with the network-enabled smart home assistant device; and

synthesizing speech, by the network-enabled smart home assistant device, to announce a generic response to the second spoken query that does not incorporate information from any calendar linked with the network-enabled smart home assistant device.

10. The method for managing scheduled events using the home assistant of claim 1, further comprising:

receiving a spoken query from an identified user;

determining that the spoken query is requesting an item be added to a stored shopping list;

in response to determining the spoken query, determining a name of the item by performing object recognition on one or more frames of the video stream; and

adding the determined name of the item to the stored shopping list.

11. A system for managing scheduled events, the system comprising:

a network-connected video camera that captures a video stream of a location; and

a cloud-based home assistant integration platform, configured to;

analyze the video stream captured using the network-connected video camera to determine an identity of a user present in the video stream;

in response to identifying the identity of the user, determine that access to a cloud-based calendar account for the user is authorized;

access the cloud-based calendar account linked with the user from a plurality of cloud-based calendar accounts accessible by the cloud-based home assistant integration platform;

determine that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;

in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, access a description linked with the next calendar event; and

transmit an indication of the next calendar event and the accessed description; and

a home assistant device that is configured to;

receive the indication of the next calendar event and the accessed description; and

synthesize speech to announce the next calendar event and the accessed description.

12. The system for managing scheduled events of claim 11, wherein the home assistant device and the network-connected video camera are incorporated as part of a single device.

13. The system for managing scheduled events of claim 11, wherein:

the cloud-based home assistant integration platform is further configured to;

determine an event location linked with the next calendar event;

access a traffic service to determine a travel time from a home location to the event location; and

transfer an indication of the travel time to the home assistant device; and

the home assistant device is further configured to;

synthesize speech that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time or the travel time.

14. The system for managing scheduled events of claim 11, wherein:

the cloud-based home assistant integration platform is configured to;

receive a spoken command, via the home assistant device, that requests a video clip be recorded; and

store, to a cloud-based video storage arrangement, the video clip from the video stream, wherein the video clip is mapped to a user account of the user who spoke the command to the home assistant device.

15. The system for managing scheduled events of claim 14, wherein:

the cloud-based home assistant integration platform is configured to;

receive, via the home assistant device, as part of the spoken command, a request that the video clip be recorded at a time in the future; and

schedule a recording event based on the spoken command and the time;

and wherein the cloud-based home assistant integration platform being configured to store the video clip comprises the cloud-based home assistant integration platform storing the video clip in response to the recording event.

16. The system for managing scheduled events of claim 11, wherein:

the cloud-based home assistant integration platform is configured to;

receive, via the network-connected smart home assistant device, a spoken command that requests an image be captured;

cause the home assistant device to synthesize speech indicating an amount of time until the image is captured; and

store the image, wherein the image is mapped to a user account of the user that spoke the spoken command; and

the network-connected video camera is further configured to capture the image.

17. The system for managing scheduled events of claim 11, wherein:

the cloud-based home assistant integration platform is further configured to;

analyze the video stream captured using the network-connected video camera to determine an unidentified user is present;

receive a second spoken query from the unidentified user;

determine the unidentified user is not authorized to access any calendar linked with the home assistant device; and

cause the smart home assistant device to synthesize speech indicating a generic response to the second spoken query that does not incorporate information from any users'"'"' calendars linked with the home assistant device.

18. A non-transitory processor-readable medium comprising processor-readable instructions configured to cause one or more processors to:

receive, from a network-enabled video camera, a video stream of a location at which a user is present;

analyze the video stream captured using the network-enabled video camera to determine an identity of the user;

in response to identifying the identity of the user, determine that a cloud-based calendar account for the user is accessible;

access the cloud-based calendar account linked with the user;

determine that a next calendar event mapped to the cloud-based calendar account is scheduled to occur within a threshold time duration;

in response to determining that the next calendar event mapped to the cloud-based calendar account is scheduled to occur within the threshold time duration, access a description linked with the next calendar event; and

cause a network-enabled smart home assistant device to output synthesized speech indicating a scheduled start time and read a description linked with the next calendar event.

19. The non-transitory processor-readable medium of claim 18, wherein the processor-readable instructions, when executed, further cause the one or more processors to:

determine that an event location is linked with the next calendar event;

access a traffic service to determine a travel time from a home location linked with the network-enabled smart home assistant device or the network-enabled video camera to the event location; and

cause the network-enabled smart home assistant device to output synthesized speech that indicates a time at which the user should leave the home location to reach the event location based on the determined travel time.

20. The non-transitory processor-readable medium of claim 18, wherein the processor-readable instructions, when executed, further cause the one or more processors to:

receive a spoken command that requests a video clip be recorded; and

store, to a cloud-based video storage platform, the video clip from the video stream, wherein the video clip is mapped to a user account of the user that spoke the spoken command.