Systems and methods for environment content sharing

• US 10,009,542 B2
• Filed: 05/08/2017
• Issued: 06/26/2018
• Est. Priority Date: 06/27/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Billinghurst, Mark Nathan, Tait, Matthew David
• Primary Examiner(s): Atala, Jamie
• Assistant Examiner(s): Abouzahra, Hesham
• Application Number:
  US15/589,927
• Publication Number:
  US 20170257564A1
• Time in Patent Office: 414 Days
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Embodiments described herein may help to provide methods for sharing and viewing part of an environment of a computing device, such as a head-mountable device (HMD). An example method involves: (a) determining a still photo panorama of an environment, (b) receiving a video stream of a first portion of the environment from a video camera on a sharing device, (c) determining a registration data stream, where the registration data stream indicates a location and an orientation of the video stream within the still photo panorama of the environment; and (d) transmitting the video stream and the registration data stream to one or more viewing devices.

Claims

1. A device comprising:

a video camera;

a processor;

a non-transitory computer readable memory; and

program instructions stored on the non-transitory computer readable memory and executable by the processor to perform operations comprising;

receiving a live video of a plurality of portions of an environment from the video camera while the video camera pans over the plurality of portions of the environment;

determining a registration data stream, wherein, for each respective image frame of the live video, the registration data stream indicates a location at which to place the respective image frame of the live video within a still photo panorama of the environment;

transmitting the live video and the registration data stream to a remote viewing device;

receiving, from the remote viewing device, live data indicative of a current location of a field of view of the remote viewing device within the still photo panorama, wherein the current location of the field of view indicates a portion of the panorama currently viewed by the remote viewing device;

determining, based on the live data and the registration data stream, a current direction of the current location of the field of view of the remote viewing device within the still photo panorama relative to a current location of the live video within the still photo panorama; and

providing for display of a visual cue indicating the determined current direction.

2. The device of claim 1, wherein the operations further comprise:

determining the still photo panorama of the environment by stitching together image frames of the plurality of portions of the environment from the live video; and

transmitting the still photo panorama to the remote viewing device.

3. The device of claim 1, wherein the location at which to place the respective image frame of the live video within the still photo panorama comprises coordinates of the respective image frame within the still photo panorama, and wherein the current location of the field of view of the remote viewing device within the still photo panorama comprises coordinates of the current location of the field of view within the still photo panorama.

4. The device of claim 1, wherein the registration data stream further indicates an orientation with which to place the respective image frame of the live video within the still photo panorama, and wherein the live data is further indicative of a current orientation of the field of view of the remote viewing device within the still photo panorama.

5. The device of claim 1, wherein the live video comprises a data stream representing the respective image frames of the live video.

6. The device of claim 1, wherein the visual cue indicating the current direction indicates a direction for a user of the device to move the video camera to align the current location of the live video with the current location of the field of view of the remote viewing device.

7. The device of claim 1, wherein the operations further comprise:

updating the still photo panorama by replacing a previously-captured image within the still photo panorama with a new image from the live video.

8. The device of claim 1, wherein the device is a head-mountable device (HMD).

9. The device of claim 8, wherein determining the registration data stream comprises using a sensor on the HMD to determine head-orientation data indicative of head orientation.

10. The device of claim 1, wherein the device further comprises an audio sensor, and wherein the operations further comprise:

receiving an audio signal from the audio sensor; and

transmitting the audio signal to the remote viewing device.

11. A device comprising:

a display;

a processor;

a non-transitory computer readable memory; and

program instructions stored on the non-transitory computer readable memory and executable by the processor to perform operations comprising;

receiving a video of a plurality of portions of an environment;

receiving a registration data stream, wherein, for each respective image frame of the video, the registration data stream indicates a location at which to place the respective image frame of the video within a still photo panorama of the environment to create an augmented panorama;

displaying a portion of the augmented panorama on the display, wherein the video is moving within the augmented panorama based on the registration data stream;

determining, based on the registration data stream, a current direction of a current location of the video within the augmented panorama relative to a current location, within the augmented panorama, of the portion of the augmented panorama displayed on the display; and

displaying, on the display, a visual cue indicating the current direction.

12. The device of claim 11, wherein the operations further comprise:

receiving the still photo panorama of the environment, wherein the still photo panorama comprises stitched-together image frames of the plurality of portions of the environment from the video.

13. The device of claim 11, wherein the operations further comprise:

moving, based on an input signal received by the processor, a field of view within the augmented panorama to control the portion of the augmented panorama displayed on the display.

14. The device of claim 13, wherein determining the current direction comprises:

determining the current direction when the video is outside the field of view.

15. The device of claim 13, wherein the display of the device comprises a display of a head-mountable device (HMD), and wherein the input signal for moving the field of view within the augmented panorama is based on had movement data received from a sensor on the HMD and indicative of head movement.

16. The device of claim 11, wherein the location at which to place the respective image frame of the video within the still photo panorama of the environment comprises coordinates of the respective image frame within the still photo panorama.

17. The device of claim 11, wherein the still photo panorama comprises a 360 degree panorama.

18. The device of claim 11, wherein the video is a live video.

19. The device of claim 11, wherein the operations further comprise:

determining a viewing direction indicating a direction of a current location, within the augmented panorama, of a field of view of another device viewing the augmented panorama relative to a current location, within the augmented panorama, of the portion of the augmented panorama displayed on the display; and

displaying, on the display, another visual cue indicating the determined viewing direction.

20. A non-transitory computer readable medium having stored therein instructions that, when executed by a computing device, cause the computing device to perform operations comprising:

receiving a video of a plurality of portions of an environment;

receiving a registration data stream, wherein, for each respective image frame of the video, the registration data stream indicates a location at which to place the respective image frame of the video within a still photo panorama of the environment to create an augmented panorama;

displaying a portion of the augmented panorama on a display of the computing device, wherein the video is moving within the augmented panorama based on the registration data stream;

determining, based on the registration data stream, a current direction of a current location of the video within the augmented panorama relative to a current location, within the augmented panorama, of the portion of the augmented panorama displayed on the display; and

displaying, on the display, a visual cue indicating the current direction.