Systems and methods for environment content sharing

• US 9,686,466 B1
• Filed: 06/27/2013
• Issued: 06/20/2017
• Est. Priority Date: 06/27/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Billinghurst, Mark Nathan, Tate, Matthew David
• Primary Examiner(s): Atala, Jamie
• Assistant Examiner(s): ABOUZAHRA, HESHAM K
• Application Number:
  US13/928,639
• Time in Patent Office: 1,454 Days
• CPC Class Codes: H04N 23/635 Region indicators; Field of view indicators
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/698 for achieving an enlarged field of view, e.g. panoramic image capture

Abstract

Embodiments described herein may help to provide methods for sharing and viewing part of an environment of a computing device, such as a head-mountable device (HMD). An example method involves: (a) determining a still photo panorama of an environment, (b) receiving a video stream of a first portion of the environment from a video camera on a sharing device, (c) determining a registration data stream, where the registration data stream indicates a location and an orientation of the video stream within the still photo panorama of the environment; and (d) transmitting the video stream and the registration data stream to one or more viewing devices.

Claims

1. A method, comprising:

receiving a continuous live video of a plurality of portions of an environment from a video camera on a sharing device while the video camera pans over the plurality of portions of the environment;

determining a still photo panorama of the environment by stitching together image frames of the plurality of portions of the environment from the continuous live video;

determining a registration data stream, wherein, for each respective image frame of the continuous live video, the registration data stream indicates coordinates at which to place the respective image frame of the continuous live video within the still photo panorama of the environment;

transmitting the still photo panorama, the continuous live video, and the registration data stream to a remote viewing device;

receiving, from the remote viewing device, live data indicative of current coordinates of a field of view of the remote viewing device within the still photo panorama, wherein the current coordinates of the field of view indicate a portion of the panorama currently viewed by the remote viewing device;

determining, based on the live data and the registration data stream, a current direction of the current coordinates of the field of view of the remote viewing device within the still photo panorama relative to current coordinates of the continuous live video within the still photo panorama; and

providing for display of visual cues indicating the determined current direction.

2. The method of claim 1, wherein determining the still photo panorama comprises using one or more previously stored images of the environment.

3. The method of claim 1, wherein determining the still photo panorama comprises updating one or more previously determined still photo panoramas of the environment.

4. The method of claim 1, wherein determining the still photo panorama comprises using one or more images of the environment captured by one or more cameras on one or more additional sharing devices.

5. The method of claim 1, wherein the still photo panorama comprises a 360 degree panorama.

6. The method of claim 1, wherein the sharing device is a head-mountable device (HMD).

7. The method of claim 6, wherein determining the still photo panorama comprises using one or more sensors on the HMD to determine head-orientation data indicative of head orientation.

8. The method of claim 6, wherein determining the registration data stream comprises using one or more sensors on the HMD to determine head-orientation data indicative of head orientation.

9. The method of claim 6, further comprising:

receiving a spatial audio signal from one or more audio sensors on the HMD;

transmitting the spatial audio signal to the remote viewing device.

10. The method of claim 6, further comprising:

receiving a non-spatial audio signal from one or more audio sensors on the HMD; and

transmitting the non-spatial audio signal to the remote viewing device.

11. The method of claim 6, further comprising receiving one or more audio signals from the remote viewing device.

12. The method of claim 1, wherein the visual cues indicating the current direction indicate a direction for a user of the sharing device to move the sharing device to align the current coordinates of the continuous live video with the current coordinates of the field of view of the remote viewing device.

13. The method of claim 1, further comprising:

receiving an additional continuous live video of an additional plurality of portions of the environment from an additional video camera on the sharing device;

determining an additional registration data stream for the additional continuous live video, wherein, for each respective image frame of the additional continuous live video, the additional registration data stream indicates coordinates at which to place the respective image frame of the additional continuous live video within the still photo panorama of the environment; and

transmitting the additional continuous live video and the additional registration data stream to the remote viewing device.

14. The method of claim 1, further comprising:

receiving an additional continuous video of an additional plurality of portions of the environment from a video camera on an additional sharing device;

determining an additional registration data stream for the additional continuous live video, wherein, for each respective image frame of the additional continuous video, the additional registration data stream indicates coordinates at which to place the respective image frame of the additional continuous video within the still photo panorama of the environment; and

transmitting the additional continuous video and the additional registration data stream to the remote viewing device.

15. A method, comprising:

receiving a continuous video of a plurality of portions of an environment;

receiving a still photo panorama of the environment, wherein the still photo panorama comprises stitched-together image frames of the plurality of portions of the environment from the continuous video;

receiving a registration data stream, wherein, for each respective image frame of the continuous video, the registration data stream indicates coordinates at which to place the respective image frame of the continuous video within the still photo panorama of the environment to create an augmented panorama;

displaying a portion of the augmented panorama in a display of a viewing device, wherein the continuous video is moving within the augmented panorama based at least in part on the registration data stream;

moving, based on an input signal to the viewing device, a field of view within the augmented panorama to control the portion of the augmented panorama displayed on the display of the viewing device;

when the video stream is outside the field of view, determining, based on the registration data stream, a current direction of current coordinates of the continuous video stream within the augmented panorama relative to current coordinates of the field of view within the augmented panorama; and

displaying, in the display of the viewing device, visual cues indicating the current direction.

16. The method of claim 15, wherein the field of view has a same size as the video stream.

17. The method of claim 15, wherein the viewing device is a first viewing device, the method further comprising:

receiving one or more viewing directions, wherein each of the one or more viewing directions indicates a direction of current coordinates of a field of view of one or more other viewing devices within the augmented panorama relative to current coordinates of the field of view of the first viewing device within the augmented panorama; and

displaying, in the display of the first viewing device, visual cues indicating the one or more viewing directions.

18. The method of claim 15, further comprising:

receiving an additional continuous video of a plurality of additional portions of the environment;

receiving an additional registration data stream, wherein, for each respective image frame of the additional continuous video, the additional registration data stream indicates coordinates at which to place the respective image frame of the additional continuous video within the augmented panorama; and

displaying the additional continuous video in the display of the viewing device, wherein the additional continuous video is moving within the augmented panorama based at least in part on the additional registration data stream.

19. The method of claim 15, wherein:

the video stream of the plurality of portions of the environment was captured during a first time period;

displaying the portion of the augmented panorama in the display of the viewing device occurs during a second time period; and

the second time period begins at a time after the first time period ends.

20. The method of claim 15, further comprising:

receiving a spatial audio signal of the environment;

receiving a non-spatial audio signal; and

playing an audio experience comprising the spatial audio signal and the non-spatial audio signal, wherein the spatial audio signal is differentiable from the non-spatial audio signal in the audio experience.

21. The method of claim 15, wherein:

the display of the viewing device comprises a display of a head-mountable device (HMD); and

the input signal to the viewing device for moving the field of view within the augmented panorama to control the portion of the augmented panorama displayed on the display of the HMD is based at least in part on head-movement data from one or more sensors on the HMD, wherein the head-movement data is indicative of head movement.

22. The method of claim 15, wherein:

the display of the viewing device comprises a display of a computer with a pointing device; and

the input signal to the viewing device for moving the field of view within the augmented panorama to control the portion of the augmented panorama displayed on the display of the computer is based at least in part on movement of the pointing device.

23. The method of claim 15, wherein:

the display of the viewing device comprises a display of a smartphone; and

the input signal to the viewing device for moving the field of view within the augmented panorama to control the portion of the augmented panorama displayed on the display of the smartphone is based at least in part on movement of the smartphone.

27. The method of claim 15, wherein the video stream is a live video stream.

28. The method of claim 27, wherein the live video stream is received from a sharing device, the method further comprising transmitting, to the sharing device, live data indicative of the current coordinates of the field of view within the augmented panorama.

24. A non-transitory computer readable medium having stored therein instructions that, when executed by a computing device, cause the computing device to perform functions comprising:

receiving a continuous live video of a plurality of portions of an environment from a video camera on the computing device while the video camera pans over the plurality of portions of the environment;

determining a still photo panorama of the environment by stitching together image frames of the plurality of portions of the environment from the continuous live video;

determining a registration data stream, wherein, for each respective image frame of the continuous live video, the registration data stream indicates coordinates at which to place the respective image frame of the continuous live video within the still photo panorama of the environment;

transmitting the still photo panorama, the continuous live video, and the registration data stream to a remote viewing device;

receiving, from the remote viewing device, live data indicative of current coordinates of a field of view of the remote viewing device within the still photo panorama, wherein the current coordinates of the field of view indicate a portion of the panorama currently viewed by the remote viewing device;

determining, based on the live data and the registration data stream, a current direction of the current coordinates of the field of view of the remote viewing device within the still photo panorama relative to current coordinates of the continuous live video within the still photo panorama; and

providing for display of visual cues indicating the determined current direction.

25. The non-transitory computer readable medium of claim 24, wherein the computing device is a head-mountable device (HMD).

26. The non-transitory computer readable medium of claim 24, wherein the visual cues indicating the current direction indicate a direction for the user of the computing device to move the computing device to align the current coordinates of the continuous live video with the current coordinates of the field of view of the remote viewing device.