Adaptive control for immersive experience delivery

• US 10,440,407 B2
• Filed: 05/09/2017
• Issued: 10/08/2019
• Est. Priority Date: 05/09/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Song, Alex, Pang, Derek, Ma, Mike, Karnad, Nikhil
• Primary Examiner(s): Lhymn, Sarah
• Application Number:
  US15/590,808
• Publication Number:
  US 20180332317A1
• Time in Patent Office: 882 Days
• Field of Search: 345619, 345632, 345633
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 30/34 Stereoscopes providing a stereoscopic pair of separated images corresponding to parallactically displaced views of the same object, e.g. 3D slide viewers
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: H04N 13/117 the virtual viewpoint locations being selected by the viewers or determined by viewer tracking
• CPC Class Codes: H04N 13/344 with head-mounted left-right displays
• CPC Class Codes: H04N 13/366 using viewer tracking
• CPC Class Codes: H04N 13/373 for tracking forward-backward translational head movements, i.e. longitudinal movements
• CPC Class Codes: H04N 13/38 for tracking vertical translational head movements
• CPC Class Codes: H04N 13/398 Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 21/21805 enabling multiple viewpoints, e.g. using a plurality of cameras
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/234345 the reformatting operation being performed only on part of the stream, e.g. a region of the image or a time segment
• CPC Class Codes: H04N 21/234363 by altering the spatial resolution, e.g. for clients with a lower screen resolution
• CPC Class Codes: H04N 21/234381 by altering the temporal resolution, e.g. decreasing the frame rate by frame skipping
• CPC Class Codes: H04N 21/23439 for generating different versions
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments

Abstract

A combined video of a scene may be generated for applications such as virtual reality or augmented reality. In one method, a data store may store video data with a first portion having a first importance metric, and a second portion having a second importance metric, denoting that viewing of the first portion is more likely and/or preferential to viewing of the second portion. The subset may be retrieved and used to generate viewpoint video from a virtual viewpoint corresponding to a viewer'"'"'s viewpoint. The viewpoint video may be displayed on a display device. One of storing the video data, retrieving the subset, and using the subset to generate the viewpoint video may include, based on the difference between the first and second importance metrics, expediting and/or enhancing performance of the step for the first portion, relative to the second portion.

Claims

1. A method for delivering video for a virtual reality or augmented reality experience, the method comprising:

at a data store, storing video data for a virtual reality or augmented reality experience, the video data comprising a first portion having a first importance metric and a second portion having a second importance metric, the first importance metric and the second importance metric being based on viewing data, the viewing data being one of;

historical viewings of the virtual reality or augmented reality experience indicating a set of portions of the virtual reality or augmented reality experience having been viewed by more viewers than other portions of the virtual reality or augmented reality experience,

or

a user input being indicative of a plurality of portions

corresponding to a set of stimuli presented as a part of the virtual reality or augmented reality experience;

at a processor, receiving viewpoint data indicative of a position and/or an orientation of a viewer'"'"'s viewpoint;

at the processor, retrieving a subset of the video data from the data store, the subset comprising at least the first portion of the video data;

at the processor, using the subset to generate viewpoint video of the virtual reality or augmented reality experience, from a virtual viewpoint corresponding to the viewer'"'"'s viewpoint; and

on a display device, displaying the viewpoint video;

wherein a difference exists between the first importance metric and the second importance metric, the difference denoting that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

and wherein performing one step selected from the group consisting of storing the video data, retrieving the subset, and using the subset to generate the view-point video comprises, based on the difference, expediting and/or enhancing performance of the step for the first portion, relative to the second portion.

2. The method of claim 1, wherein:

the video data comprises a plurality of vantage video data sets, each of which represents a view from one of a plurality of vantages within a viewing volume;

the virtual viewpoint is within the viewing volume;

the viewpoint data is indicative of the position of the viewer'"'"'s viewpoint; and

using the subset to generate the viewpoint video comprises using one or more of the vantage video data sets for one or more of the vantages positioned proximate the virtual viewpoint.

3. The method of claim 2, wherein:

the first portion comprises a first vantage video data set of the vantage video data sets, the first vantage video data set representing a view from a first vantage of the plurality of vantages;

the second portion comprises a second vantage video data set of the vantage video data sets, the second vantage video data set representing a view from a second vantage of the plurality of vantages; and

the first importance metric denotes that the first vantage video data set is more likely to be included in the subset than the second vantage video data set.

4. The method of claim 2, wherein:

the first portion represents one or more views from within a first region of the viewing volume;

the second portion represents one or more views from within a second region of the viewing volume; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises storing the video data such that at least one of a number of vantages, a density of vantages, locations of vantages, a number of vantages used to generate the viewpoint video, lighting applied to vantages, and resolution of vantages is enhanced within the first region, relative to the second region.

5. The method of claim 2, wherein:

each of the vantage video data sets comprises a plurality of tiles, each representing a view from a corresponding vantage of the plurality of vantages, along a viewing direction;

the viewpoint data is further indicative of the orientation* of the viewer'"'"'s viewpoint;

and using the subset to generate the viewpoint video comprises using one or more of the tiles, with one or more viewing directions corresponding to the orientation, for each of the one or more of the vantages positioned proximate the virtual viewpoint.

6. The method of claim 5, wherein:

the first portion comprises a first tile of the plurality of tiles of a first vantage video data set of the vantage vide data sets, the first tile representing a view from a first vantage of the plurality of vantages, along a first viewing direction;

the second portion comprises a second tile of the plurality of tiles of the first vantage video data set, the second tile representing a view from the first vantage, along a second viewing direction different from the first viewing direction; and

the first importance metric denotes that the first tile is more likely to be included in the subset than the second tile.

7. The method of claim 5, wherein:

the first portion represents a first set of tiles of the plurality of vantages, that are oriented along a first set of viewing directions of the one or more viewing directions;

the second portion represents a second set of tiles of the plurality of vantages, that are oriented along a second set of viewing directions of the one or more viewing directions; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises storing the video data such that at least one of a tile spatial resolution, tile temporal resolution, tile color depth, and tile bit rate is enhanced for the first set of tiles, relative to the second set of tiles.

8. The method of claim 1, wherein:

the subset comprises the first portion and the second portion of the video data; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises retrieving the subset of the video data such that the first portion is retrieved prior to retrieval of the second portion.

9. The method of claim 1, wherein:

the subset comprises the first portion and the second portion of the video data; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises using the subset to generate the viewpoint video such that a first segment of the viewpoint video incorporating the first portion of the video data is generated prior to generation of a second segment of the viewpoint video incorporating the second portion of the video data.

10. The method of claim 1, wherein:

the subset comprises the first portion and the second portion of the video data; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises using the subset to generate the viewpoint video such that a first segment of the viewpoint video incorporating the first portion of the video data has a higher level of quality than a second segment of the viewpoint video incorporating the second portion of the video data.

11. The method of claim 1, further comprising:

at the data store, receiving viewing data indicating that one or more viewers prefer or more frequently view the first portion of the video data over the second portion of the video data;

at the processor, based on the viewing data, assigning the first importance metric to the first portion of the video data; and

at the processor, based on the viewing data, assigning the second importance metric to the second portion of the video data.

12. The method of claim 1, further comprising:

at an input device, receiving user input indicating that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

at the processor, based on the user input, assigning the first importance metric to the first portion of the video data; and

at the processor, based on the user input, assigning the second importance metric to the second portion of the video data.

13. The method of claim 12, wherein:

the user input indicates that viewing of the first portion is more likely than viewing of the second portion; and

the method further comprises, at an output device distinct from the display device, delivering an olfactory or tactile stimulus to the viewer to prompt the viewer to position or orient the viewer'"'"'s viewpoint in a manner that causes the first portion of the video data to be included in the subset.

14. The method of claim 1, further comprising:

at the processor, carrying out analysis of one or both of the video data and audio data that accompanies the video data to determine that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

at the processor, based on the analysis, assigning the first importance metric to the first portion of the video data; and

at the processor, based on the analysis, assigning the second importance metric to the second portion of the video data.

15. A non-transitory computer-readable medium for delivering video for a virtual reality or augmented reality experience, comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing a data store to store video data for a virtual reality or augmented reality experience, the video data comprising a first portion having a first importance metric and a second portion having a second importance metric, the first importance metric and the second importance metric being based on viewing data, the viewing data being one of;

historical viewings of the virtual reality or augmented reality experience indicating a set of portions of the virtual reality or augmented reality experience having been viewed by more viewers than other portions of the virtual reality or augmented reality experience,

or

a user input being indicative of a plurality of portions

corresponding to a set of stimuli presented as a part of the virtual reality or augmented reality experience

receiving viewpoint data indicative of a position and/or an orientation of a viewer'"'"'s viewpoint;

retrieving a subset of the video data from the data store, the subset comprising at least the first portion of the video data;

using the subset to generate viewpoint video of the virtual reality or augmented reality experience, from a virtual viewpoint corresponding to the viewer'"'"'s viewpoint; and

causing a display device to display the viewpoint video;

wherein a difference exists between the first importance metric and the second importance metric, the difference denoting that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

and wherein performing one step selected from the group consisting of storing the video data, retrieving the subset, and using the subset to generate the view-point video comprises, based on the difference, expediting and/or enhancing performance of the step for the first portion, relative to the second portion.

16. The non-transitory computer-readable medium of claim 15, wherein:

the video data comprises a plurality of vantage video data sets, each of which represents a view from one of a plurality of vantages within a viewing volume;

the virtual viewpoint is within the viewing volume;

the viewpoint data is indicative of the position of the viewer'"'"'s viewpoint; and

using the subset to generate the viewpoint video comprises using one or more of the vantage video data sets for one or more of the vantages positioned proximate the virtual viewpoint.

17. The non-transitory computer-readable medium of claim 16, wherein:

the first portion comprises a first vantage video data set of the vantage video data sets, the first vantage video data set representing a view from a first vantage of the plurality of vantages;

the second portion comprises a second vantage video data set of the vantage video data sets, the second vantage video data set representing a view from a second vantage of the plurality of vantages; and

the first importance metric denotes that the first vantage video data set is more likely to be included in the subset than the second vantage video data set.

18. The non-transitory computer-readable medium of claim 16, wherein:

the first portion represents one or more views from within a first region of the viewing volume;

the second portion represents one or more views from within a second region of the viewing volume; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises storing the video data such that at least one of a number of vantages, a density of vantages, locations of vantages, a number of vantages used to generate the viewpoint video, lighting applied to vantages, and resolution of vantages is enhanced within the first region, relative to the second region.

19. The non-transitory computer-readable medium of claim 16, wherein:

each of the vantage video data sets comprises a plurality of tiles, each representing a view from a corresponding vantage of the plurality of vantages, along a viewing direction;

the viewpoint data is further indicative of the orientation of the viewer'"'"'s viewpoint; and

using the subset to generate the viewpoint video comprises using one or more of the tiles, with one or more viewing directions corresponding to the orientation, for each of the one or more of the vantages positioned proximate the virtual viewpoint.

20. The non-transitory computer-readable medium of claim 19, wherein:

the first portion comprises a first tile of the plurality of tiles of a first vantage video data set of the vantage vide data sets, the first tile representing a view from a first vantage of the plurality of vantages, along a first viewing direction;

the second portion comprises a second tile of the plurality of tiles of the first vantage video data set, the second tile representing a view from the first vantage, along a second viewing direction different from the first viewing direction; and

the first importance metric denotes that the first tile is more likely to be included in the subset than the second tile.

21. The non-transitory computer-readable medium of claim 19, wherein:

the first portion represents a first set of tiles of the plurality of vantages, that are oriented along a first set of viewing directions of the one or more viewing directions;

the second portion represents a second set of tiles of the plurality of vantages, that are oriented along a second set of viewing directions of the one or more viewing directions; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises storing the video data such that at least one of a tile spatial resolution, tile temporal resolution, tile color depth, and tile bit rate is enhanced for the first set of tiles, relative to the second set of tiles.

22. The non-transitory computer-readable medium of claim 15, wherein:

the subset comprises the first portion and the second portion of the video data; and

expediting and/or enhancing performance of the step for the first portion, relative to the second portion, comprises performing at least one selection from the group consisting of;

retrieving the subset of the video data such that the first portion is retrieved prior to retrieval of the second portion;

using the subset to generate the viewpoint video such that a first segment of the viewpoint video incorporating the first portion of the video data is generated prior to generation of a second segment of the viewpoint video incorporating the second portion of the video data; and

using the subset to generate the viewpoint video such that a first segment of the viewpoint video incorporating the first portion of the video data has a higher level of quality than a second segment of the viewpoint video incorporating the second portion of the video data.

23. The non-transitory computer-readable medium of claim 15, further comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing the data store to receive viewing data indicating that one or more viewers prefer or more frequently view the first portion of the video data over the second portion of the video data;

based on the viewing data, assigning the first importance metric to the first portion of the video data; and

based on the viewing data, assigning the second importance metric to the second portion of the video data.

24. The non-transitory computer-readable medium of claim 15, further comprising instructions stored thereon, that when executed by a processor, perform the steps of:

causing an input device to receive user input indicating that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

based on the user input, assigning the first importance metric to the first portion of the video data; and

processor, based on the user input, assigning the second importance metric to the second portion of the video data.

25. The non-transitory computer-readable medium of claim 15, further comprising instructions stored thereon, that when executed by a processor, perform the steps of:

carrying out analysis of one or both of the video data and audio data that accompanies the video data to determine that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

based on the analysis, assigning the first importance metric to the first portion of the video data; and

based on the analysis, assigning the second importance metric to the second portion of the video data.

26. A system for delivering video for a virtual reality or augmented reality experience, the system comprising:

a data store configured to store video data for a virtual reality or augmented reality experience, the video data comprising a first portion having a first importance metric and a second portion having a second importance metric, the first importance metric and the second importance metric being based on viewing data, the viewing data being one of;

historical viewings of the virtual reality or augmented reality experience indicating a set of portions of the virtual reality or augmented reality experience having been viewed by more viewers than other portions of the virtual reality or augmented reality experience,

or

a user input being indicative of a plurality of portions corresponding to a set of stimuli presented as a part of the virtual reality or augmented reality experience

a processor, communicatively connected to the data store, configured to;

receive viewpoint data indicative of a position and/or an orientation of a viewer'"'"'s viewpoint;

retrieve a subset of the video data from the data store, the subset comprising at least the first portion of the video data; and

use the subset to generate viewpoint video of the virtual reality or augmented reality experience, from a virtual viewpoint corresponding to the viewer'"'"'s viewpoint; and

a display device, communicatively coupled to the processor, configured to display the viewpoint video;

wherein a difference exists between the first importance metric and the second importance metric, the difference denoting that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

and wherein the data store and/or the processor are further configured to perform one step selected from the group consisting of storing the video data, retrieving the subset, and using the subset to generate the viewpoint video by, based on the difference, expediting and/or enhancing performance of the step for the first portion, relative to the second portion.

27. The system of claim 26, wherein:

the video data comprises a plurality of vantage video data sets, each of which represents a view from one of a plurality of vantages within a viewing volume;

the virtual viewpoint is within the viewing volume;

the viewpoint data is indicative of the position of the viewer'"'"'s viewpoint; and

the processor is further configured to use the subset to generate the viewpoint video by using one or more of the vantage video data sets for one or more of the vantages positioned proximate the virtual viewpoint.

28. The system of claim 27, wherein:

the first portion comprises a first vantage video data set of the vantage video data sets, the first vantage video data set representing a view from a first vantage of the plurality of vantages;

the second portion comprises a second vantage video data set of the vantage video data sets, the second vantage video data set representing a view from a second vantage of the plurality of vantages; and

the first importance metric denotes that the first vantage video data set is more likely to be included in the subset than the second vantage video data set.

29. The system of claim 27, wherein:

the first portion represents one or more views from within a first region of the viewing volume;

the second portion represents one or more views from within a second region of the viewing volume; and

the processor and/or the data store are further configured to expedite and/or enhance performance of the step for the first portion, relative to the second portion, by storing the video data such that at least one of a number of vantages, a density of vantages, locations of vantages, a number of vantages used to generate the viewpoint video, lighting applied to vantages, and resolution of vantages is enhanced within the first region, relative to the second region.

30. The system of claim 27, wherein:

each of the vantage video data sets comprises a plurality of tiles, each representing a view from a corresponding vantage of the plurality of vantages, along a viewing direction;

the viewpoint data is further indicative of the orientation of the viewer'"'"'s viewpoint; and

the processor is further configured to use the subset to generate the viewpoint video by using one or more of the tiles, with one or more viewing directions corresponding to the orientation, for each of the one or more of the vantages positioned proximate the virtual viewpoint.

31. The system of claim 30, wherein:

the first portion comprises a first tile of the plurality of tiles of a first vantage video data set of the vantage vide data sets, the first tile representing a view from a first vantage of the plurality of vantages, along a first viewing direction;

the second portion comprises a second tile of the plurality of tiles of the first vantage video data set, the second tile representing a view from the first vantage, along a second viewing direction different from the first viewing direction; and

the first importance metric denotes that the first tile is more likely to be included in the subset than the second tile.

32. The system of claim 30, wherein:

the first portion represents a first set of tiles of the plurality of vantages, that are oriented along a first set of viewing directions of the one or more viewing directions;

the second portion represents a second set of tiles of the plurality of vantages, that are oriented along a second set of viewing directions of the one or more viewing directions; and

the processor and/or the data store are further configured to expedite and/or enhance performance of the step for the first portion, relative to the second portion, by storing the video data such that at least one of a tile spatial resolution, tile temporal resolution, tile color depth, and tile bit rate is enhanced for the first set of tiles, relative to the second set of tiles.

33. The system of claim 26, wherein:

the subset comprises the first portion and the second portion of the video data; and

the processor and/or the data store are further configured to expedite and/or enhance performance of the step for the first portion, relative to the second portion, by performing at least one of;

retrieving the subset of the video data such that the first portion is retrieved prior to retrieval of the second portion;

using the subset to generate the viewpoint video such that a first segment of the viewpoint video incorporating the first portion of the video data is generated prior to generation of a second segment of the viewpoint video incorporating the second portion of the video data; and

using the subset to generate the viewpoint video such that a first segment of the viewpoint video incorporating the first portion of the video data has a higher level of quality than a second segment of the viewpoint video incorporating the second portion of the video data.

34. The system of claim 26, wherein:

the data store is further configured to receive viewing data indicating that one or more viewers prefer or more frequently view the first portion of the video data over the second portion of the video data; and

the processor is further configured, based on the viewing data, to;

assign the first importance metric to the first portion of the video data; and

assign the second importance metric to the second portion of the video data.

35. The system of claim 26, further comprising an input device configured to receive user input indicating that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

wherein the processor is further configured to, based on the user input;

assign the first importance metric to the first portion of the video data; and

assign the second importance metric to the second portion of the video data.

36. The system of claim 26, wherein the processor is further configured to:

carry out analysis of one or both of the video data and audio data that accompanies the video data to determine that viewing of the first portion is more likely and/or preferential to viewing of the second portion;

based on the analysis, assign the first importance metric to the first portion of the video data; and

based on the analysis, assign the second importance metric to the second portion of the video data.