Multi-view display with viewer detection

• US 10,558,264 B1
• Filed: 12/21/2016
• Issued: 02/11/2020
• Est. Priority Date: 12/21/2016
• Status: Active Grant

• Current Assignee: X Development LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Watson, Philip Edwin
• Primary Examiner(s): Wang, Yi
• Application Number:
  US15/386,691
• Time in Patent Office: 1,147 Days
• Field of Search: None
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/349 Multi-view displays for displaying three or more geometrical viewpoints without viewer tracking for viewing without the aid of special glasses using fly-eye lenses H04N13/307
• CPC Class Codes: H04N 13/368 for two or more viewers
• CPC Class Codes: H04N 13/383 for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes

Abstract

Methods, systems, and apparatus for generating images that blend an appearance of a display with an environment of the display. In some aspects, output is provided from a display that occludes an object. A gaze direction is determined for an observer located within an environment of the display. An image is generated based on the determined gaze direction of the observer. The generated image is configured to blend an appearance of the display with the environment of the display. The generated image is displayed on the display directed to the observer.

Claims

1. A computer-implemented method comprising:

providing output from a directional multi-view display that occludes an object, the multi-view display having a side comprising a display surface;

determining, for each of multiple observers located within an environment of the multi-view display, a gaze direction of the observer, wherein at least some of the multiple observers are determined to have a gaze direction that is toward the display surface;

determining, for each of the multiple observers, a priority score based at least in part on the gaze direction of the observer;

generating, for each of multiple observers determined to have a gaze direction that is toward the display surface, an image based on the determined gaze direction of the observer and the priority score for the user, each of the generated images being configured to blend an appearance of the display surface with the environment of the multi-view display for a different observer of the multiple observers, wherein at least some of the images for the observers are generated with different levels of image quality based on the priority scores for the observers; and

concurrently displaying the generated images on the display surface of the multi-view display, such that each of the generated images is directed to the corresponding observer whose gaze direction was used to generate the image.

2. The method of claim 1, wherein generating an image being configured to blend an appearance of the display surface of the multi-view display with an environment of the multi-view display comprises using data representing images taken by one or more cameras directed toward the display surface of the multi-view display to capture an image that includes at least a portion of the display surface of the multi-view display and an adjacent view of the environment.

3. The method of claim 1, further comprising determining, for each of the multiple observers, a distance between the observer and the display surface;

wherein determining the priority scores comprises determining, for each of the multiple observers, the priority score based at least in part on the gaze direction of the observer and the distance of the observer from the display surface, wherein the priority scores are determined such that the priorities of the multiple observers are inversely correlated with the distance of the multiple observers from the multi-view display.

4. The method of claim 1, further comprising determining a measure of attentiveness each of the multiple observers to the multi-view display;

wherein determining the priority scores comprises determining, for each of the multiple observers, the priority score based at least in part on the gaze direction of the observer and the measure of attentiveness of the observer to the display surface, wherein the priority scores are determined such that the priority of the observers is positively correlated with the level of attention of the multiple observers to the multi-view display.

5. The method of claim 1, further comprising determining one or more preferences associated with a particular observer of the multiple observers, and wherein generating, for each of the multiple observers, an image comprises generating an image for the particular observer based on (i) a detected eye direction of the particular observer, and (ii) the one or more preferences associated with the particular observer.

6. The method of claim 1,

wherein generating, for each of the multiple observers, an image based on the determined gaze direction of the observer comprises generating each image to provide, to the corresponding observer, a view that blends (i) an appearance of the display surface from a perspective of the observer with (ii) an appearance of the environment of the display surface from the perspective of the observer.

7. The method of claim 1, comprising:

filtering the multiple observers based on the priority scores to select a proper subset of the multiple observers based on the assigned priority scores; and

wherein generating the images comprises generating an image, for display on the display surface, for only the selected proper subset of the multiple observers.

8. The method of claim 1, wherein determining the priority scores comprises determining the priority score for each observer based on one or more of (i) a position of the observer, (ii) a distance of the observer from the display surface, or (iii) an attentiveness of the observer to the display surface.

9. The method of claim 1, further comprising dynamically changing one or more of the displayed images on the multi-view display as one or more of the gaze directions of one or more of the multiple observers changes.

10. The method of claim 1, wherein generating, for each of the multiple observers, an image based on the determined gaze direction of the observer comprises generating, for each of the multiple observers, an image that provides a view of a scene behind the occluded object from the perspective of the observer.

11. The method of claim 10, wherein generating, for each of the multiple observers, an image based on the determined gaze direction of the observer comprises augmenting, for one or more of the multiple observers, an image representing the scene to include one or more added visual elements that are not in the environment of the multi-view display.

12. The method of claim 10, wherein generating, for each of the multiple observers, an image based on the determined gaze direction of the observer comprises generating, for one or more of the multiple observers, an image that represents a simplified version of the scene, wherein the simplified version omits one or more elements of the scene behind the occluded object that would have been visible from the perspective of the observer.

13. The method of claim 1, wherein concurrently displaying the generated images on the display surface of the multi-view display comprises:

receiving data indicating one or more interactions of a particular observer of the multiple observers with a user device while the particular observer is in proximity to the multi-view display; and

generating an image based on (i) the determined gaze direction of the observer and (ii) the data indicating one or more interactions of the particular observer with the user device while the particular observer in proximity to the multi-view display.

14. The method of claim 1, wherein generating, for each of the multiple observers, an image based on the determined gaze direction of the observer comprises, for the image generated for at least one of the observers, using a trained machine learning system to complete an incomplete image of the environment.

15. The method of claim 1, wherein determining the gaze directions comprises determining the gaze directions of one or more of the multiple observers using image data from one or more cameras to detect eye positions of observers in the environment of the multi-view display;

wherein generating the images comprises;

generating a first image with a first level of image quality for a first observer based on the first observer having an assigned priority score indicating a first level of priority; and

generating a second image with a second level of image quality, which provides lower image quality than the first level of image quality, for a second observer based on the second observer having an assigned priority score indicating a second level of priority that is lower than the first level of priority.

16. The method of claim 1, wherein, to generate the images with different levels of image quality based on the priority scores, the images are generated to show the environment occluded by the multi-view display with different levels of fidelity, wherein the different levels of fidelity provide different levels of resolution, provide different levels of detail, or are updated at different rates.

17. The method of claim 1, further comprising determining, for each of the multiple observers, a distance between the observer and the display surface;

and

wherein generating the different images comprises generating the images for the observers to have different levels of resolution based on the observers being located at different distances from the display surface.

18. The method of claim 1, wherein the priority scores are determined based on gaze directions of the observers, distances of the observers from the multi-view display, and measures of attentiveness of the observers to the multi-view display.

19. The method of claim 1, comprising selecting, from among the multiple observers, a proper subset comprising observers determined to have a gaze direction that is toward the display surface, wherein the proper subset comprises observers having priority scores that exceed a predetermined priority score threshold or the proper subset comprises a predetermined number of observers selected according to the priority scores.

20. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

providing output from a directional multi-view display that occludes an object, the multi-view display having a side comprising a display surface;

determining, for each of multiple observers located within an environment of the multi-view display, a gaze direction of the observer, wherein at least some of the multiple observers are determined to have a gaze direction that is toward the display surface;

determining, for each of the multiple observers, a priority score based at least in part on the gaze direction of the observer;

generating, for each of multiple observers determined to have a gaze direction that is toward the display surface, an image based on the determined gaze direction of the observer and the priority score for the user, each of the generated images being configured to blend an appearance of the display surface with the environment of the multi-view display for a different observer of the multiple observers, wherein at least some of the images for the observers are generated with different levels of image quality based on the priority scores for the observers; and

concurrently displaying the generated images on the display surface of the multi-view display, such that each of the generated images is directed to the corresponding observer whose gaze direction was used to generate the image.

21. One or more non-transitory computer-readable media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

providing output from a directional multi-view display that occludes an object, the multi-view display having a side comprising a display surface;

determining, for each of multiple observers located within an environment of the multi-view display, a gaze direction of the observer, wherein at least some of the multiple observers are determined to have a gaze direction that is toward the display surface;

determining, for each of the multiple observers, a priority score based at least in part on the gaze direction of the observer;

generating, for each of multiple observers determined to have a gaze direction that is toward the display surface, an image based on the determined gaze direction of the observer and the priority score for the user, each of the generated images being configured to blend an appearance of the display surface with the environment of the multi-view display for a different observer of the multiple observers, wherein at least some of the images for the observers are generated with different levels of image quality based on the priority scores for the observers; and

concurrently displaying the generated images on the display surface of the multi-view display, such that each of the generated images is directed to the corresponding observer whose gaze direction was used to generate the image.