Object display with visual verisimilitude

• US 9,348,411 B2
• Filed: 05/24/2013
• Issued: 05/24/2016
• Est. Priority Date: 05/24/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Paek, Timothy S., Apacible, Johnson
• Primary Examiner(s): OLSON, JASON C
• Application Number:
  US13/901,599
• Publication Number:
  US 20140347262A1
• Time in Patent Office: 1,096 Days
• Field of Search: 345/5
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G09G 2340/04 Changes in size, position or resolution of an image
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 3/20 for presentation of an assembly of a number of characters, e.g. a page, by composing the assembly by combination of individual elements arranged in a matrix no fixed position being assigned to or needed to be assigned to the individual characters or partial characters
• CPC Class Codes: G09G 5/32 with means for controlling the display position see provisionally G09G5/42

Abstract

Described herein are technologies relating to display of a representation of an object on a display screen with visual verisimilitude to a viewer. A location of eyes of the viewer relative to a reference point on the display screen is determined. Additionally, a direction of gaze of the eyes of the viewer is determined. Based upon the location and direction of gaze of the eyes of the viewer, the representation of the object can be displayed at a scale and orientation such that it appears with visual verisimilitude to the viewer.

Claims

1. A method performed by a processor of a computing device, the method comprising:

receiving image data for display on a display, the image data comprising a plurality of views of a scene, the scene comprises an object;

receiving data from at least one sensor, the data being indicative of;

location of eyes of a viewer relative to the display; and

direction of gaze of the eyes of the viewer;

processing the image data based upon the data received from the at least one sensor, wherein processing the image data comprises;

computing a perspective and scale of the object for display on the display, the perspective and scale based upon the data received from the at least one sensor; and

generating a representation of the object based upon the computed perspective and scale and the plurality of views of the scene; and

displaying the representation of the object on the display using the perspective and the scale such that, when the eyes of the viewer are at the location and the gaze is in the direction indicated in the data received from the sensor, the representation of the object is displayed with visual verisimilitude to the viewer.

2. The method of claim 1, the image data being video data, and the processing of the image data and the displaying of the representation of the object are performed in real-time.

3. The method of claim 1, the data from the at least one sensor being an image received from a camera that is directed at the viewer, the method further comprising computing the direction of gaze of the eyes of the viewer based upon the image received from the camera.

4. The method of claim 3, wherein computing the direction of gaze of the eyes of the viewer comprises:

identifying a head of the viewer in the image;

computing a tilt of the head of the viewer in the image; and

determining the direction of the gaze of the eyes of the viewer based upon the computing of the tilt of the head of the viewer in the image.

5. The method of claim 3, wherein computing the direction of gaze of the eyes of the viewer comprises:

identifying a first feature and a second feature of a face of the viewer in the image;

computing data indicative of an orientation of the first feature relative to the second feature responsive to identifying the first feature and the second feature of the face of the viewer; and

determining the direction of the gaze of the eyes of the viewer based at least in part upon the data that is indicative of the orientation of the first feature relative to the second feature.

6. The method of claim 3, wherein the display screen is a translucent display screen, and the camera is positioned such that the display is between the viewer and the camera.

7. The method of claim 6, the display being an organic light emitting diode (OLED) display screen.

8. The method of claim 1, further comprising:

receiving audio data, the audio data corresponding in time to the image data; and

controlling operation of a directional speaker based upon the data from the at least one sensor to cause the directional speaker to output the audio data toward the viewer.

9. The method of claim 1, further comprising:

detecting a gesture set forth by the viewer based upon the data received from the at least one sensor; and

updating content displayed on the display based at least in part upon the gesture detected as being set forth by the viewer.

10. The method of claim 1, further comprising:

detecting a voice command set forth by the viewer based upon the data received from the at least one sensor; and

updating content displayed on the display based at least in part upon the voice command detected as being set forth by the viewer.

11. The method of claim 1, further comprising:

receiving data from a treadmill upon which the viewer is moving, wherein the processing of the image data is based upon the data received from the treadmill.

20. The method of claim 1, further comprising:

receiving second data from the at least one sensor;

determining, based upon the second data, that at least one of the location of eyes of the viewer relative to the display or the direction of gaze of the eyes of the viewer has changed;

computing, based upon the second data, a perspective and a distance of the viewer relative to the display; and

displaying a second representation of the object to the viewer with visual verisimilitude.

12. A system that facilitates presenting imagery on a display screen, the system comprising:

a processor; and

memory that comprises instructions that, when executed by the processor, cause the processor to perform acts comprising;

receiving an image from a camera, the camera configured to capture images of a viewer of the display screen;

computing a location of eyes of the viewer and direction of gaze of the viewer relative to the display screen based upon the image received from the camera;

processing image data based upon the location of the eyes of the viewer and the direction of the gaze of the viewer, the image data comprising a plurality of views of a scene that comprises an object, wherein processing the image data comprises;

selecting a view from the plurality of views of the scene; and

scaling the object in the view, wherein the view selected is selected and the object is scaled to cause a representation of the object to be displayed to the viewer with visual verisimilitude when the eyes of the user are at the computed location and gazing in the computed direction; and

displaying the representation of the object on the display screen.

13. The system of claim 12, wherein the image data comprises a plurality of video frames from a respective plurality of video cameras.

14. The system of claim 12 comprised by a television.

15. The system of claim 12 comprised by a mobile computing device.

16. The system of claim 12, wherein the display screen is a transparent OLED display screen, and wherein the camera is positioned such that the display screen is between the camera and the viewer.

17. The system of claim 16, the acts further comprising receiving images from a respective plurality of cameras, the display screen positioned between the viewer and the plurality of cameras.

18. The system of claim 12, the acts further comprising transmitting the image to another computing device of another viewer to facilitate creation of a portal between the viewer and the another viewer.

19. A display device, comprising:

a transparent OLED display screen;

a camera positioned to capture an image through the OLED display screen;

memory comprising instructions that, when executed by the processor, cause the processor to perform acts comprising;

receiving the image captured by the camera, the image comprising a viewer of the display device;

computing a location of eyes of the viewer based upon the image captured by the camera;

computing a direction of gaze of the eyes of the viewer based upon the image captured by the camera;

receiving image data that is to be displayed to the viewer on the OLED display screen, the image data comprising a plurality of views of an object;

responsive to computing the location of the eyes of the viewer and the direction of the gaze of the viewer, processing the image data to generate a representation of the object based upon the location of the eyes of the viewer and the direction of the gaze of the viewer, such that when the representation of the object is displayed on the OLED display screen, and when the eyes of the viewer are at the computed location and the gaze of the viewer is in the computed direction, the representation of the object is displayed to the viewer with visual verisimilitude, wherein processing the image data comprises selecting a view from the plurality of views based upon the location of the eyes of the viewer and the direction of the gaze of the viewer; and

displaying the representation of the object on the OLED display screen until one of the location of the eyes of the viewer alters or the direction of the gaze of the viewer alters.