Eye tracking enabling 3D viewing on conventional 2D display

• US 9,098,112 B2
• Filed: 03/05/2014
• Issued: 08/04/2015
• Est. Priority Date: 08/31/2010
• Status: Active Grant

• Current Assignee: Nintendo Company Limited
• Original Assignee: Nintendo Company Limited
• Inventors: Cheng, Howard, Newman, William C. Jr.
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): NAWAZ, TALHA M
• Application Number:
  US14/197,831
• Publication Number:
  US 20140184588A1
• Time in Patent Office: 517 Days
• Field of Search: 348 41-161
• US Class Current: 1/1
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 15/00 3D [Three Dimensional] image rendering
• CPC Class Codes: H04N 13/279 the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

The exemplary illustrative non-limiting technology herein enables 3D viewing on conventional 2D displays such as home television sets by tracking a person'"'"'s viewpoint. Detecting a player'"'"'s viewpoint movement to change the viewing of the displayed object gives the illusion that the object is physically present in three-dimensional space. Viewpoint movement detection can provide collision-related game logic benefits such as allowing a player to dodge projectiles, giving a game character an ability to “see” the player when not behind line-of-sight obstacles, and other advantages.

Claims

1. A method for providing three-dimensional (3D) display on a display device, the method comprising:

tracking the position and/or orientation of at least a part of a user;

using a computer processor, determining a viewpoint relative to the display device based at least in part on said tracking;

the computer processor using said determined viewpoint relative to the display device and at least one 3D virtual object modeled using real world coordinates to at least in part define the projection of the at least one 3D virtual object, including changing the viewing frustum for the projection of said at least one 3D virtual object in response to said tracking to introduce parallax for viewing the at least one 3D virtual object;

the computer processor adjusting 3D perspective transformations applied to the 3D virtual object as the determined viewpoint changes to generate new viewing perspectives; and

rendering for display, on said display device, said at least one 3D virtual object.

2. The method of claim 1 wherein the display device comprises a 2D display.

3. The method of claim 1 wherein the display device comprises a stationary display.

4. The method of claim 1 further including displaying said virtual object as part of an animated 3D video game.

5. The method of claim 1 wherein said tracking comprises tracking at least one marker disposed on the user'"'"'s body.

6. The method of claim 1 wherein said tracking comprises tracking at least three markers on the user'"'"'s body.

7. The method of claim 1 wherein said tracking comprises tracking light-emitting markers worn on the user'"'"'s head.

8. The method of claim 1 further including performing 3D collision detection for portions of said virtual object that appear to extend into the real world.

9. The method of claim 1 wherein the new viewing perspectives preserve and enhance 3D effects.

10. A system for providing three-dimensional (3D) display on a display device comprising:

a tracking device that tracks at least a portion of a user;

a processor configured to determine a viewpoint relative to the display device at least in part in response to said tracking and to use said determined viewpoint to at least in part define the projection of at least one virtual object, modeled using real world coordinates, onto an image plane, the processor further configured to change the viewing frustum for the projection of the virtual object in response to said tracking to introduce parallax for viewing the at least one virtual object;

the processor adjusting 3D perspective transformations applied to the 3D virtual object as the determined viewpoint changes to generate new viewing perspectives; and

an image stream generator configured to generate images that cause said projected at least one virtual object to be rendered for display on said display device.

11. The system of claim 10 wherein the display device is stationary.

12. The system of claim 10 wherein the display device comprises a 2D display.

13. The system of claim 10 wherein said tracking device tracks aspects of the position or orientation or both of the user'"'"'s head relative to the display device through use of at least one marker on the user'"'"'s head.

14. The system of claim 10 wherein said processor animates motion of said virtual object.

15. The system of claim 10 wherein said processor is adapted to cause display of said virtual object as part of an animated 3D video game.

16. The system of claim 10 wherein said tracking device tracks at least one point on the user'"'"'s head.

17. The system of claim 10 wherein said tracking device tracks at least three points on the user'"'"'s head.

18. The system of claim 10 wherein said tracking device comprises an infrared tracking device.

19. The system of claim 10 wherein said processor is structured to perform 3D collision detection for portions of said virtual object that appear to the user to extend out into the real world.

20. A non-transitory storage device storing computer program instructions that when executed by a computer processor, cause the computer processor to provide three-dimensional (3D) viewing display device, the computer program instructions comprising:

first instructions that track the position or orientation or both of at least a portion of a user;

second instructions that estimate the user'"'"'s viewpoint relative to the display device based at least in part on said tracking;

third instructions that use said user'"'"'s estimated viewpoint and both virtual and 3D geometry to at least in part define the two-dimensional (2D) projection of a 3D virtual object, modeled using real world coordinates, including changing the viewing frustum for the projection of the 3D virtual object in response to said tracking to introduce parallax;

fourth instructions that adjust 3D perspective transformations applied to the 3D virtual object as the determined viewpoint changes to generate new viewing perspectives; and

fifth instructions that cause display of said projected 3D virtual object on said display device.

21. The non-transitory storage device of claim 20 wherein said tracking of the position or orientation comprises at least one of a visible light camera or an infrared camera.

22. The non-transitory storage device of claim 20 wherein said tracking comprises imaging a user with a camera to determine the position or location of at least one of the head or eyes of a user with respect to the camera.