Method and apparatus for adjusting a view of a scene being displayed according to tracked head motion

• US 7,883,415 B2
• Filed: 09/15/2003
• Issued: 02/08/2011
• Est. Priority Date: 09/15/2003
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L., Deshpande, Hrishikesh R, Larsen, Eric J.
• Primary Examiner(s): Lewis; David L
• Assistant Examiner(s): Jones; Marcus D
• Application Number:
  US10/663,236
• Publication Number:
  US 20050059488A1
• Time in Patent Office: 2,703 Days
• Field of Search: 463 1- 69, 345/156, 345/7, 345/8
• US Class Current: 463/36
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6676 by dedicated player input

Abstract

A method for processing interactive user control with a scene of a video clip is provided. The method initiates with identifying a head of a user that is to interact with the scene of the video clip. Then, the identified head of the user is tracked during display of the video clip, where the tracking enables detection of a change in position of the head of the user. Next, a view-frustum is adjusted in accordance with the change in position of the head of the user. A computer readable media, a computing device and a system for enabling interactive user control for defining a visible volume being displayed are also included.

Claims

1. A method for processing interactive user control for a view of a scene displayed on a virtual window, comprising:

identifying a head of a user that is to interact with the scene;

storing an initial frame of user image data representing the head of the user, said view of the scene comprises a view-frustum initially defined by a gaze projection of a location of the head through outer edges of the virtual window when the location of the head is substantially normal to about a center point of the virtual window;

tracking the identified head of the user during display of the scene, the tracking enabling detection of a change in location of the head of the user, the tracking including,

identifying a search region within a frame of the user image data; and

comparing values within the search region to template values of the stored initial frame of image data;

adjusting the view-frustum in accordance with the change in location of the head of the user, the adjusting of the view-frustum being in response to tracking a move in the location of the head away from normal relative to the center point of the virtual window, the adjusted view-frustum defined by an updated gaze projection of the changed location of the head through the outer edges of the virtual window, such that the view-frustum moves in a direction opposite to the move in the location of the head;

adjusting a scale of the scene according to a change in a distance of the head of the user from a capture device; and

repeating the identifying the search region, the comparing, and the adjusting for successive frames of the scene, wherein the comparing is performed with the stored initial frame of image data.

2. The method of claim 1, wherein successive frames are compared to determine a relative distance of the head of the user to manipulate the scale of the scene.

3. The method of claim 1, wherein the capture device has depth capturing capability.

4. The method of claim 1, wherein the initial frame of image data is marker-less.

5. The method of claim 1, wherein the initial frame of data is maintained throughout the scene.

6. The method of claim 1, wherein the scene is of a video game.

7. The method of claim 6, wherein the interaction with the scene by tracking movement of the head of the user is independent of user hand-held controls for interacting with the video game.

8. The method of claim 1, wherein the method operation of tracking the identified head of the user during display of the scene includes,

tracking a facial portion of the head;

and

matching gray scale image data associated with the facial portion to image associated with a template of the facial portion.

9. A method for processing interactive user control with a scene, comprising:

identifying a head of a user that is to interact with the scene;

storing an initial frame of image data representing the head of the user for a duration of the scene;

tracking the identified head of the user during display of the scene, the tracking enabling detection of a change in location of the head of the user, the tracking including,

identifying a search region within a frame of the image data; and

comparing values within the search region to template values of the initial frame of image data;

translating a view-frustum in a direction opposite to the change in location of the head of the user while maintaining a focus on an object in the scene through adjustment of a view port size;

adjusting a scale of the scene according to a change in a distance of the head of the user from a capture device; and

successively updating the view frustum according to the change in location of the head of the user relative to the initial frame of image data.

10. The method of claim 9, wherein a view-frustum is defined by a gaze projection of a location of the head through outer edges of a virtual window when the location of the head is normal to a center point of the virtual window.

11. The method of claim 10, wherein translating the view-frustum maintains the virtual location of the head normal to the center point of the virtual window.

12. The method of claim 10, wherein the translating enables a change in the scene provided through the virtual window.

13. The method of claim 9, wherein the method operation of tracking the identified head of the user during display of the scene includes,

scanning a portion of each frame in the image data for the identified head.

14. The method of claim 9, wherein the method operation of translating a view-frustum in accordance with the change in location of the head of the user includes,

shifting the scene defined through the view-frustum while maintaining a lateral orientation of the head to a view port.

15. A system enabling interactive user control for defining a visible volume being displayed, comprising:

a computing device;

a display screen in communication with the computing device, the display screen configured to display image data defined through a view-frustum;

a tracking device in communication with the computing device, the tracking device capable of capturing a location change of a control object, wherein the location change of the control object effects an alignment of the view-frustum in the opposite direction relative to the display screen, wherein the computing device stores a marker-less reference image of the control object for comparison to each successive frame of image data captured through the tracking device and wherein the computing device adjusts a scale of the display image data according to a change in a distance of the control object from the tracking device, wherein the computing device is configured to adjust a view port size associated with the image data so that when the view frustum is adjusted, focus on an object within the view-frustum is maintained.

16. The system of claim 15, wherein the tracking device is a camera.

17. The system of claim 15, wherein the computing device is a video game console.

18. The system of claim 15, wherein the computing device is configured to map coordinates associated with the location change of the control object to a view change associated with a camera position.

19. The system of claim 15, wherein the computing device is configured to maintain a substantially normal gaze direction relative to a plane associated with the display screen for both the view-frustum and a view-frustum associated with the location change of the control object.

20. A method for processing interactive user control for a view of a scene displayed on a virtual window, comprising:

identifying a head of a user that is to interact with the scene;

storing an initial frame of user image data representing the head of the user, said view of the scene comprises a view-frustum initially defined by a gaze projection of a location of the head through outer edges of the virtual window when the location of the head is substantially normal to about a center point of the virtual window;

tracking the identified head of the user during display of the scene, the tracking enabling detection of a change in location of the head of the user, the tracking including;

laterally adjusting the view-frustum in a direction opposite to the change in location of the head of the user, the lateral adjusting of the view-frustum being in response to tracking a move in the location of the head away from normal relative to the center point of the virtual window, the laterally adjusted view-frustum defined by an updated gaze projection of the changed position of the head through the outer edges of the virtual window;

adjusting a scale of the scene according to a change in a distance of the head of the user from a capture device, the capture device having depth capturing capability; and

wherein the location of the head being away from normal relative to the center point of the virtual window changes an angle of the gaze projection, the change in angle of the gaze projection effects a change in viewing angle of the scene provided by a video clip.

21. The method of claim 20, wherein the change in viewing angle of the scene is a result of the detected movement of the head of the user to enable the interaction with the scene.

22. A method for processing interactive user control for a view of a scene displayed on a virtual window, comprising:

identifying a head of a user that is to interact with the scene;

storing an initial frame of user image data representing the head of the user, said view of the scene comprises a view-frustum initially defined by a gaze projection of a virtual viewpoint through outer edges of the virtual window when a location of the head is substantially normal to about a center point of the virtual window;

tracking the identified head of the user during display of the scene, the tracking enabling detection of a change in location of the head of the user, the tracking including,

laterally adjusting the virtual viewpoint in a same direction as a move in the location of the head away from normal relative to the center point of the virtual window, so as to laterally adjust the view-frustum in a direction opposite to the lateral adjustment of the virtual viewpoint, the laterally adjusted view-frustum defined by an updated gaze projection of the laterally adjusted virtual viewpoint through the outer edges of the virtual window;

adjusting a scale of the scene according to a change in a distance of the head of the user from a capture device.