Method and apparatus for adjusting a view of a scene being displayed according to tracked head motion

• US 20050059488A1
• Filed: 09/15/2003
• Published: 03/17/2005
• Est. Priority Date: 09/15/2003
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Marks, Richard L., Deshpande, Hrishikesh R., Larsen, Eric J.
• Granted Patent: US 7,883,415 B2
• Time in Patent Office: Days
• US Class Current: 463/36
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/5258 by dynamically adapting the position of the virtual camera to keep a game object or game character in its viewing frustum, e.g. for tracking a character or a ball
• CPC Class Codes: A63F 2300/1012 involving biosensors worn by the player, e.g. for measuring heart beat, limb activity
• CPC Class Codes: A63F 2300/1093 using visible light
• CPC Class Codes: A63F 2300/6661 for changing the position of the virtual camera
• CPC Class Codes: A63F 2300/6676 by dedicated player input

Abstract

A method for processing interactive user control with a scene of a video clip is provided. The method initiates with identifying a head of a user that is to interact with the scene of the video clip. Then, the identified head of the user is tracked during display of the video clip, where the tracking enables detection of a change in position of the head of the user. Next, a view-frustum is adjusted in accordance with the change in position of the head of the user. A computer readable media, a computing device and a system for enabling interactive user control for defining a visible volume being displayed are also included.

Claims

1. A method for processing interactive user control with a scene of a video clip, comprising:

identifying a head of a user that is to interact with the scene of the video clip;

tracking the identified head of the user during display of the video clip, the tracking enabling detection of a change in position of the head of the user, the tracking including,

identifying a search region within a frame of the video clip; and

comparing values within the search region to template values; and

adjusting a view-frustum in accordance with the change in position of the head of the user.

2. The method of claim 1, wherein a view-frustum is initially defined by a triangular gaze projection set between outer edges of a virtual window and a virtual position of the head when the virtual position of the head is normal to a center point of the virtual window.

3. The method of claim 2, wherein adjusting the view-frustum moves the virtual position of the head away from normal relative to the center point of the virtual window.

4. The method of claim 3, wherein the virtual position of the head being away from normal relative to the center point of the virtual window changes an angle of the triangular gaze projection, the change in angle of the triangular gaze projection displays a change in viewing angle of the scene provided by the video clip.

5. The method of claim 4, wherein the change in viewing angle of the scene is a result of the detected movement of the head of the user to enable the interaction with the scene.

6. The method of claim 1, wherein identifying the head of the user includes capturing a template of a face for the head.

7. The method of claim 6, wherein the change in position of the head of the user is continuously tracked frame-by-frame of the video clip to enable continual adjustments of the view-frustum so that a viewing angle of the scene is changed in response to the changes in position of the head of the user.

8. The method of claim 6, wherein capturing the template includes,

obtaining a marker-less image of the face for the head of the user.

9. The method of claim 6, wherein a camera is used to enable the capturing.

10. The method of claim 1, wherein the video clip is of a video game.

11. The method of claim 10, wherein the interaction with the scene by tracking movement of the head of the user is independent of user hand-held controls for interacting with the video game.

12. The method of claim 1, wherein the method operation of tracking the identified head of the user during display of the video clip includes,

tracking a facial portion of the head;

and

matching gray scale image data associated with the facial portion to image associated with a template of the facial portion.

13. The method of claim 1, wherein the method operation of adjusting a view-frustum in accordance with the change in position of the head of the user includes,

identifying a point of interest of the scene of the video clip;

and

modifying the view-frustum so that the point of interest appears at a constant position when displayed in successive video clips.

14. A method for processing interactive user control with a scene of a video clip, comprising:

identifying a head of a user that is to interact with the scene of the video clip;

tracking the identified head of the user during display of the video clip, the tracking enabling detection of a change in position of the head of the user, the tracking including,

identifying a search region within a frame of the video clip; and

comparing values within the search region to template values; and

translating a view-frustum in accordance with the change in position of the head of the user.

15. The method of claim 14, wherein a view-frustum is defined by a triangular gaze projection set between outer edges of a virtual window and a virtual position of the head when the virtual position of the head is normal to a center point of the virtual window.

16. The method of claim 15, wherein translating the view-frustum maintains the virtual position of the head normal to the center point of the virtual window.

17. The method of claim 15, wherein the translating enables a change in the scene provided through the virtual window.

18. The method of claim 14, wherein the method operation of tracking the identified head of the user during display of the video clip includes,

scanning a portion of each frame of the video clip for the identified head.

19. The method of claim 14, wherein the method operation of translating a view-frustum in accordance with the change in position of the head of the user includes,

shifting a scene defined through the view-frustum while maintaining a lateral orientation of the head to a view port.

20. The method of claim 14, wherein the method operation of translating a view-frustum in accordance with the change in position of the head of the user includes,

maintaining a focus on an object in the scene through adjustment of a view port size.

21. The method of claim 14, wherein the method operation of translating a view-frustum in accordance with the change in position of the head of the user includes,

rotating the view-frustum about the head of a user according to the change in position of the head of the user.

22. A method for managing a visible volume displayed through a view port, comprising:

locating a head of a user;

capturing image data associated with the head of the user at a different location;

scanning a search region defined within the image data;

identifying the different location of the head of the user within the search region; and

adjusting the visible volume based upon the location of the head of the user relative to the view port.

23. The method of claim 22, wherein the method operation of scanning a search region defined within the image data includes,

defining boundaries of the search region according to a maximum distance the head of the user moves between successive video frames.

24. The method of claim 22, wherein the method operation of scanning a search region defined within the image data includes,

comparing a template of a facial region of the head of the user with corresponding image areas generated through the scanning of the search region.

25. The method of claim 22, wherein the method operation of locating a head of a user includes,

capturing image data representing a facial region of the head;

and

storing the image data representing the facial region.

26. The method of claim 22, wherein the method operation of adjusting the visible volume based upon the location of the head of the user relative to the view port includes,

defining a change of the visible volume being displayed according to a degree of movement of the location of the head.

27. The method of claim 22, wherein the method operation of adjusting the visible volume based upon the location of the head of the user relative to the view port includes,

adjusting a scale associated with the visible volume according to a distance between the head and the view port.

28. A computer readable medium having program instructions for processing interactive user control with a scene of a video clip, comprising:

program instructions for identifying a head of a user that is to interact with the scene of the video clip;

program instructions for tracking the identified head of the user during display of the video clip, the tracking enabling detection of a change in position of the head of the user, the program instructions for tracking including,

program instructions for identifying a search region within a frame of the video clip; and

program instructions for comparing values within the search region to template values; and

program instructions for adjusting a view-frustum in accordance with the change in position of the head of the user.

29. The computer readable medium of claim 28, wherein the program instructions for identifying the head of the user includes,

program instructions for capturing a template of a face for the head.

30. The computer readable medium of claim 28, wherein the change in position of the head of the user is continuously tracked frame-by-frame of the video clip to enable continual adjustments of the view-frustum so that a viewing angle of the scene is changed in response to the changes in position of the head of the user.

31. The computer readable medium of claim 29, wherein the program instructions for capturing the template includes,

program instructions for obtaining a marker-less image of the face for the head of the user.

32. The computer readable medium of claim 28, wherein the program instructions for tracking the identified head of the user during display of the video clip includes,

program instructions for tracking a facial portion of the head;

and

program instructions for matching gray scale image data associated with the facial portion to image associated with a template of the facial portion.

33. The computer readable medium of claim 28, wherein the program instructions for adjusting a view-frustum in accordance with the change in position of the head of the user includes,

program instructions for identifying a point of interest of the scene of the video clip;

and

program instructions for modifying the view-frustum so that the point of interest appears at a constant position when displayed in successive video clips.

34. A computer readable medium having program instructions for processing interactive user control with a scene of a video clip, comprising:

program instructions for identifying a head of a user that is to interact with the scene of the video clip;

program instructions for tracking the identified head of the user during display of the video clip, the tracking enabling detection of a change in position of the head of the user, the program instructions for tracking including,

program instructions for identifying a search region within a frame of the video clip; and

program instructions for comparing values within the search region to template values; and

program instructions for translating a view-frustum in accordance with the change in position of the head of the user.

35. The computer readable medium of claim 34, wherein the program instructions for tracking the identified head of the user during display of the video clip include,

program instructions for scanning a portion of each frame of the video clip for the identified head.

36. The computer readable medium of claim 34, wherein the program instructions for translating a view-frustum in accordance with the change in position of the head of the user include,

program instructions for shifting a scene defined through the view-frustum while maintaining a lateral orientation of the head to a view port.

37. The computer readable medium of claim 34, wherein the program instructions for translating a view-frustum in accordance with the change in position of the head of the user include,

program instructions for maintaining a focus on an object in the scene through adjustment of a view port size.

38. The computer readable medium of claim 34, wherein the program instructions for translating a view-frustum in accordance with the change in position of the head of the user includes,

program instructions for rotating the view-frustum about the head of a user according to the change in position of the head of the user.

39. A computer readable medium having program instructions for managing a visible volume displayed through a view port, comprising:

program instructions for locating a head of a user;

capturing image data associated with the head of the user at a different location;

scanning a search region defined within the image data;

identifying the different location of the head of the user within the search region; and

program instructions for adjusting the visible volume based upon the location of the head of the user relative to the view port.

40. The computer readable medium of claim 39, wherein the program instructions for identifying the different location of the head of the user within the search region includes,

program instructions for calculating a difference between values associated with a template of the head and a template of the head at the different location.

41. The computer readable medium of claim 40, wherein the program instructions for scanning a search region defined within the image data includes,

program instructions for defining boundaries of the search region according to a maximum distance the head of the user moves between successive video frames.

42. The computer readable medium of claim 40, wherein the program instructions for scanning a search region defined within the image data includes,

program instructions for comparing a template of a facial region of the head of the user with a corresponding template generated through the scanning of the search region.

43. The computer readable medium of claim 39, wherein the program instructions for locating a head of a user includes,

program instructions for capturing image data representing a facial region of the head;

and

program instructions for storing the image data representing the facial region.

44. The computer readable medium of claim 39, wherein the program instructions for adjusting the visible volume based upon the location of the head of the user relative to the view port includes,

program instructions for defining a change of the visible volume being displayed according to a degree of movement of the location of the head.

45. The computer readable medium of claim 39, wherein the program instructions for adjusting the visible volume based upon the location of the head of the user relative to the view port includes,

program instructions for adjusting a scale associated with the visible volume according to a distance between the head and the view port.

46. A system enabling interactive user control for defining a visible volume being displayed, comprising:

a computing device;

a display screen in communication with the computing device, the display screen configured to display image data defined through a view-frustum;

a tracking device in communication with the computing device, the tracking device capable of capturing a location change of a control object, wherein the location change of the control object effects an alignment of the view-frustum relative to the display screen.

47. The system of claim 46, wherein the tracking device is a camera.

48. The system of claim 46, wherein the computing device is a video game console.

49. The system of claim 46, wherein the computing device is configured to map coordinates associated with the location change of the control object to a view change associated with a virtual camera position.

50. The system of claim 46, wherein the computing device is configured to maintain a substantially normal gaze direction relative to a plane associated with the display screen for both the view-frustum and a view-frustum associated with the location change of the control object.

51. The system of claim 46, wherein the computing device is configured to adjust a view port size associated with the image data so that when the view-frustum is adjusted, focus on an object within the view-frustum is maintained.

52. A computing device, comprising:

a memory configured to store a template of a control object;

a processor capable of receiving a video signal tracking the control object, the processor including,

logic for comparing a portion of a frame of the video signal to the template;

logic for identifying a change in a location of the control object in the portion of the frame relative to a location of the control object associated with the template; and

logic for translating the change in the location of the control object to adjust a view-frustum associated with an original location of the control object.

53. The computing device of claim 52, wherein the logic for translating the change in the location of the control object to adjust a view-frustum associated with an original location of the control object includes,

logic for shifting boundaries of a visible scene being displayed on a display screen in communication with the computing device.

54. The computing device of claim 52, wherein the computing device is a video game console.

55. The computing device of claim 52, wherein the template is stored as grayscale image data.

56. The computing device of claim 52, wherein the logic for comparing a portion of a frame of the video signal to the template includes,

logic for scanning the portion of the frame of the video signal.

57. The computing device of claim 52, wherein the control object is a head of a user.

58. The computing device of claim 57, wherein the logic for translating the change in the location of the control object to adjust a view-frustum associated with an original location of the control object includes,

logic for adjusting a scale associated with a display signal according to a distance between the head and a view port.