APPARATUS AND METHOD FOR AUGMENTED REALITY

• US 20130303285A1
• Filed: 05/07/2013
• Published: 11/14/2013
• Est. Priority Date: 05/11/2012
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Kochi, Masami, Harding, Russell, Campbell, Diarmid Archibald, Ranyard, David, Hocking, Ian Michael
• Granted Patent: US 9,724,609 B2
• Time in Patent Office: Days
• US Class Current: 463/32
• CPC Class Codes: A63F 13/00 Video games, i.e. games using an electronically generated display having two or more dimensions gaming systems which provide a financial reward G07F17/32
• CPC Class Codes: A63F 13/56 Computing the motion of game characters with respect to other game characters, game objects or elements of the game scene, e.g. for simulating the behaviour of a group of virtual soldiers or for path finding
• CPC Class Codes: A63F 13/573 using trajectories of game objects, e.g. of a golf ball according to the point of impact
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

A method of incorporating an augmented reality video image into a virtual environment comprises detecting a state of a real-world control object captured in a video image and augmenting the video image responsive to the state of the control object. The method further includes calculating trajectory parameters for an action with respect to a local co-ordinate system of the video image, responsive to the state of the control object, and embedding the augmented video image within a 3D virtual environment. The method also includes calculating the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment, and rendering a graphical effect corresponding to the action within the 3D virtual environment.

Claims

1. A method of incorporating an augmented reality video image into a virtual environment, comprising:

detecting a state of a real-world control object captured in a video image;

augmenting the video image responsive to the state of the real-world control object;

calculating trajectory parameters for an action with respect to a local co-ordinate system of the video image, responsive to the state of the real-world control object;

embedding the augmented video image within a 3D virtual environment;

calculating the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment; and

rendering a graphical effect corresponding to the action within the 3D virtual environment.

2. A method according to claim 1, comprising the steps of:

calculating, for the trajectory, a point of intersection with an image plane of the video image; and

calculating the corresponding point within the 3D virtual environment in which the augmented video image is embedded;

and in which;

the step of augmenting the video image responsive to the state of the real-world control object comprises augmenting the video image with a graphical effect corresponding to the action; and

the step of rendering the graphical effect corresponding to the action within the 3D virtual environment comprises rendering the graphical effect with a start-point at the calculated corresponding point within the 3D virtual environment.

3. A method according to claim 1, comprising the steps of:

calculating a position and orientation of the real-world control object with respect to the local co-ordinate system of the video image; and

mapping the calculated position and orientation of the real-world control object to the local co-ordinate system of the 3D virtual environment, responsive to a position and orientation of the augmented video image embedded within the 3D virtual environment;

and in which;

the step of rendering the graphical effect corresponding to the action within the 3D virtual environment comprises rendering the graphical effect with a start-point at the start of the calculated trajectory.

4. A method according to claim 3, in which the step of rendering the graphical effect corresponding to the action within the 3D virtual environment comprises modifying a depth buffer to allow display of the graphical effect when the augmented video image is positioned between the start of the calculated trajectory and the rendering viewpoint.

5. A method according to claim 1, in which the calculated trajectory is one or more selected from the list consisting of:

i. a linear vector;

ii. a parametrically defined non-linear path; and

iii. a procedurally defined non-linear path

6. A method according to claim 1, in which the step of embedding the augmented video image within the 3D virtual environment comprises scaling the augmented video image responsive to the size of the control object within the captured video image.

7. A method according to claim 1 in which the step of embedding the augmented video image within the 3D virtual environment comprises applying a graphical effect to the augmented video image that obscures the edges of the augmented video image.

8. A method according to claim 1, comprising the step of modifying a colour balance of the video image in response to one or more selected from the list consisting of:

i. a colour temperature of a virtual object used in the augmentation of the video image; and

ii. a colour temperature of the 3D virtual environment.

9. A method according to claim 1, comprising the step of modifying a brightness of the video image in response to one or more selected from the list consisting of:

i. a predefined brightness of a virtual object used in the augmentation of the video image; and

ii. a brightness value for the 3D virtual environment.

10. A non-transitory computer program product comprising computer program instructions that when performed by a processor implement a method comprising the steps of:

detecting a state of a real-world control object captured in a video image;

augmenting the video image responsive to the state of the real-world control object;

calculating trajectory parameters for an action with respect to a local co-ordinate system of the video image, responsive to the state of the real-world control object;

embedding the augmented video image within a 3D virtual environment;

calculating the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment; and

rendering a graphical effect corresponding to the action within the 3D virtual environment.

11. An entertainment device, comprising

an input configured to receive a captured video image depicting a real-world control object;

a first image analyser configured to detect a state of the real-world control object in the video image;

a first image processor configured to augment the video image responsive to the state of the control object;

a first trajectory processor configured to calculate trajectory parameters for an action with respect to a local co-ordinate system of the video image, responsive to the state of the control object;

a second image processor configured to embed the augmented video image within a 3D virtual environment;

a second trajectory processor configured to calculate the trajectory parameters for the action with respect to a local co-ordinate system of the 3D virtual environment; and

the second image processor configured to render a graphical effect corresponding to the action within the 3D virtual environment.

12. An entertainment device according to claim 11, in which:

the first trajectory processor is configured to calculate for the trajectory a point of intersection with an image plane of the video image; and

the second trajectory processor is configured to calculate the corresponding point within the 3D virtual environment in which the augmented video image is embedded;

the first image processor is configured to augment the video image with a graphical effect corresponding to the action; and

the second image processor is configured to render the graphical effect with a start-point at the calculated corresponding point within the 3D virtual environment.

13. An entertainment device according to claim 11, in which:

the first trajectory processor is configured to calculate a position and orientation of the real-world control object with respect to the local co-ordinate system of the video image; and

the second trajectory processor is configured to map the calculated position and orientation of the real-world control object to the local co-ordinate system of the 3D virtual environment, responsive to a position and orientation of the augmented video image embedded within the 3D virtual environment;

and in which;

the second image processor is configured to render the graphical effect with a start-point at the start of the calculated trajectory.

14. An entertainment device according to claim 13, comprising a depth buffer arranged to allow display of the graphical effect when the augmented video image is positioned between the start of the calculated trajectory and the rendering viewpoint.

15. An entertainment device according to claim 11, in which the second image processor is configured to scale the augmented video image responsive to the size of the control object within the captured video image when embedding the augmented video image within the 3D virtual environment.