Entertainment Device, System, and Method

• US 20100045869A1
• Filed: 08/17/2009
• Published: 02/25/2010
• Est. Priority Date: 08/19/2008
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Europe Limited (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Europe Limited (Sony Group Corp.)
• Inventors: Doucet, Nicolas, Baseley, Nathan James
• Granted Patent: US 8,542,250 B2
• Time in Patent Office: Days
• US Class Current: 348/598
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 13/00 Animation
• CPC Class Codes: G06T 15/10 Geometric effects
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

An entertainment device for combining virtual images with real images captured by a video camera so as to generate augmented reality images. The device comprises receiving means operable to receive a sequence of video images from the video camera via a communications link. The device further comprises detecting means operable to detect an augmented reality marker within the received video images, and processing means operable to generate a virtual image plane in dependence upon the detection of the augmented reality marker by the detecting means. The virtual image plane is arranged to be substantially coplanar with a real surface upon which the augmented reality marker is placed so that virtual images may be generated with respect to the real surface. The processing means is operable to generate the virtual image plane within the captured video images such that the virtual image plane is defined with respect to the detected augmented reality marker, and the virtual image plane has an area which is greater than an area which corresponds to the augmented reality marker.

Claims

1. An entertainment device for combining virtual images with real images captured by a video camera so as to generate augmented reality images, said device comprising:

a receiver operable to receive a sequence of video images from said video camera via a communications link;

a detector operable to detect an augmented reality marker within the received video images; and

a processor operable to generate a virtual image plane in dependence upon the detection of said augmented reality marker by said detector, said virtual image plane being arranged to be substantially coplanar with a real surface upon which said augmented reality marker is placed so that virtual images may be generated with respect to said real surface,

in which;

said processor is operable to generate said virtual image plane within the captured video images, and to calibrate said virtual image plane to said augmented reality marker such that said virtual image plane is defined with respect to the detected augmented reality marker, said processor being operable to generate said virtual image plane such that, once said virtual image plane has been calibrated to said augmented reality marker, said augmented reality marker can be removed from a field of view of said camera; and

said virtual image plane has an area which is greater than an area which corresponds to said augmented reality marker.

2. A device according to claim 1, in which said processor is operable to:

detect a degree of distortion between a predetermined representation of said augmented reality marker and the detected augmented reality marker; and

generate said virtual image plane in dependence upon the detected degree of distortion.

3. An entertainment device according to claim 1, in which said processor is operable to generate a virtual reality object at a virtual reality object position, said virtual reality object position being generated by said processor with reference to said virtual image plane.

4. A device according to claim 3, in which:

said virtual image plane relates to an interaction area within said augmented reality images, said interaction area being a region of said captured images in which said virtual reality object may be displayed for interaction with a user;

said processor is operable to generate said interaction area in dependence upon said virtual image plane so that said virtual reality object can be displayed within said interaction area in combination with said real images captured by said video camera.

5. A device according to claim 4, in which said processor is operable to generate said interaction area so that a pixel area of said interaction area is a predetermined percentage of a full pixel area of said video camera.

6. A device according to claim 4, in which said processor is operable to:

detect, by analysis of said received video images, interaction area pixels which surround said augmented reality marker and are colour matched to within a predetermined colour threshold with respect to a colour of one or more pixels which are detected as being adjacent to pixels corresponding to said augmented reality marker; and

generate said interaction area so that said interaction area corresponds to said detected interaction area pixels.

7. A device according to claim 4, in which:

said detector is operable to detect at least another augmented reality marker within said received video images; and

said processor is operable to generate said interaction area so that at least part of a perimeter of said interaction area corresponds with the detected augmented reality markers.

8. A device according to claim 3, in which said virtual reality object is a virtual creature.

9. An entertainment device according to claim 1, in which:

said receiver is operable to receive said sequence of video images together with depth information generated by said video camera which relates to a distance between said video camera and said augmented reality marker;

said detector is operable to detect, by analysis of said received video images and said received depth information, a three dimensional spatial position of said augmented reality marker with respect to said video camera; and

said processor is operable to generate said virtual image plane within said captured video images such that said virtual image plane is defined with respect to said detected three dimensional spatial position of said augmented reality marker.

10. A device according to claim 3, in which:

said receiver is operable to receive said sequence of video images together with depth information generated by said video camera which relates to a distance between said video camera and said augmented reality marker;

said detector is operable to detect, by analysis of said received video images and said received depth information, a three dimensional spatial position of said augmented reality marker with respect to said video camera;

said processor is operable to generate said virtual image plane within said captured video images such that said virtual image plane is defined with respect to said detected three dimensional spatial position of said augmented reality marker; and

said processor is operable to;

detect, in dependence upon said received depth information, whether said virtual reality object position is such that said virtual reality object is occluded from a point of view of said video camera by a real object; and

generate said virtual object so that said virtual object appears to be behind said real object if said virtual reality object is occluded by said real object.

11. A device according to claim 1, in which said augmented reality marker is associated with a game feature of a game executed by said entertainment device.

12. A device according to claim 1, in which said detector is operable to detect said augmented reality marker within a single frame selected from said received video images in response to an indication that detection of said marker should be performed.

13. A system for combining virtual images with real images captured by a video camera so as to generate augmented reality images, said system comprising:

a video camera operable to capture video images of a real environment and generate depth information which relates to a distance between said video camera and an augmented reality marker; and

an entertainment device according to claim 1.

14. A method of combining virtual images with real images captured by a video camera so as to generate augmented reality images, said method comprising:

receiving a sequence of video images from said video camera via a communications link;

detecting an augmented reality marker within said received video images;

generating a virtual image plane in dependence upon said detection of said augmented reality marker, said virtual image plane being arranged to be substantially coplanar with a real surface upon which said augmented reality marker is placed so that virtual images may be generated with respect to said real surface; and

calibrating said virtual image plane to said augmented reality marker,

in which;

said virtual image plane is generated within said captured video images, and said virtual image plane is calibrated to said augmented reality marker such that said virtual image plane is defined with respect to said detected augmented reality marker;

said virtual image plane is generated such that, once said virtual image plane has been calibrated to said augmented reality marker said virtual image plane to said augmented reality marker, said augmented reality marker can be removed from a field of view of said camera; and

said virtual image plane has an area which is greater than an area which corresponds to said augmented reality marker.

15. A computer program for implementing the method of claim 14.