COMPOSITE IMAGE GENERATING SYSTEM, OVERLAYING CONDITION DETERMINING METHOD, IMAGE PROCESSING APPARATUS, AND IMAGE PROCESSING PROGRAM

• US 20110090343A1
• Filed: 03/26/2009
• Published: 04/21/2011
• Est. Priority Date: 03/27/2008
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Metaio GmbH (Apple Inc.)
• Inventors: Ookura, Masahiro, Eble, Tobias, Alt, Thomas, Meier, Peter
• Granted Patent: US 8,614,747 B2
• Time in Patent Office: Days
• US Class Current: 348/164
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

A mark which at least reflects or radiates invisible light of predetermined wavelength other than visible light is added in a space of the real world. A camera apparatus comprises image capturing means for capturing a real image in which an invisible-light image may be discriminating. An image processing apparatus, comprises: mark discriminating means for discriminating at least one condition of a position of image of mark in the captured real image, a orientation of the mark, and a distance from the mark to the image capturing means, and overlaying condition determining means for determining, in correspondence with the discriminated condition, a overlaying condition which is at least one of an overlaying position which is a position of the image of the virtual object overlaid on the captured real image in the real image, an orientation of the virtual object which the age of the virtual object indicates, and a distance from a view point of viewer, of the image of the virtual object.

Claims

1. A composite image generating system, comprising a camera apparatus and an image processing apparatus, for overlaying a virtual image which is an image of a virtual object on a real image which is an image of the real world and the virtual object,

wherein at least one mark which at least reflects or radiates invisible light of a particular wavelength other than visible light is added in a space of the real world,

the image generating system comprises image capturing means for capturing the real image in which an invisible-light image may be discriminating, and

the image processing apparatus, comprises:

mark discriminating means for discriminating at least one condition of a position of an image of the mark in the captured real image, a orientation of the mark, and a distance from the mark to the image capturing means, and

overlaying condition determining means for determining, in correspondence with the discriminated condition, at least one overlaying condition which is at least one of an overlaying position which is a position of the image of a virtual object overlaid on the captured real image in the real image, an orientation of the virtual object which the image of the virtual object indicates, and a distance from a view point of a viewer, of the image of the virtual object, and

mark image erasing means for specifying an area which the mark image occupies in the captured real image, and based on the image outside the area, complementing the image of the real world in the area, to erase the mark image in the real image in a way that the user is unable to extract a hidden code from the camera image in order to copy a mark.

2. The composite image generating system according to claim 1, wherein the image capturing means of the camera apparatus captures the real image in which a visible-light image and an invisible-light image may be discriminated from each other, and the image processing apparatus further comprises composite image generating means for, based on at least one determined overlaying condition, generating the image of at least one virtual object, overlaying the image of the virtual object on the captured real image, and generating the composite image.

3. The composite image generating system according to claim 1,

wherein the overlaying condition determining means determines the overlaying position in such a way that the mark image is hidden by overlaying the image of the virtual object or another virtual object on the captured real image.

4. The composite image generating system according to claim 3,

wherein a virtual object being used to hide the mark image is similar to a known background in the real image.

5. The composite image generating system according to claim 1,

further comprising mark hiding means, which may use on of or switch between erasing the mark image in the real image, complementing the image of the real world in the area of the mark from information outside the mark area and overlaying a virtual object or another virtual object over the real image of the mark and changing the mark image by image processing means.

15. A composite image generating system according to claim 1,

wherein the at least one mark reflects or radiates invisible light of wavelength 0.7-2.5 micrometers.

6. An overlaying condition determining method in a composite image generating system for overlaying at least one virtual image which is an image of at least one virtual object on a real image which is an image of the real world to generate a composite image of the real world and the virtual object,

wherein at least one mark which at least reflects or radiates invisible light of predetermined wavelength other than visible light is added in a space of the real world,

a camera apparatus captures the real image in which an invisible-light image may be discriminating, and

an image processing apparatus discriminates at least one condition of a position of image of the mark in the captured real image, a orientation of the mark, and a distance from the mark to the image capturing means, and

determines, in correspondence with the at least one discriminated condition, at least one overlaying condition which is at least one of an overlaying position which is a position of the image of the at least one virtual object overlaid on the captured real image in the real image, an orientation of the at least one virtual object which the image of the at least one virtual object indicates, and a distance from a view point of a viewer, of the image of the at least one virtual object, and

specifies an area which the mark image occupies in the captured real image, and based on the image outside the area, complements the image of the real world in the area, to erase the mark image in the real image in a way that the user is unable to extract a hidden code from the camera image in order to copy a mark.

7. An image processing apparatus for overlaying a virtual image which is an image of at least one virtual object on a real image which is an image of the real world to generate a composite image of the real world and the virtual object,

wherein at least one mark which at least reflects or radiates invisible light of predetermined wavelength other than visible light is added in a space of the real world, the image processing apparatus comprises;

acquiring means for acquiring the real image in which an invisible-light image may be discriminating, and which has been image-captured by a camera apparatus, mark discriminating means for discriminating at least one condition of a position of image of at least one mark in the acquired real image, an orientation of the mark, and a distance from the mark to the image capturing means, and

overlaying condition determining means for determining, in correspondence with the discriminated condition, at least one overlaying condition which is at least one of an overlaying position which is a position of the image of at least one virtual object overlaid on the captured real image in the real image, an orientation of the virtual object which the image of the virtual object indicates, and a distance from a view point of viewer, of the image of the virtual object, and

mark image erasing means for specifying an area which the mark image occupies in the captured real image, and based on the image outside the area, complementing the image of the real world in the area, to erase the mark image in the real image in a way that the user is unable to extract a hidden code from the camera image in order to copy a mark.

8. An image processing program executed by a computer included in an image-processing apparatus for overlaying a virtual image which is an image of at least one virtual object on a real image which is an image of the real world to generate a composite image of the real world and the at least one virtual object,

wherein at least one mark which at least reflects or radiates invisible light of predetermined wavelength other than visible light is added in a space of the real world, and the image processing program causes the computer to function as:

acquiring means for acquiring the real image in which an invisible-light image may be discriminating, and which has been image-captured by a camera apparatus, mark discriminating means for discriminating at least one condition of a position of the image of at least one mark in the acquired real image, an orientation of the mark, and a distance from the mark, to the image capturing means, and

overlaying condition determining means for determining, in correspondence with the discriminated condition, at least one overlaying condition which is at least one of an overlaying position which is a position of the image of the virtual object overlaid on the acquired real image in the real image, an orientation of the virtual object which the image of the virtual object indicates, and a distance from a view point of a viewer, of the image of the virtual object and

mark image erasing means for specifying an area which the mark image occupies in the captured real image, and based on the image outside the area, complementing the image of the real world in the area, to erase the mark image in the real image in a way that the user is unable to extract a hidden code from the camera image in order to copy a mark.

9-14. -14. (canceled)

16. An image processing apparatus for overlaying a virtual image which is an image of at least one virtual object on a real image which is an image of the real world to generate a composite image of the real world and the virtual object,

wherein at least one mark which at least reflects or radiates invisible light of predetermined wavelength other than visible light is added in a space of the real world, the image processing apparatus comprises:

acquiring means for acquiring at least two real images which have been image-captured by a camera apparatus, with a first real image provided from a usual sensor array and a second real image in which an invisible-light image may be discriminating provided from an infrared sensor array of the camera apparatus,

mark discriminating means for discriminating at least one condition of a position of image of at least one mark in the acquired second real image, an orientation of the mark, and a distance from the mark to the image capturing means,

overlaying condition determining means for determining, in correspondence with the discriminated condition, at least one overlaying condition which is at least one of an overlaying position which is a position of the image of at least one virtual object overlaid on the captured real image in the first real image, an orientation of the virtual object which the image of the virtual object indicates, and a distance from a view point of viewer, of the image of the virtual object.

17. An image processing apparatus according to claim 16,

wherein the image processing apparatus further comprises mark image erasing means for specifying an area which the mark image occupies in the acquired second real image, and based on the image outside the area, complementing the image of the real world in the area, to erase the mark image in the first real image.

18. An overlaying condition determining method for overlaying at least one virtual image which is an image of at least one virtual object on a real image which is an image of the real world to generate a composite image of the real world and the virtual object,

wherein at least one mark which at least reflects or radiates invisible light of predetermined wavelength other than visible light is added in a space of the real world, a camera apparatus captures at least two real images with a first real image provided from a usual sensorarray and a second real image in which an invisible-light image may be discriminating provided from a infrared sensor array of the camera apparatus, and

an image processing apparatus discriminates at least one condition of a position of image of the mark in the captured second real image, an orientation of the mark, and a distance from the mark to the image capturing means, and

determines, in correspondence with the at least one discriminated condition, at least one overlaying condition which is at least one of an overlaying position which is a position of the image of the at least one virtual object indicates, and a distance from a view point of a viewer, of the image of the at least one virtual object.