IMAGE PROCESSING APPARATUS, PROJECTION CONTROL METHOD, AND PROGRAM

• US 20150029223A1
• Filed: 03/25/2013
• Published: 01/29/2015
• Est. Priority Date: 05/08/2012
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Kaino, Akihiko, Ooi, Kenichirou
• Granted Patent: US 10,366,537 B2
• Time in Patent Office: Days
• US Class Current: 345/633
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G09G 2360/04 Display device controller operating with a plurality of display units
• CPC Class Codes: G09G 3/002 to project the image of a two-dimensional display, such as an array of light emitting or modulating elements or a CRT

Abstract

An information processing apparatus and method acquires an image, performs an image recognition process on the acquired image to recognize a physical object in that image, and then generates a virtual image based on the results of the image recognition process. The virtual image includes a virtual object positioned relative to the physical object that was recognized in the acquired image. A display then displays the virtual image, and a projector projects at least part of the virtual image. The apparatus and method also include modes in which the display displays the virtual image, but the projector does not project the virtual image, where the projector projects the virtual image but the display does not display the virtual image, and where the display displays the virtual image and the projector projects the virtual image.

Claims

1. An information processing apparatus, comprising:

an image pickup unit configured to acquire an image;

a processor configured to

perform an image recognition process on the image acquired by the image pickup unit to recognize a physical object in the image, and generate a virtual image including a virtual object positioned relative to the physical object recognized in the image;

a display unit configured to display the virtual image; and

a projection unit configured to project at least a portion of the virtual image.

2. The information processing apparatus according to claim 1, wherein in a first mode, the processor controls the display unit to display the virtual image;

and

in a second mode, the processor controls the projection unit to project at least the portion of the virtual image.

3. The information processing apparatus according to claim 2, wherein in the first mode, the processor controls the projection unit to inhibit projection of the virtual image, and

in the second mode, the processor controls the display to inhibit display of the virtual image.

4. The information processing apparatus according to claim 3, wherein in a third mode, the processor controls the display unit to display the virtual image and controls the projection unit to project at least the portion of the virtual image.

5. The information processing apparatus according to claim 1, wherein a version of the virtual object projected by the projection unit is projected with lower detail than a version of the virtual object displayed by the display unit as part of the virtual image.

6. The information processing apparatus according to claim 1, wherein a resolution of the virtual object projected by the projection unit is adjusted based on a distance between the projection unit and a target of a projected virtual image.

7. The information processing apparatus according to claim 1, wherein an attribute of the virtual object projected by the projection unit is modified based on an accuracy of the image recognition process.

8. The information processing apparatus according to claim 7, wherein the attribute of the virtual object projected by the projection unit is modified to increase the accuracy of the image recognition process.

9. The information processing apparatus according to claim 8, wherein the attribute is a position of projection of the virtual object.

10. The information processing apparatus according to claim 9, wherein the position of projection of the virtual object is changed to increase the accuracy of the image recognition process.

11. The information processing apparatus according to claim 9, wherein the position of projection of the virtual object is changed based on a user input.

12. The information processing apparatus according to claim 11, wherein the display includes a touch screen, and the user input is a gesture on the touch screen.

13. The information processing apparatus according to claim 12, wherein the gesture includes a flick or a drag along the touch screen.

14. The information processing apparatus according to claim 1, further comprising:

an input unit to receive a user input,

wherein the virtual object projected by the projection unit is changed based on a privacy level received at the input unit.

15. The information processing apparatus according to claim 1, wherein the processor adjusts a projection size and a projection resolution of the virtual object projected by the projection unit based on a distance from the projection unit to a projection target.

16. The information processing apparatus according to claim 1, wherein the physical object is recognized in three-dimensional space and the virtual object is generated at a three-dimensional position relative to the physical object in the three-dimensional space.

17. The information processing apparatus according to claim 16, wherein the physical object is a three-dimensional physical object and the virtual object is a three-dimensional virtual object.

18. An information processing system, comprising:

a plurality of information processing devices, each including

an image pickup unit configured to acquire an image;

a processor configured to

perform an image recognition process on the image acquired by the image pickup unit to recognize a physical object in the image, and generate a virtual image including a virtual object positioned relative to the physical object recognized in the image,

a display unit configured to display the virtual image under control of the processor, and

a projection unit configured to project at least a portion of the virtual image under control of the processor,

wherein one of the plurality of information processing devices determines an interest level in the physical object based on a light intensity of virtual images projected by other information processing devices.

19. An information processing method, comprising:

acquiring an image in an image pickup unit;

performing, in a processor, an image recognition process on the image acquired to recognize a physical object in the image;

generating, in the processor, a virtual image including a virtual object positioned relative to the physical object recognized in the image;

displaying the virtual image on a display unit; and

projecting at least a portion of the virtual image by a projection unit.

20. A non-transitory computer-readable medium storing computer readable instructions thereon, the computer-readable instructions when executed by a computer cause the computer to perform a method comprising:

acquiring an image;

performing an image recognition process on the image acquired to recognize a physical object in the image;

generating a virtual image including a virtual object positioned relative to the physical object recognized in the image;

displaying the virtual image; and

projecting at least a portion of the virtual image.