IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND COMPUTER-READABLE STORAGE MEDIUM

• US 20170076638A1
• Filed: 11/22/2016
• Published: 03/16/2017
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: TAKEUCHI, Yuichiro
• Granted Patent: US 10,636,326 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G09B 29/102 using electrical means
• CPC Class Codes: G09B 29/106 using electronic means
• CPC Class Codes: G09B 29/12 Relief maps relief models G09B25/06

Abstract

A method is provided for displaying physical objects. The method comprises capturing an input image of physical objects, and matching a three-dimensional model to the physical objects. The method further comprises producing a modified partial image by at least one of modifying a portion of the matched three-dimensional model, or modifying a partial image extracted from the input image using the matched three-dimensional model. The method also comprises displaying an output image including the modified partial image superimposed over the input image.

Claims

1. An image processing apparatus, comprising:

circuitry configured to;

acquire an input image of a real world;

acquire a measurement result associated with a position and a posture of a portable device;

acquire a three-dimensional model, wherein the three-dimensional model indicates a shape and a location of at least one object of interest of a plurality of objects within the real world by a plurality of feature points;

generate at least one three-dimensional virtual object as a counterpart the at least one object of interest of the plurality of objects within the real world, wherein a shape and a location of the at least one three-dimensional virtual object is based on the shape and the location of each object of interest in the three-dimensional model; and

cause a display to display the at least one three-dimensional virtual object onto the at least one object of interest so as to emphasize the at least one object of interest.

2. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to extract the plurality of feature points for each object of interest of the plurality of objects.

3. The image processing apparatus according to claim 2,

wherein the circuitry is further configured to match at least one feature point of the plurality of extracted feature points for each object with one or more of the shape and the location of the at least one generated three-dimensional. virtual object.

4. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to emphasize the at least one object of interest using a predetermined animation.

5. The image processing apparatus according to claim 4,

wherein the predetermined animation extends a size of the at least one three-dimensional virtual object displayed onto the at least one object of interest,

6. The image processing apparatus according to claim 4,

wherein the predetermined animation shakes the at least one three-dimensional virtual object displayed onto the at least one object of interest.

7. The image processing apparatus according to claim 1,

wherein the at least one object of interest is selected from the plurality of objects based on respective ratings of the plurality of objects within the real world.

8. The image processing apparatus according to claim 7,

wherein the respective ratings of the plurality of objects are determined based on a numerical value of each object.

9. The image processing apparatus according to claim 8,

wherein numerical value of each object is used to determine a rank order of the plurality of objects.

10. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to extract at least one partial image from the input image of the real world.

11. The image processing apparatus according to claim 10,

wherein, when it is determined that there is a moving object that overlaps the at least one extracted partial image, the circuitry is further configured to average a plurality of extracted partial images to generate a single partial object in which an effect of the moving object is reduced.

12. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to initiate a storage into a computer-readable medium of the three-dimensional model.

13. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to acquire the measurement result based on a detected movement of the portable device.

14. An image processing method, performed via at least one processor, the method comprising:

acquiring an input image of a real world;

acquiring a measurement result associated with a position and a posture of a portable device;

acquiring a three-dimensional model, Wherein the three-dimensional model indicates a shape and a location of at least one object of interest of a plurality of objects within the real world by a plurality of feature points;

generating at least one three-dimensional virtual object as a counterpart to the at least one object of interest of the plurality of objects within the real world, wherein a shape and a location of the at least one three-dimensional virtual object is based on the shape and the location of each object of interest in the three-dimensional model; and

causing a display to display the at least one three-dimensional virtual object onto the at least one object of interest so as to emphasize the at least one object of interest.

15. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes the computer to execute a method, the method comprising:

acquiring an input image of a real world;

acquiring a measurement result associated with a position and a posture of a portable device;

acquiring a three-dimensional model, wherein the three-dimensional model indicates a shape and a location of at least one object of interest of a plurality of objects within the real world by a plurality of feature points;

generating at least one three-dimensional virtual object as a counterpart to the at least one object of interest of the plurality of objects within the real world, wherein a shape and a location of the at least one three-dimensional virtual object is based on the shape and the location of each object of interest in the three-dimensional model; and

causing a display to display the at least one three-dimensional virtual object onto the at least one object of interest so as to emphasize the at least one object of interest.