Image processing apparatus, image processing method, and computer-readable storage medium for displaying three-dimensional virtual objects to modify display shapes of objects of interest in the real world

• US 10,636,326 B2
• Filed: 11/22/2016
• Issued: 04/28/2020
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Takeuchi, Yuichiro
• Primary Examiner(s): Entezari, Michelle M
• Application Number:
  US15/358,480
• Publication Number:
  US 20170076638A1
• Time in Patent Office: 1,253 Days
• Field of Search: None
• CPC Class Codes: G06T 13/20 3D [Three Dimensional] animation
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2215/12 Shadow map, environment map
• CPC Class Codes: G06T 2215/16 Using real world measurements to influence rendering
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G09B 29/102 using electrical means
• CPC Class Codes: G09B 29/106 using electronic means
• CPC Class Codes: G09B 29/12 Relief maps relief models G09B25/06

Abstract

A method is provided for displaying physical objects. The method comprises capturing an input image of physical objects, and matching a three-dimensional model to the physical objects. The method further comprises producing a modified partial image by at least one of modifying a portion of the matched three-dimensional model, or modifying a partial image extracted from the input image using the matched three-dimensional model. The method also comprises displaying an output image including the modified partial image superimposed over the input image.

Claims

1. An image processing apparatus, comprising:

circuitry configured to;

acquire an input image of a real world;

acquire a measurement result associated with a position and a posture of a portable device;

acquire a three-dimensional model, wherein the three-dimensional model indicates a shape and a location of at least one object of interest of a plurality of objects within the real world by a plurality of feature points;

generate at least one three-dimensional virtual object as a counterpart to the at least one object of interest of the plurality of objects within the real world, wherein a shape and a location of the at least one three-dimensional virtual object is based on the shape and the location of each object of interest in the three-dimensional model; and

cause a display to display the at least one three-dimensional virtual object onto the at least one object of interest so as to modify display of the shape of the at least one object of interest,

wherein the shape of each object of the at least one object of interest is modified in response to a sensed movement of the display with an acceleration greater than a predetermined threshold acceleration while the at least one object of interest is displayed on the display, and

wherein the shape of each object of the at least one object of interest is modified by using a predetermined animation including at least one of shaking, elongating and contracting, or jumping.

2. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to extract the plurality of feature points for each object of interest of the plurality of objects.

3. The image processing apparatus according to claim 2,

wherein the circuitry is further configured to match at least one feature point of the plurality of extracted feature points for each object with one or more of the shape and the location of the at least one generated three-dimensional virtual object.

4. The image processing apparatus according to claim 1,

wherein the predetermined animation extends a size of the at least one three-dimensional virtual object displayed onto the at least one object of interest.

5. The image processing apparatus according to claim 1,

wherein the predetermined animation shakes the at least one three-dimensional virtual object displayed onto the at least one object of interest.

6. The image processing apparatus according to claim 1,

wherein the at least one object of interest is selected from the plurality of objects based on respective ratings of the plurality of objects within the real world.

7. The image processing apparatus according to claim 6,

wherein the respective ratings of the plurality of objects are determined based on a numerical value of each object.

8. The image processing apparatus according to claim 7,

wherein numerical value of each object is used to determine a rank order of the plurality of objects.

9. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to extract at least one partial image from the input image of the real world.

10. The image processing apparatus according to claim 9,

wherein, when it is determined that there is a moving object that overlaps the at least one extracted partial image, the circuitry is further configured to average a plurality of extracted partial images to generate a single partial object in which an effect of the moving object is reduced.

11. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to initiate a storage into a computer-readable medium of the three-dimensional model.

12. The image processing apparatus according to claim 1,

wherein the circuitry is further configured to acquire the measurement result based on a detected movement of the portable device.

15. The image processing apparatus according to claim 1,

wherein the at least one object of interest comprises a plurality of objects of interest, and

wherein the shape of each object of the plurality of objects of interest is modified such that a size of each object corresponds to a respective rating of the object.

16. The image processing apparatus according to claim 1,

wherein the sensed movement of the display comprises an oscillating movement that indicates that the display is shaken by a user.

13. An image processing method, performed via at least one processor, the method comprising:

acquiring an input image of a real world;

acquiring a measurement result associated with a position and a posture of a portable device;

acquiring a three-dimensional model, wherein the three-dimensional model indicates a shape and a location of at least one object of interest of a plurality of objects within the real world by a plurality of feature points;

generating at least one three-dimensional virtual object as a counterpart to the at least one object of interest of the plurality of objects within the real world, wherein a shape and a location of the at least one three-dimensional virtual object is based on the shape and the location of each object of interest in the three-dimensional model; and

causing a display to display the at least one three-dimensional virtual object onto the at least one object of interest so as to modify display of the shape of the at least one object of interest,

wherein the shape of each object of the at least one object of interest is modified in response to a sensed movement of the display with an acceleration greater than a predetermined threshold acceleration while the at least one object of interest is displayed on the display, and

wherein the shape of each object of the at least one object of interest is modified by using a predetermined animation including at least one of shaking, elongating and contracting, or jumping.

14. A non-transitory computer-readable medium having embodied thereon a program, which when executed by a computer, causes the computer to execute a method, the method comprising:

acquiring an input image of a real world;

acquiring a measurement result associated with a position and a posture of a portable device;

acquiring a three-dimensional model, wherein the three-dimensional model indicates a shape and a location of at least one object of interest of a plurality of objects within the real world by a plurality of feature points;

generating at least one three-dimensional virtual object as a counterpart to the at least one object of interest of the plurality of objects within the real world, wherein a shape and a location of the at least one three-dimensional virtual object is based on the shape and the location of each object of interest in the three-dimensional model; and

causing a display to display the at least one three-dimensional virtual object onto the at least one object of interest so as to modify display of the shape of the at least one object of interest,

wherein the shape of each object of the at least one object of interest is modified in response to a sensed movement of the display with an acceleration greater than a predetermined threshold acceleration while the at least one object of interest is displayed on the display, and

wherein the shape of each object of the at least one object of interest is modified by using a predetermined animation including at least one of shaking, elongating and contracting, or jumping.