Automatic scaling of objects based on depth map for image editing

• US 9,542,722 B2
• Filed: 12/29/2014
• Issued: 01/10/2017
• Est. Priority Date: 12/29/2014
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Gharavi-Alkhansari, Mohammad, Abdollahian, Golnaz, Berestov, Alexander, Ji, Wei, Takeda, Hiroyuki, Usikov, Daniel
• Primary Examiner(s): Yang, Ryan R
• Application Number:
  US14/584,756
• Publication Number:
  US 20160189342A1
• Time in Patent Office: 743 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 7/50 Depth or shape recovery
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof

Abstract

Automatic scaling of image objects being moved or copied to a target object location is presented. An original image and its depth map are received within which the user selects an original object and a target location for the object to be moved or copied. A center of scaling is found and a base location selected at which depth value is to be obtained at the target location for use in scaling. Target object is then scaled in response to a ratio between original object depth and target object depth. An occlusion check is made, after which non-occluded pixels are pasted to the target location to complete the move/copy which is in proper proportion and accounts for surrounding structures.

Claims

1. An apparatus for automatically scaling image objects being moved or copied from an original object to a target object, comprising:

(a) a computer processor configured for receiving and processing images; and

(b) programming executable on the computer processor for performing steps comprising;

(i) receiving an original object selection in an image and a target location for the original object to be moved/copied as a target object at said target location;

(ii) obtaining or receiving a depth map for the image containing the original object and the target object location; and

(iii) determining interdependent scaling factor and positioning for the target object, in response to;

(iii)(A) determining a center of scaling for the target object;

(iii)(B) selecting a base location on the original object at which depth value is to be obtained at the target location for use in scaling; and

(iii)(C) scaling target object at target location about the center of scaling in response to a ratio between original object depth and target object depth and pasting pixels of said target object into said target location;

(iv) whereby the target object is shown in a proper size in relation to its surroundings.

2. The apparatus recited in claim 1, wherein programming executable on the computer processor further comprises determining any occlusions on target object at target location and moving only non-occluded pixels of target object to target location.

3. The apparatus recited in claim 1, wherein programming executable on the computer processor further comprises receiving an original object selection and a target location in response to a user performing selection of original object and target location through a touch screen interface.

4. The apparatus recited in claim 1, wherein said programming executable on the computer processor further comprises generating an object mask specifying each pixel belonging to the original object.

5. The apparatus recited in claim 1, wherein said depth comprises distance between a lens plane of an image capture device utilized to capture said original image and points in a scene being captured as pixels.

6. The apparatus recited in claim 1, wherein said depth map contains a depth value for image pixels.

7. The apparatus recited in claim 1, wherein said programming executable on the computer processor further comprises tying said target object location to its background at a depth associated with that location thus assuming said target object rests upon another object at said target object location.

8. The apparatus recited in claim 1, wherein said center of scaling comprises a point in said target object about which expansion or shrinking is performed when scaling said original object to said target object.

9. The apparatus recited in claim 1, wherein programming executable on the computer processor further comprises selecting a base location on the original object in response to determining a bottom of the original object.

10. The apparatus recited in claim 1, wherein programming executable on the computer processor is configured for scaling target object at target location, by determining two different depth values for any point P, as depth obtained from the depth map as D₁(x), and as a computed depth assuming that object is scaled by a factor s(P)=x/b, wherein depth is Dc(x)=DB/s(P)=DB b/x, and solving for value x as scaling factor, wherein value b is distance between object origin and its bottom.

11. The apparatus recited in claim 1, wherein said apparatus comprises a personal computer device, laptop, tablet computer, palmtops configured for receiving images, as well as in cameras and mobile phones configured for image capturing.

12. An apparatus for automatically scaling image objects being moved or copied from an original object to a target object, comprising:

(a) a computer processor configured for receiving and processing images; and

(b) programming executable on the computer processor for performing steps comprising;

(i) receiving an original object selection in an image and a target location for the original object to be moved/copied as a target object at said target location;

(ii) obtaining or receiving a depth map for the image containing the original object and the target object location; and

(iii) determining interdependent scaling factor and positioning for location of the target object, in response to;

(iii)(A) determining a center of scaling for the target object;

(iii)(B) selecting a base location on the original object at which depth value is to be obtained at the target location for use in scaling; and

(iii)(C) scaling target object at target location about center of scaling in response to a ratio between original object depth and target object depth and pasting pixels of said target object into said target location;

(iv) wherein determination of interdependent scaling factor and positioning for the target object includes determining two different depth values for any point P, as depth obtained from the depth map as D1(x), and as a computed depth assuming that object is scaled by a factor s(P)=x/b, wherein depth is Dc(x)=DB/s(P)=DB b/x, and determining value x as scaling factor, wherein value b is distance between object origin and its bottom; and

(v) whereby the target object is shown in a proper size in relation to its surroundings.

13. The apparatus recited in claim 12, wherein programming executable on the computer processor further comprises determining any occlusions on target object at target location and moving only non-occluded pixels of target object to target location.

14. The apparatus recited in claim 12, wherein programming executable on the computer processor further comprises receiving an original object selection and a target location in response to a user performing selection of original object and target location through a touch screen interface.

15. The apparatus recited in claim 12, wherein said programming executable on the computer processor further comprises generating an object mask specifying each pixel belonging to the original object.

16. The apparatus recited in claim 12, wherein said depth comprises distance between a lens plane of an image capture device utilized to capture said original image and points in the scene being captured as pixels.

17. The apparatus recited in claim 12, wherein said depth map contains a depth value for image pixels.

18. The apparatus recited in claim 12, wherein said programming executable on the computer processor further comprises tying said target object location to its background at depth associated with that location thus assuming said target object rests upon another object at said target object location.

19. The apparatus recited in claim 12, wherein programming executable on the computer processor further comprises selecting a base location on the original object in response to determining the bottom of the original object.

20. A method of automatically scaling image objects being moved or copied from an original object to a target object, comprising:

(a) receiving an original object selection in an image and a target location for the original object to be moved/copied as a target object at said target location within an image processing device having a computer processor and programming executable on the computer processor;

(b) obtaining or receiving a depth map for the image containing the original object and the target object location; and

(c) determining interdependent scaling factor and positioning for the target object, in response to;

(c)(i) determining a center of scaling for the target object;

(c)(ii) selecting a base location on the original object at which depth value is to be obtained at the target location for use in scaling; and

(c)(iii) scaling target object at target location about the center of scaling in response to a ratio between original object depth and target object depth and pasting pixels of said target object into said target location;

(d) whereby the target object is shown in a proper size in relation to its surroundings.