Digital image manipulation

• US 9,196,069 B2
• Filed: 09/26/2013
• Issued: 11/24/2015
• Est. Priority Date: 02/15/2010
• Status: Active Grant

• Current Assignee: Nokia Technologies Oy (Nokia Corporation)
• Original Assignee: Mobile Imaging in Sweden AB
• Inventors: Lindskob, Alexander, Pettersson, Gustaf, Holmstedt, Ulf, Windmark, Johan, Niemi, Sami
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US14/037,563
• Publication Number:
  US 20140177975A1
• Time in Patent Office: 789 Days
• Field of Search: 382/100, 382/201, 382/218, 382/254, 382/282, 382/284, 382/290, 345/619, 345/620, 345/652, 345/653, 345/654, 345/663, 345/664, 345/665, 345/689, 348169-172, 348/222.1, 348/352
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

There is disclosed a method for seamlessly replacing areas in a digital image with corresponding data from temporally close digital images depicting substantially the same scene. The method uses localized image registration error minimization over a fixed preliminary boundary. A least cost closed path which constitutes a boundary for the area to be replaced is calculated using dynamic programming. The replacement area is blended such that image data information from one image is seamlessly replaced with image data information from another image.

Claims

1. A method comprising:

receiving at least first image data information representing a first image, second image data information representing a second image; and

third image data information representing a third image;

generating fourth image data information representing a fourth image by;

replacing a first portion of the first image data information representing a first image area with a first portion of the second image data information representing the first image area in the second image;

maintaining at least a second portion of the first image data information representing a second image area;

replacing a third portion of the first image data information representing a third image area with a third portion of the third image data information representing the third image area in the third image.

2. The method according to claim 1, comprising:

processing the first image data information to determine the first image area;

processing the first image data information to determine the second image area; and

processing the third image data information to determine the third image area.

3. The method according to claim 1, comprising:

enabling a user to control replacement of the first portion of the first image data information representing the first image area, replacement of the second portion of the first image data information representing the second image area, and replacement of the third portion of the first image data information representing the third image area.

4. The method according to claim 1, comprising:

enabling user identification and selection, via a man-machine interface, of a source for image data information that replaces the first portion of the first image data information representing the first image area and of a source of image data information that replaces the third portion of the first image data information representing the third image area.

5. The method according to claim 4, wherein the man-machine interface comprises a touch sensitive display.

6. The method according to claim 1, comprising:

enabling user identification and selection, via a man-machine interface, of a source for image data information for replacing the second portion of the first image data information representing the second image area.

7. The method according to claim 1, wherein the first image area, the second image area and the third image area are identified to a user using a geometrical object.

8. The method according to claim 1, wherein replacing the first portion of the first image data information representing the first image area with the first portion of the second image data information representing the first image area in the second image, replaces an area depicting a human being with closed eyes with a corresponding area from another similar image where the eyes of the same human being are open.

9. The method according to claim 1, wherein the first image area, the second image area and the third image area are determined by feature analysis.

10. The method according to claim 9, wherein the parts of the persons are faces.

11. The method according to claim 1, wherein the first image is a group photo of a first person, a second person and a third person wherein the first image area corresponds to a part of the first person, the second image area corresponds to a part of the second person, and the third image area corresponds to a part of the third person.

12. The method according to claim 11, wherein the first image, the second image and the third image have been captured in succession within a time interval with a small temporal difference between captures and represent substantially the same scene.

13. The method according to claim 1, wherein the first image is selected from a plurality of possible first images and wherein each one of said plurality of possible first images is associated with a quality measure, and wherein the source image is selected from the plurality of possible source image as the image having smallest quality measure.

14. The method according to claim 1, wherein the first image, the second image and the third image are captured in any order.

15. The method according to claim 1, wherein areas of digital images are replaced seamlessly.

16. The method according to claim 15, wherein identifying said target area further comprises finding a compensation movement between at least a part of a source image comprising the source area and at least a part of a target image comprising the target area, such that said compensation movement minimizes a difference between the source image and the target image.

17. The method according to claim 15, further comprising determining a detail measure, error function or cost between a candidate source area in a source image and a candidate target area in a target image, said source area and said target area being identified based on the determined detail measure, error function or cost.

18. The method according to claim 1, comprising:

identifying a source area in one of the first image, the second image and the third image, the source area having a set of coordinates;

using the set of coordinates for the source area to, in response to said identifying said source area, identify a target area in the second image and the third image.

19. The method according to claim 1, wherein the first image, the second image and the third image have been encoded, said method further comprising in one or more of the first image, the second image and the third image decoding only image data information representing the first area, the second area and the third area.

20. The method according to claim 1, wherein the first image area corresponds to a first object in different images and wherein the first image area is determined based on properties of the object.

21. A method as claimed in claim 20, wherein the first object in different images is same part of same human being captured at different times.

22. A non-transitory computer-readable storage medium encoded with instructions that, when performed by a processor, cause performance of the method of claim 1.

23. An apparatus comprising:

at least one processor; and

at least one memory including a computer program

the at least one memory and the computer program configured to, with the at least one processor, cause the apparatus at least to perform;

receiving at least first image data information representing a first image, second image data information representing a second image; and

third image data information representing a third image;

generating fourth image data information representing a fourth image by;

replacing a first portion of the first image data information representing a first image area with a first portion of the second image data information representing the first image area in the second image;

maintaining at least a second portion of the first image data information representing a second image area;

replacing a third portion of the first image data information representing a third image area with a third portion of the third image data information representing the third image area in the third image.

24. An apparatus according to claim 23, wherein the at least one memory and the

computer program configured to, with the at least one processor, cause the apparatus at least to perform:

processing the first image data information to determine the first image area;

processing the first image data information to determine the second image area; and

processing the third image data information to determine the third image area.

25. An apparatus according to claim 23 wherein the at least one memory and the

computer program configured to, with the at least one processor, cause the apparatus at least to perform:

enabling a user to control replacement of the first portion of the first image data information representing the first image area, replacement of the second portion of the first image data information representing the second image area, and replacement of the third portion of the first image data information representing the third image area.

26. An apparatus according to claim 23 comprising a man-machine interface, wherein the at least one memory and the computer program configured to, with the at least one processor, cause the apparatus at least to perform:

enabling user identification and selection, via the man-machine interface, of a source for image data information for replacing the first portion of the first image data information representing the first image area, a source for image data information for replacing the second portion of the first image data information representing the second image area and of a source of image data information for replacing the third portion of the first image data information representing the third image area.

27. An apparatus according to claim 23, comprising a man-machine interface, wherein the at least one memory and the computer program configured to, with the at least one processor, cause the apparatus at least to perform:

identifying the first image area, the second image area and the third image area to a user using a geometrical object.

28. An apparatus according to claim 23, wherein the at least one memory and the computer program configured to, with the at least one processor, cause the apparatus at least to perform:

determining, using feature analysis, the first image area, the second image area and the third image area.

29. An apparatus according to claim 28 wherein the at least one memory and the feature analysis identifies different faces in a group photo as the first image area, the second image area and the third image area.

30. An apparatus according to claim 23 wherein the first image, the second image and the third image have been captured in succession within a time interval with a small temporal difference between captures.

31. An apparatus according to claim 30, configured as a mobile communications apparatus comprising a man-machine interface and a transceiver.