Portable electronic devices with integrated image/video compositing

• US 9,350,924 B2
• Filed: 08/25/2014
• Issued: 05/24/2016
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Posa, John G.
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US14/467,687
• Publication Number:
  US 20160057363A1
• Time in Patent Office: 638 Days
• US Class Current: 1/1
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/725 Cordless telephones user interfaces specially adapted therefor H04M1/724
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/2125 Display of information relating to the still picture recording
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Digital imagery is composed or composited within a smartphone or other portable electronic device to merge foreground and background subject matter acquired simultaneously with different cameras or acquired at different times with the same camera. A processor stores the digital imagery along with information derived or used by an operation to differentiate the foreground subject matter from the background scene, which may be a digital matting operation, and/or may involve finding an edge or an outline of the foreground subject matter. The foreground subject matter and the background scene may be stored as different graphical layers in the memory. A user control such as a touchscreen may enable a user to select and resize or reposition the foreground subject matter on a background scene of a separately acquired image. The digital imagery may constitute a still image or a sequence of images representative of motion video.

Claims

1. A portable electronic device, comprising:

a user control;

a camera operative to acquire a digital image in response to the user control, the digital image including foreground subject matter and a background scene;

a memory for storing the digital image; and

a processor within the portable electronic device, the processor being operative to perform the following functions;

a) receive the digital image acquired by the camera, and

b) perform an operation on the digital image to differentiate the foreground subject matter from the background scene, thereby enabling the foreground subject matter to be resized or repositioned separately from the background scene.

2. The portable electronic device of claim 1, where the operation is a digital matting operation.

3. The portable electronic device of claim 2, wherein the processor is further operative to store the digital image in the memory along with information derived or used by the digital matting operation to differentiate the foreground subject matter from the background scene.

4. The portable electronic device of claim 2, wherein the information derived or used by the digital matting operation includes a trimap.

5. The portable electronic device of claim 1, wherein the operation includes finding an edge or an outline of the foreground subject matter.

6. The portable electronic device of claim 1, wherein the foreground subject matter and the background scene are stored as different graphical layers in the memory.

7. The portable electronic device of claim 1, wherein the user control enables a user to select and resize or reposition the foreground subject matter on a background scene of a separately acquired image.

8. The portable electronic device of claim 7, wherein the separately acquired image is a previously acquired image retrieved from the memory.

9. The portable electronic device of claim 7, including a second camera;

and

wherein the separately acquired image is acquired at the same time as the digital image using the second camera.

10. The portable electronic device of claim 1, including a display screen;

and

wherein the digital image is displayed on the display screen prior to, or following, the storage thereof.

11. The portable electronic device of claim 1, wherein:

the user control includes a touchscreen display operative to display the digital image; and

a user is capable of performing the following operations in conjunction with the processor;

a) select the differentiated foreground subject matter of the digital image;

b) retrieve a different digital image from the memory; and

c) superimpose the foreground subject matter of the digital image onto the different digital image retrieved from the memory.

12. The portable electronic device of claim 1, further including:

a second camera operative to acquire a second digital image in response to the user control, the second digital image including foreground subject matter and a background scene; and

the processor is operative to automatically insert the foreground subject matter differentiated from the digital image into the background scene of the second digital image.

13. The portable electronic device of claim 12, wherein the digital cameras point in opposite directions.

14. The portable electronic device of claim 12, wherein:

one of the digital cameras is a rear camera that points at the user during operation of the device;

the other digital camera is a front camera that points away from the user during operation of the device; and

wherein the processor is operative to automatically insert an image of the user taken by the rear camera into a background scene taken by the front camera.

15. The portable electronic device of claim 12, further including a display screen to display the composite image for acceptance by a user;

and

wherein a user is capable of adjusting one or more of the following before or after the acceptance of the composite image;

the zoom levels of one or both of the cameras,

the size or position of the foreground subject matter relative to the background scene, and

luminance or chrominance levels of the foreground subject matter relative to the background scene.

16. The portable electronic device of claim 1, wherein the device is a smartphone further including cellular telephone communications circuitry.

17. A method of composing a digital image, comprising the steps of:

providing a portable electronic device including one or more digital cameras, a memory, a display and a processor;

acquiring background imagery with one of the cameras of the portable electronic device;

acquiring foreground imagery with one of the cameras of the portable electronic device;

performing a matting operation on the foreground imagery to extract foreground subject matter from the foreground imagery;

automatically generating composite imagery by inserting the foreground subject matter into the background imagery using the processor within the portable electronic device; and

displaying the composite imagery on the display of the portable electronic device.

18. The method of claim 17, including the step of generating and using an alpha mask to insert the foreground subject matter into the background imagery.

19. The method of claim 17, wherein the portable electronic device includes a front-viewing camera and a rear-viewing camera, and wherein the method includes the steps of:

using one of the cameras to acquire the background imagery; and

using the other camera to acquire the foreground imagery.

20. The method of claim 17, including the step of using the front-viewing and rear-viewing cameras to acquire the background and foreground imagery simultaneously.

21. The method of claim 17, including the step of using the same camera on the portable electronic device to capture the background imagery and the foreground imagery at different times.

22. The method of claim 17, including the steps of:

storing the background imagery and the foreground imagery in the memory; and

retrieving the stored background imagery and the foreground imagery to generate the composite imagery.

23. The method of claim 17, including the step of storing the composite imagery in the memory.

24. The method of claim 17, including the step of storing the background imagery and the foreground imagery as separate graphical layers in the memory.

25. The method of claim 17, including the steps of:

selecting particular subject matter in the foreground imagery; and

generating the composite imagery by superimposing the selected subject matter onto the background imagery.

26. The method of claim 25, wherein the display is a touch-screen display, and the method includes the steps of:

displaying the selected subject matter superimposed over the background imagery; and

reducing, enlarging or repositioning the selected subject matter relative to the background imagery using the touch-screen display.

27. The method of claim 25, wherein the particular subject matter in the foreground imagery is automatically selected based upon the distance of the particular subject matter from the portable electronic device.

28. The method of claim 27, wherein the distance of the particular subject matter from the portable electronic device is based upon one or more of the following:

a predetermined value,

facial recognition,

focus,

depth of field,

an optical transfer function, and

spatial frequency.

29. The method of claim 17, wherein the composite imagery comprises a still photograph or motion video.