Portable Electronic Devices with Integrated Image/Video Compositing

• US 20180278737A1
• Filed: 04/06/2018
• Published: 09/27/2018
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Posa, John G.
• Granted Patent: US 10,477,005 B2
• Time in Patent Office: Days
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/725 Cordless telephones user interfaces specially adapted therefor H04M1/724
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/2125 Display of information relating to the still picture recording
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Digital imagery is composed or composited within a smartphone or other portable electronic device to merge foreground and background subject matter acquired simultaneously with different cameras or acquired at different times with the same camera. A processor stores the digital imagery along with information derived or used by an operation to differentiate the foreground subject matter from the background scene, which may be a digital matting operation, and/or may involve finding an edge or an outline of the foreground subject matter. The foreground subject matter and the background scene may be stored as different graphical layers in the memory. A user control such as a touchscreen may enable a user to select and resize or reposition the foreground subject matter on a background scene of a separately acquired image. The digital imagery may constitute a still image or a sequence of images representative of motion video.

Claims

1. An apparatus, comprising:

a display;

a first digital camera;

a second digital camera;

a memory; and

a processor programmed to perform instructions to cause the apparatus to;

acquire, using the first digital camera, a first image comprising;

foreground subject matter; and

a background scene;

perform a segmentation operation on the first image to segment the first image into at least;

a foreground region comprising at least a portion of the foreground subject matter; and

a background region comprising at least a portion of the background scene;

extract a second image from the first image, wherein the second image comprises the foreground region of the first image;

acquire a third image from the second digital camera;

generate a composite image by compositing the second image onto the third image; and

display the composite image on the display.

2. The apparatus of claim 1, wherein the instructions to cause the apparatus to perform the segmentation operation further comprise instructions to cause the apparatus to perform at least one of the following operations:

a focus-based algorithm;

a facial recognition algorithm; and

a digital matting algorithm.

3. The apparatus of claim 1, wherein the processor is further programmed to perform instructions to cause the apparatus to:

generate a temporary outline around the foreground region.

4. The apparatus of claim 1, wherein the instructions to cause the apparatus to perform the segmentation operation further comprise instructions to cause the apparatus to:

find an edge or an outline of a user in the foreground subject matter.

5. The apparatus of claim 1, wherein the processor is further programmed to perform instructions to cause the apparatus to:

resize or reposition the foreground region relative to the background region.

6. The apparatus of claim 1, wherein the first image and the third image are acquired at the same time.

7. The apparatus of claim 1, wherein the processor is further programmed to perform instructions to cause the apparatus to perform one or more of the following operations:

adjust the zoom levels of one or both of the first and second digital cameras; and

adjust the luminance or chrominance levels of the foreground region relative to the background region.

8. A method, comprising:

acquiring, using a first digital camera, a first image comprising;

foreground subject matter; and

a background scene;

performing a segmentation operation on the first image to segment the first image into at least;

a foreground region comprising at least a portion of the foreground subject matter; and

a background region comprising at least a portion of the background scene;

extracting a second image from the first image, wherein the second image comprises the foreground region of the first image;

acquiring a third image from a second digital camera;

generating a composite image by compositing the second image onto the third image; and

displaying the composite image on a display.

9. The method of claim 8, wherein performing the segmentation operation further comprises performing at least one of the following operations:

a focus-based algorithm;

a facial recognition algorithm; and

a digital matting algorithm.

10. The method of claim 8, further comprising:

generating a temporary outline around the foreground region.

11. The method of claim 8, wherein performing the segmentation operation further comprises:

finding an edge or an outline of a user in the foreground subject matter.

12. The method of claim 8, further comprising:

resizing or repositioning the foreground region relative to the background region.

13. The method of claim 8, wherein the first image and the third image are acquired at the same time.

14. The method of claim 8, wherein the first digital camera, the second digital camera, and the display are part of an electronic device.

15. A non-transitory computer-readable medium storing instructions executable by a processor, the instructions comprising instructions that, when executed, cause the processor to:

acquire, using a first digital camera, a first image comprising;

foreground subject matter; and

a background scene;

perform a segmentation operation on the first image to segment the first image into at least;

a foreground region comprising at least a portion of the foreground subject matter; and

a background region comprising at least a portion of the background scene;

extract a second image from the first image, wherein the second image comprises the foreground region of the first image;

acquire a third image from a second digital camera;

generate a composite image by compositing the second image onto the third image; and

display the composite image on a display.

16. The non-transitory computer-readable medium of claim 15, wherein the instructions that, when executed, cause the processor to perform the segmentation operation further comprise instructions that, when executed, cause the processor to perform at least one of the following operations:

a focus-based algorithm;

a facial recognition algorithm; and

a digital matting algorithm.

17. The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed, cause the processor to:

generate a temporary outline around the foreground region.

18. The non-transitory computer-readable medium of claim 15, wherein the instructions that, when executed, cause the processor to perform the segmentation operation further comprise instructions that, when executed, cause the processor to:

find an edge or an outline of a user in the foreground subject matter.

19. The non-transitory computer-readable medium of claim 15, further comprising instructions that, when executed, cause the processor to:

resize or reposition the foreground region relative to the background region.

20. The non-transitory computer-readable medium of claim 15, wherein the first image and the third image are acquired at the same time.