Portable electronic devices with integrated image/video compositing

• US 9,942,375 B2
• Filed: 05/23/2016
• Issued: 04/10/2018
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Posa, John G.
• Primary Examiner(s): HO, TUAN V
• Application Number:
  US15/162,071
• Publication Number:
  US 20160269526A1
• Time in Patent Office: 687 Days
• CPC Class Codes: H04M 1/0264 for a camera module assembly
• CPC Class Codes: H04M 1/72403 with means for local support of applications that increase the functionality
• CPC Class Codes: H04M 1/725 Cordless telephones user interfaces specially adapted therefor H04M1/724
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector digitizers for touch screens G06F3/041
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04N 1/00307 with a mobile telephone apparatus constructional features of portable telephone sets H04M1/0202; mobile radio systems H04W84/00
• CPC Class Codes: H04N 1/2112 using still video cameras
• CPC Class Codes: H04N 1/2125 Display of information relating to the still picture recording
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability
• CPC Class Codes: H04N 5/272 Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

Digital imagery is composed or composited within a smartphone or other portable electronic device to merge foreground and background subject matter acquired simultaneously with different cameras or acquired at different times with the same camera. A processor stores the digital imagery along with information derived or used by an operation to differentiate the foreground subject matter from the background scene, which may be a digital matting operation, and/or may involve finding an edge or an outline of the foreground subject matter. The foreground subject matter and the background scene may be stored as different graphical layers in the memory. A user control such as a touchscreen may enable a user to select and resize or reposition the foreground subject matter on a background scene of a separately acquired image. The digital imagery may constitute a still image or a sequence of images representative of motion video.

Claims

1. An apparatus, comprising:

a display;

a first digital camera;

a second digital camera;

a memory; and

a processor

programmed to perform instructions to cause the apparatus to;

acquire a first video of a user in front of background imagery using the first digital camera;

perform an operation on the acquired first video to extract a second video of the user from the background imagery;

acquire a new background scene;

generate a composite video by inserting the second video of the user into the new background scene; and

display the composite video on the display.

2. The apparatus of claim 1, wherein the processor is further programmed to cause the apparatus to perform a digital matting operation on the acquired first video to extract the second video of the user from the background imagery.

3. The apparatus of claim 2, wherein the processor is further programmed to cause the apparatus to store the acquired first video in the memory along with a trimap used by the digital matting operation to extract the second video of the user from the background imagery.

4. The apparatus of claim 1, wherein the operation performed on the acquired first video to extract a video of the user without the background imagery comprises finding an edge or an outline of the user.

5. The apparatus of claim 1, wherein the acquired first video and the acquired new background scene are stored as different graphical layers in the memory.

6. The apparatus of claim 1, further comprising a user control that enables a user to select, resize, or reposition the second video of the user relative to the acquired new background scene.

7. The apparatus of claim 1, wherein the acquired new background scene comprises a still image or video acquired by the second digital camera.

8. The apparatus of claim 7, wherein the acquired first video and the acquired new background scene are captured simultaneously.

9. The apparatus of claim 1, wherein the acquired new background scene comprises a previously-acquired image or previously-acquired video retrieved from the memory.

10. The apparatus of claim 1, wherein the display comprises a touchscreen display configured to enable a user to perform the following functions in conjunction with the processor:

select the first video of the user; and

superimpose the first video of the user onto the acquired new background scene.

11. The apparatus of claim 10, wherein the processor is further programmed to cause the apparatus to perform one or more of the following operations:

adjust the zoom levels of one or both of the cameras;

adjust the size or position of the video of the user relative to the new background scene; and

adjust the luminance or chrominance levels of the video of the user relative to the acquired new background scene.

12. A method comprising:

acquiring a first video of a user in front of background imagery using a first digital camera of a first electronic device;

performing an operation on the acquired first video to create a second video of the user that excludes the background imagery;

acquiring a third representation of a scene using a second digital camera of the first electronic device;

generating a composite video by inserting the second video of the user into the third representation of the scene acquired by the second digital camera; and

displaying the composite video on a display of the electronic device.

13. The method of claim 12, further comprising:

using an alpha mask to insert the first video of the user into the scene.

14. The method of claim 12, wherein the fields of view of the first and second digital cameras are oriented in substantially opposing directions.

15. The method of claim 12, wherein the first video of the user in front of background imagery and the third representation of the scene are acquired simultaneously.

16. The method of claim 12, wherein the third representation of the scene comprises a video.

17. The method of claim 12, further comprising:

storing the composite video in a memory.

18. The method of claim 12, further comprising:

transmitting the composite video to a remote location using a transceiver.

19. The method of claim 12, further comprising:

storing the second video of the user in a memory; and

retrieving the second video of the user from the memory so that it may be inserted into the third representation of the scene.

20. The method of claim 12, further comprising:

storing the first video of the user in front of the background imagery and the third representation of the scene as separate graphical layers in a memory.

21. The method of claim 12, wherein performing an operation on the acquired first video comprises receiving user input at the first electronic device.

22. The method of claim 12, wherein the display comprises a touch-screen display, and the method further comprises:

displaying the acquired first video of the user in front of background imagery on the touch-screen display; and

reducing, enlarging, or repositioning the user relative to the background imagery using the touch-screen display.

23. The method of claim 12, wherein the second video of the user is automatically extracted from the acquired first video of the user in front of background imagery based, at least in part, upon a distance of the user from the first electronic device.

24. The method of claim 23, wherein the distance of the user from the first electronic device is determined based, at least in part, upon one or more of the following:

a predetermined value, facial recognition, focus, depth of field, an optical transfer function, and spatial frequency.