Metadata-Assisted Image Filters

• US 20120307096A1
• Filed: 07/13/2011
• Published: 12/06/2012
• Est. Priority Date: 06/05/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Ford, Bradley D., Johnson, Garrett M., Bray, Cedric, Cieplinski, Avi, Khoe, May-Li, Victor, B. Michael, Bernstein, Jeffrey Traer, Costanzo, Bianca Cheng
• Granted Patent: US 8,854,491 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

This disclosure pertains to devices, methods, systems, and computer readable media for generating and/or interpreting image metadata to determine input parameters for various image processing routines, e.g., filters that distort or enhance an image, in a way that provides an intuitive experience for both the user and the software developer. Such techniques may attach the metadata to image frames and then send the image frames down an image processing pipeline to one or more image processing routines. Image metadata may include face location information, and the image processing routine may include an image filter that processes the image metadata in order to keep the central focus (or foci) of the image filter substantially coincident with one or more of the faces represented in the face location information. The generated and/or interpreted metadata may also be saved to a metadata track for later application to unfiltered image data.

Claims

1. A non-transitory program storage device, readable by a programmable control device comprising instructions stored thereon for causing the programmable control device to perform a method comprising the acts of:

obtaining a first image of a first scene, the first image stored in a first memory;

characterizing the first age;

generating metadata information for the first image based at least in part on the act of characterizing, the metadata information identifying a plurality of faces in the first image;

programmatically selecting one of the identified plurality of faces based at least in part on the generated metadata information;

applying an image filter to the first image to generate a second image, wherein an effect of the filter is centered at the selected face; and

storing the second image in a second memory.

2. The non-transitory program storage device of claim 1, wherein the act of selecting comprises selecting a largest face from the plurality of faces based, at least in part, on the metadata information.

3. The non-transitory program storage device of claim 1, wherein the act of selecting comprises selecting a face from the plurality of faces that is most likely to include open eyes based, at least in part, on the metadata information.

4. The non-transitory program storage device of claim 1, wherein the act of selecting comprises selecting a face from the plurality of faces that has a longest persistence in the scene based, at least in part, on the metadata information.

5. The non-transitory program storage device of claim 1, wherein the act of selecting comprises selecting a face from the plurality of faces that is closest to a center of the first image based, at least in part on the metadata information.

6. The non-transitory program storage device of claim 1, wherein the method further comprises the act of storing the metadata information in the second memory.

7. The non-transitory program storage device of claim 1, wherein the act of generating further comprises associating the metadata information to the first image using a timestamp.

8. The non-transitory program storage device of claim 1, wherein the image filter comprises a distortion filter.

9. An electronic device, comprising:

memory;

an image sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform a method comprising the acts of;

obtaining a first image of a first scene, the first it age stored in a first memory;

characterizing the first image;

generating metadata information for the first image based at least in part on the act of characterizing, the metadata information identifying a plurality of faces in the first image;

programmatically selecting one of the identified plurality of faces based at least in part on the generated metadata information;

applying an image filter to the first image to generate a second image, wherein an effect of the filter is centered at the selected face; and

storing the second image in a second memory.

10. A non-transitory program storage device, readable by a programmable control device comprising instructions stored thereon for causing the programmable control device to perform a method comprising the acts of:

obtaining a first image, the first image stored in a first memory;

characterizing the first image;

generating metadata information for the first image based at least in part on the act of characterizing, the metadata information identifying a plurality of faces in the first image;

programmatically selecting a plurality of the identified faces based at least in part on the generated metadata information;

applying an image filter to the first image to generate a second image, wherein an effect of the filter is centered at each of the plurality of selected faces; and

storing the second image in a second memory.

11. The non-transitory program storage device of claim 10, wherein the image filter comprises one or more of the following image filters:

Kaleidoscope, Light Tunnel, Squeeze, Twirl, and Stretch.

12. The non-transitory program storage device of claim 10, wherein the method further comprises the act of storing the metadata information in the second memory.

13. The non-transitory program storage device of claim 10, wherein the act of generating further comprises associating the metadata information to the first image using a timestamp.

14. The non-transitory program storage device of claim 10, wherein the image filter comprises a distortion filter.

15. An image processing method comprising:

(a) receiving image data;

(b) receiving metadata describing aspects of the image;

(c) identifying a face in the image based on the metadata;

(d) determining one or more input parameters to an image filter based on the metadata;

(e) applying the image filter centered at the identified face using the one or more input parameters to generate a filtered image; and

(f) storing the filtered image in a memory.

16. The method of claim 15, further comprising:

repeating steps (a)-(f) in real-time on a video stream of received images.

17. The method of claim 15, wherein the image filter comprises a distortion filter.

18. The method of claim 15, further comprising the act of storing the metadata in the memory.

19. An image processing system comprising:

a first server process for receiving and processing an image; and

a first memory for storing the processed image,

wherein the first server process is configured to generate metadata related to the image,

wherein the metadata comprises information representative of the image, format description information for the image, and a dictionary of attachments, and

wherein the first server process is further configured to associate the metadata and the image with a buffer frame and send the buffer frame to a client application.

20. The system of claim 19, wherein the client application comprises an image distortion filter.

21. The system of claim 20, wherein the image distortion filter has a central focus.

25. The system of claim 19, wherein the server process and the client application each are in communication with the first memory.

22. The system of dam 19, wherein a location of the central focus of the image distortion filter is determined at least in part by the metadata related to the image being filtered.

23. The system of dam 22, wherein the dictionary of attachments comprises face location information.

24. The system of claim 23, wherein the location of the central focus of the image distortion filter is further determined by a location of a face specified in the face location information.