Metadata-assisted image filters

• US 8,854,491 B2
• Filed: 07/13/2011
• Issued: 10/07/2014
• Est. Priority Date: 06/05/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Ford, Bradley D., Johnson, Garrett M., Bray, Cdric, Cieplinski, Avi, Khoe, May-Li, Victor, B. Michael, Costanzo, Bianca Cheng, Bernstein, Jeffrey Trr
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): TEJANO, DWIGHT ALEX C
• Application Number:
  US13/181,628
• Publication Number:
  US 20120307096A1
• Time in Patent Office: 1,182 Days
• Field of Search: 348/222.1, 348/239, 386278-290, 382276-308
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

This disclosure pertains to devices, methods, systems, and computer readable media for generating and/or interpreting image metadata to determine input parameters for various image processing routines, e.g., filters that distort or enhance an image, in a way that provides an intuitive experience for both the user and the software developer. Such techniques may attach the metadata to image frames and then send the image frames down an image processing pipeline to one or more image processing routines. Image metadata may include face location information, and the image processing routine may include an image filter that processes the image metadata in order to keep the central focus (or foci) of the image filter substantially coincident with one or more of the faces represented in the face location information. The generated and/or interpreted metadata may also be saved to a metadata track for later application to unfiltered image data.

Claims

1. A non-transitory program storage device, readable by a programmable control device comprising instructions stored thereon for causing the programmable control device to perform operations comprising:

obtaining a first image of a first scene, the first image stored in a first memory and including a timestamp to uniquely identify the first image among a plurality of images obtained from an incoming video stream;

characterizing the first image;

generating metadata information for the first image based at least in part on the act of characterizing, the metadata information identifying a plurality of faces in the first image;

attaching the metadata information to the first image based on the timestamp corresponding to the first image;

programmatically selecting one of the identified plurality of faces based at least in part on the generated metadata information;

applying an image filter to the first image to generate a second image, wherein an effect of the filter is centered at the selected face; and

storing the second image in a second memory.

2. The non-transitory program storage device of claim 1, wherein the operation for selecting comprises selecting a largest face from the plurality of faces based, at least in part, on the metadata information.

3. The non-transitory program storage device of claim 1, wherein the operation for selecting comprises selecting a face from the plurality of faces that is most likely to include open eyes based, at least in part, on the metadata information.

4. The non-transitory program storage device of claim 1, wherein operation for of selecting comprises selecting a face from the plurality of faces that has a longest persistence in the scene based, at least in part, on the metadata information.

5. The non-transitory program storage device of claim 1, wherein the operation for selecting comprises selecting a face from the plurality of faces that is closest to a center of the first image based, at least in part on the metadata information.

6. The non-transitory program storage device of claim 1, wherein the operations further comprise storing the metadata information in the second memory.

7. The non-transitory program storage device of claim 1, wherein the operations further comprise storing the metadata information in a metadata track separate from the first image or the second image.

8. The non-transitory program storage device of claim 1, wherein the image filter comprises a distortion filter.

26. The non-transitory program storage device of claim 1, wherein the operation generating the metadata includes an operation for time-tagging the metadata with the timestamp corresponding to the first image.

9. An electronic device, comprising:

memory;

an image sensor;

a display communicatively coupled to the memory; and

a programmable control device communicatively coupled to the memory, display, and image sensor, wherein the memory includes instructions for causing the programmable control device to perform operations comprising;

obtaining a first image of a first scene, the first image stored in a first memory and including a timestamp to uniquely identify the first image among a plurality of images obtained from an incoming video stream;

characterizing the first image;

generating metadata information for the first image based at least in part on the act of characterizing, the metadata information identifying a plurality of faces in the first image;

coupling the metadata information to the first image based on the timestamp corresponding to the first image;

programmatically selecting one of the identified plurality of faces based at least in part on the generated metadata information;

applying an image filter to the first image to generate a second image, wherein an effect of the filter is centered at the selected face; and

storing the second image in a second memory.

10. A non-transitory program storage device, readable by a programmable control device comprising instructions stored thereon for causing the programmable control device to perform operations comprising:

obtaining a first image of a first scene, the first image stored in a first memory and including a timestamp to uniquely identify the first image among a plurality of images obtained from an incoming video stream;

characterizing the first image;

generating metadata information for the first image based at least in part on the act of characterizing, the metadata information identifying a plurality of faces in the first image;

time-tagging the metadata with the timestamp corresponding to the first image;

relating, via the time-tagging, the metadata information to the first image based on the timestamp corresponding to the first image;

programmatically selecting one of the identified plurality of faces based at least in part on the generated metadata information;

applying an image filter to the first image to generate a second image, wherein an effect of the filter is centered at the selected face; and

storing the second image in a second memory.

11. The non-transitory program storage device of claim 10, wherein the image filter comprises one or more of the following image filters:

Kaleidoscope, Light Tunnel, Squeeze, Twirl, and Stretch.

12. The non-transitory program storage device of claim 10, wherein the operations further comprise storing the metadata information in the second memory.

13. The non-transitory program storage device of claim 10, wherein the operations further comprise storing the metadata information in a metadata track separate from the first image or the second image.

14. The non-transitory program storage device of claim 10, wherein the image filter comprises a distortion filter.

15. An image processing method comprising:

receiving image data from an incoming video stream;

generating a timestamp to uniquely identify the image data within the incoming video stream;

receiving metadata describing aspects of an image represented by the image data;

attaching the metadata to the image data based on the timestamp corresponding to the image data;

identifying a face in the image based on the metadata;

determining one or more input parameters to an image filter based on the meta data;

applying the image filter centered at the identified face using the one or more input parameters to generate a filtered image; and

storing the filtered image in a memory.

16. The method of claim 15, further comprising:

repeating the operations recited in claim 15 in real-time on the incoming video stream, the incoming video stream including a plurality of images.

17. The method of claim 15, wherein the image filter comprises a distortion filter.

18. The method of claim 15, further comprising the act of storing the metadata in the memory separate from the image data.

19. An image processing system comprising:

a first server process for receiving and processing an image from an incoming video stream;

a second server process for generating a timestamp to uniquely identify the image;

a third server process for generating metadata related to the image and storing the metadata in a metadata track using the timestamp to correlate the metadata with the image; and

a first memory for storing the processed image,

wherein the metadata comprises information representative of the image, format description information for the image, and a dictionary of attachments,

wherein the first server process is further configured to associate the metadata and the image with a buffer frame and send the buffer frame and the metadata track to a client application, and

wherein the metadata is associated with the buffer frame via the timestamp associated with the image and the buffer frame.

20. The system of claim 19, wherein the client application comprises an image distortion filter.

21. The system of claim 20, wherein the image distortion filter has a central focus.

22. The system of claim 21, wherein a location of the central focus of the image distortion filter is determined at least in part by the metadata related to the image being filtered.

23. The system of claim 22, wherein the dictionary of attachments comprises face location information.

24. The system of claim 23, wherein the location of the central focus of the image distortion filter is further determined by a location of a face specified in the face location information.

25. The system of claim 19, wherein the server process and the client application each are in communication with the first memory.