Method of gathering visual meta data using a reference image

• US 8,330,831 B2
• Filed: 06/16/2008
• Issued: 12/11/2012
• Est. Priority Date: 08/05/2003
• Status: Active Grant

• Current Assignee: FotoNation Limited (Adeia Inc.)
• Original Assignee: DigitalOptics Corporation Europe Limited (Adeia Inc.)
• Inventors: Steinberg, Eran, Prilutsky, Yury, Pososin, Alexei, Bigioi, Petronel, Zamfir, Adrian, Drimbarean, Alexandru, Corcoran, Peter
• Primary Examiner(s): GILES, NICHOLAS G
• Application Number:
  US12/140,125
• Publication Number:
  US 20080317357A1
• Time in Patent Office: 1,639 Days
• Field of Search: 348/77, 348/78, 348/231.2, 348/231.3, 348/169
• US Class Current: 348/231.3
• CPC Class Codes: G06T 1/0007 Image acquisition
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06T 2207/30216 Redeye defect
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: G06T 7/0002 Inspection of images, e.g. flaw detection
• CPC Class Codes: G06V 10/28 Quantising the image, e.g. histogram thresholding for discrimination between background and foreground patterns
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: G06V 40/193 Preprocessing; Feature extraction
• CPC Class Codes: H04N 1/62 Retouching, i.e. modification of isolated colours only or in isolated picture areas only
• CPC Class Codes: H04N 1/624 Red-eye correction control of camera based on recognised facial parts H04N23/611
• CPC Class Codes: H04N 2201/3252 Image capture parameters, e.g. resolution, illumination conditions, orientation of the image capture device H04N2201/3256 and subgroups take precedence
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 5/76 Television signal recording
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8233 the additional signal being a character code signal

Abstract

A digital image processing technique gathers visual meta data using a reference image. A main image and one or more reference images are captured on a hand-held or otherwise portable or spatial or temporal performance-based image capture device. The reference images are analyzed based on predefined criteria in comparison to the main image. Based on said analyzing, supplemental meta data are created and added to the main image at a digital data storage location.

Claims

1. A digital image processing method for gathering visual meta data using a separately-captured reference image, comprising:

capturing separately, on a hand-held or otherwise portable or spatial or temporal performance-based image capture device, an original main image and a plurality of reference images, including a stream of multiple relatively lower resolution reference images having a temporal and spatial overlap or proximity with the original image, or combinations thereof;

detecting and tracking a face within the stream of reference images;

based on the detecting and tracking, adding a last known location data to said main image at a digital data storage location;

applying face recognition to the face;

based on the applying of face recognition, adding a personal identifier to said main image at said digital data storage location;

analyzing one or more of said plurality of reference images based on predefined criteria in comparison to said main image;

based on said analyzing, creating supplemental meta data; and

adding said supplemental meta data to said main image at said digital data storage location.

2. The method of claim 1, further comprising:

modifying said main image using said supplemental meta data; and

rendering the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device.

3. The method of claim 1, comprising capturing at least one reference image that comprises one or more hi-speed versions to freeze motion of a substantially same scene.

4. The method of claim 1, comprising capturing at least one reference image that comprises an infra-red version of substantially the same scene to enhance a heat emitting object in the frame.

5. The method of claim 4, wherein said heat emitting object comprises a human body, or portion thereof.

6. The method of claim 5, wherein said heat emitting object comprises an exposed skin region of said human body.

7. The method of claim 6, wherein said heat emitting object comprises a face.

8. The method of claim 4, wherein said heat emitting object includes an animal.

9. The method of claim 1, wherein the original main image and at least one reference image comprise different focal distances.

10. The method of claim 9, wherein said reference image comprises a fixed focal length.

11. The method of claim 9, wherein said reference image includes two or more focal lengths organized in a grid pattern.

12. The method of claim 1, wherein the reference image comprises a plurality of images organized as a video clip.

13. The method of claim 1, comprising capturing a reference image that comprises a different focal length than said main image, thereby capturing a different angle of view relative to said main image.

14. The method of claim 13, wherein the reference image comprises an enlarged resolution of at least a portion of said main image.

15. The method of claim 13, further comprising dynamically determining said reference image.

16. The method of claim 15, wherein said dynamically determining comprises a function of content analysis of the image scene.

17. The method of claim 16, wherein said content analysis includes face detection, exposure analysis, color segmentation, local motion, eye detection, or edge detection, or combinations thereof.

18. The method of claim 13, wherein said different angle of view is achieved by selecting a portion of the image as captured by the sensor that is used to capture said main image.

19. The method of claim 1, comprising capturing a reference image that comprises a non-translucent light integrator for determining exposure and color balance.

20. The method of claim 1, wherein said predefined criteria includes facial instantiations such as frontal view, eyes wide open, smile, or a different angle of view than the main image, or combinations thereof.

21. A hand-held or otherwise portable or spatial or temporal performance-based image capture device, comprising:

(a) one or more lenses, an aperture and a photodetector configured for separately capturing an original main image and a stream of multiple relatively lower resolution reference images having a temporal and spatial overlap or proximity with the original image, or combinations thereof;

(b) a processor;

(c) one or more processor-readable media having embedded therein code configured to program the processor to create supplemental meta-data based on an analysis using predefined criteria of said stream of multiple relatively lower resolution reference images in comparison to said main image, and to detect and track a face within the stream of reference images, and to apply face recognition to the face, thereby to generate respectively last known location and personal identifier data; and

to add said supplemental meta data and said last known location and personal identifier data to said main image at a digital data storage location.

22. The image capture device of claim 21, wherein the code is further configured to program the processor to modify said main image using said supplemental meta data;

and to render the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device.

23. The image capture device of claim 21, comprising capturing at least one reference image that comprises one or more hi-speed versions to freeze motion of a substantially same scene.

24. The image capture device of claim 21, comprising capturing at least one reference image comprises an infra-red version of substantially the same scene to enhance a heat emitting object in the frame.

25. The image capture device of claim 24, wherein said heat emitting object comprises a human body, or portion thereof.

26. The image capture device of claim 25, wherein said heat emitting object comprises an exposed skin region of said human body.

27. The image capture device of claim 26, wherein said heat emitting object comprises a face.

28. The image capture device of claim 24, wherein said heat emitting object includes an animal.

29. The image capture device of claim 21, comprising capturing at least one reference image comprising a different focal distance than the main image.

30. The image capture device of claim 29, wherein said at least one reference image comprises a fixed focal length.

31. The image capture device of claim 29, wherein said at least one reference image includes two or more focal lengths organized in a grid pattern.

32. The image capture device of claim 21, wherein the stream of multiple relatively lower resolution reference images comprises a plurality of images organized as a video clip.

33. The image capture device of claim 21, comprising capturing at least one reference image that comprises a different focal length than said main image, thereby capturing a different angle of view relative to said main image.

34. The image capture device of claim 33, wherein the at least one reference image comprises an enlarged resolution of at least a portion of said main image.

35. The image capture device of claim 33, wherein the code is further configured to program the processor to dynamically determine said at least one reference image.

36. The image capture device of claim 35, wherein said dynamically determining comprises a function of content analysis of the image scene.

37. The image capture device of claim 36, wherein said content analysis includes face detection, exposure analysis, color segmentation, local motion, eye detection, or edge detection, or combinations thereof.

38. The image capture device of claim 33, wherein said different angle of view is achieved by selecting a portion of the image as captured by the sensor that is used to capture said main image.

39. The image capture device of claim 21, comprising capturing at least one reference image that comprises a non-translucent light integrator for determining exposure and color balance.

40. The image capture device of claim 21, wherein said predefined criteria includes facial instantiations such as frontal view, eyes wide open, smile, or a different angle of view than the main image, or combinations thereof.

41. One or more non-transitory processor-readable storage devices having embedded therein code for programming a processor to perform a digital image processing method for gathering visual meta data using a stream of multiple relatively lower resolution reference images having a temporal and spatial overlap or proximity with a separately-captured main digital image, wherein the code is configured to program the processor to create supplemental meta-data based an analysis using predefined criteria of said stream of reference images in comparison to said main image, and to detect and track a face within the stream of reference images, and to apply face recognition to the face, thereby to generate respectively last known location and personal identifier data;

and to add said supplemental meta data and said last known location and personal identifier data and to said main image at a digital data storage location.

42. The one or more non-transitory processor-readable storage devices of claim 41, wherein the code is configured to program the processor to modify said main image using said supplemental meta data;

and to render the modified image at a digital rendering device, display or printer, or combinations thereof, as output from the image capture device.