METHOD AND APPARATUS FOR BACKGROUND SUBTRACTION USING FOCUS DIFFERENCES

• US 20140118570A1
• Filed: 03/05/2013
• Published: 05/01/2014
• Est. Priority Date: 10/31/2012
• Status: Active Grant

• Current Assignee: West Texas Technology Partners LLC
• Original Assignee: Atheer, Inc.
• Inventors: Chehade, Mohamed Nabil Hajj, Fateh, Sina, Itani, Sleiman, Yang, Allen Yang
• Granted Patent: US 9,894,269 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06T 7/194 involving foreground-background segmentation
• CPC Class Codes: G06T 7/557 from light fields, e.g. from plenoptic cameras
• CPC Class Codes: G06T 7/571 from focus
• CPC Class Codes: H04N 13/239 using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/959 by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics

Abstract

First and second images are captured at first and second focal lengths, the second focal length being longer than the first focal length. Element sets are defined with a first element of the first image and a corresponding second element of the second image. Element sets are identified as background if the second element thereof is more in-focus than or as in-focus as the first element. Background elements are subtracted from further analysis. Comparisons are based on relative focus, e.g. whether image elements are more or less in-focus. Measurement of absolute focus is not necessary, nor is measurement of absolute focus change; images need not be in-focus. More than two images, multiple element sets, and/or multiple categories and relative focus relationships also may be used.

Claims

1. A method, comprising steps of:

receiving a first image, said first image having a first focal length and a first field of view;

receiving a second image, said second image having a second focal length longer than said first focal length, said second image having a second field of view substantially identical to said first field of view;

defining in said first and second images a plurality of element sets, each said element set comprising a first element of said first image and a substantially corresponding second element of said second image;

comparing a relative focus of said first and second elements in said element sets;

assigning to a background category those element sets wherein said second element therein is more in-focus than or substantially equally in-focus as said first element therein; and

subtracting said element sets assigned to said background category from further analysis.

2. A method, comprising steps of:

receiving a plurality of images substantially sharing a field of view, each of said images having a unique focal length;

determining a sequence of said images of increasing focal length, such that an n+1th image has a longer focal length than an nth image;

defining in said plurality of images at least one element set, each said element set comprising substantially corresponding elements in said plurality of images;

comparing a relative focus of said element set; and

assigning to one of at least one category those element sets wherein said relative focus of said sequence of images satisfies a corresponding one of at least one relationship.

3. The method of claim 2, further comprising:

defining in said plurality of images a plurality of said element sets.

4. The method of claim 2, wherein:

comparing said relative focus of said element set comprises comparing a focus of consecutive elements in said element set.

5. The method of claim 2, wherein:

said plurality of images comprises two images.

6. The method of claim 2, wherein:

said plurality of images comprises at least three images.

7. The method of claim 2, wherein:

said at least one category comprises a background category representing a background of said field of view.

8. The method of claim 2, wherein:

said at least one relationship comprises a background relationship such that all n+1th elements in said element set are more in-focus than or substantially equally in-focus as corresponding nth elements in said element set.

9. The method of claim 2, wherein:

said first at least one relationship comprises a background relationship such that all n+1th elements in said element set are more in-focus than corresponding nth elements in said element set.

10. The method of claim 2, wherein:

said at least one category comprises a foreground category representing a foreground of said field of view.

11. The method of claim 2, wherein:

said at least one relationship comprises a foreground relationship such that all n+1th elements in said element set are less in-focus than or substantially equally in-focus as corresponding nth elements in said element set.

12. The method of claim 2, wherein:

said at least one relationship comprises a foreground relationship such that all n+1th elements in said element set are less in-focus than corresponding nth elements in said element set.

13. The method of claim 6, wherein:

said at least one category comprises a midground category representing a midground of said field of view.

14. The method of claim 6, wherein:

said at least one relationship comprises a midground relationship such that an n+1th element in said element set is more in-focus than or substantially equally in-focus as a corresponding nth element in said element set, and an n+2th element in said element set is less in-focus than or substantially equally in-focus as said n+1th element in said element set.

15. The method of claim 6, wherein:

said at least one relationship comprises a midground relationship such that an n+1th element in said element set is more in-focus than a corresponding nth element in said element set, and an n+2th element in said element set is less in-focus than said n+1th element in said element set.

16. The method of claim 2, wherein:

said plurality of images comprises at least four images;

said at least one category comprises a first midground category representing a first midground of said field of view;

said at least one category comprises a second midground category representing a second midground of said field of view;

said at least one relationship comprises a first midground relationship corresponding with said first midground category, said first midground relationship being such that an n+1th element in said element set is more in-focus than or substantially equally in-focus as a corresponding nth element in said element set, and an n+2th element in said element set is less in-focus than or substantially equally in-focus as said n+1th element in said element set; and

said at least one relationship comprises a second midground relationship corresponding with said second midground category, said second midground relationship being such that an m+1th element in said element set is more in-focus than or substantially equally in-focus as a corresponding mth element in said element set, and an m+2th element in said element set is less in-focus than or substantially equally in-focus as said m+1th element in said element set;

wherein n is not equal to m.

17. The method of claim 2, wherein:

said plurality of images comprises at least four images;

said at least one category comprises a first midground category representing a first midground of said field of view;

said at least one category comprises a second midground category representing a second midground of said field of view;

said at least one relationship comprises a first midground relationship corresponding with said first midground category, said first midground relationship being such that an n+1th element in said element set is more in-focus than a corresponding nth element in said element set, and an n+2th element in said element set is less in-focus than said n+1th element in said element set; and

said at least one relationship comprises a second midground relationship corresponding with said second midground category, said second midground relationship being such that an m+1th element in said element set is more in-focus than a corresponding mth element in said element set, and an m+2th element in said element set is less in-focus than said m+1th element in said element set;

wherein n is not equal to m.

18. The method of claim 2, wherein:

a shortest focal length of said plurality of images is approximately at least an anticipated distance to a foreground object in said field of view.

19. The method of claim 2, wherein:

a shortest focal length of said plurality of images corresponds to a length of a user'"'"'s arm.

20. The method of claim 2, wherein:

a shortest focal length of said plurality of images corresponds to a maximum extension of a user'"'"'s hand.

21. The method of claim 2, wherein:

a shortest focal length of said plurality of images is at least 0.5 meters.

22. The method of claim 2, wherein:

comparing a relative focus of said element set comprises comparing a relative degree of edge definition of said element set.

23. The method of claim 22, wherein:

comparing a relative degree of edge definition comprises comparing a relative spatial rate of change of an image property.

24. The method of claim 23, wherein:

said image property comprises pixel gradient.

25. The method of claim 24, wherein:

said image property comprises brightness.

26. The method of claim 24, wherein:

said image property comprises color.

27. The method of claim 24, wherein:

said image property comprises color channel value.

28. The method of claim 24, wherein:

said image property comprises at least one of the group consisting of red channel value, green channel value, and blue channel value.

29. The method of claim 24, wherein:

said image property comprises at least one of the group consisting of cyan channel value, yellow channel value, magenta channel value, and black channel value.

30. The method of claim 2, wherein:

said images are captured substantially simultaneously.

31. The method of claim 2, wherein:

said images are captured at different times.

32. The method of claim 2, wherein:

at least a first of said plurality of images is captured by a first sensor, and at least a second of said plurality of images is captured by a second sensor.

33. The method of claim 32, wherein:

said field of view comprises an intersection of a first field of view of said first sensor and a second field of view of said second sensor.

34. The method of claim 33, wherein:

said first and second fields of view are substantially identical.

35. The method of claim 2, wherein:

said plurality of images are derived from a plenoptic data set.

36. The method of claim 2, further comprising:

subtracting said element sets assigned to at least one of said at least one category from further analysis.

37. An apparatus, comprising:

a first image sensor having a first focal length and a first field of view;

a second image sensor having a second focal length longer than said first focal length and a second field of view substantially identical to said first field of view;

a body with said first and second image sensors disposed thereon, said body being configured to be wearable by a user on a head of said user, such that said first and second fields of view comprise a region of space in front of said user, said region of space being with a reach of hands of said user; and

a processor in communication with said sensors;

wherein;

said first image sensor is adapted to capture a first image, and said second sensor is adapted to substantially simultaneously capture a second image;

said processor is adapted to define in said first and second images a plurality of element sets, each said element set comprising a first element of said first image and a substantially corresponding second element of said second image;

said processor is adapted to compare a relative focus said first and second elements in said element sets;

said processor is adapted to assign to a background category those element sets wherein said second element therein is more in-focus than or substantially equally in-focus as said first element therein; and

said processor is adapted to subtract said element sets assigned to said background category from further analysis.

38. An apparatus, comprising:

at least one image sensor; and

a processor in communication with said sensor;

wherein;

said image sensor is adapted to capture a plurality of images substantially sharing a field of view, each of said images having a unique focal length;

said processor is adapted to determine a sequence of said images of increasing focal length, such that an n+1th image has a longer focal length than an nth image;

said processor is adapted to define in said plurality of images at least one element set, said element set comprising substantially corresponding elements in said plurality of images;

said processor is adapted to compare a relative focus of said element set; and

said processor is adapted to assign to one of at least one category those element sets wherein said relative focus of said sequence of images satisfies a corresponding one of at least one relationship.

39. The apparatus of claim 38, wherein:

said image sensor is disposed in a head mounted display.

40. The apparatus of claim 38, further comprising:

first and second image sensors, said first sensor being adapted to capture at least a first of said plurality of images, said second sensor being adapted to capture at least a second of said plurality of images.

41. The apparatus of claim 40, wherein:

said first sensor is preset to a first focal length, and said second sensor is preset to a second focal length.

42. The apparatus of claim 40, wherein:

said first sensor has a fixed first focal length, and said second sensor has a fixed second focal length.

43. The apparatus of claim 40, wherein:

said first and second sensor are arranged so as to generate stereo image pairs.

44. The apparatus of claim 40, further comprising:

a beam splitter;

wherein said beam splitter and said first and second sensors are arranged such that a first field of view of said first sensor is substantially identical to a second field of view of said second sensor.

45. The apparatus of claim 40, wherein:

said first and second sensors are arranged proximate one another and substantially aligned, such that a first field of view of said first sensor is substantially identical to a second field of view of said second sensor.

46. The apparatus of claim 40, wherein:

said first and second sensors are adapted to capture said first and second images substantially simultaneously.

47. The apparatus of claim 38, wherein:

said image sensor comprises a plenoptic imager, said plenoptic imager being adapted to capture said plurality of images having unique focal lengths in a plenoptic data set.

48. The apparatus of claim 38, wherein:

said image sensor is a variable focal-length sensor.

49. The apparatus of claim 48, wherein:

said image sensor captures all of said plurality of images.

50. An apparatus, comprising:

means for receiving a plurality of images substantially sharing a field of view, each of said images having a unique focal length;

means for determining a sequence of said images of increasing focal length, such that an n+1th image has a longer focal length than an nth image;

means for defining in said plurality of images at least one element set, said element set comprising substantially corresponding elements in said plurality of images;

means for comparing a relative focus of said element set throughout said sequence of images of increasing focal length;

means for assigning to one of at least one category those element sets wherein said relative focus of said sequence of images satisfies a corresponding one of at least one relationship.