Proximity object tracker

• US 8,547,327 B2
• Filed: 10/13/2009
• Issued: 10/01/2013
• Est. Priority Date: 10/07/2009
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Clarkson, Ian, Hildreth, Evan
• Primary Examiner(s): Agustin, Peter Vincent
• Application Number:
  US12/578,530
• Publication Number:
  US 20110080490A1
• Time in Patent Office: 1,449 Days
• Field of Search: None
• US Class Current: 345/156
• CPC Class Codes: E05Y 2800/106 Lighting
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: G06F 3/0428 by sensing at the edges of the touch surface the interruption of optical paths, e.g. an illumination plane, parallel to the touch surface which may be virtual sensing beam interruptions in a planar beam grid of an optical touch-screen G06F3/0421
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16
• CPC Class Codes: H04N 23/56 provided with illuminating means

Abstract

Object tracking technology, in which controlling an illumination source is controlled to illuminate while a camera is capturing an image to define an intersection region within the image captured by the camera. The image captured by the camera is analyzed to detect an object within the intersection region. User input is determined based on the object detected within the intersection region and an application is controlled based on the determined user input.

Claims

1. An electronic system comprising:

a camera having a field of view of a first area;

a display screen configured to display a graphical user interface, wherein

the camera is positioned at a first side of the display screen, is angled with respect to the display screen, and the field of view of the camera is of the first area in front of the display screen;

an illumination source that is angled with respect to the camera and that is configured to illuminate a second area, the second area intersecting the first area to define an intersection region within the field of view of the camera, wherein

the illumination source is positioned at a second side of the display screen, is angled with respect to the display screen, and is configured to illuminate the second area in front of the display screen, the second area in front of the display screen intersecting the first area in front of the display screen to define the intersection region; and

a processing unit configured to perform operations comprising;

capturing an image from the camera;

analyzing the image captured by the camera to detect an object within the intersection region;

determining user input based on the object detected within the intersection region; and

controlling an application based on the determined user input.

2. The system of claim 1,

wherein the second side of the display screen is opposite of the first side of the display screen.

3. The electronic system of claim 1 wherein analyzing the image captured by the camera to detect an object within the intersection region comprises:

comparing pixels of the image captured by the camera to a brightness threshold to produce a binary image, wherein pixels in the binary image indicate whether or not the corresponding pixels in the image captured by the camera meet the brightness threshold;

grouping pixels within the binary image into one or more blobs;

grouping the one or more blobs into one or more clusters; and

determining a position of one or more objects in the binary image based on the one or more clusters.

4. The electronic system of claim 3 wherein grouping the one or more blobs into one or more clusters comprises clustering blobs within the binary image into one or more clusters based on a tracking mode of the electronic system.

5. The electronic system of claim 4:

wherein clustering blobs within the binary image into one or more clusters based on a tracking mode of the electronic system comprises, when the electronic system is configured in a single object tracking mode, clustering blobs within the binary image into a single cluster;

wherein determining a position of one or more objects in the binary image based on the one or more clusters comprises determining a position of the single object based on the single cluster; and

wherein determining user input based on the object detected within the intersection region comprises determining user input based on the position of the single object.

6. The electronic system of claim 4:

wherein clustering blobs within the binary image into one or more clusters based on a tracking mode of the electronic system comprises, when the electronic system is configured in a two object adjacent tracking mode, clustering blobs in a horizontal direction from an outer edge of first and second sides of the binary image to a center of the binary image to identify a first cluster at the first side of the image and a second cluster at the second side of the image;

wherein determining a position of one or more objects in the binary image based on the one or more clusters comprises determining a position of a first object based on the first cluster and determining a position of a second object based on the second cluster; and

wherein determining user input based on the object detected within the intersection region comprises determining user input based on the position of the first object and the position of the second object.

7. The electronic system of claim 6 wherein clustering blobs in the horizontal direction from the outer edge of first and second sides of the binary image to the center of the binary image to identify the first cluster at the first side of the image and the second cluster at the second side of the image comprises weighting proximity of blobs in the horizontal direction higher than proximity of blobs in a vertical direction in clustering blobs together.

8. The electronic system of claim 4:

wherein clustering blobs within the binary image into one or more clusters based on a tracking mode of the electronic system comprises, when the electronic system is configured in a two object stacked tracking mode, clustering blobs in a vertical direction from an outer edge of a top and a bottom of the binary image to a center of the binary image to identify a first blob at a top portion of the image and a second blob at a bottom portion of the image;

wherein determining a position of one or more objects in the binary image based on the one or more clusters comprises determining a position of a first object based on the first cluster and determining a position of a second object based on the second cluster; and

wherein determining user input based on the object detected within the intersection region comprises determining user input based on the position of the first object and the position of the second object.

9. The electronic system of claim 8 wherein clustering blobs in the vertical direction from the outer edge of the top and the bottom of the binary image to the center of the binary image to identify the first cluster at the top portion of the image and the second cluster at the bottom portion of the image comprises weighting proximity of blobs in the vertical direction higher than proximity of blobs in a horizontal direction in clustering blobs together.

10. The electronic system of claim 4 wherein clustering blobs within the binary image into one or more clusters based on a tracking mode of the electronic system and determining the position of one or more objects in the binary image based on the one or more clusters comprises:

determining a tracking mode of the electronic system from among at least a single hand tracking mode, a two hand adjacent tracking mode, and a two hand stacked tracking mode;

in response to the determined tracking mode of the electronic system being the single hand tracking mode;

clustering blobs within the binary image into a single cluster;

computing a position of the single object based on the single cluster;

in response to the determined tracking mode of the electronic system being the two hand adjacent tracking mode;

clustering blobs in a horizontal direction from an outer edge of first and second sides of the binary image to a center of the binary image to identify a first cluster at the first side of the image and a second cluster at the second side of the image;

computing a position of a first object based on the first cluster; and

computing a position of a second object based on the second cluster; and

in response to the determined tracking mode of the electronic system being the two hand stacked tracking mode;

clustering blobs in a vertical direction from an outer edge of a top and a bottom of the binary image to a center of the binary image to identify a first cluster at a top portion of the image and a second cluster at a bottom portion of the image;

computing a position of a first object based on the first cluster; and

computing a position of a second object based on the second cluster.

11. The electronic system of claim 1 wherein determining user input based on the object detected within the intersection region comprises:

mapping a position of the detected object to an interface displayed by the application being controlled; and

determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled.

12. The electronic system of claim 11 wherein determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled comprises determining whether the mapped position of the detected object corresponds to an element displayed in the interface displayed by the application being controlled.

13. The electronic system of claim 11:

wherein mapping the position of the detected object to the interface displayed by the application being controlled comprises mapping the position of the detected object to a cursor position in the interface displayed by the application being controlled; and

wherein determining user input based on the mapped position of the detected object to the interface displayed by the application being controlled comprises determining user input based on the cursor position in the interface displayed by the application being controlled.

14. The electronic system of claim 1 wherein determining user input based on the object detected within the intersection region comprises:

detecting performance of a gesture by the detected object based on positions of the detected object determined within a series of images captured by the camera; and

determining user input based on the detected gesture.

15. The electronic system of claim 14:

wherein detecting performance of the gesture by the detected object based on positions of the detected object determined within the series of images captured by the camera comprises detecting a swipe gesture; and

wherein determining user input based on the detected gesture comprises determining user input based on the detected swipe gesture.

16. The electronic system of claim 14:

wherein detecting performance of the gesture by the detected object based on positions of the detected object determined within the series of images captured by the camera comprises detecting a gesture in which two detected objects are moving horizontally together or apart; and

wherein determining user input based on the detected gesture comprises determining user input based on the detected gesture in which two detected objects are moving horizontally together or apart.

17. The electronic system of claim 1 wherein the illumination source is a first illumination source, further comprising:

a second illumination source that is angled with respect to the camera differently than the first illumination source and that is configured to illuminate a third area, wherein;

the third area intersects the first area,

the third area is different than the second area, and

the third area in combination with the second area defines a combined intersection region within the field of view of the camera;

wherein the processing unit is configured to perform operations further comprising;

controlling the first and second illumination sources to illuminate in sequence such that images captured by the camera are in an alternating pattern;

identifying a first image captured when the first illumination source was illuminated and the second illumination sources was not illuminated;

identifying a second image captured when the first illumination source was not illuminated and the second illumination sources was illuminated; and

analyzing the first and second images in combination to determine a position of an object within the combined intersection region defined by the first and second illumination sources.

18. The electronic system of claim 1:

wherein capturing the image from the camera comprises capturing a grayscale image;

wherein analyzing the image captured by the camera to detect the object within the intersection region comprises;

comparing pixels of the grayscale image to a brightness threshold to produce a binary image, wherein pixels in the binary image indicate whether or not the corresponding pixels in the grayscale image captured by the camera meet the brightness threshold;

grouping pixels within the binary image into blobs;

referencing the grayscale image in clustering blobs within the binary image into one or more clusters; and

determining a position of one or more objects in the binary image based on results of the clustering.

19. The electronic system of claim 1 wherein the illumination source is an infrared emitter.

20. The electronic system of claim 1 wherein the processing unit is configured to perform operations further comprising controlling the illumination source to illuminate while the camera is capturing the image to define the intersection region within the image captured by the camera.

21. The electronic system of claim 20 wherein controlling the illumination source to illuminate while the camera is capturing the image to define the intersection region within the image captured by the camera comprises controlling the illumination source to turn on prior to capturing the image from the camera.

22. The electronic system of claim 20:

wherein controlling the illumination source to illuminate while the camera is capturing the image to define the intersection region within the image captured by the camera comprises controlling the illumination source to illuminate in sequence with images captured by the camera in an alternating pattern such that a first image captured by the camera is captured when the illumination source is illuminated and a second image captured by the camera is captured when the illumination source is not illuminated; and

wherein analyzing the image captured by the camera to detect the object within the intersection region comprises;

subtracting the second image from the first image to produce a resulting image; and

analyzing the resulting image to detect the object within the intersection region.

23. The electronic system of claim 1 wherein analyzing the image captured by the camera to detect the object within the intersection region comprises ignoring objects that are within the camera'"'"'s field of view and outside of the intersection region.

24. The electronic system of claim 1 wherein analyzing the image captured by the camera to detect an object within the intersection region comprises using motion information to detect a moving object within the intersection region.

25. At least one computer-readable storage medium encoded with executable instructions that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

capturing an image of an intersection region from a camera

positioned at a first side of a display screen and angled with respect to the display screen the intersection region being defined within a field of view of the camera of a first area in front of the display screen

by an illumination source

positioned at a second side of the display screen

and angled with respect to the display screen

so as to illuminate a second area in front of the display screen intersecting the first area in front of the display screen to define the intersection region in front of the display screen;

analyzing the image from the camera to detect an object within an intersection region;

determining user input based on the object detected within the intersection region; and

controlling an application based on the determined user input.

26. A method comprising:

capturing an image from a camera, wherein

the camera is positioned at a first side of a display screen, the camera is angled with respect to the display screen, and the field of view of the camera is of a first area in front of the display screen;

analyzing the image captured by the camera to detect an object within an intersection region defined within the camera'"'"'s field of view by an illumination source, wherein

the illumination source is positioned at a second side of the display screen, the illumination source is angled with respect to the display screen, and the illumination source is configured to illuminate the second area in front of the display screen, the second area in front of the display screen intersecting the first area in front of the display screen to define the intersection region in front of the display screen;

determining user input based on the object detected within the intersection region; and

controlling an application based on the determined user input.

27. An apparatus comprising:

means for capturing an image;

means for display, wherein

the means for capturing the image is positioned at a first side of the means for display, the means for capturing the image is angled with respect to the means for display, and the field of view of the means for capturing the image is of a first area in front of the means for display;

means for illumination, wherein

the means for illumination is positioned at a second side of the means for display, the means for illumination is angled with respect to the means for display, and the means for illumination is configured to illuminate a second area in front of the means for display, the second area in front of the means for display intersecting the first area in front of the means for display to define and intersection region in front of the means for display;

means for analyzing the image captured by the means for capturing the image to detect an object within the intersection region;

means for determining user input based on the object detected within the intersection region; and

means for controlling an application based on the determined user input.