OBJECT DETECTION AND TRACKING WITH AUDIO AND OPTICAL SIGNALS

• US 20140192024A1
• Filed: 01/07/2014
• Published: 07/10/2014
• Est. Priority Date: 01/08/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: HOLZ, David
• Granted Patent: US 9,465,461 B2
• Time in Patent Office: Days
• US Class Current: 345/175
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/043 using propagating acoustic waves

Abstract

The technology disclosed addresses these problems by supplementing optical gesture recognition with the ability to recognize touch gestures. This capability allows the user to execute intuitive gestures involving contact with a surface. For example, in low-light situations where free-form gestures cannot be recognized optically with a sufficient degree of reliability, a device implementing the invention may switch to a touch mode in which touch gestures are recognized. In implementations, two contact microphones or other vibratory or acoustical sensors, that are coupled to an optical motion-capture and are in contact with a surface that a user touches, are monitored. When the contact microphones detect audio signals (or other vibrational phenomena) generated by contact of an object with the surface that the user touches, a position of the object traveling across and in contact with the surface is tracked.

Claims

1. A method of operating an optical motion-capture system in low-light situations, the method including:

monitoring at least two contact microphones coupled to an optical motion-capture system, wherein the contact microphones are in contact with a surface that a user touches; and

in response to detection of audio signals generated by contact of an object with the surface that the user touches, tracking a position of the object by;

measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping surface locations that satisfy the TDOA;

analyzing at least one image, captured by a camera of the optical motion-capture system, of the object in contact with the surface; and

using the image analysis to select among the mapped TDOA surface locations as a surface location of the contact.

2. The method of claim 1, further including calculating a path of the object traveling across and in contact with the surface by:

repeatedly measuring TDOAs of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface.

3. The method of claim 2, further including comparing the path to a plurality of path templates and identifying a template that best matches the path.

4. The method of claim 1, further including constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.

5. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.

6. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.

7. A method of tracking a path of an object of interest traveling across and in contact with a surface, the method comprising:

capturing, with a camera, a sequence of images of a field of view that includes at least a portion of the surface;

capturing audio signals through the surface using at least two microphones; and

computationally constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.

8. The method of claim 7, wherein the 3D model includes a position and shape of the object.

9. The method of claim 7, wherein tracking movement includes calculating a location and direction of the object by computationally analyzing the audio signals.

10. The method of claim 9, wherein the location and direction of the object is calculated using a time difference of arrival.

11. The method of claim 7, further including filtering the audio signals to exclude frequencies outside a defined range.

12. The method of claim 7, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory or a gesture of the tracked object.

13. The method of claim 12, wherein the trajectory or the gesture dictates a change in operational mode.

14. An optical motion-capture system, including:

a camera for capturing images of an object within a field of view thereof;

a contact microphone coupled to the camera and in contact with a surface that a user touches;

a sensor for monitoring the contact microphone;

a control module for tracking a position of the object, in response to detection of audio signals generated by contact of an object with the surface that the user touches, by;

measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping surface locations that satisfy the TDOA;

analyzing at least one image, captured by the camera of the optical motion-capture system, of the object in contact with the surface; and

using the image analysis to select among the mapped TDOA surface locations as a surface location of the contact.

15. The system of claim 14, further configured to calculate a path of the object traveling across and in contact with the surface by:

repeatedly measuring TDOAs of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface.

16. The system of claim 14, further configured to construct a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.

17. The system of claim 14, further configured to recognize at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.

18. The system of claim 14, further configured to recognize at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.