Object detection and tracking with audio and optical signals

• US 9,465,461 B2
• Filed: 01/07/2014
• Issued: 10/11/2016
• Est. Priority Date: 01/08/2013
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David
• Primary Examiner(s): Sitta, Grant
• Assistant Examiner(s): BOGALE, AMEN W
• Application Number:
  US14/149,709
• Publication Number:
  US 20140192024A1
• Time in Patent Office: 1,008 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/04166 Details of scanning methods, e.g. sampling time, grouping of sub areas or time sharing with display driving Synchronisation with the driving of the display or the backlighting unit to avoid interferences generated internally G06F3/04184
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/043 using propagating acoustic waves

Abstract

The technology disclosed addresses these problems by supplementing optical gesture recognition with the ability to recognize touch gestures. This capability allows the user to execute intuitive gestures involving contact with a surface. For example, in low-light situations where free-form gestures cannot be recognized optically with a sufficient degree of reliability, a device implementing the invention may switch to a touch mode in which touch gestures are recognized. In implementations, two contact microphones or other vibratory or acoustical sensors, that are coupled to an optical motion-capture and are in contact with a surface that a user touches, are monitored. When the contact microphones detect audio signals (or other vibrational phenomena) generated by contact of an object with the surface that the user touches, a position of the object traveling across and in contact with the surface is tracked.

Claims

1. A method of operating an optical motion-capture system in low-light situations, the method including:

monitoring at least two contact microphones coupled to the optical motion-capture system, wherein the contact microphones are in contact with a surface; and

in response to detection of audio signals generated by contact of an object with the surface determining a surface location by;

measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping a curve, including a plurality of possible surface locations of the contact, using the measured time difference of arrival;

analyzing at least one image, captured by a camera of the optical motion-capture system, of the object in contact with the surface; and

using the image analysis, selecting one of the plurality of possible surface locations included in the mapped curve, as the surface location of the contact;

repeatedly measuring time difference of arrival of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface;

thereby calculating a path of the object traveling across and in contact with the surface.

2. The method of claim 1, further including comparing the path to a plurality of path templates and identifying a template that best matches the path.

3. The method of claim 1, further including constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.

4. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the object.

5. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of the object.

6. A method of tracking a path of an object of interest traveling across and in contact with a surface, the method comprising:

capturing, with a camera, a sequence of images of a field of view that includes at least a portion of the surface;

capturing audio signals through the surface using at least two microphones;

computationally constructing a 3D model of the object and tracking movement of the object over the surface based on the sequence of images and the audio signals, including;

measuring a time difference of arrival (TDOA) of the audio signals at the microphones and mapping a curve, including a plurality of possible surface locations of the object, using the captured audio signals;

analyzing at least one of the sequence of images; and

using the image analysis, selecting one of the plurality of possible surface locations, included in the mapped curve, as a position of the object; and

repeatedly measuring time difference of arrival of the audio signals at the microphones and performing the mapping, the analyzing and the selecting responsive to additional contacts of the object with the surface, thereby determining a path for the object traveling across the surface.

7. The method of claim 6, wherein the 3D model includes a shape of the object.

8. The method of claim 6, wherein tracking movement includes calculating a direction of the object by computationally analyzing the audio signals.

9. The method of claim 8, wherein the position and direction of the object are calculated using a time difference of arrival of the audio signals at the microphones.

10. The method of claim 6, further including filtering the audio signals to exclude frequencies outside a defined range.

11. The method of claim 6, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory or a gesture of the tracked object.

12. The method of claim 11, wherein the trajectory or the gesture dictates a change in operational mode.

13. An optical motion-capture system, including:

a camera for capturing images of an object within a field of view thereof;

two contact microphones coupled to the camera and in contact with a surface; and

a sensor for monitoring the contact microphones;

a control module for tracking a position of the object, in response to detection of audio signals generated by contact of the object with the surface, by;

measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping a curve, including a plurality of possible surface locations of the contact, using the measured time difference of arrival;

analyzing at least one image, captured by the camera of the optical motion-capture system, of the object in contact with the surface; and

using the image analysis, selecting one of the plurality of possible surface locations included in the mapped curve, as the surface location of the contact;

repeatedly measuring time difference of arrival of the audio signals at the contact microphones responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface;

thereby calculating a path of the object traveling across and in contact with the surface.

14. The system of claim 13, further configured to construct a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the audio signals.

15. The system of claim 13, further configured to recognize at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.

16. The system of claim 13, further configured to recognize at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.