OBJECT DETECTION AND TRACKING FOR PROVIDING A VIRTUAL DEVICE EXPERIENCE

• US 20150199025A1
• Filed: 01/15/2015
• Published: 07/16/2015
• Est. Priority Date: 01/15/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: HOLZ, David S.
• Granted Patent: US 9,613,262 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/043 using propagating acoustic waves
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed can provide capabilities such as using vibrational sensors and/or other types of sensors coupled to a motion-capture system to monitor contact with a surface that a user can touch. A virtual device can be projected onto at least a portion of the surface. Location information of a user contact with the surface is determined based at least in part upon vibrations produced by the contact. Control information is communicated to a system based in part on a combination of the location on the surface portion of the virtual device and the detected location information of the user contact. The virtual device experience can be augmented in some implementations by the addition of haptic, audio and/or visual projectors.

Claims

1. A method of operating a motion-capture system, the method including:

monitoring at least three vibration sensors coupled to a motion-capture system; and

in response to detection of vibrations generated by contact of an object with a surface of a solid medium, tracking a position of the object by;

detecting a first vibration through a first medium by a first sensor;

detecting a second vibration through a second medium by at least a second and a third sensor;

measuring a time difference of arrival (TDOA) of the second vibration at the at least second and third sensors with respect to a TDOA of the first vibration at the first sensor and mapping surface locations that satisfy the TDOA;

calculating a distance from the object to the second and third sensor based on the respective TDOAs; and

determining location information for the object relative to the sensors based on the mapped TDOA surface locations.

2. The method of claim 1, further including calculating a path of the object traveling across and in contact with the surface by:

repeatedly determining location information responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface.

3. The method of claim 2, further including comparing the path to a plurality of path templates and identifying a template that best matches the path.

4. The method of claim 2, further including constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the vibrations.

5. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.

6. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.

7. The method of claim 1, further including:

projecting a virtual device onto at least a portion of the surface;

detecting location information of a user contact with the surface based at least in part upon vibrations produced by the contact; and

communicating control information to a system based in part on a combination of the location on the surface portion of the virtual device and the detected location information of the user contact.

8. The method of claim 1, wherein the first medium is at least one of a solid medium or an air medium.

9. The method of claim 1, wherein the second medium is at least one of a solid medium or an air medium.

10. A method of creating a virtual device experience in a three-dimensional (3D) sensory environment, the method including:

monitoring at least two contact microphones coupled to an optical motion-capture system in a three-dimensional (3D) sensory environment, wherein the contact microphones are in contact with a surface that a user touches;

projecting a virtual device onto at least a portion of a surface; and

in response to detection of audio signals generated by contact of an object with the surface that the user touches, tracking a position of the object in with contact with the virtual device by;

measuring a time difference of arrival (TDOA) of the audio signals at the contact microphones and mapping surface locations that satisfy the TDOA;

analyzing at least one image, captured by a camera of the optical motion-capture system, of the object in contact with the virtual device;

using the image analysis to select among the mapped TDOA surface locations as location on the surface portion of the virtual device;

detecting location information of the user contact with the surface based at least in part upon audio signals produced by the contact; and

communicating control information to a system based in part on a combination of the location on the surface portion of the virtual device and the detected location information of the user contact.

11. The method of claim 10, wherein the projecting includes projecting an image of a virtualized object onto the surface portion.

12. The method of claim 11, wherein the image includes an image of a book.

13. The method of claim 10, further including filtering the audio signals to exclude frequencies outside a defined range.

14. The method of claim 10, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory or a gesture of the tracked object.

15. The method of claim 14, wherein the trajectory or the gesture dictates a change in operational mode.

16. An acoustical motion-capture system, the system including:

a plurality of vibrational sensors in contact with a surface that a user touches; and

a control module for tracking a position of an object responsive to detection of vibrations generated by contact of the object with the surface that the user touches, by;

measuring a time difference of arrival (TDOA) of the vibrations at the vibrational sensors and mapping surface locations that satisfy the TDOA;

analyzing at least one contact, captured by the sensors, of the object in contact with the surface; and

using the analysis to select among mapped TDOA surface locations as a surface location of the contact.

17. The system of claim 16, further including:

an imaging system configured to capture a sequence of images, wherein the control module is further configured to calculate a path of the object traveling across and in contact with the surface by;

repeatedly determining location information responsive to successive contacts of the object with the surface; and

analyzing the sequence of images of the successive contacts of the object with the surface.

18. The system of claim 17, further configured to construct a 3D model of the object and track movement thereof over the surface based on the sequence of images and the vibrations.

19. The system of claim 17, further configured to recognize at least one of an image sequence or a vibration that indicates a trajectory of the tracked object.

20. The system of claim 19, further configured to recognize at least one of an image sequence or a vibration that indicates a gesture of the tracked object.