Object detection and tracking for providing a virtual device experience

• US 9,613,262 B2
• Filed: 01/15/2015
• Issued: 04/04/2017
• Est. Priority Date: 01/15/2014
• Status: Active Grant

• Current Assignee: Lmi Liquidating Co., LLC
• Original Assignee: Leap Motion Incorporated (Ultraleap Limited)
• Inventors: Holz, David S
• Primary Examiner(s): Sasinowski, Andrew
• Assistant Examiner(s): Lin, Chun-Nan
• Application Number:
  US14/598,149
• Publication Number:
  US 20150199025A1
• Time in Patent Office: 810 Days
• CPC Class Codes: G06F 2203/04106 Multi-sensing digitiser, i.e. digitiser using at least two different sensing technologies simultaneously or alternatively, e.g. for detecting pen and finger, for saving power or for improving position detection
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/043 using propagating acoustic waves
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

The technology disclosed can provide capabilities such as using vibrational sensors and/or other types of sensors coupled to a motion-capture system to monitor contact with a surface that a user can touch. A virtual device can be projected onto at least a portion of the surface. Location information of a user contact with the surface is determined based at least in part upon vibrations produced by the contact. Control information is communicated to a system based in part on a combination of the location on the surface portion of the virtual device and the detected location information of the user contact. The virtual device experience can be augmented in some implementations by the addition of haptic, audio and/or visual projectors.

Claims

1. A method of operating a motion-capture system, the method including:

monitoring at least three vibration sensors coupled to a motion-capture system; and

in response to detection of vibrations generated by contact of an object with a surface of a first medium, tracking a position of the object by;

detecting a first vibration through the first medium by a first vibration sensor;

detecting a second vibration through a second medium by at least a second and a third vibration sensor, wherein the first medium is different from the second medium;

measuring a time difference of arrival (TDOA) of the second vibration at the second and third vibration sensors with respect to the TDOA of the first vibration at the first vibration sensor and mapping surface locations that satisfy the TDOA, whereby the TDOA of the second vibration at the second and third vibration sensors differs with respect to the TDOA of the first vibration at the first vibration sensor because the second vibration travels through the second medium at a different velocity that the first vibration travels through the first medium;

calculating a distance from the object to the second and third vibration sensors based on the respective TDOAs; and

determining location information for the object relative to the vibration sensors based on the mapped TDOA surface locations.

2. The method of claim 1, further including calculating a path of the object traveling across and in contact with the surface by:

repeatedly determining location information responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface.

3. The method of claim 2, further including comparing the path to a plurality of path templates and identifying a template that best matches the path.

4. The method of claim 2, further including constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the vibrations.

5. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.

6. The method of claim 1, further including recognizing at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.

7. The method of claim 1, further including:

projecting a virtual device onto at least a portion of the surface;

detecting location information of a user contact with the surface based at least in part upon vibrations produced by the contact; and

communicating control information to a system based in part on a combination of the location on the surface portion of the virtual device and the detected location information of the user contact.

8. The method of claim 1, wherein the first medium is at least one of a solid medium or an air medium.

9. The method of claim 1, wherein the second medium is at least one of a solid medium or an air medium.

10. A motion-capture system, including:

at least three vibration sensors; and

a control module comprising (i) an interface coupling to the at least three vibration sensors and operative to receive information indicating detection of vibrations generated by contact of an object with a surface of a first medium, (ii) a processor coupled to the interface and (iii) a non-transitory computer readable storage medium coupled to the processor and storing instructions, wherein the instructions when executed on the processors, implement actions comprising;

monitoring the at least three vibration sensors coupled to the control module; and

in response to detection of vibrations generated by contact of an object with a surface of a first medium, tracking a position of the object by;

detecting a first vibration through the first medium by a first vibration sensor;

detecting a second vibration through a second medium by at least a second and a third vibration sensor, wherein the first medium is different from the second medium;

measuring a time difference of arrival (TDOA) of the second vibration at the second and third vibration sensors with respect to the TDOA of the first vibration at the first vibration sensor and mapping surface locations that satisfy the TDOA, whereby the TDOA of the second vibration at the second and third vibration sensors differs with respect to the TDOA of the first vibration at the first vibration sensor because the second vibration travels through the second medium at a different velocity that the first vibration travels through the first medium;

calculating a distance from the object to the second and third vibration sensors based on the respective TDOAs; and

determining location information for the object relative to the vibration sensors based on the mapped TDOA surface locations.

11. The system of claim 10, further implementing calculating a path of the object traveling across and in contact with the surface by:

repeatedly determining location information responsive to successive contacts of the object with the surface; and

analyzing a sequence of images of the successive contacts of the object with the surface.

12. The system of claim 11, further implementing comparing the path to a plurality of path templates and identifying a template that best matches the path.

13. The system of claim 11, further implementing constructing a 3D model of the object and tracking movement thereof over the surface based on the sequence of images and the vibrations.

14. The system of claim 10, further implementing recognizing at least one of an image sequence or an audio signal that indicates a trajectory of the tracked object.

15. The system of claim 10, further implementing recognizing at least one of an image sequence or an audio signal that indicates a gesture of the tracked object.

16. The system of claim 10, further implementing:

projecting a virtual device onto at least a portion of the surface;

detecting location information of a user contact with the surface based at least in part upon vibrations produced by the contact; and

communicating control information to a system based in part on a combination of the location on the surface portion of the virtual device and the detected location information of the user contact.

17. The system of claim 10, wherein the first medium is at least one of a solid medium or an air medium.

18. The system of claim 10, wherein the second medium is at least one of a solid medium or an air medium.

19. A non-transitory computer readable storage medium impressed with computer program instructions to operate a motion-capture system, wherein the instructions, when executed on a processor, implement a method comprising:

monitoring at least three vibration sensors coupled to the motion-capture system; and

in response to detection of vibrations generated by contact of an object with a surface of a first medium, tracking a position of the object by;

detecting a first vibration through the first medium by a first vibration sensor;

detecting a second vibration through a second medium by at least a second and a third vibration sensor, wherein the first medium is different from the second medium;

measuring a time difference of arrival (TDOA) of the second vibration at the second and third vibration sensors with respect to the TDOA of the first vibration at the first vibration sensor and mapping surface locations that satisfy the TDOA, whereby the TDOA of the second vibration at the second and third vibration sensors differs with respect to the TDOA of the first vibration at the first vibration sensor because the second vibration travels through the second medium at a different velocity that the first vibration travels through the first medium;

calculating a distance from the object to the second and third vibration sensors based on the respective TDOAs; and

determining location information for the object relative to the vibration sensors based on the mapped TDOA surface locations.