Correlating movement information received from different sources

• US 9,939,888 B2
• Filed: 09/15/2011
• Issued: 04/10/2018
• Est. Priority Date: 09/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Wilson, Andrew D., Benko, Hrvoje
• Primary Examiner(s): ZALALEE, SULTANA MARCIA
• Application Number:
  US13/233,018
• Publication Number:
  US 20130069931A1
• Time in Patent Office: 2,399 Days
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06V 40/20 Movements or behaviour, e.g. gesture recognition recognition of facial expressions G06V40/16

Abstract

A system is described herein which receives internal-assessed (IA) movement information from a mobile device. The system also receives external-assessed (EA) movement information from at least one monitoring system which captures a scene containing the mobile device. The system then compares the IA movement information with the EA movement information with respect to each candidate object in the scene. If the IA movement information matches the EA movement information for a particular candidate object, the system concludes that the candidate object is associated with the mobile device. For example, the object may correspond to a hand that holds the mobile device. The system can use the correlation results produced in the above-indicated manner to perform various environment-specific actions.

Claims

1. A method, implemented by computing functionality, for assessing movement of a mobile device associated with a subject in a scene, comprising:

receiving internal-assessed (IA) movement information regarding the movement of the mobile device from at least one position-determining device provided by the mobile device;

receiving external-assessed (EA) movement information from at least one monitoring system which captures images of the scene containing the subject and the mobile device;

comparing the IA movement information with the EA movement information for a candidate object associated with the subject using a candidate orientation of the mobile device;

repeating, at least one time, said comparing with respect to another candidate object, each iteration of said comparing producing correlation results;

determining, based on the correlation results for each candidate object, at least one matching object which is associated with the mobile device by selecting a candidate object with the strongest correlation results as a matching object; and

fusing the IA movement information with the EA movement information to yield more precise IA and EA movement information comprising a more precise location in space and a more precise orientation in space for the at least one matching object;

performing an environment-specific action using the fused IA and EA information for the matching object based on an outcome of said determining,

said receiving of the IA movement information, receiving of the EA movement information, comparing, repeating, determining, fusing and performing being implemented by the computing functionality.

2. The method of claim 1, wherein the subject comprises a human being, and where each candidate object corresponds to a part of the human being.

3. The method of claim 1, wherein the correlation results for a given candidate object are obtained by using a candidate orientation of the mobile device and a comparison of the IA movement information with the EA movement for the given candidate object with respect to the candidate orientation.

4. The method of claim 1, wherein the strongest correlation results for a candidate object are obtained by using, at a given time, an acceleration of the mobile device obtained from the EA movement information, an acceleration of the mobile device obtained from the IA movement information, an acceleration caused by gravity and the orientation of the acceleration of the mobile device with respect to gravity.

5. The method of claim 1, wherein the mobile device corresponds to a device that is associated with part of an anatomy of a human being.

6. The method of claim 1, wherein the monitoring system comprises an apparatus which generates the EA movement information by:

generating a depth image of the scene;

identifying each candidate object within the scene based on the depth image; and

assessing movement of each candidate object in the scene.

7. The method of claim 6, wherein the monitoring system identifies each candidate object by comparing the depth image with a multi-part skeletal model of the subject.

8. The method of claim 1, further comprising:

removing a contribution of gravity from the IA movement information, based on an orientation of the mobile device, to provide IA-corrected movement information,

wherein said comparing comprises comparing said IA-corrected movement information with the EA movement information.

9. The method of claim 8, wherein the orientation of the mobile device comprises an actual orientation of the mobile device that is assessed based on the IA movement information received from the mobile device.

10. The method of claim 8, wherein the orientation of the mobile device comprises a guessed orientation of the mobile device.

11. The method of claim 10, wherein the guessed orientation of the mobile device comprises an actual orientation of the mobile device if a matching object is identified based on use of the guessed orientation.

12. The method of claim 8, wherein said comparing comprises comparing said IA-corrected movement information with the EA movement information over a time window.

13. The method of claim 1, wherein said comparing comprises:

extracting at least one feature from the IA movement information, to provide at least one IA feature;

extracting at least one feature from the EA movement information, to provide at least one EA feature; and

comparing said at least one IA feature with said at least one EA feature.

14. The method of claim 1, wherein said performing comprising:

supplementing the EA movement information with the IA movement information to provide enhanced control information with respect to at least one matching object in the scene; and

using the enhanced control information to control operation of a target system.

15. The method of claim 14, wherein said supplementing comprises adding orientation information that is obtained from the IA movement information to the EA movement information, the orientation information pertaining to an orientation of said at least one matching object.

16. The method of claim 1, wherein said fusing the EA movement information with the IA movement information to provide more precise EA movement information and more precise IA movement information comprises providing a precise position in space and a precise orientation at that position for at least one matching object in the scene.

17. A computer readable storage device for storing computer readable instructions, the computer readable instructions providing a movement processing system when executed by one or more processing devices, the computer readable instructions comprising:

logic configured to receive first movement information of a mobile device from at least one position-determining device provided by the mobile device;

logic configured to receive second movement information of the mobile device from at least one other source external to the mobile device that captures images of the mobile device and objects as part of a scene;

logic configured to identify one or more candidate objects associated with the mobile device by comparing a depth image with a model of a human being;

logic configured to compare the first movement information with the second movement information for each of the candidate objects using a candidate orientation of the mobile device, producing correlation results; and

logic configured to identify a matching object which is associated with the mobile device, based on the correlation results.

18. The computer readable storage medium of claim 17, further comprising logic configured to remove a contribution of gravity from the first movement information prior to said correlating, based on an orientation of the mobile device, wherein the orientation of the mobile device comprises an actual orientation of the mobile device that is assessed based on the first movement information received from the mobile device.

19. The computer readable storage medium of claim 17, further comprising logic configured to remove a contribution of gravity from the first movement information prior to said correlating, based on an orientation of the mobile device, wherein the orientation of the mobile device comprises a guessed orientation of the mobile device.

20. A movement processing system, implemented by computing functionality, for assessing movement of a mobile device in a scene, comprising:

a correlation processing module, comprising;

logic configured to receive internal-assessed (IA) movement information of the mobile device from at least one position-determining device provided by the mobile device;

logic configured to receive external-assessed (EA) movement information of the mobile device from at least one monitoring system which captures at least one image of the scene containing a human being and the mobile device;

logic configured to identify the location of a plurality of candidate parts of a human being associated with the mobile device;

logic configured to compare the IA movement information of the mobile device with the EA movement information with respect to a plurality of candidate parts of the human being, to produce correlation results for each candidate part using a candidate orientation of the mobile device; and

logic configured to determine, based on the correlation results for each candidate part, at least one matching part which is associated with the mobile device by using the correlation results for each candidate part and selecting the candidate part with the greatest correlation as the matching part associated with the mobile device; and

at least one application module, comprising;

logic configured to supplement the EA movement information of the mobile device with the IA movement information of the mobile device to provide enhanced control information with respect to at least one matching candidate part in the scene; and

logic configured to use the enhanced control information to control operation of a target system.