Monitoring interactions between two or more objects within an environment

• US 8,619,049 B2
• Filed: 05/17/2011
• Issued: 12/31/2013
• Est. Priority Date: 05/17/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Harrison, Chris, Benko, Hrvoje, Wilson, Andrew David
• Primary Examiner(s): GIESY, ADAM
• Application Number:
  US13/109,598
• Publication Number:
  US 20120293402A1
• Time in Patent Office: 959 Days
• Field of Search: None
• US Class Current: 345/173
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0421 by interrupting or reflecting a light beam, e.g. optical touch-screen
• CPC Class Codes: H04M 1/724 User interfaces specially adapted for cordless or mobile telephones
• CPC Class Codes: H04M 2250/52 including functional features of a camera constructional features concerning the integration of cameras in portable phones H04M1/0264
• CPC Class Codes: H04M 2250/54 including functional features of a projector or beamer module assembly constructional features concerning the integration of projectors in portable phones H04M1/0272

Abstract

One or more techniques and/or systems are provided for monitoring interactions by an input object with an interactive interface projected onto an interface object. That is, an input object (e.g., a finger) and an interface object (e.g., a wall, a hand, a notepad, etc.) may be identified and tracked in real-time using depth data (e.g., depth data extracted from images captured by a depth camera). An interactive interface (e.g., a calculator, an email program, a keyboard, etc.) may be projected onto the interface object, such that the input object may be used to interact with the interactive interface. For example, the input object may be tracked to determine whether the input object is touching or hovering above the interface object and/or a projected portion of the interactive interface. If the input object is in a touch state, then a corresponding event associated with the interactive interface may be invoked.

Claims

1. A method for monitoring interactions with a projected interactive interface, comprising:

tracking an input object and an interface object in real-time within an environment, the tracking comprising iteratively;

identifying the input object and the interface object within the environment based upon depth maps associated with captured images of the environment;

projecting an interactive interface onto the interface object; and

determining whether the input object is in a hover state or a touch state in relation to at least one of the interface object or the interactive interface based upon the tracking.

2. A method for monitoring interactions between an input object and an interface object, comprising:

receiving a depth map comprising depth data for pixels within a captured image of an environment;

generate a depth derivative image from the depth map based upon computing depth derivatives for pixel clusters within the captured image using the depth data;

iterating over the depth derivative image to identify an input object having depth derivative characteristics within predefined input object parameters; and

iterating over the depth derivative image to identify an interface object having depth derivative characteristics within predefined interface object parameters, the input object distinguishable from the interface object.

3. The method of claim 2, comprising:

tracking the input object and the interface object in real-time within the environment, the tracking comprising iteratively;

receiving depth maps comprising depth data for pixels within captured images of the environment;

generating depth derivative images from the depth maps; and

distinguishing between the input object and the interface object based upon depth data.

4. The method of claim 2, comprising:

determining whether the input object is in a hover state or a touch state in relation to the interface object based upon depth data.

5. The method of claim 4, comprising at least one of:

if the input object is in the touch state, then invoking a corresponding touch event;

or

if the input object is in the hover state, then invoking a corresponding hover event.

6. The method of claim 4, the determining whether the input object is in a hover state or a touch state comprising:

flood filling the input object based upon depth data; and

determining whether the flood filling filled at least a portion of the interface object.

7. The method of claim 4, comprising:

tracking the input object while in the touch state to determine a stroke input.

8. The method of claim 2, comprising:

projecting an interactive interface onto the interface object.

9. The method of claim 8, comprising:

dynamically adjusting the projection of the interactive interface based upon distortion of the interface object detected from depth data associated with the interface object.

10. The method of claim 8, comprising:

upon determining that the input object is in a touch state in relation to the interface object at a location upon which the interactive interface is projected, at least one of;

invoking an event corresponding to the interactive interface;

or

altering the projection of the interactive interface to provide real-time feedback.

11. The method of claim 2, comprising:

iterating over the depth derivative image to identify a second interface object; and

determining whether the input object is in a hover state or a touch state in relation to the second interface object.

12. The method of claim 2, comprising:

iterating over the depth derivative image to identify a second input object; and

determining whether the second input object is in a hover state or a touch state in relation to the interface object.

13. A system for monitoring interactions between an input object and an interface object, comprising:

a projection component configured to project an interactive interface onto an interface object; and

a tracking component configured to;

track an input object and the interface object in real-time within an environment by iteratively identifying the input object and the interface object within the environment based upon depth maps associated with captured images of the environment; and

determine whether the input object is in a hover state or a touch state in relation to at least one of the interface object or the interactive interface based upon the tracking.

14. The system of claim 13, the environment comprising a physical environment.

15. The system of claim 13, the tracking component configured to:

detect that the input object is in the touch state in relation to the interface object at a location upon which the interactive interface is projected; and

alter the projection of the interactive interface to provide real-time feedback.

16. The system of claim 13, the tracking component configured to:

generate a depth derivative image from a depth map based upon computing depth derivatives for pixel clusters within a captured image using depth data of the depth map;

flood fill the input object within the depth derivative image based upon depth data; and

determine whether the flood filling filled at least a portion of the interface object to determine whether the input object is in a touch state in relation to the interface object.

17. The system of claim 13, the tracking component configured to:

detect that the input object is in a touch state in relation to the interface object at a location upon which the interactive interface is projected; and

invoke a event corresponding to the interactive interface.

18. The system of claim 13, the tracking component configured to:

identify the interface object based at least in part upon a user hand gesture defining an interface area.

19. The system of claim 13, the tracking component configured to:

identify a finger as the input object based upon cylindrical depth derivative characteristics.

20. The system of claim 13, the projection component configured to:

dynamically adjust the projection of the interactive interface based upon distortion of the interface object detected from depth data associated with the interface object.