Trigger zones for objects in projected surface model

• US 10,306,193 B2
• Filed: 04/27/2015
• Issued: 05/28/2019
• Est. Priority Date: 04/27/2015
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Weber, Karon, Zannier, Federico, Villecroze, Jean-Louis
• Primary Examiner(s): Lee, Y
• Application Number:
  US14/697,453
• Publication Number:
  US 20160316185A1
• Time in Patent Office: 1,492 Days
• Field of Search: 348164
• CPC Class Codes: G03B 17/54 with projector
• CPC Class Codes: G06F 2203/014 Force feedback applied to GUI
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 10/147 Details of sensors, e.g. sensor lenses fingerprint or palmprint sensors G06V40/13; vascular sensors G06V40/145; eye sensors G06V40/19
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: G06V 40/107 Static hand or arm
• CPC Class Codes: H04N 23/12 with one sensor only
• CPC Class Codes: H04N 23/51 Housings
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 9/3179 Video signal processing therefor

Abstract

An integrated processing and projection device suitable for use on a supporting surface includes a processor and a projector designed to provide a display on the supporting surface proximate to the device. Various sensors enable object and gesture detection in a detection area in the display area. Trigger zones are defined in the detection area such that interaction of an object or human limb in the detection zone provides object and zone specific feedback by the integrated processing and projection device. The feedback can be provided in the projection area or may be provided as audible or active feedback to a device having active feedback capabilities.

Claims

1. A processor implemented method of displaying feedback related to real objects and trigger zones in a projected display area, the processor implemented method comprising:

rendering, via an interaction device, a display area comprising a user interface on a surface supporting the interaction device, the interaction device comprising the processor and a projector;

monitoring the display area utilizing sensors of the interaction device, each of the sensors having a field of view defining a detection area including at least the display area;

establishing one or more trigger zones in a corresponding one or more regions of the detection area, the one or more trigger zones comprising inputs to the user interface;

responsive to a real object entering at least a portion of a trigger zone of the one or more trigger zones detected by said monitoring, causing object feedback to emanate from the interaction device using one or more output elements of the interaction device, the object feedback including visible light display feedback projected onto the real object by the projector, the real object comprising at least an appendage a user;

detecting movement of the real object within the display area; and

responsive to detecting the movement of the real object within the display area, causing the visible light display feedback projected onto the real object to move with the real object.

2. The processor implemented method of claim 1 wherein the establishing comprises defining a two dimensional area of the display area and the trigger zone includes the two dimensional area and an area bounded by the two dimensional area having a height above the surface.

3. The processor implemented method of claim 1 wherein the establishing comprises defining a three dimensional area of the display area not including any portion of the surface.

4. The processor implemented method of claim 1, wherein the sensors include at least an RGB camera and an infrared detector, the infrared detector adjacent to the surface when the housing rests on the surface, and said monitoring comprises comparing image data from the RGB camera and the infrared detector to determine said interaction.

5. A computer storage device including code operable to instruct a processor to control an interaction device, the interaction device including the processor, a communication interface, and a projector provided within a housing that supports the interaction device on a surface, the code operable to:

using a first sensor and a second sensor, monitor the surface in a display area defined by the projector, each of the first sensor and the second sensor configured to provide images of a field of view comprising a detection area encompassing the display area;

establish one or more trigger zones in a corresponding one or more regions of the detection area, the one or more trigger zones comprising inputs to a user interface displayed on the surface via the projector;

cause object feedback relative to an active real object having controllable features to emanate from the interaction device using one or more output elements of the interaction device, the object feedback including visible light display feedback projected via the projector onto a portion of the display area where the active real object is located and not projected onto one or more portions of the display area where the active real object is not located; and

output a signal via the communication interface of the interaction device to activate a controllable feature of the active real object.

6. The computer storage device of claim 5, wherein the object feedback comprises a projection of a reaction to the active real object entering into the trigger zone based on a projected display area and related to object interaction with the projected display area.

7. The computer storage device of claim 5 wherein a trigger zone includes a three dimensional area of the display area attached to an object in the display area and movable with the object in the display area.

8. An integrated processing and projection system, comprising:

a display projector adapted to display a user interface in a display area on a surface supporting a housing of the system;

sensors each having a field of view encompassing a detection area including at least the display area, the sensors configured to monitor the detection area;

a processor; and

memory storing instructions executable by the processor to

render the user interface;

establish one or more trigger zones in a corresponding one or more regions of the detection area, the one or more trigger zones comprising inputs to the user interface;

detect, via the sensors, a real object entering at least a portion of a trigger zone of the one or more trigger zones, the real object comprising at least an appendage of a user;

in response to detecting the real object entering at least the portion of the trigger zone, output object feedback including visible light display feedback projected onto the real object by the display projector;

detect movement of the real object within the display area; and

in response to detecting the movement of the real object within the display area, cause the visible light display feedback projected onto the real object to move with the real object.

9. The integrated processing and projection system of claim 8, wherein the instructions are executable to establish the one or more trigger zones by defining a two-dimensional area of the display area, and establishing a trigger zone including the two-dimensional area and an area bonded by the two-dimensional area having a height above the surface.

10. The integrated processing and projection system of claim 8, wherein the instructions are executable to establish the one or more trigger zones by defining a three-dimensional area of the display area that does not include any portion of the surface.

11. The integrated processing and projection system of claim 8, wherein the sensors comprise at least a RGB camera and an infrared detector, the infrared detector being adjacent to the surface when the housing rests on the surface, and wherein the sensors are configured to monitor the detection area via image data obtained from the RGB camera and the infrared detector.