MULTI-MODAL GESTURE BASED INTERACTIVE SYSTEM AND METHOD USING ONE SINGLE SENSING SYSTEM

• US 20160364007A1
• Filed: 01/30/2015
• Published: 12/15/2016
• Est. Priority Date: 01/30/2015
• Status: Active Grant

• Current Assignee: Sony Depthsensing Solutions SA/NV (Sony Group Corp.)
• Original Assignee: SoftKinetic Software
• Inventors: KAMOVICH, ALIAKSANDR, THOLLOT, JULIEN, GUIGUES, LAURENT
• Granted Patent: US 10,534,436 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/042 by opto-electronic means
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text

Abstract

Described herein is a method and a system for providing efficient and complementary natural multi-modal gesture based interaction with a computerized system which displays visual feedback information on a graphical user interface on an interaction surface. The interaction surface is within the frustum of an imaging device comprising a single sensing system. The system uses the single sensing system for detecting both touch gesture interactions with the interaction surface (120) and three-dimensional touch-less gesture interactions in areas or volumes above the interaction surface performed by hands of a user. Both types of interaction are associated contextually with an interaction command controlling the computerized system when the gesture has been detected. The system comprises preferably a projection system for displaying the graphical user interface and visual feedback on the interaction surface, the projection system being locatable on the same side or on the opposite side of the interaction surface to the sensing system.

Claims

1. A method for multi-modal touch and touch-less interaction with a computerized system in which said multi-modal touch and touch-less interaction is performed using data information from a single sensing system, the single sensing system being a three-dimensional imaging device, the method comprising the steps of:

a) detecting and tracking at least one portion of at least one object within a frustum of the three-dimensional imaging device;

b) initiating the interaction by determining if said at least one portion of said at least one object being tracked is performing at least one of;

a predetermined touch gesture on a predetermined interactive area of an interaction surface and, a predetermined touch-less tree-dimensional gesture in a predetermined interactive volume on the vector axis normal to a predetermined interactive area;

c) interacting with the computerized system by detecting and recognizing the gestures performed by said at least one portion of said at least one object within the frustum of the three-dimensional imaging device, the detected and recognized gestures being at least one of;

a predetermined touch gesture on a predetermined interactive area of the interaction surface and a predetermined touch-less tree-dimensional gesture in a predetermined interactive volume on the vector axis normal to a predetermined interactive area.

2. A method according to claim 1, wherein detecting the performance of a touch gesture on a predetermined interactive area of the interaction surface corresponds to detecting when said at least one portion of said at least one object being tracked is positioned in three-dimensional space at the same location in space as the predetermined interactive area on the interaction surface.

3. A method according to claim 1, wherein detecting if a touch gesture has been performed is determined when the distance from the interaction surface of said at least one portion of said at least one object being tracked, in three-dimensional space, is below a predetermined threshold.

4. A method according to claim 1, wherein determining a multi-touch gesture is performed when the positions of at least two portions of said at least one object being tracked in space reach at least two predetermined interactive areas belonging to the interaction surface.

5. A method according to claim 1, further comprising the step of determining touch gesture interaction controls as a function of successive positions and duration of at least one of:

a touch gesture and a multi-touch gesture.

6. A method according to claim 1, further comprising the step of determining three-dimensional touch-less gesture interaction controls as function of the three-dimensional gesture performed by said at least one portion of said at least one object being tracked.

7. A method according to claim 1, wherein step c) further comprises the step of ending the interaction when a predetermined event is triggered, said predetermined event comprising at least one of:

the elapsing of a predetermined time period, the recognition of a predetermined touch gesture on the interaction surface, the recognition of a predetermined three-dimensional touch-less gesture in the frustum of the three-dimensional imaging device, and an exit of said at least one object from a predetermined volume in the three-dimensional space.

8. A method according to claim 7, wherein step b) comprises using a first detected and tracked portion of said at least one object, and step c) comprises using a second detected and tracked portion of said at least one object.

9. A method according to claim 7, wherein steps b) and c) are sequentially controlled using a single detected and tracked portion of one object.

10. A method according to claim 1, further comprises displaying a predetermined visual feedback on a graphical user interface on at least a portion of the interaction surface, the visual feedback being relative to at least one of:

the position of said at least one portion of said at least one object and the recognized gestures of said at least one portion of said at least one object being tracked.

11. A method according to claim 10, wherein the interaction with the graphical user interface displayed on the interaction surface further comprises operating the steps of:

d) determining the position in space and the topology of the interaction surface using the three-dimensional imaging device;

e) determining a predetermined set of interactive areas on the interaction surface as function of its topology and size; and

f) associating with each interactive area at least one touch gesture interaction control.

12. A method according to claim 11, further comprising the steps of:

g) associating with the predetermined set of interactive areas, at least one interactive volume, each interactive volume being located above said interactive area along a normal vector to that interactive area; and

h) associating to each interactive volume with a predetermined three-dimensional touch-less gesture interaction control.

13. A method according to claim 11, wherein the interaction surface is at least one of:

a portion of a body of a user, a desk table, a wall, an infrared translucent surface, and an object, the graphical user interface being projected onto the interaction surface.

17. A system according to claim 14, being configured for performing the method of claim 1.

18. A non-transitory computer medium configured for storing executable instructions, which when loaded on a computerised system, perform the method according to claim 1.

14. A system for interacting with a graphical user interface, the system comprising:

a display system for displaying the graphical user interface onto an interaction surface;

a three-dimensional imaging system being operated for at least tracking at least one portion of at least one hand of a user within its frustum; and

a computer system being configured for controlling the display system and the three-dimensional imaging device, and for determining gesture based interaction controls using data output from the three-dimensional imaging device;

the system being characterized in that the display surface for displaying the graphical user interface comprises at least a portion of the frustum of the three-dimensional imaging device and being substantially aligned therewith.

15. A system according to claim 14, wherein the display system comprises a projector element which is located on the same side of the interaction surface onto which the graphical user interface is displayed as the three-dimensional imaging device.

16. A system according to claim 14, wherein the three-dimensional imaging system and the projector element are located on opposite sides of the interaction surface onto which the graphical user interface is projected, the interaction surface being operable for diffusing radiation having a wavelength in a range which substantially corresponds to the visible part of the electromagnetic spectrum and for transmitting radiation having a wavelength in a range which substantially corresponds to the infrared part of the electromagnetic spectrum with limited diffusion, the transmission coefficient of the interaction surface being higher than 50% and the limited diffusion being lower than 20 degrees.