Systems and methods for remapping three-dimensional gestures onto a finite-size two-dimensional surface

• US 9,619,042 B2
• Filed: 09/30/2016
• Issued: 04/11/2017
• Est. Priority Date: 05/05/2014
• Status: Expired due to Fees

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Dal Mutto, Carlo, Pasqualotto, Giuliano, Murali, Giridhar, Stoppa, Michele, Khalili, Amir hossein, Kamal, Ahmed Tashrif, Hummel, Britta
• Primary Examiner(s): Nguyen, Kevin M
• Application Number:
  US15/282,219
• Publication Number:
  US 20170017307A1
• Time in Patent Office: 193 Days
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method for operating a real-time gesture based interactive system includes: obtaining a sequence of frames of data from an acquisition system; comparing successive frames of the data for portions that change between frames; determining whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data; defining a 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data; tracking a movement of the interaction medium to generate a plurality of 3D positions of the interaction medium; detecting movement of the interaction medium from inside to outside the 3D interaction zone at a boundary 3D position; shifting the 3D interaction zone relative to the boundary 3D position; computing a plurality of computed positions based on the 3D positions; and supplying the computed positions to control an application.

Claims

1. A real-time gesture based interactive system comprising:

a processor;

an acquisition system configured to capture a sequence of frames of data for constructing a depth map of a field of view of the acquisition system;

memory storing an interaction medium tracking application, the interaction medium tracking application configuring the processor to;

obtain the data from the acquisition system;

compare successive frames of the frames of data for portions that change from one frame to the next;

determine whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data;

define an inner 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data, where the inner 3D interaction zone corresponds to a bounded region that is less than the frame of data and that contains the part of the interaction medium detected in the sequence of frames of data;

track a movement of the interaction medium in the data to generate a plurality of 3D positions of the interaction medium;

detect movement of the interaction medium from inside to outside the inner 3D interaction zone at a boundary 3D position;

shift the inner 3D interaction zone relative to the boundary 3D position;

compute a plurality of computed positions based on the 3D positions; and

supply the computed positions to control an application.

2. The system of claim 1, wherein each of the computed positions includes an x component, a y component, and a z component.

3. The system of claim 1, wherein the acquisition system comprises at least one of:

a plurality of cameras in a stereo arrangement having overlapping fields of view;

an infrared camera;

a color visible light camera; and

an illumination source configured to generate at least one of visible light, infrared light, ultrasonic waves, and electromagnetic signals.

4. The system of claim 1, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to:

compute a 3D velocity of the interaction medium within the inner 3D interaction zone based on the 3D positions; and

compute two-dimensional movement data corresponding to the 3D positions and the 3D velocity, differences in the two-dimensional movement data being non-linear with respect to differences in the 3D positions.

5. The system of claim 1, wherein the interaction medium is a portion of a human body.

6. The system of claim 1, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to detect the movement of the interaction medium from the inside to the outside of the inner 3D interaction zone based on the 3D positions.

7. The system of claim 1, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to detect the movement of the interaction medium from the inside to the outside of the inner 3D interaction zone by detecting a coarse movement of the interaction medium within an entire portion of the field of view of the acquisition system.

8. The system of claim 1, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to:

detect a disengagement event; and

stop tracking the movement of the interaction medium in response to the disengagement event.

9. The system of claim 8, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to detect the disengagement event by:

comparing the interaction medium detected in the frames of data to a target configuration to generate a compatibility confidence level; and

detecting the disengagement event when the confidence level is below a threshold level.

10. The system of claim 8, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to detect the disengagement event by:

comparing the interaction medium detected in the frames of data to a disengagement configuration; and

detecting the disengagement event when the disengagement configuration is detected.

11. The system of claim 1, wherein the memory further stores instructions of the interaction medium tracking application to configure the processor to:

track the movement of the interaction medium along a first direction toward a boundary of the inner interaction zone and concurrently update the plurality of computed positions in accordance with the movement along the first direction;

shift the inner interaction zone along the first direction and concurrently stop updating the plurality of computed positions in accordance with movement along the first direction; and

track the movement of the interaction medium along a second direction opposite the first direction and concurrently start updating the plurality of computed positions in accordance with the movement along the second direction.

12. A real-time gesture based interactive system comprising:

a processor;

an acquisition system configured to capture a sequence of frames of data for constructing a depth map of a field of view of the acquisition system;

memory storing an interaction medium tracking application, the interaction medium tracking application configuring the processor to;

obtain the data from the acquisition system;

compare successive frames of the frames of data for portions that change from one frame to the next;

determine whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data;

define an inner 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data, where the inner 3D interaction zone corresponds to a bounded region that is less than the frame of data and that contains the part of the interaction medium detected in the sequence of frames of data;

track a movement of the interaction medium in the data to generate a plurality of 3D positions of the interaction medium;

compute a 3D velocity of the interaction medium within the inner 3D interaction zone based on the 3D positions;

compute a plurality of computed positions based on the 3D positions and the 3D velocity, differences in the computed positions being non-linear with respect to differences in the 3D positions; and

supply the computed positions to control an application.

13. The system of claim 12, wherein each of the computed positions includes an x component, a y component, and a z component.

14. The system of claim 12, wherein the 3D velocity comprises a horizontal component v_x and a vertical component v_y and wherein the 3D velocity is thresholded by a minimum threshold T_m and a maximum threshold T_M to compute thresholded 3D velocities a and b, where:

a=min(max(|v_x^t|,T_m),T_M)

b=min(max(|v_y^t|,T_m),T_M).

15. The system of claim 14, wherein the real-time gesture based interactive system further comprises a display device having S_w columns and S_h rows,

wherein the plurality of computed positions comprises column data c and row data r corresponding to coordinates of the display device, and

wherein the 3D velocity and the thresholded 3D velocities are mapped onto the column data c and the row data r, where:

c_t=min(max(c_t-1−

v_x^t*K*a,0),S_w)

r_t=min(max(r_t-1−

v_y^t*K*b,0),S_h),

and where K is a sensitivity parameter.

16. A method for operating a real-time gesture based interactive system, the method comprising:

obtaining a sequence of frames of data from an acquisition system configured to capture data for constructing a depth map of a field of view of the acquisition system;

comparing, by a processor, successive frames the frames of data for portions that change from one frame to the next;

determining, by the processor, whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data;

defining, by the processor, an inner 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data, where the inner 3D interaction zone corresponds to a bounded region that is less than the frame of data and that contains the part of the interaction medium detected in the sequence of frames of data;

tracking, by the processor, a movement of the interaction medium in the data to generate a plurality of 3D positions of the interaction medium;

detecting, by the processor, movement of the interaction medium from inside to outside the inner 3D interaction zone at a boundary 3D position;

shifting, by the processor, the inner 3D interaction zone relative to the boundary 3D position;

computing, by the processor, a plurality of computed positions based on the 3D positions; and

supplying the computed positions to control an application.

17. The method of claim 16, wherein each of the computed positions includes an x component, a y component, and a z component.

18. The method of claim 16, wherein the acquisition system comprises at least one of:

a plurality of cameras in a stereo arrangement having overlapping fields of view;

an infrared camera;

a color visible light camera; and

an illumination source configured to generate at least one of visible light, infrared light, ultrasonic waves, and electromagnetic signals.

19. The method of claim 16, further comprising:

computing a 3D velocity of the interaction medium within the inner 3D interaction zone based on the 3D positions; and

computing two-dimensional movement data corresponding to the 3D positions and the 3D velocity, differences in the two-dimensional movement data being non-linear with respect to differences in the 3D positions.

20. The method of claim 16, wherein the interaction medium is a portion of a human body.

21. The method of claim 16, further comprising detecting the movement of the interaction medium from the inside to the outside of the inner 3D interaction zone based on the 3D positions.

22. The method of claim 16, further comprising detecting the movement of the interaction medium from the inside to the outside of the inner 3D interaction zone by detecting a coarse movement of the interaction medium within an entire portion of the field of view of the acquisition system.

23. The method of claim 16, further comprising:

detecting a disengagement event; and

stopping tracking the movement of the interaction medium in response to the disengagement event.

24. The method of claim 23, wherein the detecting the disengagement event comprises:

comparing the interaction medium detected in the frames of data to a target configuration to generate a compatibility confidence level; and

detecting the disengagement event when the confidence level is below a threshold level.

25. The method of claim 23, wherein the detecting the disengagement event comprises:

comparing the interaction medium detected in the frames of data to a disengagement configuration; and

detecting the disengagement event when the disengagement configuration is detected.

26. The method of claim 16, further comprising:

tracking the movement of the interaction medium along a first direction toward a boundary of the inner interaction zone and concurrently update the plurality of computed positions in accordance with the movement along the first direction;

shifting the inner interaction zone along the first direction and concurrently stop updating the plurality of computed positions in accordance with movement along the first direction; and

tracking the movement of the interaction medium along a second direction opposite the first direction and concurrently start updating the plurality of computed positions in accordance with the movement along the second direction.

27. A method for tracking a gesture comprising:

obtaining a sequence of frames of data from an acquisition system configured to capture data for constructing a depth map of a field of view of the acquisition system;

comparing, by a processor, successive frames of the frames of data for portions that change from one frame to the next;

determining, by the processor, whether any of the portions that changed are part of an interaction medium detected in the sequence of frames of data;

defining, by the processor, an inner 3D interaction zone relative to an initial position of the part of the interaction medium detected in the sequence of frames of data, where the inner 3D interaction zone corresponds to a bounded region that is less than the frame of data and that contains the part of the interaction medium detected in the sequence of frames of data;

tracking, by the processor, a movement of the interaction medium in the data to generate a plurality of 3D positions of the interaction medium;

computing, by the processor, a 3D velocity of the interaction medium within the inner 3D interaction zone based on the 3D positions;

computing, by the processor, a plurality of computed positions based on the 3D positions and the 3D velocity, differences in the computed positions being non-linear with respect to differences in the 3D positions; and

supplying, by the processor, the computed positions to control an application.

28. The method of claim 27, wherein each of the computed positions includes an x component, a y component, and a z component.

29. The method of claim 27, wherein the 3D velocity comprises a horizontal component v_x and a vertical component v_y and wherein the 3D velocity is thresholded by a minimum threshold T_m and a maximum threshold T_M to compute thresholded 3D velocities a and b, where:

a=min(max(|v_x^t|,T_m),T_M)

b=min(max(|v_y^t|,T_m),T_M).

30. The method of claim 29, wherein the application includes a user interface displayed on a display device having S_w columns and S_h rows,

wherein the plurality of computed positions comprises column data c and row data r corresponding to coordinates of the display device, and

wherein the 3D velocity and the thresholded 3D velocities are mapped onto the column data c and the row data r, where:

c_t=min(max(c_t-1−

v_x^t*K*a,0),S_w)

r_t=min(max(r_t-1−

v_y^t*K*b,0),S_h),

and where K is a sensitivity parameter.