Vibrotactile system to augment 3D input systems

• US 9,798,388 B1
• Filed: 10/25/2013
• Issued: 10/24/2017
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: KAYA Dynamics LLC
• Original Assignee: Aquifi, Inc.
• Inventors: Murali, Giridhar
• Primary Examiner(s): Mandeville, Jason
• Application Number:
  US14/064,033
• Time in Patent Office: 1,460 Days
• Field of Search: 345156-158
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0487 using specific features provided by the input device, e.g. functions controlled by the rotation of a mouse with dual sensing arrangements, or of the nature of the input device, e.g. tap gestures based on pressure sensed by a digitiser

Abstract

A system and method for providing a 3D gesture based user interface with haptic feedback is disclosed. A processing system providing the 3D system provides haptic feedback by capturing image data. A gesture is then detected. The detected gesture is then used to determine an appropriate haptic feedback is determined. A signal that indicates the appropriate haptic feedback is generated and provided to a haptic feedback device. The haptic feedback devices then provides the appropriate haptic feedback.

Claims

1. An image processing system, configured to conduct three-dimensional (3D) gesture based interactive sessions, that provides haptic feedback comprising:

a memory containing an image processing application; and

a processor configured via the image processing application to;

receive captured image data and process the captured image data to detect a target object, the target object being a human hand, the captured image being captured by at least one camera,

detect, based upon locations and poses of the detected target object, a 3D initialization gesture,

define, based upon a detection of the initialization gesture, a 3D interaction zone having an interior and a boundary, the boundary of the 3D interaction zone being defined based on spatial regions occupied by the target object during the initialization gesture, the 3D interaction zone corresponding to a portion of the captured image data smaller than a field of view of the at least one camera,

detect, constrained to the portion of the captured image data corresponding to the interior of the 3D interaction zone, a 3D gesture comprising a plurality of 3D spatial positions of the target object in the 3D interaction zone from the portion of the captured image data,

determine a spatial position of the 3D spatial positions of the target object in the 3D interaction zone during the 3D gesture;

determine whether the target object has moved from a location in the interior of the 3D interaction zone to a location proximate the boundary of the 3D interaction zone during the 3D gesture from the spatial position of the 3D spatial positions of the target object in the 3D interaction zone, the portion of the captured image data corresponding to the 3D interaction zone being constant during the 3D gesture,

determine a haptic feedback response based on a determination that the target object is proximate the boundary of the 3D interaction zone,

generate a haptic response signal that indicates the determined haptic feedback response, and

provide the haptic response signal to a haptic feedback device, wherein the haptic feedback device is a vibrotactile ring comprising a plurality of vibrating motors and a controller is configured to receive the haptic response signal and generate signals that selectively drive the plurality of vibrating motors to generate the determined haptic feedback response.

2. The image processing system of claim 1 wherein the processor is further configured to define the 3D interaction zone based upon motions of the target object.

3. The image processing system of claim 1 wherein the processor is configured to define the 3D interaction zone based upon motions of the target object during the initialization gesture.

4. The image processing system of claim 1 wherein the processor is further configured to determine a size of the 3D interaction zone based upon a scale of at least one of a plurality of templates that matches a part of the target object visible in a sequence of frames of video data captured during detection of the initialization gesture and a distance of the part of the target object visible in the sequence of frames of video data from the at least one camera used to capture the sequence of frames of video data.

5. The image processing system of claim 1 wherein the processor is further configured to determine a size of the 3D interaction zone based upon a region in 3D space in which motion of the target object is observed during the initialization gesture.

6. The image processing system of claim 1 wherein the 3D interaction zone is a predetermined size determined based upon human physiology.

7. The image processing system of claim 1 wherein the 3D gesture detected is a pointing gesture that is interacting with a gesture responsive interface object.

8. The image processing system of claim 1 wherein the 3D gesture detected is a pointing gesture that is interacting with an affordance.

9. The image processing system of claim 1 wherein the processor is further configured to determine an interactive response to the detected 3D gesture and wherein a haptic feedback response is determined based upon the interactive response.

10. The image processing system of claim 9 wherein the interactive response to the detected 3D gesture is an interaction with a gesture reactive interface object.

11. The image processing system of claim 9 wherein the detected 3D gesture targets a gesture reactive interface object and the interactive response is to unlock a new set of 3D gestures.

12. The image processing system of claim 9 wherein the detected 3D gesture is a tapping motion and the interactive response is interacting with a gesture response interface object.

13. The image processing system of claim 9 wherein the detected 3D gesture is a pointing motion, the interactive response is a tap, and a determined haptic feedback is a specific vibration that simulates the sense of tapping on a surface.

14. The image processing system of claim 9 wherein the detected 3D gesture is a movement along a substantially vertical axis, the interactive response is determined to be a scrolling action and a haptic feedback response is a specific vibration that simulates a sense of moving a finger through an interior surface of a pipe.

15. The image processing system of claim 9 wherein the detected 3D gesture is a movement along a substantially horizontal axis, the interactive response is determined to be a scrubbing action and a haptic feedback response is a specific vibration that simulates a sense of rubbing a finger across a surface.

16. The image processing system of claim 1 wherein the processor is configured to provide the haptic response signal to the haptic feedback device by transmitting the haptic response signal to the haptic feedback device via a wireless connection.

17. The image processing system of claim 1 wherein the processor is configured to provide the haptic response signal to the haptic feedback device via a haptic feedback device interface.

18. The image processing system of claim 1 wherein

the plurality of vibrating motors are situated around a circumference of the vibrotactile ring;

and

the controller is configured to receive the haptic response signal and generate signals that selectively drive each of the plurality of vibrating motors to generate the determined haptic response.

19. The image processing system of claim 1 wherein the controller generates specific signals to drive each of the plurality of vibrating motors to provide a specific vibration pattern as the haptic response.

20. The image processing system of claim 1 wherein the processor is configured to provide the haptic response signal via a wireless connection to the vibrotactile ring and the vibrotactile ring further comprises a wireless receiver that receives the haptic response signal via the wireless connection and provides the haptic response signal to the controller.

21. The image processing system of claim 1 wherein the processor is configured to detect a hand pose associated with the 3D gesture in the captured image data and determine the orientation of the vibrotactile ring relative to a 3D interaction zone based upon the hand pose, and generate a haptic response signal based upon the orientation of the vibrotactile ring relative to the 3D interaction zone.

22. The image processing system of claim 1, wherein the haptic feedback response indicates the direction in which the boundary is located relative to the target object.

23. A method for providing haptic feedback for three-dimensional (3D) gesture based interactive sessions that provides haptic feedback comprising:

receiving captured image data and processing the captured image data to detect a target object in an image processing system, the target object being a human hand, the captured image being captured by at least one camera;

detecting, based upon locations and poses of the detected target object, a 3D initialization gesture;

defining, based upon a detection of the initialization gesture, a 3D interaction zone having an interior and a boundary, the boundary of the 3D interaction zone being defined based on spatial regions occupied by the target object during the initialization gesture, the 3D interaction zone corresponding to a portion of the captured image data smaller than a field of view of the at least one camera;

detecting a 3D gesture comprising a plurality of 3D spatial positions of the target object in the 3D interaction zone from the portion of the captured image data using the image processing system, the detecting being constrained to the portion of the captured image data corresponding to the interior of the 3D interaction zone;

determining a spatial position of the 3D spatial positions of the target object in the 3D interaction zone during the 3D gesture;

determining whether the target object has moved from a location in the interior of the 3D interaction zone to a location proximate the boundary of the 3D interaction zone during the 3D gesture from the spatial position of the 3D spatial positions of the target object in the 3D interaction zone, the portion of the captured image data corresponding to the 3D interaction zone being constant during the 3D gesture;

determining a haptic feedback response based on a determination that the target object is proximate the boundary of the 3D interaction zone;

generating a haptic response signal that indicates the determined haptic feedback response using the image processing system; and

providing the haptic response signal from the image processing system to a haptic feedback device, wherein the haptic feedback device is a vibrotactile ring comprising a plurality of vibrating motors and a controller is configured to receive the haptic response signal and generate signals that selectively drive the plurality of vibrating motors to generate the determined haptic feedback response.

24. The method of claim 23, wherein the haptic feedback response indicates the direction in which the boundary is located relative to the target object.

25. An image processing system, configured to conduct three-dimensional (3D) gesture based interactive sessions, that provides haptic feedback comprising:

a memory containing an image processing application; and

a processor configured via the image processing application to;

receive captured image data and process the captured image data to detect a target object, the target object being a human hand, the captured image being captured by at least one camera,

detect, based upon locations and poses of the detected target object, a 3D initialization gesture,

define, based upon a detection of the initialization gesture, a 3D interaction zone having an interior and a boundary, the boundary of the 3D interaction zone being defined based on spatial regions occupied by the target object during the initialization gesture, the 3D interaction zone corresponding to a portion of the captured image data smaller than a field of view of the at least one camera,

detect, constrained to the portion of the captured image data corresponding to the interior of the 3D interaction zone, a 3D gesture comprising a plurality of 3D spatial positions of the target object in the 3D interaction zone from the portion of the captured image data,

determine a spatial position of the 3D spatial positions of the target object in the 3D interaction zone during the 3D gesture;

determine whether the target object has moved from a location in the interior of the 3D interaction zone to a location proximate the boundary of the 3D interaction zone during the 3D gesture from the spatial position of the 3D spatial positions of the target object in the 3D interaction zone, the portion of the captured image data corresponding to the 3D interaction zone being constant during the 3D gesture,

determine a haptic feedback response based on a determination that the target object is proximate the boundary of the 3D interaction zone,

generate a haptic response signal that indicates the determined haptic feedback response, and

provide the haptic response signal to a haptic feedback device, wherein the haptic feedback device is embedded in a steering wheel of an automobile comprising a plurality of vibrating motors and a controller is configured to receive the haptic response signal and generate signals that selectively drive the plurality of vibrating motors to generate the determined haptic feedback response.