Pinch And Hold Gesture Navigation On A Head-Mounted Display

• US 20160306431A1
• Filed: 03/15/2016
• Published: 10/20/2016
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Computer Entertainment Incorporated (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Marks, Richard L.
• Granted Patent: US 10,156,908 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0185 Displaying image at variable distance
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A system for performing a pinch and hold gesture is described. The system includes a head-mounted display (HMD) and a glove, which is worn by a hand of the user. Each finger segment of the glove includes a sensor for detecting positions of the finger segment when moved by the hand. The system includes a computing device interfaced with the HMD and the glove. The computing device analyzes data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by at least two of the finger segments. Moreover, the computing device generates image data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location in the scene at which the pinch and hold gesture is associated.

Claims

1. A system including:

a head-mounted display (HMD) configured to be worn by a user;

a glove configured to be worn by a hand of the user, the glove having finger segments for each finger of the hand, each finger segment includes a sensor for detecting positions of the finger segment when moved by the hand;

one or more cameras for capturing images of the HMD and images of the glove;

a computing device connected to the one or more cameras and interfaced with the HMD and the glove,

wherein the computing device is configured to analyze data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by at least two of the finger segments,

wherein the computing device is configured for generating image data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location in the scene at which the pinch and hold gesture is associated,

wherein while the pinch and hold gesture is active, the computing device is configured for coupling movement of the glove with movement of the scene, such that the scene moves with detected movement of the glove.

2. The system of claim 1, wherein the computing device is configured to release the coupling between the movement of the glove and the movement of the scene when the pinch and hold gesture is determined to be deactivated.

3. The system of claim 1, wherein the computing device is configured to ignore an effect of movement of the HMD on the movement of the scene during a time period in which the pinch and hold gesture is active.

4. The system of claim 1, wherein the computing device is configured to determine whether a rotate gesture is performed by the glove while the pinch and hold gesture is active, wherein the computing device is configured to send data for rotating the scene in response to determining that the rotate gesture is performed.

5. The system of claim 1, wherein the computing device is configured to determine whether a slide gesture is performed by the glove while the pinch and hold gesture is active, wherein the computing device is configured to send data for translating the scene in response to determining that the slide gesture is performed, the translation of the scene rendering a non-displayed portion of the scene, the portion not rendered on the HMD before the translation motion occurs.

6. The system of claim 1, wherein while the pinch and hold gesture is active, the computing device is configured to ignore movement of a head of the user to avoid changing the scene according to the movement of the head.

7. The system of claim 1, wherein the computing device is configured to determine a position of each of the finger segments from the images of the glove, wherein the computing device is configured to determine from the positions of the finger segments that a portion of one of the at least two finger segments is in contact with a portion of another one of the at least two finger segments to determine that the pinch and hold gesture is performed by the at least two finger segments.

8. The system of claim 1, wherein the computing device is configured to send data for displaying an image of the hand of the user in the scene, wherein the visual cue is generated at the location in the scene at which the image of the hand performs the pinch and hold gesture.

9. A method for translating a viewable portion of a scene viewed via a head-mounted display (HMD), the method comprising:

capturing images of the HMD configured to be worn by a user in a real space to determine a view direction and changes in the view direction;

capturing images of a hand of the user;

tracking the hand of the user from the captured images of the hand;

activating a pinning of the scene as viewed via the HMD, such that the pinning fixes the scene to a location of the hand being tracked, the pinning being activated upon detecting a pinch and hold gesture performed by the hand;

enabling translation of the scene in response to movement of the hand while the pinning is active, the translation enabling a view beyond a current view of the scene;

disabling the translation when the pinning is detected to be inactive, the pinning is inactive when the pinch and hold gesture is no longer detected.

10. The method of claim 9, further comprising:

tracking the HMD from the captured images of the HMD; and

enabling a change in the viewable portion of the scene when the pinning is detected to be inactive and when movement of the HMD is tracked.

11. The method of claim 9, further comprising generating a cue within the viewable portion of the scene to indicate that the pinning is activated.

12. The method of claim 9, wherein activating the pinning of the scene is executed when the pinch and hold is determined to be performed for greater than a pre-determined amount of time.

13. The method of claim 9, further comprising:

receiving gesture data regarding an additional gesture performed by the user while the pinch and hold gesture is performed; and

modifying the viewable portion of the scene according to the gesture data.

14. The method of claim 9, further comprising:

capturing images of another hand of the user;

tracking the other hand of the user from the captured images of the other hand;

facilitating rotation of the scene when from tracking the other hand, it is determined that a rotate gesture is performed by the other hand of the user, wherein the rotation of the scene is facilitated while the scene is fixed to the location of the hand being tracked.

15. The method of claim 9, wherein the pinch and hold gesture is no longer detected when a thumb of the hand is not is contact with a finger of the hand.

16. A method for changing a size of a scene viewed via a head-mounted display (HMD), the method comprising:

capturing images of the HMD configured to be worn by a user in a real space to determine a view direction and changes in the view direction;

capturing images of hands of the user;

determining from the captured images that each hand performs a pinch and hold gesture;

fixing the scene to locations in the scene at which the captured images indicate the pinch and hold gestures are being performed;

enabling change in a size of the scene in response to relative changes in separation between the hands while the scene is fixed; and

disabling change in size of the scene when one of the pinch and hold gestures from one of the hands is detected from the captured images to have been released.

17. The method of claim 16, further comprising:

generating data for zooming-in the scene in response to determining that the separation is increasing; and

generating data for zooming-out the scene in response to determining that the separation is decreasing.

18. The method of claim 16, further comprising:

determining whether the hands move away or towards the HMD;

facilitating zooming-in of the scene upon determining that the hands move towards the HMD; and

facilitating zooming-out of the scene upon determining that the hands move away from the HMD.

19. The method of claim 16, further comprising:

determining whether the hands move away or towards the HMD in a direction substantially parallel to a width of a torso of the user;

facilitating zooming-out of the scene upon determining that the hands move towards the HMD in the direction;

facilitating zooming-in of the scene upon determining that the hands move away from the HMD in the direction.

20. The method of claim 16, wherein tracking the hands comprises tracking the hands to determine positions and orientations of the hands, the method further comprising:

analyzing the positions and orientations to determine whether the hands perform rotate gestures;

providing data for enabling tilting of objects in the scene in response to determining that the hands perform the rotate gestures.

21. The method of claim 16, further comprising:

determining from the captured images whether the separation between the hands is changing while the hands move in a direction substantially perpendicular to a width of a torso of the user;

identifying from the captured images a dominant motion between a motion in which the separation changes and the movement in the direction substantially perpendicular to the width of the torso of the user;

determining a submissive motion from the dominant motion; and

enabling change in the size of the scene according to the dominant motion while ignoring the submissive motion.

22. The method of claim 16, further comprising:

determining from the captured images whether the separation between the hands is changing while the hands rotate around an axis substantially parallel to a width of a torso of the user;

identifying from the captured images a dominant motion between a motion in which the separation changes and a motion in which the hands rotate;

determining a submissive motion from the dominant motion; and

enabling change in the scene according to the dominant motion while ignoring the submissive motion.

23. The method of claim 16, further comprising:

determining a first relative position between the HMD and the one of the hands;

determining a second relative position between the HMD and another one of the hands; and

sending for display image data for the hands to the HMD upon determining that the hands are within a field-of-view of the HMD, the determination that the hands are within the field-of-view is made from the first and second relative positions.