Pinch and hold gesture navigation on a head-mounted display

• US 10,156,908 B2
• Filed: 03/15/2016
• Issued: 12/18/2018
• Est. Priority Date: 04/15/2015
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Stafford, Jeffrey Roger, Marks, Richard L.
• Primary Examiner(s): Edwards, Mark
• Application Number:
  US15/071,127
• Publication Number:
  US 20160306431A1
• Time in Patent Office: 1,008 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0147 comprising a device modifying the resolution of the displayed image
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0185 Displaying image at variable distance
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G02B 27/0179 Display position adjusting means not related to the information to be displayed
• CPC Class Codes: G06F 2203/04806 Zoom, i.e. interaction techniques or interactors for controlling the zooming operation
• CPC Class Codes: G06F 3/005 Input arrangements through a video camera
• CPC Class Codes: G06F 3/014 Hand-worn input/output arrangements, e.g. data gloves
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0485 Scrolling or panning

Abstract

A system for performing a pinch and hold gesture is described. The system includes a head-mounted display (HMD) and a glove, which is worn by a hand of the user. Each finger segment of the glove includes a sensor for detecting positions of the finger segment when moved by the hand. The system includes a computing device interfaced with the HMD and the glove. The computing device analyzes data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by at least two of the finger segments. Moreover, the computing device generates image data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location in the scene at which the pinch and hold gesture is associated.

Claims

1. A system including:

a head-mounted display (HMD) configured to be worn by a user;

a glove configured to be worn by a hand of the user, the glove having finger segments for each finger of the hand, each finger segment includes a sensor for detecting positions of the finger segment when moved;

one or more cameras for capturing images of the HMD and images of the glove;

a computing device connected to the one or more cameras and interfaced with the HMD and the glove,

wherein the computing device is configured to analyze data from the sensors of the finger segments to determine that a pinch and hold gesture is performed by two of the finger segments,

wherein the computing device is configured to generate visual cue data that is communicated to the HMD, such that a scene rendered on the HMD is modified to render a visual cue indicative of a location associated with the pinch and hold gesture, wherein the location is within the scene, wherein the scene includes a virtual object and a background,

wherein while the pinch and hold gesture is active, the computing device is configured for coupling movement of the glove with movement of the virtual object and background, such that the virtual object and background move together with movement of the glove,

wherein during a time period in which the pinch and hold gesture is active, the computing device does not move the scene with any movement of the HMD.

2. The system of claim 1, wherein the computing device is configured to release the coupling between the movement of the glove and the movement of the scene when the pinch and hold gesture is determined to be deactivated.

3. The system of claim 1, wherein the computing device is configured to determine whether a rotate gesture is performed by the glove while the pinch and hold gesture is active, wherein the computing device is configured to send data for rotating the scene in response to determining that the rotate gesture is performed.

4. The system of claim 1, wherein the computing device is configured to determine whether a slide gesture is performed by the glove while the pinch and hold gesture is active, wherein the computing device is configured to send data for translating the scene in response to determining that the slide gesture is performed, the translation of the scene rendering a non-displayed portion of the scene, the non-displayed portion not rendered on the HMD before the translation motion occurs.

5. The system of claim 1, wherein the computing device is configured to determine a position of each of the finger segments from the images of the glove, wherein the computing device is configured to determine from the positions of the finger segments that a portion of one of the two finger segments is in contact with a portion of another one of the two finger segments to determine that the pinch and hold gesture is performed by the two finger segments.

6. The system of claim 1, wherein the computing device is configured to send data for displaying an image of the hand of the user in the scene, wherein the visual cue is generated at the location in the scene at which the image of the hand performs the pinch and hold gesture.

22. The system of claim 1, wherein the virtual object and the background move together to include an additional virtual object within the scene, wherein the additional virtual object is not present within the scene before the movement of the virtual object and the background.

23. The system of claim 1, wherein the captured images of the HMD include image data regarding the movement of the HMD, wherein the computing device is configured to receive the image data regarding the movement of the HMD, wherein the computing device is configured to determine that the movement of the HMD occurred during the time period in which the pinch and hold gesture is active, wherein the computing device is configured to ignore the image data regarding the movement of the HMD upon determining that the movement of the HMD occurred during the time period in which the pinch and hold gesture is active.

7. A method for translating a viewable portion of a scene viewed via a head-mounted display (HMD), the method comprising:

capturing images of the HMD configured to be worn by a user in a real space to determine a view direction and changes in the view direction;

capturing images of a hand of the user;

tracking the hand of the user from the captured images of the hand;

activating a pinning of the scene as viewed via the HMD, such that the pinning fixes the scene to a location based on the hand being tracked, the pinning being activated upon detecting a pinch and hold gesture performed by the hand;

enabling a translation of the scene in response to movement of the hand while the pinning is active, the translation enabling a view beyond a current view of the scene to include a virtual object absent from the current view,

wherein during a time period in which the pinch and hold gesture is being performed, the scene does not move with any movement of the HMD;

disabling a translation of the scene when the pinning is detected to be inactive, the pinning is inactive when the pinch and hold gesture is no longer detected.

8. The method of claim 7, further comprising:

tracking the HMD from the captured images of the HMD; and

enabling a change in the viewable portion of the scene when the pinning is detected to be inactive and when movement of the HMD is tracked.

9. The method of claim 7, further comprising generating a cue within the viewable portion of the scene to indicate that the pinning is activated.

10. The method of claim 7, wherein said activating the pinning of the scene is executed when the pinch and hold is determined to be performed for greater than a pre-determined amount of time.

11. The method of claim 7, further comprising:

receiving gesture data regarding an additional gesture performed by the user while the pinch and hold gesture is performed; and

modifying the viewable portion of the scene according to the gesture data.

12. The method of claim 7, further comprising:

capturing images of another hand of the user;

tracking the other hand of the user from the captured images of the other hand;

facilitating rotation of the scene when from tracking the other hand, it is determined that a rotate gesture is performed by the other hand of the user, wherein the rotation of the scene is facilitated while the scene is fixed to the location.

13. The method of claim 7, wherein the pinch and hold gesture is no longer detected when a thumb of the hand is not is contact with a finger of the hand.

24. The method of claim 7, wherein the captured images of the HMD include image data regarding the movement of the HMD, the method further comprising:

receiving the image data regarding the movement of the HMD;

determining from the image data regarding the movement of the HMD that the movement of the HMD occurred during the time period in which the pinch and hold gesture is active;

ignoring the image data regarding the movement of the HMD upon determining that the movement of the HMD occurred during the time period in which the pinch and hold gesture is active.

14. A method for changing a size of a scene viewed via a head-mounted display (HMD), the method comprising:

capturing images of the HMD configured to be worn by a user in a real space to determine a view direction and changes in the view direction;

capturing images of hands of the user;

determining from the captured images of the hands that each hand performs a pinch and hold gesture;

fixing the scene to locations in the scene at which the captured images of the hands indicate the pinch and hold gestures are being performed;

enabling change in a size of the scene in response to relative changes in separation between the hands while the scene is fixed,

wherein during a time period in which the pinch and hold gesture is being performed, the scene does not move with any movement of the HMD; and

disabling change in the size of the scene when one of the pinch and hold gestures from one of the hands is detected from the captured images of the hands to have been released.

15. The method of claim 14, further comprising:

generating data for zooming-in the scene in response to determining that the separation is increasing; and

generating data for zooming-out the scene in response to determining that the separation is decreasing.

16. The method of claim 14, further comprising:

determining whether the hands move away or towards the HMD;

facilitating zooming-in of the scene upon determining that the hands move towards the HMD; and

facilitating zooming-out of the scene upon determining that the hands move away from the HMD.

17. The method of claim 14, further comprising:

determining whether the hands move away or towards the HMD in a direction substantially parallel to a width of a torso of the user;

facilitating zooming-out of the scene upon determining that the hands move towards the HMD in the direction;

facilitating zooming-in of the scene upon determining that the hands move away from the HMD in the direction.

18. The method of claim 14, wherein the images of the hands are captured to determine positions and orientations of the hands, the method further comprising:

analyzing the positions and orientations to determine whether the hands perform rotate gestures;

providing data for enabling tilting of an object in the scene in response to determining that the hands perform the rotate gestures.

19. The method of claim 14, further comprising:

determining from the captured images of the hands whether the separation between the hands is changing while the hands move in a direction substantially perpendicular to a width of a torso of the user;

identifying from the captured images of the hands a dominant motion between a motion in which the separation changes and the movement in the direction substantially perpendicular to the width of the torso of the user;

determining a submissive motion from the dominant motion; and

enabling change in the size of the scene according to the dominant motion while ignoring the submissive motion.

20. The method of claim 14, further comprising:

determining from the captured images of the hands whether the separation between the hands is changing while the hands rotate around an axis substantially parallel to a width of a torso of the user;

identifying from the captured images of the hands a dominant motion between a motion in which the separation changes and a motion in which the hands rotate;

determining a submissive motion from the dominant motion; and

enabling change in the scene according to the dominant motion while ignoring the submissive motion.

21. The method of claim 14, further comprising:

determining a first relative position between the HMD and the one of the hands;

determining a second relative position between the HMD and another one of the hands; and

sending for display image data for the hands to the HMD upon determining that the hands are within a field-of-view of the HMD, the determination that the hands are within the field-of-view is made from the first and second relative positions.

25. The method of claim 14, wherein the captured images of the HMD include image data regarding the movement of the HMD, the method further comprising:

receiving the image data regarding the movement of the HMD;

determining from the image data regarding the movement of the HMD that the movement of the HMD occurred during the time period in which the pinch and hold gesture is active;

ignoring the image data regarding the movement of the HMD upon determining that the movement of the HMD occurred during the time period in which the pinch and hold gesture is active.