Information processing apparatus and information processing method

• US 10,514,767 B2
• Filed: 01/24/2014
• Issued: 12/24/2019
• Est. Priority Date: 04/02/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.)
• Inventors: Ishiwata, Katsutoshi, Komori, Akihiro
• Primary Examiner(s): Tran, Tam T
• Application Number:
  US14/776,906
• Publication Number:
  US 20160041619A1
• Time in Patent Office: 2,160 Days
• Field of Search: None
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/01 Input arrangements or combined input and output arrangements for interaction between user and computer G06F3/16 takes precedence
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04812 Interaction techniques based on cursor appearance or behaviour, e.g. being affected by the presence of displayed objects
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Provided is an information processing apparatus including: a hand-shape recognition unit configured to recognize a hand shape of an operator on a basis of measured information related to a hand of the operator; and a display control unit configured to move a display position of an operation object on a display screen according to a change in a position of the hand of the operator when the hand shape of the operator is a predetermined shape.

Claims

1. A head mounted display, comprising:

at least one sensor configured to obtain at least one of depth information and a captured image of an operator;

a display device configured to;

display a first operation object at a first display position, wherein the first operation object is an operation surface that includes a plurality of icons; and

display a second operation object at a second display position that corresponds to an operator'"'"'s point of view; and

at least one processor configured to;

recognize a shape of an operator'"'"'s hand, based on the at least one of the depth information or the captured image; and

control the display device to;

move, when the operator'"'"'s hand is closed and has a holding shape, the first operation object from the first display position to a third display position in a depth direction based on a first movement to change a position of the operator'"'"'s hand in the depth direction,

wherein at the third display position, a fourth display position of a first icon of the plurality of icons corresponds to the second display position of the second operation object; and

display the first icon.

2. The head mounted display according to claim 1, wherein the at least one processor is further configured to:

keep the first operation object displayed until a second movement of the operator'"'"'s hand is recognized, wherein the second movement of the operator'"'"'s hand includes a change in a direction of an operator'"'"'s finger; and

delete the first operation object based on the second movement,

wherein the second movement of the operator'"'"'s hand further includes rotation of the operator'"'"'s hand; and

control, based on a recognition of the rotation of the operator'"'"'s hand in a first direction, the display device to display the first operation object at a depth position that corresponds to a depth position of the operator'"'"'s hand after the rotation.

3. The head mounted display according to claim 2,

wherein the at least one processor is further configured to control the display device to delete the first operation object, based on a recognition of rotation of the operator'"'"'s hand in a second direction opposite to the first direction.

4. The head mounted display according to claim 1, wherein the at least one processor is further configured to:

recognize an operator'"'"'s first hand and an operator'"'"'s second hand simultaneously;

move the first display position of the first operation object based on a position of the operator'"'"'s first hand in the depth direction, wherein the first display position is moved based on a recognition of the holding shape of the operator'"'"'s first hand; and

determine a tap operation on the first operation object, corresponding to a position of a finger of the operator'"'"'s second hand, wherein the tap operation is determined based on a determination that a shape of the operator'"'"'s second hand is a finger pointing shape.

5. The head mounted display according to claim 4, wherein the at least one processor is further configured to determine the tap operation on the first operation object, based on a determination that the position of the finger of the operator'"'"'s second hand is in front of the position of the operator'"'"'s first hand in the depth direction.

6. The head mounted display according to claim 5, wherein the at least one processor is further configured to

set a ratio of a movement amount of the first operation object to a movement amount of the operator'"'"'s first hand to be different from a ratio of the movement amount of the first operation object to a movement amount of the operator'"'"'s second hand.

7. The head mounted display according to claim 5, wherein the at least one processor is further configured to change a ratio of a movement amount of the first operation object to a movement amount of the operator'"'"'s first hand, based on a distance between the operator'"'"'s first hand and the operator'"'"'s second hand in the depth direction.

8. The head mounted display according to claim 7, wherein the at least one processor is further configured to decrease the ratio of the movement amount of the first operation object to the movement amount of the operator'"'"'s first hand, based on a decrease in the distance between the operator'"'"'s first hand and the operator'"'"'s second hand in the depth direction.

9. The head mounted display according to claim 5, wherein the at least one processor is further configured to prevent the first operation object from moving, based on a movement amount of the operator'"'"'s first hand while the operator'"'"'s second hand has contact with the first operation object.

10. The head mounted display according to claim 1, wherein

the first operation object includes the first icon and a second icon having a display content different from that of the first icon, and

when the operator'"'"'s hand is open, the at least one processor is further configured to control the display device to delete the first icon and keep the second icon displayed until a second movement of the operator'"'"'s hand is recognized.

11. The head mounted display according to claim 1, wherein

the second operation object corresponds a focus frame, and

the at least one processor is further configured to:

control the display device to display the focus frame at the second display position corresponding to the operator'"'"'s point of view; and

start an application corresponding to the first icon based on a determination that the focus frame is on the first icon for a threshold time period.

12. The head mounted display according to claim 1, wherein

the second operation object corresponds a focus frame, and

the at least one processor is further configured to control the display device to:

display the focus frame on the first operation object,

wherein a position of the focus frame on the first operation object corresponds to the operator'"'"'s point of view; and

select the first icon of the plurality of icons, based on the first movement of the first operation object such that the first icon is inside the focus frame after the first movement of the first operation object.

13. The head mounted display according to claim 12, wherein the second display position of the focus frame is fixed.

14. The head mounted display according to claim 1, wherein the second operation object is placed at a height same as a height of operator'"'"'s eyes.

15. An information processing method, comprising:

in a head mounted display;

obtaining at least one of depth information and a captured image of an operator;

displaying on a display device, a first operation object at a first display position, wherein the first operation object is an operation surface that includes a plurality of icons; and

displaying on the display device, a second operation object at a second display position that corresponds to an operator'"'"'s point of view;

recognizing, a shape of an operator'"'"'s hand, based on the at least one of the depth information or the captured image; and

moving, on the display device, when the operator'"'"'s hand is closed and has a holding shape, the first operation object from the first display position to a third display position in a depth direction based on a first movement to change a position of the operator'"'"'s hand in the depth direction, wherein at the third display position, a fourth display position of a first icon of the plurality of icons corresponds to the second display position of the second operation object; and

displaying the first icon.

16. A non-transitory computer-readable medium having stored thereon, computer-executable instructions which when executed by at least one processor of a head mounted display, cause the at least one processor to execute operations, the operations comprising:

obtaining, from at least one sensor, at least one of depth information and a captured image of an operator;

displaying, on a display device, a first operation object at a first display position, wherein the first operation object is an operation surface that includes a plurality of icons; and

displaying on the display device, a second operation object at a second display position that corresponds to an operator'"'"'s point of view;

recognizing a shape of an operator'"'"'s hand, based on a basis of the at least one of the depth information or the captured image;

moving, on the display device, when the operator'"'"'s hand is closed and has a holding shape, the first operation object from the first display position to a third display position in a depth direction based on a first movement to change a position of the operator'"'"'s hand in the depth direction, wherein at the third display position, a fourth display position of a first icon of the plurality of icons corresponds to the second display position of the second operation object; and

displaying the first icon.