Instruction input device, instruction input method, program, recording medium, and integrated circuit

• US 8,933,886 B2
• Filed: 06/16/2011
• Issued: 01/13/2015
• Est. Priority Date: 06/17/2010
• Status: Active Grant

• Current Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Original Assignee: Panasonic Intellectual Property Corporation of America (Panasonic Holdings Corporation)
• Inventors: Imoto, Hiroyasu, Maeda, Shigenori
• Primary Examiner(s): Awad, Amr
• Assistant Examiner(s): WILLIS, RANDAL L
• Application Number:
  US13/634,183
• Publication Number:
  US 20130002551A1
• Time in Patent Office: 1,307 Days
• US Class Current: 345/158
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors

Abstract

An instruction input device includes: a first direction detection unit detecting the first direction in which the user is looking; a second direction detection unit detecting a second direction in which the user is performing a pointing operation; a gaze position calculation unit calculating a gaze position of the user on the screen; a reference coordinate group calculation unit calculating a reference line in space corresponding to the gaze position and connecting the user and the screen; an offset amount calculation unit calculating a distance of the second direction between the reference line and the input coordinate indicating the user'"'"'s hand as an offset amount with respect to the gaze position; and a pointer display position calculation unit calculating a position in which a distance of the first screen predetermined direction between the position and the gaze position is the offset amount on the screen.

Claims

1. An instruction input device which calculates a display position of a pointer to be displayed on a two-dimensional screen, based on a pointing operation of movement of a hand of a user in a three-dimensional space, the instruction input device comprising:

a user position detection unit configured to detect a three-dimensional position of the user;

an input coordinate detection unit configured to detect, as a three-dimensional input coordinate, a coordinate indicating a three-dimensional position of the hand of the user;

a first direction detection unit configured to detect, based on a body posture of the user, a first direction which is determined with respect to a screen vertical direction and is a direction in which the user is looking;

a second direction detection unit configured to detect, based on the body posture of the user, a second direction which is determined with respect to a screen horizontal direction and is a direction in which the user is performing the pointing operation;

a gaze position calculation unit configured to calculate a gaze position of the user on the two-dimensional screen, based on the three-dimensional position of the user and the first direction;

a reference line calculation unit configured to calculate a reference line in a three-dimensional space between the user and the two-dimensional screen, based on the three-dimensional position of the user and the first direction, the reference line being a line corresponding to the gaze position;

an offset amount calculation unit configured to (i) calculate, as a reference point, a point from among a plurality of points on the reference line, the reference point being calculated such that a direction between the three-dimensional input coordinate and the reference point is parallel to the second direction, and (ii) calculate a positional relationship between the reference point and the three-dimensional input coordinate as an offset amount with respect to the gaze position in a first screen predetermined direction along the two-dimensional screen; and

a pointer display position calculation unit configured to calculate, as the display position of the pointer to be displayed on the two-dimensional screen, a position in which a distance of the first screen predetermined direction between the position and the gaze position is the offset amount on the two-dimensional screen.

2. The instruction input device according to claim 1,

wherein the three-dimensional position of the user is a face position or a head position of the user.

3. The instruction input device according to claim 1, further comprising a three-dimensional coordinate input device which outputs a self position when held by the user,

wherein the input coordinate detection unit is configured to detect, as the three-dimensional input coordinate, a three-dimensional coordinate of the self position output by the three-dimensional coordinate input device.

4. The instruction input device according to claim 1, further comprising

an imaging unit configured to capture an image of the user,

wherein the input coordinate detection unit is configured to detect the three-dimensional input coordinate indicating the three-dimensional position of the hand of the user, based on an image captured by the imaging unit.

5. The instruction input device according to claim 4,

wherein the input coordinate detection unit is further configured to recognize right and left hands of the user based on an image captured by the imaging unit, and detect the three-dimensional input coordinate for each of the recognized right and left hands.

6. The instruction input device according to claim 1,

wherein the first direction detection unit is configured to detect the first direction based on the body posture of the user which includes at least one of a face orientation and a gaze.

7. The instruction input device according to claim 1,

wherein the second direction detection unit is configured to detect the second direction based on the body posture of the user which includes at least a body orientation.

8. The instruction input device according to claim 7,

wherein the second direction detection unit is configured to detect the second direction based on the body posture of the user and a direction of the two-dimensional screen with respect to the three-dimensional position of the user.

9. The instruction input device according to claim 1,

wherein the reference line calculation unit is configured to calculate the reference line based on the three-dimensional position of the user, the first direction, and the second direction, in the case where an angle between the first direction and the second direction is smaller than a predetermined angle.

10. The instruction input device according to claim 1, further comprising

a user information database which stores information indicating at least a correspondence relationship between a head position and a chest position of the user,

wherein the reference line calculation unit is configured to calculate the reference line based on the three-dimensional position of the user, the first direction, and information indicating the correspondence relationship obtained from the user information database.

11. The instruction input device according to claim 1, further comprising:

a user information database which stores information indicating at least a chest position of each user; and

a user recognition unit configured to recognize the user,

wherein the reference line calculation unit is configured to calculate the reference line based on the three-dimensional position of the user, the first direction, and information which indicates at least the chest position of the user and is obtained from the user information database based on user recognition information indicating the user recognized by the user recognition unit.

12. The instruction input device according to claim 10,

wherein the reference line calculation unit is configured to calculate the reference line based on the three-dimensional position of the user, the first direction, information indicating the correspondence relationship obtained from the user information database, and the second direction, in the case where an angle between the first direction and the second direction is smaller than a predetermined angle.

13. The instruction input device according to claim 1, further comprising

a user information database which stores information indicating at least a correspondence relationship between a head position and an arm length of the user,

wherein the offset amount calculation unit is configured to calculate the offset amount based on the second direction and information indicating the correspondence relationship obtained from the user information database.

14. The instruction input device according to claim 1, further comprising:

a user information database which stores information indicating at least an arm length for each user; and

a user recognition unit configured to recognize the user,

wherein the offset amount calculation unit is configured to calculate the offset amount based on the second direction and information which indicates at least the arm length of the user, and

the information is obtained from the user information database based on user recognition information indicating the user recognized by the user recognition unit.

15. The instruction input device according to claim 1, further comprising

a display unit configured to display the pointer on the display position calculated by the pointer display position calculation unit.

16. The instruction input device according to claim 1,

wherein the offset amount calculation unit is configured to calculate a distance in a third direction between the reference line and the input coordinate as another offset amount with respect to the gaze position in a second screen predetermined direction along the screen,

the pointer display position calculation unit is configured to calculate, as a display position of the pointer displayed on the two-dimensional screen, a position in which a distance in the second screen predetermined direction between the position and the gaze position is the other offset amount on the two-dimensional screen, and

the third direction is a direction different from the first direction and the second direction, and the second screen predetermined direction is a direction different from the first screen predetermined direction.

17. An instruction input method for calculating a display position of a pointer to be displayed on a two-dimensional screen, based on a pointing operation of movement of a hand of a user in a three-dimensional space, the instruction input method comprising:

detecting a three-dimensional position of the user;

detecting, as a three-dimensional input coordinate, a coordinate indicating a three-dimensional position of the hand of the user;

detecting, based on a body posture of the user, a first direction which is determined with respect to a screen vertical direction and is a direction in which the user is looking;

detecting, based on the body posture of the user, a second direction which is determined with respect to a screen horizontal direction and is a direction in which the user is performing the pointing operation;

calculating a gaze position of the user on the screen, based on the three-dimensional position of the user and the first direction;

calculating a reference line in a three-dimensional space between the user and the two-dimensional screen, based on the three-dimensional position of the user and the first direction, the reference line being a line corresponding to the gaze position;

calculating, as a reference point, a point from among a plurality of points on the reference line, the reference point being calculated such that a direction between the three-dimensional input coordinate and the reference point is parallel to the second direction;

calculating a positional relationship between the reference point and the three-dimensional input coordinate as an offset amount with respect to the gaze position in a first screen predetermined direction along the two-dimensional screen; and

calculating, as a display position of the pointer to be displayed on the two-dimensional screen, a position in which a distance in the first screen predetermined direction between the position and the gaze position is the offset amount on the two-dimensional screen.

18. A non-transitory recording medium having stored thereon a program for calculating a display position of a pointer to be displayed on a two-dimensional screen, based on a pointing operation of movement of a hand of a user in a three-dimensional space, the program causing a computer to execute:

detecting a three-dimensional position of the user;

detecting, as a three-dimensional input coordinate, a coordinate indicating a three-dimensional position of the hand of the user;

detecting, based on a body posture of the user, a first direction which is determined with respect to a screen vertical direction and is a direction in which the user is looking;

detecting, based on the body posture of the user, a second direction which is determined with respect to a screen horizontal direction and is a direction in which the user is performing the pointing operation;

calculating a gaze position of the user on the two-dimensional screen, based on the three-dimensional position of the user and the first direction;

calculating a reference line in a three-dimensional space between the user and the two-dimensional screen, based on the three-dimensional position of the user and the first direction, the reference line being a line corresponding to the gaze position;

calculating, as a reference point, a point from among a plurality of points on the reference line, the reference point being calculated such that a direction between the three-dimensional input coordinate and the reference point is parallel to the second direction;

calculating a positional relationship between the reference point and the three-dimensional input coordinate as an offset amount with respect to the gaze position in a first screen predetermined direction along the two-dimensional screen; and

calculating, as a display position of the pointer to be displayed on the two-dimensional screen, a position in which a distance in the first screen predetermined direction between the position and the gaze position is the offset amount on the two-dimensional screen.

19. An integrated circuit which calculates a display position of a pointer to be displayed on a two-dimensional screen, based on a pointing operation of movement of a hand of a user in a three-dimensional space, the integrated circuit comprising:

a user position detection unit configured to detect a three-dimensional position of the user;

an input coordinate detection unit configured to detect, as a three-dimensional input coordinate, a coordinate indicating a three-dimensional position of the hand of the user;

a first direction detection unit configured to detect, based on a body posture of the user, a first direction which is determined with respect to a screen vertical direction and is a direction in which the user is looking;

a second direction detection unit configured to detect, based on the body posture of the user, a second direction which is determined with respect to a screen horizontal direction and is a direction in which the user is performing the pointing operation;

a gaze position calculation unit configured to calculate a gaze position of the user on the two-dimensional screen, based on the three-dimensional position of the user and the first direction;

a reference line calculation unit configured to calculate a reference line in a three-dimensional space between the user and the two-dimensional screen, based on the three-dimensional position of the user and the first direction, the reference line being a line corresponding to the gaze position;

an offset amount calculation unit configured to (i) calculate, as a reference point, a point from among a plurality of points on the reference line, the reference point being calculated such that a direction between the three-dimensional input coordinate and the reference point is parallel to the second direction, and (ii) calculate a positional relationship between the reference point and the three-dimensional input coordinate as an offset amount with respect to the gaze position in a first screen predetermined direction along the two-dimensional screen; and

a pointer display position calculation unit configured to calculate, as a display position of the pointer to be displayed on the two-dimensional screen, a position in which a distance of the first screen predetermined direction between the position and the gaze position is the offset amount on the two-dimensional screen.