Three-dimensional user interface apparatus and three-dimensional operation method

• US 9,541,997 B2
• Filed: 03/28/2013
• Issued: 01/10/2017
• Est. Priority Date: 07/27/2012
• Status: Active Grant

• Current Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Original Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Inventors: Morishita, Koji, Nagai, Katsuyuki, Noda, Hisashi
• Primary Examiner(s): Lee, Jr., Kenneth B
• Application Number:
  US14/416,846
• Publication Number:
  US 20150277555A1
• Time in Patent Office: 1,384 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

A three-dimensional user interface apparatus includes a calculation unit that calculates three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using three-dimensional information acquired from a three-dimensional sensor, a generation unit that generates virtual three-dimensional object data indicating a virtual three-dimensional object disposed in the three-dimensional coordinate space, a state acquisition unit that acquires state information of the specific part of the target person, an operation specifying unit that specifies a predetermined process to be performed from among a plurality of predetermined processes on the basis of a combination of the state information and a change in the three-dimensional position information, a processing unit that performs the predetermined process specified by the operation specifying unit on the virtual three-dimensional object data, and a display processing unit that displays a virtual three-dimensional object corresponding to the virtual three-dimensional object data on which the predetermined process has been performed, on a display unit.

Claims

1. A three-dimensional user interface apparatus comprising:

a three-dimensional information acquisition unit that acquires three-dimensional information from a three-dimensional sensor;

a position calculation unit that calculates three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the three-dimensional information acquired by the three-dimensional information acquisition unit;

a virtual data generation unit that generates virtual three-dimensional object data indicating a virtual three-dimensional object disposed in the three-dimensional coordinate space;

a state acquisition unit that acquires state information of the specific part of the target person;

an operation specifying unit that specifies a predetermined process to be performed from among a plurality of predetermined processes on the basis of a combination of the state information acquired by the state acquisition unit and a change in the three-dimensional position information;

an object processing unit that performs the predetermined process specified by the operation specifying unit on the virtual three-dimensional object data; and

a display processing unit that displays a virtual three-dimensional object corresponding to the virtual three-dimensional object data on which the predetermined process has been performed, on a display unit,

wherein the operation specifying unit determines whether the specific part of the target person is present within a predetermined three-dimensional range with reference to the virtual three-dimensional object on the basis of the three-dimensional position information calculated by the position calculation unit, and determines whether the predetermined process is to be performed by the object processing unit according to a result of the determination,

wherein the operation specifying unit detects a movement of the specific part of the target person from within the predetermined three-dimensional range to the outside of the predetermined three-dimensional range, and specifies, as the predetermined process, a movement process or a rotation process corresponding to a distance and a direction between a position within the predetermined three-dimensional range and a position outside the predetermined three-dimensional range before and after the movement.

2. The three-dimensional user interface apparatus according to claim 1,

wherein the position calculation unit calculates the three-dimensional position information of one hand of the target person as the specific part of the target person,

wherein the state acquisition unit acquires the state information of the one hand of the target person as the specific part of the target person, and

wherein the operation specifying unit specifies, as the predetermined process, a process of performing a movement by a distance corresponding to a linear movement amount of the one hand while the one hand of the target person is maintained in the specific state.

3. The three-dimensional user interface apparatus according to claim 2,

wherein, in a case where a distance from a specific point of the virtual three-dimensional object to the one hand of the target person is not changed before and after the one hand is moved while the one hand is maintained in the specific state, the operation specifying unit specifies, as the predetermined process, a rotation process with the specific point of the virtual three-dimensional object as a reference point on the basis of a solid angle change amount of a line segment joining the one hand of the target person and the specific point of the virtual three-dimensional object.

4. The three-dimensional user interface apparatus according to claim 1,

wherein the position calculation unit calculates respective pieces of three-dimensional position information on a three-dimensional coordinate space regarding a plurality of specific parts of the target person,

wherein the state acquisition unit acquires respective pieces of state information regarding the plurality of specific parts of the target person, and

wherein the operation specifying unit calculates a positional relationship between the plurality of specific parts on the basis of the plurality of pieces of three-dimensional position information regarding the plurality of specific parts calculated by the position calculation unit, and specifies the predetermined process from among the plurality of predetermined processes on the basis of a change in the calculated positional relationship and the plurality of pieces of state information acquired by the state acquisition unit.

5. The three-dimensional user interface apparatus according to claim 4,

wherein the position calculation unit calculates three-dimensional position information of both hands of the target person as the plurality of specific parts,

wherein the state acquisition unit acquires state information of both the hands of the target person as the plurality of specific parts, and

wherein the operation specifying unit specifies, as the predetermined process, an enlargement process or a reduction process with a position of one hand of the target person as a reference point at a magnification ratio or a reduction ratio corresponding to a change amount of a distance between both the hands while both the hands are maintained in a specific state, or a rotation process with a position of one hand of the target person as a reference point on the basis of a solid angle change amount of a line segment joining both the hands while both the hands are maintained in a specific state.

6. The three-dimensional user interface apparatus according to claim 1,

wherein the operation specifying unit measures a period in which state information acquired by the state acquisition unit and the three-dimensional position information are not changed, and in a case where the measured period exceeds a predetermined period, the operation specifying unit specifies, as the predetermined process, a process of adding display data of a function menu to data of a virtual three-dimensional space in which the virtual three-dimensional object is disposed.

7. The three-dimensional user interface apparatus according to claim 1, further comprising:

a first object detection unit that detects a known common real object from the three-dimensional information;

a first reference setting unit that sets the three-dimensional coordinate space on the basis of the common real object detected by the first object detection unit and calculates a position and a direction of the three-dimensional sensor;

a visual line image acquisition unit that acquires a visual line image in which the specific part of the target person is reflected from an imaging unit that is disposed at a position and in a direction different from the position and direction of the three-dimensional sensor;

a second object detection unit that detects the known common real object from the visual line image acquired by the visual line image acquisition unit;

a second reference setting unit that shares the three-dimensional coordinate space on the basis of the common real object detected by the second object detection unit and calculates a position and a direction of the imaging unit; and

an image synthesizing unit that synthesizes the virtual three-dimensional object and the visual line image captured by the imaging unit on the basis of the position and the direction of the imaging unit calculated by the second reference setting unit and the three-dimensional coordinate space,

wherein the position calculation unit calculates the three-dimensional position information on the three-dimensional coordinate space by converting three-dimensional position information regarding the specific part of the target person acquired from the three-dimensional information which is acquired by the three-dimensional information acquisition unit on the basis of the position and the direction of the three-dimensional sensor calculated by the first reference setting unit and the three-dimensional coordinate space, and

wherein the display processing unit displays the image obtained by the image synthesizing unit on the display unit.

8. A three-dimensional operation method executed by at least one computer, comprising:

acquiring three-dimensional information from a three-dimensional sensor;

calculating three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the acquired three-dimensional information;

generating virtual three-dimensional object data indicating a virtual three-dimensional object disposed in the three-dimensional coordinate space;

acquiring state information of the specific part of the target person;

specifying a predetermined process to be performed from among a plurality of predetermined processes on the basis of a combination of the acquired state information and a change in the three-dimensional position information;

performing the specified predetermined process on the virtual three-dimensional object data; and

displaying a virtual three-dimensional object corresponding to the virtual three-dimensional object data on which the predetermined process has been performed, on a display unit,

wherein specifying the predetermined process comprises determining whether the specific part of the target person is present within a predetermined three-dimensional range with reference to the virtual three-dimensional object on the basis of the calculated three-dimensional position information, and determining whether the predetermined process is to be performed according to a result of the determination,

wherein specifying the predetermined process comprises detecting a movement of the specific part of the target person from within the predetermined three-dimensional range to the outside of the predetermined three-dimensional range, and specifying, as the predetermined process, a movement process or a rotation process corresponding to a distance and a direction between a position within the predetermined three-dimensional range and a position outside the predetermined three-dimensional range before and after the movement.

9. The three-dimensional operation method according to claim 8, further comprising:

determining whether or not the specific part of the target person is present within a predetermined three-dimensional range with reference to the virtual three-dimensional object on the basis of the calculated three-dimensional position information; and

determining whether or not the predetermined process is to be performed according to a result of the determination.

10. The three-dimensional operation method according to claim 8,

wherein the calculating of the three-dimensional position information includes calculating respective pieces of three-dimensional position information on a three-dimensional coordinate space regarding a plurality of specific parts of the target person,

wherein the acquiring of the state information includes acquiring respective pieces of state information regarding the plurality of specific parts of the target person, and

wherein the specifying of the predetermined process includes calculating a positional relationship between the plurality of specific parts on the basis of the plurality of pieces of calculated three-dimensional position information regarding the plurality of specific parts, and specifying the predetermined process from among the plurality of predetermined processes on the basis of a change in the calculated positional relationship and the plurality of pieces of acquired state information.

11. A non-transitory computer-readable storage medium storing a program for causing at least one computer to execute a three-dimensional operation method, the three-dimensional operation method comprising:

acquiring three-dimensional information from a three-dimensional sensor;

calculating three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the acquired three-dimensional information;

generating virtual three-dimensional object data indicating a virtual three-dimensional object disposed in the three-dimensional coordinate space;

acquiring state information of the specific part of the target person;

specifying a predetermined process to be performed from among a plurality of predetermined processes on the basis of a combination of the acquired state information and a change in the three-dimensional position information;

performing the specified predetermined process on the virtual three-dimensional object data; and

displaying a virtual three-dimensional object corresponding to the virtual three-dimensional object data on which the predetermined process has been performed, on a display unit,

wherein specifying the predetermined process comprises determining whether the specific part of the target person is present within a predetermined three-dimensional range with reference to the virtual three-dimensional object on the basis of the calculated three-dimensional position information, and determining whether the predetermined process is to be performed according to a result of the determination,

wherein specifying the predetermined process comprises detecting a movement of the specific part of the target person from within the predetermined three-dimensional range to the outside of the predetermined three-dimensional range, and specifying, as the predetermined process, a movement process or a rotation process corresponding to a distance and a direction between a position within the predetermined three-dimensional range and a position outside the predetermined three-dimensional range before and after the movement.

12. The non-transitory computer-readable storage medium according to claim 11, wherein the three-dimensional operation method further includes:

determining whether or not the specific part of the target person is present within a predetermined three-dimensional range with reference to the virtual three-dimensional object on the basis of the calculated three-dimensional position information; and

determining whether or not the predetermined process is to be performed according to a result of the determination.

13. The non-transitory computer-readable storage medium according to claim 11,

wherein the calculating of the three-dimensional position information includes calculating respective pieces of three-dimensional position information on a three-dimensional coordinate space regarding a plurality of specific parts of the target person,

wherein the acquiring of the state information includes acquiring respective pieces of state information regarding the plurality of specific parts of the target person, and

wherein the specifying of the predetermined process includes calculating a positional relationship between the plurality of specific parts on the basis of the plurality of pieces of calculated three-dimensional position information regarding the plurality of specific parts, and specifying the predetermined process from among the plurality of predetermined processes on the basis of a change in the calculated positional relationship and the plurality of pieces of acquired state information.

14. A three-dimensional user interface apparatus comprising:

a three-dimensional information acquisition unit that acquires three-dimensional information from a three-dimensional sensor;

a position calculation unit that calculates three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the three-dimensional information acquired by the three-dimensional information acquisition unit;

a virtual data generation unit that generates virtual three-dimensional object data indicating a virtual three-dimensional object disposed in the three-dimensional coordinate space;

a state acquisition unit that acquires state information of the specific part of the target person;

an operation specifying unit that specifies a predetermined process to be performed from among a plurality of predetermined processes on the basis of a combination of the state information acquired by the state acquisition unit and a change in the three-dimensional position information;

an object processing unit that performs the predetermined process specified by the operation specifying unit on the virtual three-dimensional object data; and

a display processing Unit that displays a virtual three-dimensional object corresponding to the virtual three-dimensional object data on which the predetermined process has been performed, on a display unit,

wherein the operation specifying, unit measures a period in which state information acquired by the state acquisition unit and the three-dimensional position information are not changed, and in a case where the measured period exceeds a predetermined period, the operation specifying unit specifies, as the predetermined process, a process of adding display data of a function menu to data of a virtual three-dimensional space in which the virtual three-dimensional object is disposed.