Three-dimensional user interface apparatus and three-dimensional operation method

• US 9,495,068 B2
• Filed: 03/08/2013
• Issued: 11/15/2016
• Est. Priority Date: 07/27/2012
• Status: Active Grant

• Current Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Original Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Inventors: Morishita, Koji, Nagai, Katsuyuki, Noda, Hisashi
• Primary Examiner(s): Cunningham, Gregory F
• Application Number:
  US14/416,763
• Publication Number:
  US 20150205484A1
• Time in Patent Office: 1,348 Days
• Field of Search: 348/46, 382/103
• US Class Current: 1/1
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

A three-dimensional user interface apparatus includes a three-dimensional information acquisition unit acquiring three-dimensional information from a three-dimensional sensor; a position calculation unit calculating three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the three-dimensional information; a virtual data generation unit generating virtual three-dimensional space data indicating a virtual three-dimensional space which is disposed in the three-dimensional coordinate space and at least part of which is set in a display region; a space processing unit performing a predetermined process corresponding to a change in the three-dimensional position information regarding the specific part of the target person, on the three-dimensional coordinate space or the virtual three-dimensional space data; and a display processing unit displaying the virtual three-dimensional space within the display region on a display unit on the basis of virtual three-dimensional space data obtained through the predetermined process.

Claims

1. A three-dimensional user interface apparatus comprising:

a three-dimensional information acquisition unit that acquires three-dimensional information from a three-dimensional sensor;

a position calculation unit that calculates three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the three-dimensional information acquired by the three-dimensional information acquisition unit;

a virtual data generation unit that generates virtual three-dimensional space data indicating a virtual three-dimensional space which is disposed in the three-dimensional coordinate space and at least part of which is set in a display region, the virtual data generation unit further generating a virtual object to be displayed in the display region, the virtual object being other than the virtual three-dimensional space;

a display processing unit that displays, on a display unit, the virtual object and the virtual three-dimensional space within the display region;

a state acquisition unit that acquires state information of the specific part of the target person;

a space processing unit that performs a predetermined process on the virtual three-dimensional space; and

an operation specifying unit that performs;

a first process of determining a predetermined process to be performed by the space processing unit from among a plurality of predetermined processes, on the basis of a combination of a change in a location of the specific part of the target person being relative to a location of the virtual object and a change in a state of the specific part of the target person;

a second process of determining whether or not the specific part of the target person is present within a predetermined three-dimensional range being defined relative to the location of the virtual object, on the basis of the three-dimensional position information calculated by the position calculation unit; and

a third process of determining whether or not to make the space processing unit perform the predetermined process on the basis of a result of the second process,

wherein the space processing unit performs the predetermined process determined by the operation specifying unit on the virtual three-dimensional space being displayed on the display unit, when the operation specifying unit determines to make the space processing unit perform the predetermined process, and

wherein, after the predetermined process is performed on the virtual three-dimensional space being displayed on the display unit, the display processing unit displays the virtual three-dimensional unit on which the predetermined process has been performed.

2. The three-dimensional user interface apparatus according to claim 1,

wherein the position calculation unit calculates three-dimensional position information of one hand of the target person as the specific part of the target person,

wherein the state acquisition unit acquires state information of the one hand of the target person as the specific part of the target person, and

wherein the operation specifying unit specifies, as the predetermined process, a process of performing a movement by a distance corresponding to a linear movement amount of the one hand of the target person while the one hand is maintained in a specific state.

3. The three-dimensional user interface apparatus according to claim 1,

wherein the position calculation unit calculates respective pieces of three-dimensional position information on the three-dimensional coordinate space regarding a plurality of specific parts of the target person,

wherein the state acquisition unit acquires respective pieces of state information regarding the plurality of specific parts of the target person, and

wherein the operation specifying unit calculates a positional relationship between the plurality of specific parts on the basis of the plurality of pieces of three-dimensional position information regarding the plurality of specific parts calculated by the position calculation unit, and specifies the predetermined process from among the plurality of predetermined processes on the basis of a change in the calculated positional relationship and the pieces of state information acquired by the state acquisition unit.

4. The three-dimensional user interface apparatus according to claim 3,

wherein the position calculation unit calculates the respective pieces of three-dimensional position information of both hands of the target person as the plurality of specific parts,

wherein the state acquisition unit acquires the respective pieces of state information of both the hands of the target person as the plurality of specific parts, and

wherein the operation specifying unit specifies, as the predetermined process, an enlargement process or a reduction process with a position of one hand of the target person as a reference point at a magnification ratio or a reduction ratio corresponding to a change amount of a distance between both the hands, or a rotation process with a position of one hand of the target person as a reference point in a solid angle change amount corresponding to a solid angle change amount of a line segment joining both the hands.

5. The three-dimensional user interface apparatus according claim 1,

wherein the operation specifying unit determines whether or not the state information acquired by the state acquisition unit indicates a specific state, and determines whether or not the predetermined process is to be performed by the space processing unit according to a result of the determination.

6. The three-dimensional user interface apparatus according to claim 1,

wherein, the operation specifying unit specifies, as the predetermined process, a rotation process with a specific point of the virtual object as a reference point in a solid angle change amount corresponding to a solid angle change amount of a line segment joining the specific part of the target person and the specific point of the virtual object.

7. The three-dimensional user interface apparatus according to claim 1,

wherein the operation specifying unit detects a movement of the specific part of the target person from within the predetermined three-dimensional range to outside of the predetermined three-dimensional range, and specifies, as the predetermined process, a rotation process corresponding to a distance and a direction between a position within the predetermined three-dimensional range and a position outside the predetermined three-dimensional range before and after the movement.

8. The three-dimensional user interface apparatus according to claim 1,

wherein, the operation specifying unit measures a period in which state information acquired by the state acquisition unit and the three-dimensional position information are not changed, and specifies, as the predetermined process, a process of adding display data of a function menu in a case where the measured period exceeds a predetermined period.

9. The three-dimensional user interface apparatus according to claim 1, further comprising:

a first object detection unit that detects a known common real object from the three-dimensional information;

a first reference setting unit that sets the three-dimensional coordinate space on the basis of the common real object detected by the first object detection unit and calculates a position and a direction of the three-dimensional sensor;

a visual line image acquisition unit that acquires a visual line image in which the specific part of the target person is reflected from an imaging unit that is disposed at a position and in a direction different from the position and direction of the three-dimensional sensor;

a second object detection unit that detects the known common real object from the visual line image acquired by the visual line image acquisition unit;

a second reference setting unit that shares the three-dimensional coordinate space on the basis of the common real object detected by the second object detection unit and calculates a position and a direction of the imaging unit; and

an image synthesizing unit that synthesizes the virtual three-dimensional space within the display region with the visual line image captured by the imaging unit on the basis of the position and the direction of the imaging unit calculated by the second reference setting unit and the three-dimensional coordinate space,

wherein the position calculation unit calculates the three-dimensional position information on the three-dimensional coordinate space by converting the three-dimensional position information regarding the specific part of the target person acquired from the three-dimensional information which is acquired by the three-dimensional information acquisition unit on the basis of the position and the direction of the three-dimensional sensor calculated by the first reference setting unit and the three-dimensional coordinate space, and

wherein the display processing unit displays the image obtained by the image synthesizing unit on the display unit.

10. A three-dimensional operation method executed by at least one computer, the method comprising:

acquiring three-dimensional information from a three-dimensional sensor;

calculating three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the acquired three-dimensional information;

generating virtual three-dimensional space data indicating a virtual three-dimensional space which is disposed in the three-dimensional coordinate space and at least part of which is set in a display region, and generating a virtual object to be displayed in the display region, the virtual object being other than the virtual three-dimensional space;

performing a predetermined process on the virtual three-dimensional space;

displaying, on a display unit, the virtual object and the virtual three-dimensional space within the display region on a display unit on the basis of virtual three-dimensional space data which is obtained by performing the predetermined process;

acquiring state information of the specific part of the target person;

performing;

a first process of determining a predetermined process to be performed by the space processing unit from among a plurality of predetermined processes, on the basis of a combination of a change in a location of the specific part of the target person being relative to a location of the virtual object and a change in a state of the specific part of the target person;

a second process of determining whether or not the specific part of the target person is present within a predetermined three-dimensional range being defined relative to the location of the virtual object, on the basis of the three-dimensional position information calculated by the position calculation unit; and

a third process of determining whether or not to make the space processing unit perform the predetermined process on the basis of a result of the second process,

wherein performing the predetermined process includes performing the predetermined process determined by the operation specifying unit on the virtual three-dimensional space being displayed on the display unit, when the operation specifying unit determines to make the space processing unit perform the predetermined process, and

wherein, after the predetermined process is performed on the virtual three-dimensional space being displayed on the display unit, the virtual three-dimensional unit on which the predetermined process has been performed is displayed.

11. A non-transitory computer-readable storage medium storing a program for causing at least one computer to perform a three-dimensional operation method, the three-dimensional operation method comprising:

acquiring three-dimensional information from a three-dimensional sensor;

calculating three-dimensional position information on a three-dimensional coordinate space regarding a specific part of a target person by using the acquired three-dimensional information;

generating virtual three-dimensional space data indicating a virtual three-dimensional space which is disposed in the three-dimensional coordinate space and at least part of which is set in a display region, and generating a virtual object to be displayed in the display region, the virtual object being other than the virtual three-dimensional space;

performing a predetermined process on the virtual three-dimensional;

displaying, on a display unit, the virtual object and the virtual three-dimensional space within the display region on a display unit on the basis of virtual three-dimensional space data which is obtained by performing the predetermined process;

acquiring state information of the specific part of the target person;

performing;

a first process of determining a predetermined process to be performed by the space processing unit from among a plurality of predetermined processes, on the basis of a combination of a change in a location of the specific part of the target person being relative to a location of the virtual object and a change in a state of the specific part of the target person;

a second process of determining whether or not the specific part of the target person is present within a predetermined three-dimensional range being defined relative to the location of the virtual object, on the basis of the three-dimensional position information calculated by the position calculation unit; and

a third process of determining whether or not to make the space processing unit perform the predetermined process on the basis of a result of the second process,

wherein performing the predetermined process includes performing the predetermined process determined by the operation specifying unit on the virtual three-dimensional space being displayed on the display unit, when the operation specifying unit determines to make the space processing unit perform the predetermined process, and

wherein, after the predetermined process is performed on the virtual three-dimensional space being displayed on the display unit, the virtual three-dimensional unit on which the predetermined process has been performed is displayed.