Three-dimensional user interface device and three-dimensional operation processing method

• US 10,007,351 B2
• Filed: 09/06/2013
• Issued: 06/26/2018
• Est. Priority Date: 03/11/2013
• Status: Active Grant

• Current Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Original Assignee: NEC Solution Innovators, Ltd (NEC Corporation)
• Inventors: Nagai, Katsuyuki, Noda, Hisashi, Morishita, Koji
• Primary Examiner(s): Mandeville, Jason
• Application Number:
  US14/773,567
• Publication Number:
  US 20160018897A1
• Time in Patent Office: 1,754 Days
• Field of Search: 345156, 345157, 345 7, 345 8
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04847 Interaction techniques to control parameter settings, e.g. interaction with sliders or dials

Abstract

A three-dimensional user interface device includes a coordinate setting unit setting a three-dimensional coordinate space (3DCS) on the basis of a line-of-sight image, a virtual data generation unit generating three-dimensional area data representing a transparent virtual three-dimensional operation area (V3DOA) arranged in an arbitrary position in the 3DCS in a state in which at least a boundary of the area is visible, a display processing unit displaying a V3DOA represented by the generated three-dimensional area data by using a visible space in the 3DCS corresponding to a space shown on the line-of-sight image as a display reference, and an operation detection unit detecting an operation performed by the operator with the specific region in the V3DOA on the basis of the three-dimensional position acquired with respect to the specific region of the operator in the 3DCS and a position of the V3DOA in the 3DCS.

Claims

1. A three-dimensional user interface device, comprising:

one or more processors configured to

acquire a line-of-sight image from an operator, the line-of-sight image being captured by an image capturing device mounted on a head of the operator,

set three-dimensional coordinate space on the basis of the acquired line-of-sight image, and

generate three-dimensional area data representing a transparent virtual three-dimensional operation area arranged in an arbitrary position in the three-dimensional coordinate space in a state in which at least a boundary of the virtual three-dimensional operation area is visible, and generate display data of a function menu arranged in the three-dimensional coordinate space, the function menu representing a plurality of shapes taken by the virtual three-dimensional operation area; and

a display processor that causes a display mounted on the head of the operator to display the virtual three-dimensional operation area represented by the three-dimensional area data which is generated by the one or more processors by using a visible space in the three-dimensional coordinate space corresponding to a space shown on the line-of-sight image as a display reference, and causes the display to display the function menu,

wherein the one or more processors are configured to

acquire a three-dimensional position of a specific region of the operator in the three-dimensional coordinate space,

detect an operation performed by the operator with the specific region in the virtual three-dimensional operation area on the basis of the three-dimensional position which is acquired by the one or more processors and a position of the virtual three-dimensional operation area in the three-dimensional coordinate space, and detect an operation performed by the operator with the specific region to the function menu to cause the three-dimensional area data representing the virtual three-dimensional operation area which is arranged in a position in the three-dimensional space to be generated, the position determined according to a detected position of the operation and which has a shape selected by the operation, and

set a start point and an end point in the position in the three-dimensional coordinate space which is determined according to the detected position of the operation to the function menu, and set a shape of an area, where two spherical bodies intersect, to the virtual three-dimensional operation area which is indicated by new three-dimensional area data, the two spherical bodies having the start point and the end point as each center point and each having a spherical surface including a point to which a distance from the center point is half of a straight line connecting the start point and the end point.

2. The three-dimensional user interface device according to claim 1, wherein the one or more processors are configured to apply at least one of resizing processing, rotation processing, modification processing, and movement processing corresponding to a specific operation which is detected by the one or more processors to the three-dimensional area data, when the one or more processors detect the specific operation to the virtual three-dimensional operation area itself.

3. The three-dimensional user interface device according to claim 2, wherein

the one or more processors generate a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and

specify a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas, and acquires operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.

4. The three-dimensional user interface device according to claim 1, wherein

the one or more processors generate a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and

specify a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas, and acquires operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.

5. The three-dimensional user interface device according to claim 1, wherein the one or more processors synthesize the virtual three-dimensional operation area represented by the three-dimensional area data with the line-of-sight image showing the space which is the display reference on the basis of the three-dimensional coordinate space,

wherein the display processor causes the display to display an image obtained by the one or more processors.

6. A three-dimensional operation processing method executed by at least one computer, the method comprising:

acquiring a line-of-sight image from an operator, the line-of-sight image being captured by an image capturing device mounted on a head of the operator;

setting a three-dimensional coordinate space on the basis of the acquired line-of-sight image;

generating three-dimensional area data representing a transparent virtual three-dimensional operation area arranged in an arbitrary position in the three-dimensional coordinate space in a state in which at least a boundary of the virtual three-dimensional operation area is visible;

causing a display mounted on the head of the operator to display the virtual three-dimensional operation area represented by the generated three-dimensional area data by using a visible space in the three-dimensional coordinate space corresponding to a space shown on the line-of-sight image as a display reference;

acquiring a three-dimensional position of a specific region of the operator in the three-dimensional coordinate space;

detecting an operation performed by the operator with the specific region in the virtual three-dimensional operation area on the basis of the acquired three-dimensional position and a position of the virtual three-dimensional operation area in the three-dimensional coordinate space;

generating display data of a function menu arranged in the three-dimensional coordinate space, the function menu representing a plurality of shapes taken by the virtual three-dimensional operation area;

causing the display to display the function menu; and

detecting an operation performed by the operator with the specific region to the function menu,

wherein said generating the three-dimensional area data includes generating the three-dimensional area data representing the virtual three-dimensional operation area which is arranged in a position in the three-dimensional space, the position determined according to a detected position of the operation to the function menu, and which has a shape selected by the operation, and

said generating the three-dimensional area data includes setting a start point and an end point in the position in the three-dimensional coordinate space which is determined according to the detected position of the operation to the function menu and generating the three-dimensional area data representing the virtual three-dimensional operation area having a shape of an area, where two spherical bodies intersect, the two spherical bodies having the start point and the end point as each center point and each having a spherical surface including a point to which a distance from the center point is half of a straight line connecting the start point and the end point.

7. The three-dimensional operation processing method according to claim 6, further comprising:

detecting a specific operation of the operator to the virtual three-dimensional operation area itself; and

applying at least one of resizing processing, rotation processing, modification processing, and movement processing corresponding to the detected specific operation to the three-dimensional area data.

8. The three-dimensional operation processing method according to claim 6,

wherein said generating the three-dimensional area data includes generating a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and

said detecting the operation includes specifying a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas and acquiring operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.

9. The three-dimensional operation processing method according to claim 6, further comprising:

synthesizing the virtual three-dimensional operation area represented by the three-dimensional area data with the line-of-sight image showing the space which is the display reference on the basis of the three-dimensional coordinate space,

wherein said causing displaying the virtual three-dimensional operation area includes causing the display to display an image obtained by the synthesizing.

10. A non-transitory computer-readable storage medium storing a program for causing at least one computer to execute a three-dimensional operation processing method, the method comprising:

acquiring a line-of-sight image from an operator, the line-of-sight image being captured by an image capturing device mounted on a head of the operator;

setting a three-dimensional coordinate space on the basis of the acquired line-of-sight image;

generating three-dimensional area data representing a transparent virtual three-dimensional operation area arranged in an arbitrary position in the three-dimensional coordinate space in a state in which at least a boundary of the virtual three-dimensional operation area is visible;

causing a display mounted on the head of the operator to display the virtual three-dimensional operation area represented by the generated three-dimensional area data by using a visible space in the three-dimensional coordinate space corresponding to a space shown on the line-of-sight image as a display reference;

acquiring a three-dimensional position of a specific region of the operator in the three-dimensional coordinate space;

detecting an operation performed by the operator with the specific region in the virtual three-dimensional operation area on the basis of the acquired three-dimensional position and a position of the virtual three-dimensional operation area in the three-dimensional coordinate space;

generating display data of a function menu arranged in the three-dimensional coordinate space, the function menu representing a plurality of shapes taken by the virtual three-dimensional operation area;

causing the display to display the function menu; and

detecting an operation performed by the operator with the specific region to the function menu,

wherein said generating the three-dimensional area data includes generating the three-dimensional area data representing the virtual three-dimensional operation area which is arranged in a position in the three-dimensional space, the position determined according to a detected position of the operation to the function menu, and which has a shape selected by the operation, and

said generating the three-dimensional area data includes setting a start point and an end point in the position in the three-dimensional coordinate space which is determined according to the detected position of the operation to the function menu and generating the three-dimensional area data representing the virtual three-dimensional operation area having a shape of an area, where two spherical bodies intersect, the two spherical bodies having the start point and the end point as each center point and each having a spherical surface including a point to which a distance from the center point is half of a straight line connecting the start point and the end point.

11. The non-transitory computer-readable storage medium according to claim 10, further comprising:

detecting a specific operation of the operator to the virtual three-dimensional operation area itself; and

applying at least one of resizing processing, rotation processing, modification processing, and movement processing corresponding to the detected specific operation to the three-dimensional area data.

12. The non-transitory computer-readable storage medium according to claim 10, wherein

said generating the three-dimensional area data includes generating a plurality of the three-dimensional area data respectively representing a position in the three-dimensional coordinate space and an operation type with respect to each of a plurality of the virtual three-dimensional operation areas, and

said detecting the operation includes specifying a virtual three-dimensional operation area which is operated among the plurality of virtual three-dimensional operation areas and acquiring operation information corresponding to an operation type which is set in the specified virtual three-dimensional operation area.

13. The non-transitory computer-readable storage medium according to claim 10, further comprising:

synthesizing the virtual three-dimensional operation area represented by the three-dimensional area data with the line-of-sight image showing the space which is the display reference on the basis of the three-dimensional coordinate space,

wherein said causing to display the virtual three-dimensional operation area includes causing the display to display an image obtained by the synthesizing.