INFORMATION DISPLAY SYSTEM, INFORMATION DISPLAY METHOD, AND RECORDING MEDIUM

• US 20140292653A1
• Filed: 09/05/2012
• Published: 10/02/2014
• Est. Priority Date: 09/06/2011
• Status: Active Grant

• Current Assignee: NEC Biglobe Ltd (KDDI Corporation)
• Original Assignee: NEC Biglobe Ltd (KDDI Corporation)
• Inventors: Kamba, Tomonari, Hirata, Kyoji
• Granted Patent: US 9,459,706 B2
• Time in Patent Office: Days
• US Class Current: 345/157
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/03545 Pens or stylus
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

In an AR device, when display control of additional information is performed for an object, a display device acquires a picture including an object in real space. An pointing device outputs, when pointing the object in real space by an operation of a user, information showing a feature for identifying the pointed object. A control device analyzes the picture by the display device and identifies objects in the picture. The control device outputs position information in the picture for the identified object. The control device outputs position information in the picture for the pointed object based on the outputted information from the pointing device. The control device extracts additional information of a target object from an additional information group of objects in real space. The control device performs display control of the additional information based on the calculated position information. The display device displays the additional information.

Claims

1. An information display system comprising:

at least one display device;

at least one pointing device; and

a control device configured to communicate with said at least one display device and said at least one pointing device,

wherein said control device identifies an object which can be seen through said at least one display device based on information not from said at least one display device but from said at least one pointing device which points said object in a real space, and performs display control of additional information with respect to said object to on said at least one display device.

2. The information display system according to claim 1, wherein said control device performs display control of additional information with respect to only an object which is pointed by said at least one pointing device.

3. The information display system according to claim 1, wherein said control device checks a visual feature of a picture from said at least one display device with a visual feature at a center of a picture from said at least one pointing device and calculates position information in a picture from said at least one display device with respect to an object which is pointed by said at least one pointing device.

4. The information display system according to claim 1, wherein said control device checks information related to a position component and a direction component of said at least one display device with information related to a position component and a direction component of said at least one pointing device and calculates position information in a picture from said at least one display device with respect to an object which is pointed by said at least one pointing device.

5. The information display system according to claim 1, wherein said control device performs display control of a marker for confirming an object which is pointed by said at least one pointing device based on position information calculated by said control device.

6. The information display system according to claim 1, wherein said control device performs display control of an optical axis showing a line segment of a direction which is pointed by said at least one pointing device based on information related to a position component and a direction component of said at least one pointing device.

7. The information display system according to claim 6, wherein said control device changes a depth value of an optical axis showing a line segment of a direction which is pointed by said at least one pointing device in accordance with input pointing from said at least one pointing device based on information related to a position component and a direction component of said at least one display device and information related to a position component and a direction component of said at least one pointing device.

8. The information display system according to claim 1, wherein said control device performs, when a position of a marker displayed in said at least one display device is adjusted from said at least one pointing device, in a case that said adjusted position of a marker overlaps with a position of said object in a picture from said at least one display device, display control of additional information related to said object.

9. The information display system according to claim 8, wherein said control device performs adjustment such that a position of a marker overlaps with said pointed object by moving a marker in up and down directions and right and left directions based on input pointing from said at least one pointing device.

10. The information display system according to claim 8, wherein said control device performs adjustment, while fixing a position of a marker, such that said pointed object overlaps with said fixed position of a marker.

11. The information display system according to claim 8, wherein said control device performs adjustment such that a position of a marker overlaps with said pointed object by moving a marker along a line segment direction of an optical axis showing a line segment of a direction which is pointed by said at least one pointing device.

12. The information display system according to claim 1, wherein said control device relates an image and additional information of a virtual object to display position information of a virtual object to store them, detects that a position component and a direction component of said at least one display device is coincide with a position component and a direction component of display position information of a virtual object, and performs display control on said at least one display device for displaying an image of a virtual object in a display region corresponding to a display position information of a virtual object.

13. The information display system according to claim 1, wherein said control device relates an image and additional information of a virtual object to a specific object or a specific mark to store them, detects that a specific object or a specific mark exists in a picture from said at least one display device, and performs display control on said at least one display device for displaying an image of a virtual object in a display region including a specific object or a specific mark.

14. The information display system according to claim 12, wherein said control device recognizes, when making said at least one display device display a virtual object, a display position of a virtual object in the picture, and in a case that calculated position information in a picture is coincide with a display position of a virtual object in a picture, said control device performs display control for displaying additional information of a virtual object near a display position of said virtual object.

15. The information display system according to claim 1, wherein said at least one display device includes:

a picture inputting section configured to acquire a picture including an object in real space, and

an additional information outputting section configured to display additional information related to said object in real space,

wherein said at least one pointing device includes;

an object pointing section configured to be used by a user to point said object in real space, and

pointed object identification information acquiring section configured to output information showing a feature for identifying an object pointed by said object pointing section,

wherein said control device includes;

a picture analyzing section configured to identify, by analyzing a picture acquired from said picture inputting section, an object in said acquired picture,

a target object position calculating section configured to output position information in said acquired picture for said pointed object and output position information in said acquired picture for said pointed object based on information outputted from said pointed object identification information acquiring section,

a storage section configured to store additional information related to said object in real space,

an additional information extracting section configured to extract at least one of additional information related to said identified object and additional information related to said pointed object, from said storage section, and

an additional information output controlling section configured to perform display control of additional information extracted by said additional information extracting section based on position information calculated by said target object position calculating section.

16. An information displaying method comprising:

arranging at least one display device, at least one pointing device and a control device configured to communicate with said at least one display device and said at least one pointing device, and

said control device identifying an object which can be seen through said at least one display device based on information not from said at least one display device but from said at least one pointing device which points said object in a real space, and performing display control of additional information with respect to said object to on said at least one display device.

17. A non-transitory computer-readable recording medium in which a computer-readable information display program code is stored for executing an information display method on a computing machine as a control device, said method comprising:

arranging at least one display device, at least one pointing device and said control device configured to communicate with said at least one display device and said at least one pointing device; and

said control device identifying an object which can be seen through said at least one display device based on information not from said at least one display device but from said at least one pointing device which points said object in a real space, and performing display control of additional information with respect to said object to on said at least one display device.