INFORMATION PROCESSING APPARATUS, INFORMATION PROCESSING METHOD, AND PROGRAM

• US 20120320046A1
• Filed: 07/06/2012
• Published: 12/20/2012
• Est. Priority Date: 03/30/2010
• Status: Active Grant

• Current Assignee: NS Solutions Corporation (Nippon Steel Corporation)
• Original Assignee: NS Solutions Corporation (Nippon Steel Corporation)
• Inventors: Yokoyama, Masaru, Okamoto, Ricardo Musashi, IHARA, Noboru, Sasao, Kazuhiro, Sakurai, Arata
• Granted Patent: US 9,030,494 B2
• Time in Patent Office: Days
• US Class Current: 345/419
• CPC Class Codes: G05B 2219/32014 Augmented reality assists operator in maintenance, repair, programming, assembly, use of head mounted display with 2-D 3-D display and voice feedback, voice and gesture command
• CPC Class Codes: G05B 23/0216 Human interface functionality, e.g. monitoring system providing help to the user in the selection of tests or in its configuration
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00

Abstract

An information processing apparatus including an imaged image input unit inputting an imaged image of a facility imaged in an imaging device to a display control unit, a measurement information input, unit inputting measurement information measured by a sensor provided in the facility from the sensor to a creation unit, a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information input by the measurement information input unit, and a display control unit overlaying and displaying the virtual image created in the creation unit and the imaged image input by the imaged image input unit on a display device.

Claims

1. An information processing apparatus, comprising:

an imaged image input unit inputting an imaged image of a facility imaged in an imaging device;

a measurement information input, unit inputting measurement information measured by a sensor provided in the facility from the sensor;

a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information inputted by the measurement information input unit; and

a display control unit overlaying and displaying the virtual image created in the creation unit and the imaged image inputted by the imaged image input unit on a display device.

3. The information processing apparatus according to claim 1, further comprising:

a control information input unit inputting control information transmitted to the facility from a control device controlling the facility,

wherein the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the control information inputted by the control information input unit and the measurement information inputted by the measurement information input unit.

4. The information processing apparatus according to claim 1, further comprising:

a control schedule information input unit inputting control schedule information representing contents of control scheduled to be performed on the facility; and

a simulation unit performing simulation to estimate the status of the outside or inside of the facility when control represented by the control schedule information is performed based on the control schedule information inputted by the control schedule information input unit and the measurement information inputted by the measurement information input unit,

wherein the creation unit creates from numeric data representing results of the simulation a three-dimensional virtual image representing the results of the simulation.

5. The information processing apparatus according to claim 1, further comprising:

a determination unit determining an order number corresponding to a product in the facility,

wherein the creation unit creates a virtual image representing the order number determined by the determination unit.

9. The information processing apparatus according to claim 1, wherein

the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the measurement information inputted by the measurement information input unit when the facility is in operation, and

the display control unit overlays and displays on the display device the virtual image created in the creation unit and the imaged image inputted by the imaged image input unit when the facility is in operation, or overlays and displays on the display device a virtual image created in advance and the imaged image inputted by the imaged image input unit when the facility is not in operation.

11. The information processing apparatus according to claim 3, wherein

the facility is a facility in a steel plant,

the sensor is a plurality of temperature sensors, an air pressure sensor, or a pressure sensor inside or outside the facility, and

the control information is blowing of oxygen or gas into the facility, or charging of material into the facility.

2. An information processing apparatus, comprising:

a measurement information input unit inputting measurement information measured by a sensor provided in the facility from the sensor;

a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information inputted by the measurement information input unit; and

a display control unit overlaying and displaying the virtual image created in the creation unit on the facility which is seen through a display device.

6. The information processing apparatus according to claim 2, further comprising

a control information input unit inputting control information transmitted to the facility from a control device controlling the facility,

wherein the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the control information inputted by the control information input unit and the measurement information inputted by the measurement information input unit.

7. The information processing apparatus according to claim 2, further comprising:

a control schedule information input, unit inputting control schedule information representing contents of control scheduled to be performed on the facility; and

a simulation unit performing simulation to estimate the status of the outside or inside of the facility when control represented by the control schedule information is performed based on the control schedule information inputted by the control schedule information input unit and the measurement information inputted by the measurement information input unit,

wherein the creation unit creates from numeric data representing results of the simulation a three-dimensional virtual image representing the results of the simulation.

8. The information processing apparatus according to claim 2, further comprising:

a determination unit determining an order number corresponding to a product in the facility,

wherein the creation unit creates a virtual image representing the order number determined by the determination unit.

10. The information processing apparatus according to claim 2 wherein

the creation unit creates the virtual image representing the status of the outside or inside of the facility based on the measurement information inputted by the measurement information input unit when the facility is in operation, and

the display control unit overlays and displays the virtual image created in the creation unit on the facility which is seen through the display device when the facility is in operation, or overlays and displays a virtual image created in advance on the facility which is seen through the display device when the facility is not in operation.

12. An information processing method executed by an information processing apparatus, the information processing method comprising:

an imaged image input, step of inputting an imaged image of a facility imaged in an imaging device;

a measurement information input step of inputting measurement information measured by a sensor provided in the facility from the sensor;

a creation step of creating a virtual image representing a status of an outside or inside of the facility based on the measurement information inputted in the measurement information input step; and

a display control step of overlaying and displaying the virtual image created in the creation step and the imaged image inputted in the imaged image input step on a display device.

13. An information processing method executed by an information processing apparatus, the information processing method comprising:

a measurement information input step of inputting measurement information measured by a sensor provided in the facility from the sensor;

a creation step of creating a virtual image representing a status of an outside or inside of the facility based on the measurement information inputted in the measurement information input step; and

a display control step of overlaying and displaying the virtual image created in the creation step on the facility which is seen through a display device.

14. A program causing a computer to function as:

an imaged image input unit inputting an imaged image of a facility imaged in an imaging device;

a measurement information input unit inputting measurement information measured by a sensor provided in the facility from the sensor to a creation unit;

a creation unit creating a virtual image representing a status of an outside or inside of the facility used on the measurement information inputted by the measurement information input unit; and

a display control unit overlaying and displaying the virtual image created in the creation unit and the imaged image inputted by the imaged image input unit on a display device.

15. A program causing a computer to function as:

a measurement information input unit inputting measured measurement information by a sensor provided in the facility from the sensor;

a creation unit creating a virtual image representing a status of an outside or inside of the facility based on the measurement information inputted by the measurement information input unit; and

a display control unit overlaying and displaying the virtual image created in the creation unit on the facility which is seen through a display device.

16. An information providing apparatus having a display unit and connected communicably to a storage unit, the information providing apparatus comprising:

a reading unit reading dangerous range information indicating a dangerous range of each state of a dangerous target and dangerous target position information indicating a position of the dangerous target, which are stored in the storage unit;

a position information obtaining unit obtaining apparatus position information calculated from information indicating a position of the information providing apparatus which is detected in a position detection device;

a direction information obtaining unit obtaining direction information calculated from information indicating a direction of the information providing apparatus which is detected in a direction detection device;

a posture information obtaining unit obtaining posture information calculated from information indicating a posture of the information providing apparatus which is detected in a posture detection device;

a determination unit determining a visual field of the information providing apparatus based on the apparatus position information obtained in the position information obtaining unit, the direction information obtained in the direction information obtaining unit, the posture information obtained in the posture information obtaining unit, and visual field information of the information providing apparatus which is defined in advance; and

a display control unit determining a dangerous target included in the visual field based on the apparatus position information, the direction information, the posture information, and the dangerous range information and the dangerous target position information which are read in the reading unit, and generating a dangerous range image for the determined dangerous target and displaying the image on the display unit.

17. The information providing apparatus according to claim 16, wherein

the display control unit calculates conversion information correlating conversion between a coordinate system of a real space and a coordinate system of a display space based on the apparatus position information, the direction information, and the posture information, calculates range coordinates corresponding to the dangerous range information based on coordinates of the dangerous target position information in the coordinate system of the real space, and applies the conversion information to the calculated range coordinates to generate the dangerous range image.

18. The information providing apparatus according to claim 16, further comprising

an input unit inputting a video imaged in an imaging device,

wherein the display control unit calculates conversion information correlating conversion between a coordinate system of a real space and a coordinate system of a display space by analysing the video inputted by the input, unit, calculates range coordinates corresponding to the dangerous range information based on coordinates of the dangerous target position information in the coordinate system of the real space, and applies the conversion information to the calculated range coordinates to generate the dangerous range image.

19. The information providing apparatus according to claim 16, further comprising:

an input unit inputting a video imaged in an imaging device,

wherein the display control unit calculates conversion information correlating conversion between a coordinate system of a real space and a coordinate system of a display space by analyzing the video inputted by the input, unit, determines coordinates of a marker indicating a reference position added to the determined dangerous target in the coordinate system of the real space, calculates range coordinates corresponding to the dangerous range information originating at the determined coordinates, and applies the conversion information to the calculated range coordinates to generate the dangerous range image.

20. The information providing apparatus according to claim 17, wherein

the display control unit controls transmittance of the display unit to make the dangerous target visible via the display unit, and displays the dangerous range image.

21. The information providing apparatus according to claim 18, wherein

the display control unit overlays and displays the dangerous range image on the video inputted by the input unit.

22. The information providing apparatus according to claim 16, wherein

the reading unit further reads movement schedule information indicating a schedule of the dangerous target to move, which is stored in the storage unit, and

when the display control unit determines that the dangerous target does not exist in the visual field, if the display control unit determines that the dangerous target is approaching the information providing apparatus based on the movement schedule information read in the reading unit, and the apparatus position information and a current time, the display control unit generates an image notifying a danger due to the dangerous target and displays the image on the display unit.

23. The information providing apparatus according to claim 16, wherein

when the display control unit generates dangerous range images for plural dangerous targets, the display control unit generates a dangerous range image for each of the plural dangerous targets, and analyzes generated plural dangerous range images to determine a safety zone or an evacuation route where dangers due to the plural dangerous targets do not reach, and further generates an image indicating the determined safety zone or evacuation route and displays the image on the display unit.

24. The information providing apparatus according to claim 16, wherein

the display control unit generates the dangerous range image in a planar form or a stereoscopic form.

25. The information providing apparatus according to claim 16, wherein

the display control unit generates the dangerous range image and displays the image on the display unit when a distance to the dangerous target is equal to or less than a predetermined threshold.

26. An information providing method executed by an information providing apparatus having a display unit and connected communicably to a storage unit, the information providing method comprising:

a reading step of reading dangerous range information indicating a dangerous range of each state of a dangerous target and dangerous target position information indicating a position of the dangerous target, which are stored in the storage unit;

a position information obtaining step of obtaining apparatus position information calculated from information indicating a position of the information providing apparatus which is detected in a position detection device;

a direction information obtaining step of obtaining direction information calculated from information indicating a direction of the information providing apparatus which is detected in a direction detection device;

a posture information obtaining step of obtaining posture information calculated from information indicating a posture of the information providing apparatus which is detected in a posture detection device;

a determination step of determining a visual field of the information providing apparatus based on the apparatus position information obtained in the position information obtaining step, the direction information obtained in the direction information obtaining step, the posture information obtained in the posture information obtaining step, and visual field information of the information providing apparatus which is defined in advance; and

a display control step of determining a dangerous target included in the visual field based on the apparatus position information, the direction information, the posture information, and the dangerous range information and the dangerous target position information which are read in the reading step, and generating a dangerous range image for the determined dangerous target and displaying the image on the display unit.

27. A program causing a computer having a display unit and connected communicably to a storage unit to function as:

a reading unit reading dangerous range information indicating a dangerous range of each state of a dangerous target and dangerous target position information indicating a position of the dangerous target, which are stored in the storage unit;

a position information obtaining unit obtaining apparatus position information calculated from information indicating a position of the computer which is detected in a position detection device;

a direction information obtaining unit obtaining direction information calculated from information indicating a direction of the computer which is detected in a direction detection device;

a posture information obtaining unit obtaining posture information calculated from information indicating a posture of the computer which is detected in a posture detection device;

a determination unit determining a visual field of the computer based on the apparatus position information obtained in the position information obtaining unit, the direction information obtained in the direction information obtaining unit, the posture information obtained in the posture information obtaining unit, and visual field information of the computer which is defined in advance; and

a display control unit determining a dangerous target included in the visual field based on the apparatus position information, the direction information, the posture information, and the dangerous range information and the dangerous target position information which are read in the reading unit, and generating a dangerous range image for the determined dangerous target and displaying the image on the display unit.