Maintenance support method, maintenance support system, and maintenance support program

• US 10,217,088 B2
• Filed: 03/08/2017
• Issued: 02/26/2019
• Est. Priority Date: 03/08/2016
• Status: Expired due to Fees

• Current Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Mihara, Shinya, Nakanishi, Ayaka
• Primary Examiner(s): Bhatnagar, Anand P
• Application Number:
  US15/453,072
• Publication Number:
  US 20170262817A1
• Time in Patent Office: 720 Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06K 7/1417 2D bar codes
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 40/19 Sensors therefor

Abstract

According to one embodiment, a maintenance support method is performed by a wearable device including a first camera and a second camera, and an information processing apparatus. The method includes causing the wearable device to detect identification information of a maintenance target position from a first image shot by the first camera and transmit the identification information and a second image shot by the second camera to the information processing apparatus in correspondence with the detection of the identification information. The method includes causing the information processing apparatus to determine, based on the second image, a situation of a user who wears the wearable device, and record the situation of the user in association with the identification information.

Claims

1. A maintenance support method performed by a wearable device including a first camera and a second camera, which shoot different directions, and an information processing apparatus, the method comprising:

causing the wearable device to detect identification information of a maintenance target position from a first image shot by the first camera and transmit the identification information and a second image shot by the second camera to the information processing apparatus in correspondence with the detection of the identification information; and

causing the information processing apparatus to receive the identification information and the second image, determine, based on the second image, a situation of a user who wears the wearable device, and record the situation of the user in association with the identification information,

wherein the wearable device detects a two dimensional barcode of the maintenance target position from the first image, and transmits an ID of the two dimensional barcode, a detection start time and a detection end time of the two dimensional barcode, the second image, and a shooting time of the second image to the information processing apparatus, and

the information processing apparatus receives the ID, the detection start time, the detection end time, the second image, and the shooting time, determines the situation of the user based on the detection start time, the detection end time, the second image, and the shooting time, and records the situation of the user in association with the identification information.

2. The method according to claim 1, wherein the information processing apparatus detects a pupil of the user from the second image, detects a viewing direction of the user from the pupil, determines a visual confirmation situation of the maintenance target position by the user based on the viewing direction, and records the visual confirmation situation of the user in association with the identification information.

3. The method according to claim 1, wherein the wearable device transmits the second image shot between the detection start time and the detection end time to the information processing apparatus.

4. A non-transitory computer-readable medium comprising a maintenance support program that causes a computer to execute:

detecting identification information of a maintenance target position from a first image shot by a first camera out of the first camera and a second camera of the wearable device;

transmitting the identification information and a second image shot by the second camera in correspondence with the detection of the identification information;

receiving the identification information and the second image;

determining, based on the second image, a situation of a user who wears the wearable device; and

recording the situation of the user in association with the identification information,

wherein the detecting comprises detecting a two dimensional barcode of the maintenance target position from the first image,

the transmitting comprises transmitting an ID of the two dimensional barcode, a detection start time and a detection end time of the two dimensional barcode, the second image, and a shooting time of the second image,

the receiving comprises receiving the ID, the detection start time, the detection end time, the second image, and the shooting time,

the determining comprises determining the situation of the user based on the detection start time, the detection end time, the second image, and the shooting time, and

the recording comprises recording the situation of the user in association with the identification information.

5. The medium according to claim 4, wherein the determining comprises detecting a pupil of the user from the second image, detecting a viewing direction of the user from the pupil, and determining a visual confirmation situation of the maintenance target position by the user based on the viewing direction, and

the recording comprises recording the visual confirmation situation of the user in association with the identification information.

6. The medium according to claim 4, wherein the transmitting comprises transmitting the second image shot between the detection start time and the detection end time.

7. A maintenance support system comprising a wearable device including a first camera and a second camera, which shoot different directions, and an information processing apparatus,

wherein the wearable device

is configured to detect identification information of a maintenance target position from a first image shot by the first camera, and

transmit the identification information and a second image shot by the second camera to the information processing apparatus in correspondence with the detection of the identification information, and

the information processing apparatus:

is configured to receive the identification information and the second image,

determine, based on the second image, a situation of a user who wears the wearable device, and

a recording unit configured to record the situation of the user in association with the identification information;

wherein the detecting comprises detecting a two dimensional barcode of the maintenance target position from the first image,

the transmitting comprises transmitting an ID of the two dimensional barcode, a detection start time and a detection end time of the two dimensional barcode, the second image, and a shooting time of the second image,

the receiving comprises receiving the ID, the detection start time, the detection end time, the second image, and the shooting time,

the determining comprises determining the situation of the user based on the detection start time, the detection end time, the second image, and the shooting time, and

the recording comprises recording the situation of the user in association with the identification information.

8. The system according to claim 7, wherein the information processing apparatus detects a pupil of the user from the second image, detects a viewing direction of the user from the pupil, determines a visual confirmation situation of the maintenance target position by the user based on the viewing direction, and records the visual confirmation situation of the user in association with the identification information.

9. The system according to claim 7, wherein the wearable device transmits the second image shot between the detection start time and the detection end time to the information processing apparatus.