Hands-free and non-visually occluding object information interaction system

• US 8,600,761 B2
• Filed: 09/09/2008
• Issued: 12/03/2013
• Est. Priority Date: 09/09/2008
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Marsh, Joseph C., Smith, Eric M.
• Primary Examiner(s): AZAD, ABUL K
• Application Number:
  US12/206,902
• Publication Number:
  US 20100063821A1
• Time in Patent Office: 1,911 Days
• Field of Search: 704270-275
• US Class Current: 704/275
• CPC Class Codes: G10L 13/00 Speech synthesis; Text to speech systems
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence

Abstract

Technologies are described herein for providing a hands-free and non-visually occluding interaction with object information. In one method, a visual capture of a portion of an object is received through a hands-free and non-visually occluding visual capture device. An audio capture is also received from a user through a hands-free and non-visually occluding audio capture device. The audio capture may include a request for information about a portion of the object in the visual capture. The information is retrieved and is transmitted to the user for playback through a hands-free and non-visually occluding audio output device.

Claims

1. A method for providing a hands-free and non-visually occluding object information interface, comprising:

receiving a direct visual capture of a portion of an object through a hands-free and non-visually occluding visual capture device configured for attachment to a user at the same location as the object;

receiving an audio capture from the user through a hands-free and non-visually occluding audio capture device, the audio capture comprising a request for information about a portion of the object in the visual capture;

retrieving the information; and

transmitting the information to the user for playback through a hands-free and non-visually occluding output device.

2. The method of claim 1, wherein receiving a visual capture of a portion of an object through a hands-free and non-visually occluding visual capture device comprises receiving a video capture of a portion of the object through a video capture device.

3. The method of claim 1, wherein receiving a visual capture of a portion of an object through a hands-free visual capture device comprises receiving an image capture of a portion of the object through a hands-free image capture device.

4. The method of claim 1, wherein the hands-free and non-visually occluding visual capture device is operated through voice commands spoken into the hands-free and non-visually occluding audio capture device.

5. The method of claim 1, wherein the hands-free and non-visually occluding visual capture device mimics an actual view from the user operating the hands-free and non-visually occluding visual capture device.

6. The method of claim 1, wherein retrieving the information comprises:

converting the audio capture into text comprising the request for information; and

retrieving the information about a portion of the object according to the request for information in the text.

7. The method of claim 1, wherein transmitting the information to the user for playback through a hands-free and non-visually occluding output device comprises:

converting the information to a spoken dialog through a speech synthesizer; and

transmitting the spoken dialog to the user for playback through the hands-free and non-visually occluding audio output device.

8. The method of claim 1, further comprising determining a field of view of the visual capture device with respect to the object.

9. The method of claim 8, further comprising aligning the field of view with a known representation of the object to determine the field of view of the visual capture device with respect to the object.

10. The method of claim 9, wherein the known representation of the object comprises a computer aided design (CAD) generated representation of the object.

11. The method of claim 8, further comprising mapping the portion of the object in the visual capture utilizing a simultaneous location and mapping (SLAM) analysis of the visual capture.

12. The method of claim 1, wherein retrieving the information comprises retrieving information identifying the portion of the object.

13. The method of claim 1, wherein retrieving the information comprises retrieving instructions for performing a maintenance task on the portion of the object.

14. A method for providing a hands-free and non-visually occluding object interface, comprising:

receiving a video capture of a portion of an object through a hands-free and non-visually occluding video capture device attached to a user operating the hands-free and non-visually occluding video capture device at the same location as the object, the hands-free and non-visually occluding video capture device configured to directly capture video of the portion of the object as actually viewed by the user;

receiving an audio capture from the user through a hands-free and non-visually occluding audio capture device attached to the user, the audio capture comprising a request for information about the portion of the object in the video capture;

converting the audio capture into text comprising the request for information;

retrieving the information according to the request for information in the text;

converting the information into a spoken dialog through a speech synthesizer; and

transmitting the information to the user for playback through a hands-free and non-visually occluding audio output device attached to the user.

15. The method of claim 14, wherein the hands-free and non-visually occluding video capture device comprises a head-mounted video camera or a helmet-mounted video camera.

16. The method of claim 14, wherein the hands-free and non-visually occluding audio capture device and the hands-free and non-visually occluding audio output device comprise a headset attached to the user.

17. The method of claim 14, wherein the request for information about the portion of the object in the video capture comprises predefined commands for processing the video capture.

18. The method of claim 14, wherein the hands-free and non-visually occluding video capture device is controlled by instructions spoken into the hands-free audio capture device.

19. A non-transitory computer-readable medium having computer-executable instructions stored thereon which, when executed by a computer, cause the computer to:

receive a video capture of a portion of an object through a hands-free and non-visually occluding video capture device attached to a user operating the hands-free and non-visually occluding video capture device at the same location as the object, the hands-free and non-visually occluding video capture device being configured to directly capture video of the portion of the object as actually viewed by the user;

receive an audio capture from the user through a hands-free and non-visually occluding audio capture device attached to the user, the audio capture comprising a request for instructions for performing a maintenance task on the portion of the object in the video capture;

convert the audio capture into text comprising the request for instructions;

retrieve the instructions according to the request for instructions in the text;

convert the instructions into a spoken dialog through a speech synthesizer; and

transmit the instructions to the user for playback through the a hands-free and non-visually occluding audio output device attached to the user.

20. The non-transitory computer-readable storage medium of claim 19, wherein the object comprises an aircraft component or assembly.