Wire harness assembly

• US 9,589,390 B2
• Filed: 05/13/2015
• Issued: 03/07/2017
• Est. Priority Date: 05/13/2015
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: DeStories, Jason Gerald, Edman, Frederick C., Crocco, James R.
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US14/711,294
• Publication Number:
  US 20160335800A1
• Time in Patent Office: 664 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2113/16 Cables, cable trees or wire harnesses
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06F 30/15 Vehicle, aircraft or watercraft design
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A method and apparatus for an augmented reality vision system. A first input selecting a branch is received. Task information associated with a wire harness containing the branch is retrieved based on the first input. A second input requesting information related to inserting a wire is received. A plug map of a plug of the branch of the wire harness is generated based on the second input and using the task information associated with the wire harness. The plug map is displayed on a display system. A location of a selected wire is indicated in the plug map.

Claims

1. An augmented reality vision system for a wire harness assembly environment, the augmented reality vision system comprising:

a display system having a viewing area configured to display an image, wherein the display system is head-mounted; and

a processor running a mobile application to provide data regarding the wire harness, in which the mobile application receives a first input selecting a branch, retrieves task information associated with the wire harness containing the branch based on the first input, receives a second input requesting information related to inserting a wire, generates a plug map of a plug of the branch of the wire harness based on the second input and using the task information associated with the wire harness, displays the plug map on the display system, and indicates a location of a selected wire in the plug map, wherein the mobile application is further configured to generate video data of a combination of a display of the plug map on the display system and live video feed of the wire harness, and wherein the mobile application is further configured to provide the video data to a second display system viewable by a second person other than a first person forming the wire harness.

2. The augmented reality vision system of claim 1 further comprising:

a computer system configured to send the task information regarding the wire harness to the display system to display on the display system.

3. The augmented reality vision system of claim 2, wherein the camera system is configured to generate image data related to a number of tasks performed in the wire harness assembly environment.

4. The augmented reality vision system of claim 2, wherein the camera system is configured to capture an image of an indicator of a shop order.

5. The augmented reality vision system of claim 1, wherein the mobile application is further configured to generate a record of operations of tasks performed while forming the wire assembly.

6. The augmented reality vision system of claim 1 further comprising:

an input device associated with the display system, and is configured to receive the first input, wherein the input device is a touch pad.

7. The augmented reality vision system of claim 1, wherein the mobile application uses additional input from a camera system to verify that the wire has been inserted, and wherein the mobile application additionally prevents display of a subsequent task to be performed with respect to the wire assembly until inserting the wire has been verified as being completed correctly.

8. The augmented reality vision system of claim 1, wherein the mobile application further provides, simultaneously with a display of the plug map on the display system, a second display of instructions and checklists associated with forming the wire harness on the display system.

9. An augmented reality vision system for a wire harness manufacturing environment, the augmented reality vision system comprising:

a display system having a viewing area that is substantially transparent, wherein the viewing area is configured to display an image, wherein the display system takes the form of augmented reality glasses;

an input device associated with the display system, wherein the input device is configured to receive at least one of a first input or a second input;

a camera system associated with the display system; and

a processor associated with the display system and running a mobile application to provide data regarding the wire harness in which the mobile application receives the first input selecting a branch, retrieves task information associated with the wire harness containing the branch based on the first input, receives the second input requesting information related to inserting a wire, generates a plug map of a plug of the branch of the wire harness based on the second input and using the task information associated with the wire harness, displays the plug map on the display system, and indicates a location of a selected wire in the plug map, and wherein the mobile application is further configured to generate a record of operations of tasks performed while forming the wire assembly.

18. The augmented reality vision system of claim 9, wherein the record of operations include image data taken by the camera system representing what is seen on the display system.

19. The augmented reality vision system of claim 9, wherein augmented reality system further comprises:

a microphone that is part of the augmented reality glasses, and wherein the record of operations include notes recorded by voice input captured by the microphone.

20. The augmented reality vision system of claim 9, wherein the record of operations include image data taken by the camera system representing what is seen on the display system, and wherein augmented reality system further comprises:

a microphone that is part of the augmented reality glasses, wherein the record of operations further includes notes recorded by voice input captured by the microphone; and

a communication system that is part of the augmented reality glasses, the communication system configured to transmit the image data and the notes to a second display system for display to a second user.

10. A method of displaying an image using an augmented reality vision system, the method comprising:

receiving a first input selecting a branch;

retrieving task information associated with a task regarding a wire harness containing the branch based on the first input;

receiving a second input requesting information related to inserting a wire;

generating a plug map of a plug of the branch of the wire harness based on the second input and using the task information associated with the wire harness;

displaying the plug map on a display system;

indicating a location of a selected wire in the plug map;

using the camera system to take a second image of the wire harness after the first task is performed;

comparing the first image to the second image to form a comparison;

verifying that the first task was completed correctly based on the comparison to form a verification;

responsive to the verification indicating successful completion of the first task, displaying an indication that the first task was completed successfully;

responsive to the verification indicating unsuccessful completion of the first task, displaying an alert that the first task was not completed successfully; and

preventing display of a second task to be performed with respect to the wire assembly until the first task is completed successfully.

11. The method of claim 10, wherein the augmented reality vision system comprises augmented reality glasses, and wherein retrieving the task information is performed by retrieving the task information from a data storage that is part of the augmented reality vision system.

12. The method of claim 11 further comprising:

indicating the selected wire has been completed in response to receiving a third input; and

indicating a location of a next wire in the number of wires after indicating the selected wire has been completed.

13. The method of claim 10, further comprising:

receiving, from the second person via the communication device, assistance feedback that is based on the video data, the assistance feedback communicating assistance in forming the wire harness by the first person.

14. The method of claim 10 further comprising:

displaying strip and crimp information related to the selected wire prior to displaying the plug map.

15. The method of claim 14, wherein the strip and crimp information includes at least one of a wire type, a wire length, or a tool type.

16. The method of claim 10 further comprising:

creating a recommended order of installation of the wires into the plug of the branch of the wire harness based on a knowledge base.

17. The method of claim 10, further comprising:

generating, by the augmented reality vision system, video data of a combination of a display of the plug map on the display system and live video feed of the wire harness; and

communicating, via a communication device that is part of the augmented reality vision system, the video data to a second display system viewable by a second person other than a first person forming the wire harness while using the augmented reality vision system.

21. A method of displaying an image using an augmented reality vision system, the method comprising:

receiving a first input selecting a branch;

retrieving task information associated with a task regarding a wire harness containing the branch based on the first input;

receiving a second input requesting information related to inserting a wire;

generating a plug map of a plug of the branch of the wire harness based on the second input and using the task information associated with the wire harness;

displaying the plug map on a display system;

indicating a location of a selected wire in the plug map;

using the camera system to take a second image of the wire harness after the first task is performed;

comparing the first image to the second image to form a comparison;

verifying that the first task was completed correctly based on the comparison to form a verification;

responsive to the verification indicating successful completion of the first task, displaying an indication that the first task was completed successfully;

responsive to the verification indicating unsuccessful completion of the first task, displaying an alert that the first task was not completed successfully;

preventing display of a second task to be performed with respect to the wire assembly until the first task is completed successfully;

generating a record of operations of tasks performed while forming the wire assembly;

generating video data of a combination of a display of the plug map on the display system and live video feed of the wire harness;

providing the video data to a second display system viewable by a second person other than a first person forming the wire harness; and

receiving, from the second person via the communication device, assistance feedback that is based on the video data, the assistance feedback communicating assistance in completing the first task by the first person.