Augmented reality based interactive troubleshooting and diagnostics for a vehicle

• US 9,472,028 B2
• Filed: 06/23/2014
• Issued: 10/18/2016
• Est. Priority Date: 06/23/2014
• Status: Active Grant

• Current Assignee: GM Global Technology Operations LLC (General Motors Company)
• Original Assignee: GM Global Technology Operations LLC (General Motors Company)
• Inventors: Abdel-Rahman, Akram M., Gautama, Neeraj R., Eng, Donald
• Primary Examiner(s): Alharbi, Adam
• Application Number:
  US14/311,529
• Publication Number:
  US 20150371455A1
• Time in Patent Office: 848 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06Q 10/20 Administration of product repair or maintenance
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/0808 Diagnosing performance data testing of vehicles G01M17/00; testing of electrical installation on vehicles G01R31/005

Abstract

An augmented reality based interactive troubleshooting and diagnostic system and related operating methods are presented here, including a diagnostic communication method for a vehicle having an onboard diagnostics subsystem. An exemplary embodiment of the method uses a mobile user device to obtain self-diagnostic information generated by the onboard diagnostics subsystem. The method continues by sending a query to a solution database system, wherein the query includes the obtained self-diagnostic information and vehicle configuration data descriptive of the vehicle. The mobile user device receives solution data provided by the solution database system in response to the query. The solution data addresses at least one topic associated with the obtained self-diagnostic information. The method continues by operating the camera and the display element in response to the received solution data to perform an augmented reality procedure associated with the at least one topic.

Claims

1. A diagnostic communication method for a vehicle comprising an onboard diagnostics subsystem, the method comprising:

obtaining, with a mobile user device comprising a camera and a display element, self diagnostic information generated by the onboard diagnostics subsystem of the vehicle;

sending a query to a solution database system, the query comprising the obtained self diagnostic information and vehicle configuration data descriptive of the vehicle;

receiving solution data at the mobile user device, the solution data provided by the solution database system in response to the query, and the solution data addressing at least one topic associated with the obtained self diagnostic information; and

operating the camera and the display element of the mobile user device in response to the received solution data to perform an augmented reality procedure associated with the at least one topic, wherein, during the augmented reality procedure, the camera captures real time image data of an object of interest on the vehicle, the display element renders the captured real time image data, and the mobile user device concurrently renders supplemental audio, visual, or haptic feedback information corresponding to the received solution data.

2. The method of claim 1, wherein the sending comprises:

sending the query from the mobile user device to the solution database system.

3. The method of claim 1, wherein the sending comprises:

sending the query from an onboard communication subsystem of the vehicle to the solution database system.

4. The method of claim 1, wherein the sending comprises:

sending the query to a network-based solution database system that is remotely located relative to both the vehicle and the mobile user device.

5. The method of claim 1, wherein the sending comprises:

sending the query to a locally maintained solution database system that is resident at the mobile user device.

6. The method of claim 1, wherein the sending comprises:

sending the query to an onboard solution database system that is resident at the vehicle.

7. The method of claim 1, wherein the sending comprises:

sending the query to an external computing device that comprises the solution database system, wherein the external computing device is communicatively coupled to the mobile user device to receive the query.

8. The method of claim 1, wherein the sending comprises:

sending the query to an external computing device that comprises the solution database system, wherein the external computing device is communicatively coupled to an onboard communication subsystem of the vehicle to receive the query.

9. The method of claim 1, further comprising:

communicating the self-diagnostic information and the vehicle configuration data from an onboard communication subsystem of the vehicle to the solution database system.

10. The method of claim 1, further comprising:

acquiring, at the mobile user device, user feedback related to quality of the augmented reality procedure; and

communicating the user feedback from the mobile user device to the solution database system for use in updating the solution database system.

11. The method of claim 1, further comprising:

capturing, with the camera of the mobile user device, an image of a potentially problematic component, feature, or element of the vehicle;

generating a diagnostic check request based on the captured image; and

communicating the diagnostic check request from the mobile user device to the onboard diagnostics subsystem, wherein the self diagnostic information is generated by the onboard diagnostics subsystem in response to the diagnostic check request.

12. A computer readable storage media comprising executable instructions capable of performing a method comprising:

obtaining self diagnostic information generated by an onboard diagnostics subsystem of a vehicle;

sending a query to a solution database system, the query comprising the obtained self diagnostic information and vehicle configuration data descriptive of the vehicle;

receiving solution data, the solution data provided by the solution database system in response to the query, and the solution data addressing at least one topic associated with the obtained self diagnostic information; and

operating a camera and a display element of a mobile user device in response to the received solution data to perform an augmented reality procedure associated with the at least one topic, wherein, during the augmented reality procedure, the camera captures real time image data of an object of interest on the vehicle, the display element renders the captured real time image data, and the mobile user device concurrently renders supplemental audio, visual, or haptic feedback information corresponding to the received solution data.

13. The computer readable storage media of claim 12, wherein the method performed by the executable instructions further comprises:

acquiring user feedback related to quality of the augmented procedure; and

communicating the user feedback to the solution database system for use in updating the solution database system.

14. The computer readable storage media of claim 12, wherein the method performed by the executable instructions further comprises:

capturing, with the camera of the mobile user device, an image of a potentially problematic component, feature, or element of the vehicle;

generating a diagnostic check request based on the captured image; and

communicating the diagnostic check request from the mobile user device to the onboard diagnostics subsystem, wherein the self diagnostic information is generated by the onboard diagnostics subsystem in response to the diagnostic check request.

15. A diagnostic communication method for a vehicle comprising an onboard diagnostics subsystem, the method comprising:

obtaining, with a mobile user device comprising a camera and a display element, self diagnostic information generated by the onboard diagnostics subsystem and vehicle configuration data descriptive of the vehicle;

retrieving initial solution data based on the obtained self diagnostic information and the obtained vehicle configuration data, the solution data addressing at least one topic associated with the obtained diagnostic information; and

operating the camera and the display element of the mobile user device in response to the retrieved initial solution data to perform an initial augmented reality procedure associated with the at least one topic, wherein, during the initial augmented reality procedure, the camera captures real time image data of an object of interest on the vehicle, the display element renders the captured real time image data, and the mobile user device concurrently renders supplemental audio, visual, or haptic feedback information corresponding to the received solution data.

16. The method of claim 15, wherein the retrieving comprises:

retrieving the initial solution data from a locally maintained solution database system that is resident at the mobile user device.

17. The method of claim 15, wherein the retrieving comprises:

retrieving the initial solution data from a network-based solution database system that is remotely located relative to both the vehicle and the mobile user device.

18. The method of claim 15, further comprising:

sending a query to a network-based solution database system that is remotely located relative to both the vehicle and the mobile user device, the query comprising the obtained self diagnostic information and the vehicle configuration data;

receiving additional solution data at the mobile user device, the additional solution data provided by the network-based solution database system in response to the query, and the additional solution data addressing the at least one topic associated with the obtained self diagnostic information; and

operating the camera and the display element in response to the received additional solution data to perform an additional augmented reality procedure associated with the at least one topic.

19. The method of claim 15, further comprising:

acquiring, at the mobile user device, user feedback related to quality of the initial augmented reality procedure, the additional augmented reality procedure, or both the initial augmented reality procedure and the additional augmented reality procedure; and

communicating the user feedback from the mobile user device to the solution database system for use in updating the solution database system.

20. The method of claim 15, further comprising:

capturing, with the camera of the mobile user device, an image of a potentially problematic component, feature, or element of the vehicle;

generating a diagnostic check request based on the captured image; and

communicating the diagnostic check request from the mobile user device to the onboard diagnostics subsystem, wherein the self diagnostic information is generated by the onboard diagnostics subsystem in response to the diagnostic check request.