Providing vehicle owner's manual information using object recognition in a mobile device

• US 9,552,519 B2
• Filed: 06/02/2014
• Issued: 01/24/2017
• Est. Priority Date: 06/02/2014
• Status: Active Grant

• Current Assignee: General Motors LLC (General Motors Company)
• Original Assignee: General Motors LLC (General Motors Company)
• Inventors: Gurovich, Danilo E., Fry, Keith A., McCutchen, Shane M., Rudman, Daniel E., Goree, Joseph M. II
• Primary Examiner(s): HEIDEMANN, JASON E
• Application Number:
  US14/293,656
• Publication Number:
  US 20150347848A1
• Time in Patent Office: 967 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/30 in albums, collections or shared content, e.g. social network photos or video
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/223 Execution procedure of a spoken command

Abstract

There is provided a communication system that includes a vehicle and a mobile device, and a method uses that system to provide vehicle owner'"'"'s manual information. The method includes the steps of: receiving a digital image of a portion of a vehicle; identifying an object of the vehicle in the digital image; associating the object with at least one reference object image stored in an object library comprising a plurality of reference object images; displaying owner'"'"'s manual information based on the association; and storing at least a portion of the digital image in the object library as a new reference object image for future identifications.

Claims

1. A computer program product, comprising:

a non-transitory computer readable medium having stored thereon program instructions for use by a camera-enabled mobile device, wherein, upon execution by the mobile device, the program instructions stored on the computer readable medium cause the mobile device to carry out the steps of;

receiving a first digital image of a portion of a vehicle, wherein the portion of the vehicle is in a stationary state at a time the first digital image is received;

identifying a primary object of the vehicle in the first digital image;

associating the primary object with at least one reference object image previously stored in an object library comprising a plurality of reference object images;

displaying owner'"'"'s manual information based on the association;

in response to associating the primary object with the at least one reference object image, adding at least a portion of the first digital image to the object library by storing the at least a portion of the first digital image in the object library as one of the plurality of reference object images, wherein the at least a portion includes the identified primary object; and

thereafter, receiving a second digital image and repeating the identifying, associating, and displaying steps with respect to the second digital image,

wherein repeating the associating step includes;

parsing a plurality of reference object images in the object library to associate at least one of the plurality of reference object images with a primary object in the second digital image, wherein the parsed plurality of reference object images includes the added at least a portion of the first digital image; and

using one or more of the plurality of reference object images as a basis for displaying owner'"'"'s manual information associated with the second digital image, wherein the used one or more of the plurality of reference object images includes the added at least a portion of the first digital image.

2. The computer program product of claim 1, wherein the identifying step includes determining the identity of the primary object based on the primary object'"'"'s location within a centralized region of the first digital image.

3. The computer program product of claim 1, wherein the object library includes more than one reference object image for at least some vehicle primary objects.

4. The computer program product of claim 3, wherein for at least some vehicle primary objects, the more than one reference object image includes one or more of the reference object image captured at a first photographic angle, the reference object image captured at a second photographic angle different than the first photographic angle, and a non-photographic version of the reference object image.

5. The computer program product of claim 1, wherein the storing step further comprises one of:

storing the entire first digital image in the object library, or storing only a portion of the first digital image in the object library.

6. The computer program product of claim 1, wherein the added at least a portion of the first digital image stored in the object library as one of the plurality of reference object images has at least one different lighting or color characteristic than the previously stored reference object images.

7. The computer program product of claim 1, wherein the displaying step further comprises displaying one or more menu items directly-associated with the primary object, or displaying one or more predictive menu items, wherein the predictive menu items are indirectly-associated with the primary object, or both.

8. The computer program product of claim 1, further comprising automatically highlighting at least one object in the digital scan to suggest owner'"'"'s manual information that is available to the user.

9. The computer program product of claim 1, wherein the program instructions further comprise identifying a secondary object of the vehicle in the first digital image while concurrently identifying the primary object.

10. The computer program product of claim 9, wherein the program instructions further comprise:

associating the secondary object with at least one reference object image stored in an object library,

wherein the displaying step includes;

displaying information directly-associated with both the first and second objects, or

displaying information pertaining to a classification of objects to which both the first and second objects are associated.

11. The computer program product of claim 10, wherein the step of identifying the secondary object includes determining the identity of the secondary object based the secondary object'"'"'s location outside the centralized region of the first digital image.

12. The computer program product of claim 1, wherein the program instructions further comprise:

receiving speech audio containing one or more identifying terms associated with at least one object identified in the first digital image;

determining the one or more identifying terms using an automatic speech recognition (ASR) engine; and

determining the owner'"'"'s manual information to be displayed based on the one or more identifying terms and the identified and associated primary object of the first digital image.

13. A computer program product, comprising:

a non-transitory computer readable medium having stored thereon program instructions for use by a camera-enabled mobile device, wherein, upon execution by the mobile device, the program instructions stored on the computer readable medium cause the mobile device to carry out the steps of;

receiving a digital scan containing at least one vehicle object;

receiving speech audio containing one or more words associated with vehicle object(s);

processing the received speech audio using an automatic speech recognition (ASR) engine in the mobile device;

identifying the at least one vehicle object based on the digital scan and based on the one or more words of the processed speech audio, wherein the identifying step further comprises associating the at least one vehicle object with at least one reference object image stored in an object library of the mobile device that comprises a plurality of reference object images; and

in response to the identification, displaying owner'"'"'s manual information via a display of the mobile device, wherein the owner'"'"'s manual information is associated with the at least one vehicle object while displaying the at least one vehicle object in the digital scan,

wherein, when the received and processed speech audio includes a query associated with the digital scan, the displaying step includes displaying refined owner'"'"'s manual information that is pertinent to the query.

14. The computer program product of claim 13, further comprising storing the digital scan in the object library as a new reference object image for future identifications, or storing the speech audio in an audio library for future identifications, or both.

15. The computer program product of claim 13, the identifying step further comprising automatically highlighting at least one object in the digital scan to suggest owner'"'"'s manual information that is available to the user.

16. The computer program product of claim 13, wherein the at least one reference object image includes one or more of a reference object image captured at a first photographic angle, the same reference object image captured at a second photographic angle different than the first photographic angle, and a graphic-design version of the same reference object image.

17. The computer program product of claim 13, wherein the at least one vehicle object includes a first vehicle object and a second vehicle object, wherein the displaying step further comprises concurrently displaying owner'"'"'s manual information pertaining to both the first and second vehicle objects.

18. The computer program product of claim 17, wherein the first vehicle object is located within a centralized region of the digital scan and the second vehicle object is located at least partially outside of the centralized region, wherein the owner'"'"'s manual information pertaining to the first vehicle object is displayed more prominently than the owner'"'"'s manual information pertaining to the second vehicle object.

19. The computer program product of claim 13, wherein the at least one vehicle object is in a stationary state at a time the digital scan is received.

20. A method of providing vehicle owner'"'"'s manual information, comprising the steps of:

receiving a first digital image of a portion of a vehicle, wherein the portion of the vehicle is in a stationary state at a time the first digital image is received;

identifying an object of the vehicle in the first digital image;

associating the object with at least one reference object image previously stored in an object library comprising a plurality of reference object images;

displaying owner'"'"'s manual information based on the association;

in response to associating the object with the at least one reference object image, adding at least a portion of the first digital image to the object library by storing the at least a portion of the first digital image in the object library as one of the plurality of reference object images, wherein the at least a portion includes the identified object; and

thereafter, receiving a second digital image and repeating the identifying, associating, and displaying steps with respect to the second digital image,

wherein repeating the associating step includes;

parsing a plurality of reference object images in the object library to associate at least one of the plurality of reference object images with an object in the second digital image, wherein the parsed plurality of reference object images includes the added at least a portion of the first digital image; and

using one or more of the plurality of reference object images as a basis for displaying owner'"'"'s manual information associated with the second digital image, wherein the used one or more of the plurality of reference object images includes the added at least a portion of the first digital image.