SYSTEMS METHODS AND COMPUTER PROGRAM PRODUCTS FOR PROVIDING AUGMENTED SHOPPING INFORMATION

• US 20090287587A1
• Filed: 05/15/2008
• Published: 11/19/2009
• Est. Priority Date: 05/15/2008
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.)
• Original Assignee: Sony Ericsson Mobile Communications USA Incorporated (Sony Group Corp.)
• Inventors: Bloebaum, L. Scott, Rutledge, Melissa
• Granted Patent: US 7,707,073 B2
• Time in Patent Office: Days
• US Class Current: 705/27
• CPC Class Codes: G06Q 10/08 Logistics, e.g. warehousing, loading or distribution; Inventory or stock management
• CPC Class Codes: G06Q 30/06 Buying, selling or leasing transactions
• CPC Class Codes: G06Q 30/0633 Lists, e.g. purchase orders, compilation or processing
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 4/33 for indoor environments, e.g. buildings
• CPC Class Codes: H04W 4/80 Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Methods for providing augmented shopping information to a user of a mobile terminal in a store include providing a shopping list identifying a plurality of items and obtaining information regarding physical locations of respective ones of the items in the store. The location of the mobile terminal in the store is established. An image of a view from the location is obtained. An item having an associated location in the obtained image is identified based on the obtained information regarding physical locations. A position within the image corresponding to the respective physical location of the identified item is determined. A graphic indication of the identified item is added to the obtained image at the determined position within the image to provide an augmented image of the view that is displayed to the user on a display of the mobile terminal.

Claims

1. A method for providing augmented shopping information to a user of a mobile terminal in a store, comprising:

providing a shopping list identifying a plurality of items;

obtaining information regarding physical locations of respective ones of the items in the store;

establishing a location of the mobile terminal in the store;

obtaining an image of a view from the location, wherein a perspective of the view is defined by the user pointing a camera used to obtain the image;

identifying at least one of the items having an associated location in the obtained image based on the obtained information regarding physical locations;

determining a position within the image corresponding to the respective physical location of the identified at least one of the items;

adding a graphic indication of the identified at least one of the items to the obtained image at the determined position within the image to provide an augmented image of the view; and

displaying the augmented image of the view to the user on a display of the mobile terminal.

2. The method of claim 1, wherein establishing the location comprises receiving an input from the user designating the location and/or receiving data from a known physical location wireless transmitter within the store.

3. The method of claim 2, wherein establishing the location includes receiving the data from the known physical location wireless transmitter within the store and wherein the known physical location wireless transmitter comprises a near field communication transmitter.

4. The method of claim 2, wherein establishing the location includes receiving the data from the known physical location wireless transmitter within the store and wherein the known physical location wireless transmitter comprises a plurality of wireless local area network transmitters and establishing the location includes determining the location based on triangulation using signals received from the plurality of wireless local area network transmitters.

5. The method of claim 2, wherein establishing the location includes establishing an orientation of the mobile terminal based on a magnetic sensor within the mobile terminal, wherein the orientation corresponds to a direction in which the mobile terminal is aligned as determined by the magnetic sensor in the mobile terminal.

6. The method of claim 1, wherein establishing the location comprises receiving an input from the user designating the location, wherein the input comprises designation of the obtained image as the view from the location and wherein establishing the location comprises:

detecting an element in the obtained image; and

establishing the location based on the detected element.

7. The method of claim 1, wherein the image comprises a video image and obtaining the image includes obtaining a sequence of images to provide the video image and wherein identifying at least one of the items having an associated location in the obtained image, determining the position within the image, adding the graphic indication and displaying the augmented image are repeatedly performed for respective ones of the sequence of images responsive to detecting a change in the view from the location.

8. The method of claim 1, wherein obtaining information further comprises obtaining supplemental information regarding the respective ones of the items in the store and wherein adding the graphic indication includes modifying the graphic indication based on the supplemental information.

9. The method of claim 8, wherein the supplemental information includes pricing information associated with the respective items and wherein modifying the graphic indication includes including the pricing information in the graphic indication.

10. The method of claim 1, wherein obtaining information further comprises obtaining an identification and physical location of a substitute item for at least one of the items that has an associated location in the obtained image and wherein the method further comprises including a graphic indication of the identified substitute item in the augmented image at a position in the image corresponding to the physical location of the substitute item.

11. The method of claim 1, wherein obtaining information regarding physical locations comprises:

establishing a communication connection between the mobile terminal and a store inventory server;

providing the shopping list to the store inventory server using the established communication connection; and

obtaining the information from the store inventory server responsive to providing the shopping list to the store inventory server.

12. The method of claim 1, further comprising providing a user profile for the store for the user and wherein establishing a location comprises establishing the location based on the user profile.

13. The method of claim 12, wherein obtaining information further comprises obtaining supplemental information regarding the respective ones of the items in the store based on the user profile.

14. The method of claim 1, further comprising:

detecting a change in orientation of the mobile terminal to a specified orientation, wherein the change in orientation corresponds to a change in an alignment direction of the mobile terminal; and

displaying an overhead view of a layout of the store on the display of the mobile terminal responsive to detection of the change in orientation of the mobile terminal.

15. The method of claim 1, wherein displaying the augmented image is preceded by obtaining a selection from the user of a subset of the items and adding a graphic indication comprises adding a graphic indication only for ones of the selected subset of the items having an associated location in the obtained image.

16. A wireless mobile terminal configured to carry out the method of claim 1.

17. A computer program product for providing augmented shopping information to a user of a mobile terminal in a store, the computer program product comprising computer program code embodied in a computer readable medium, the computer program code comprising program code configured to carry out the method of claim 1.

21. The method of claim 1, wherein the camera is in the mobile terminal.

18. A wireless mobile terminal, comprising:

a shopping list identifying a plurality of items desired by a user;

a wireless communication circuit configured to obtain information regarding physical locations of items in a store;

a camera configured to obtain an image of a view from a location of the mobile terminal;

a location determination module configured to establish a location of the mobile terminal in the store;

a shopping aide module configured to identifying at least one of the items on the shopping list having an associated location in the obtained image based on the obtained information regarding physical locations, to determine a position within the image corresponding to the respective physical location of the identified at least one of the items and to add a graphic indication of the identified at least one of the items to the obtained image at the determined position within the image to provide an augmented image of the view; and

a display configured to display the augmented image of the view to the user.

19. The wireless mobile terminal of claim 18, wherein the wireless communication circuit is further configured to obtain supplemental information regarding the items in the store and wherein the shopping aide module is further configured to modify the graphic indication based on the supplemental information.

20. The wireless mobile terminal of claim 18, further comprising a housing and wherein the wireless communication circuit, the camera, the location determination module and the shopping aide module are positioned in the housing and wherein the display is coupled to the housing.