System and method for selecting targets in an augmented reality environment

• US 10,388,070 B2
• Filed: 11/12/2015
• Issued: 08/20/2019
• Est. Priority Date: 05/01/2012
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Spivack, Nova T.
• Primary Examiner(s): Gray, Ryan M
• Application Number:
  US14/939,889
• Publication Number:
  US 20160071326A1
• Time in Patent Office: 1,377 Days
• Field of Search: None
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06Q 30/0623 Item investigation
• CPC Class Codes: G06Q 30/0639 Item locations
• CPC Class Codes: G06Q 30/0641 Shopping interfaces
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

Techniques are disclosed for facilitating electronic commerce in an augmented reality environment. In some embodiments, a method comprises detecting, by a mobile device, presence of the physical product or the real life service; and presenting, on the mobile device, information to conduct the transaction of a physical product or a real life service via the augmented reality environment. In some embodiments, a method comprises detecting one or more targets in the augmented reality platform using a select area in a perspective of a user, the perspective being captured by a mobile device; and prompting the user to choose an object of interest from the one or more detected targets.

Claims

1. A method for selecting, in an augmented reality platform, a target representing a physical object, via a mobile device, the method comprising:

detecting presence of one or more physical objects in the augmented reality platform in a perspective of a user, the perspective being captured by the mobile device;

obtaining identifying information associated with the one or more physical objects in the augmented reality platform;

prompting the user to select the target from the one or more physical objects detected in the augmented reality platform;

prompting the user to enter a mode in which the mobile device receives an assistance from the user to perform the detection;

in response to receiving a user input for selecting the target, transmitting the identifying information associated with the selected target to an external device; and

based on information of a product corresponding to the selected target received from the external device, displaying information for conducting a transaction of the product,

wherein the displayed information comprises an advertisement regarding the product and another similar product related to the product.

2. The method of claim 1, further comprising:

obtaining views of the physical object to generate a recognition pattern.

3. The method of claim 1, further comprising:

entering a learning mode in which the mobile device receives assistance from the user to perform the detection,

wherein the assistance includes taking snapshots from multiple views of the physical object to develop a recognition pattern.

4. The method of claim 1, further comprising conducting a series of tests to improve an image recognition of the physical object.

5. The method of claim 1, further comprising:

providing, via a display of the mobile device, a lasso tool to specify a select area in the perspective.

6. The method of claim 5, further comprising:

responsive to detecting, via the mobile device, one or more of movement and selection by a selection tool or a pointer, or movement and selection represented by a gesture; and

receiving the selection of the target.

7. The method of claim 1, further comprising:

capturing, via the mobile device, a gesture from one or more of movements or nonmovements of an eye of the user, locations of a focal point of an eye of the user, or movements of an eye lid of the user,

wherein, the gesture indicates selection of the target by the user.

8. The method of claim 1, further comprising:

capturing, via the mobile device, a gesture from one or more of movements of hand or finger as recognized by a camera of the mobile device, movements of a virtual pointer controlled by a wireless peripheral device, movements of a virtual pointer controlled by a touch-sensitive display of the mobile device, or movements of the mobile device itself,

wherein, the gesture indicates selection of the target by the user.

9. The method of claim 8, further comprising capturing the gestures based on a speed or velocity of the movements.

10. The method of claim 1, further comprising:

confirming, via the mobile device, with the user of the selection of the target,

wherein the confirmation comprises highlighting or outlining the target in the augmented reality platform.

11. The method of claim 5, wherein the target is further detected using metadata defining the target.

12. The method of claim 1, wherein the mobile device comprises an electronic eyewear or a head-mounted device.

13. The method of claim 1, wherein the target represents the physical product or real world services.

14. A system which provides an augmented reality platform, the system comprising:

a processor; and

a memory coupled to the processor and having stored thereon instructions which, when executed by the processor, configure the processor to;

detect presence of one or more physical objects in the augmented reality platform in a perspective of a user,

obtain identifying information associated with the one or more physical objects in the augmented reality platform,

prompt the user to select the target from the one or more physical objects detected in the augmented reality platform,

prompt the user to enter a mode in which the mobile device receives an assistance from the user to perform the detection,

in response to receiving a user input for selecting the target, transmit the identifying information associated with the selected target to an external device, and

based on information of a product corresponding to the selected target received from the external device, display information for conducting a transaction of the product,

wherein the displayed information comprises an advertisement regarding the product and another similar product related to the product.

15. The system of claim 14, wherein the processor is further configured to:

provide, via a display of the mobile device, the user with a lasso tool in the perspective to form the select area.

16. The system of claim 14, wherein the processor is further configured to:

prompt, via a display of the mobile device, the user to enter a learning mode in which the system receives assistance from the user to perform the detection.

17. The system of claim 14, wherein, when entering the learning mode in which the assistance is received to perform the detection, the assistance includes one or more of a name, metadata, a boundary, or snapshots or images from multiple views to develop a recognition pattern, for the target.

18. The system of claim 14, wherein the target is detected further using metadata stored in a repository defining the target.

19. The system of claim 18, wherein the metadata for use in detection of the target includes one or more of an item name, an alias, an attribute, a shape, a size, a dimension, or augmented reality markers.

20. The system of claim 18, wherein the metadata for use in detection of the target includes photos or videos.

21. The system of claim 14, wherein the processor is further configured to:

receive the user'"'"'s selection of the target,

detect one or more of movement and selection by a selection tool or a pointer, or movement and selection represented by a gesture,

capture the gesture from one or more of movements or non-movements of an eye of the user, locations of a focal point of an eye of the user, or movements of an eye lid oft he user, and

capture the gesture from one or more of movements of hand or finger as recognized by a camera, movements of a virtual pointer controlled by a wireless peripheral device, movements of a virtual pointer controlled by a touch-sensitive display, or movements of the system itself.

22. The system of claim 21, wherein the processor is further configured to:

capture based on a speed or velocity of the movements.

23. The system of claim 14, wherein the device is in a form of an electronic eyewear or a head-mounted device.

24. The system of claim 14,

wherein the target is detected using a recognition pattern stored in a pattern recognition repository, and

wherein multiple views of the target are used to generate the recognition pattern s.

25. A non-transitory machine-readable storage medium, having stored thereon instructions which when executed by a processor, cause the processor to perform a method for selecting, in an augmented reality platform, a target representing a physical object, the method comprising:

detecting presence of one or more physical objects in the augmented reality platform in a perspective of a user;

obtaining identifying information associated with the one or more physical objects in the augmented reality platform;

prompting the user to select the target among the one or more physical objects detected in the augmented reality platform;

prompting the user to enter a mode in which a mobile device receives an assistance from the user to perform the detection;

in response to receiving a user input for selecting the target, transmitting the identifying information associated with the selected target to an external device; and

based on information of a product corresponding to the selected target received from the external device, displaying information for conducting a transaction of the product,

wherein the displayed information comprises an advertisement regarding the product and another similar product related to the product.

26. The non-transitory machine-readable storage medium of claim 25,

wherein the target is detected further using metadata stored in a repository defining the target, and

wherein the metadata for use in detection of the target includes one or more of an item name, an alias, an attribute, a shape, a size, a dimension, or an augmented reality marker.

27. A mobile device comprising:

a touch-sensitive display;

a camera;

a processor; and

a memory coupled to the processor and storing instructions which, when executed by the processor, cause the mobile device to;

display images, being captured via the camera, in an augmented reality user interface;

detect presence of one or more physical objects in the images being captured in the augmented reality user interface;

obtain identifying information associated with the one or more physical objects present in the images being captured in the augmented reality user interface;

prompt the user to select the target from the one or more physical objects present in the images being captured in the augmented reality user interface;

prompt the user to enter a mode in which a mobile device receives an assistance from the user to perform the detection,

in response to receiving a user input for selecting the target, transmit the identifying information associated with the selected target to an external device; and

based on information of a product corresponding to the selected target received from the external device, display information for conducting a transaction of the product,

wherein the displayed information comprises an advertisement regarding the product and another similar product relating to the product.