Cross-reality select, drag, and drop for augmented reality systems

• US 10,025,486 B2
• Filed: 03/15/2013
• Issued: 07/17/2018
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Elwha LLC (Intellectual Ventures LLC)
• Original Assignee: Elwha LLC (Intellectual Ventures LLC)
• Inventors: Fein, Gene, Levien, Royce A., Lord, Richard T., Lord, Robert W., Malamud, Mark A., Rinaldo, Jr., John D., Tegreene, Clarence T.
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Nguyen, Nhat Huy T
• Application Number:
  US13/840,712
• Publication Number:
  US 20140282162A1
• Time in Patent Office: 1,950 Days
• Field of Search: None
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06Q 30/0643 Graphical representation of items or shoppers

Abstract

Methods, apparatuses, computer program products, devices and systems are described that carry out detecting a first action of a user at a location in a real world field of view of an augmented reality device; displaying an augmented reality representation in response to at least one of a user input or detecting a first action of a user at a location in a real world field of view of an augmented reality device; moving the displayed augmented reality representation on a display of the augmented reality device according to at least one detected second action of the user; and registering the displayed augmented reality representation at a location in the display of the augmented reality device in response to at least one of a user input or moving a displayed augmented reality representation on a display of the augmented reality device according to at least one detected second action of the user.

Claims

1. A system comprising:

circuitry configured for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device;

circuitry configured for recognizing one or more objects in a field of view that satisfy the at least one selection constraint;

circuitry configured for constraining selection to the one or more objects in the field of view;

circuitry configured for selecting at least one of the one or more objects in the field of view based at least partly on detection of a first action of a user at a location that corresponds to the at least one of the one or more objects; and

circuitry configured for displaying an augmented reality representation on a display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects.

2. The system of claim 1, further comprising:

circuitry configured for detecting a pointing gesture or a touch of a user at the location that corresponds to the at least one of the one or more objects.

3. The system of claim 1, further comprising:

circuitry configured for highlighting on the display of the augmented reality device the one or more objects for possible selection.

4. The system of claim 1, wherein the circuitry configured for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device comprises:

circuitry configured for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device and based at least partly on one or more of a time of day or a geographic location.

5. The system of claim 1, further comprising:

circuitry configured for detecting using image data the first action of the user at the location that corresponds to the at least one of the one or more objects.

6. The system of claim 1, further comprising:

circuitry configured for detecting a hand, finger, or pointing gesture of the user at the location that corresponds to the at least one of the one or more objects.

7. The system of claim 1, further comprising:

circuitry configured for detecting a voice command indicating selection of the at least one of the one or more objects.

8. The system of claim 1, further comprising:

circuitry configured for detecting the first action that matches at least one of a stored gesture or a stored image.

9. The system of claim 1, further comprising:

circuitry configured for dragging the augmented reality representation across the display of the augmented reality device according to a second action of the user.

10. The system of claim 1, further comprising:

circuitry configured for registering the augmented reality representation at a destination on the display of the augmented reality device at least partly in response to at least one of a user gesture, a user voice command, or a user touch command indicating that the destination has been reached.

11. The system of claim 1, further comprising:

circuitry configured for registering the augmented reality representation at an icon on the display of the augmented reality device at least partly in response to moving the augmented reality representation.

12. The system of claim 1, further comprising:

circuitry configured for performing image recognition with respect to the at least one of the one or more objects in the field of view.

13. The system of claim 1, further comprising:

circuitry configured for detecting a finger of the user within the field of view at a location that corresponds to the at least one of the one or more objects.

14. The system of claim 1, further comprising:

circuitry configured for obtaining from memory an enhanced depiction of the at least one of the one or more objects for display as the augmented reality representation.

15. The system of claim 1, further comprising:

circuitry configured for moving the augmented reality representation on the display of the augmented reality device to track movement of a finger of the user.

16. The system of claim 1, wherein the augmented reality device comprises a smartphone including a camera that defines the field of view.

17. The system of claim 1, wherein the circuitry configured for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device comprises:

circuitry configured for determining at least one selection constraint based at least partly on one or more media applications running on one or more display devices in proximity to an augmented reality device.

18. The system of claim 1, further comprising:

circuitry configured for initiating playback of content associated with the at least one of the one or more objects using at least one of the one or more devices in proximity to the augmented reality device.

19. The system of claim 1, wherein the augmented reality device comprises one or more of augmented reality eyeglasses, a smartphone, a tablet computer, a smartwatch, a digital camera, or a digital video device.

20. The system of claim 1, wherein the circuitry configured for displaying an augmented reality representation on a display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects comprises:

circuitry configured for displaying an augmented reality representation including a list of one or more menu items on a display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects.

21. The system of claim 1, wherein the circuitry configured for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device comprises:

circuitry configured for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device and based at least partly on a context proximate to the augmented reality device.

22. The system of claim 1, further comprising:

circuitry configured for detecting touch of a touch sensitive portion of the augmented reality device.

23. A computer processor configured to perform operations comprising:

determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device;

recognizing one or more objects in a field of view that satisfy the at least one selection constraint;

constraining selection to the one or more objects in the field of view;

selecting at least one of the one or more objects in the field of view based at least partly on detection of a first action of a user at a location that corresponds to the at least one of the one or more objects; and

displaying an augmented reality representation on a display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects.

24. An augmented reality device comprising:

a display; and

a computer processor programmed to perform operations including at least;

determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to the augmented reality device;

recognizing one or more objects in a field of view that satisfy the at least one selection constraint;

constraining selection to the one or more objects in the field of view;

selecting at least one of the one or more objects in the field of view based at least partly on detection of a first action of a user at a location that corresponds to the at least one of the one or more objects; and

displaying an augmented reality representation on the display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects.

25. A system comprising:

at least one computing device; and

one or more instructions that when executed on the at least one computing device program the at least one computing device to perform operations including at least;

determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device;

recognizing one or more objects in a field of view that satisfy the at least one selection constraint;

constraining selection to the one or more objects in the field of view;

selecting at least one of the one or more objects in the field of view based at least partly on detection of a first action of a user at a location that corresponds to the at least one of the one or more objects; and

displaying an augmented reality representation on a display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects.

26. A system comprising:

means for determining at least one selection constraint based at least partly on one or more applications running on one or more devices in proximity to an augmented reality device;

means for recognizing one or more objects in a field of view that satisfy the at least one selection constraint;

means for constraining selection to the one or more objects in the field of view;

means for selecting at least one of the one or more objects in the field of view based at least partly on detection of a first action of a user at a location that corresponds to the at least one of the one or more objects; and

means for displaying an augmented reality representation on a display of the augmented reality device at least partly in response to selection of the at least one of the one or more objects.