Utilizing real world objects for user input

• US 10,013,083 B2
• Filed: 04/28/2014
• Issued: 07/03/2018
• Est. Priority Date: 04/28/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Amariutei, Alexandru, Hildreth, Evan Robbert
• Primary Examiner(s): Trapanese, William C
• Application Number:
  US14/263,885
• Publication Number:
  US 20150309629A1
• Time in Patent Office: 1,527 Days
• Field of Search: None
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042

Abstract

Methods, systems, apparatuses and computer-readable media for utilizing real world objects to interact with a user interface are presented. The method may comprise a device processing image data to identify an interactive surface area and an interacting object. Subsequently, the device may determine at least two regions of the interactive surface area. In addition, the device may map commands of a user interface to the at least two regions of the interactive surface area. Subsequently, the device may determine a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area. In addition, the device may perform a mapped command of the user interface, wherein the mapped command is determined based on the selected region.

Claims

1. A method of a device, comprising:

processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

determining at least two regions of the interactive surface area;

tracking each of the at least two regions of the interactive surface area;

mapping commands of a user interface to the at least two regions of the interactive surface area, wherein the mapping commands of a user interface to regions of the interactive surface area further comprises;

selecting a number of commands of the user interface from a plurality of user commands based on how many tracked regions comprise the tracked regions of the interactive surface area, wherein a number of commands of the plurality of commands exceeds how many tracked regions comprise the tracked regions; and

correlating each tracked region with a different command based in part on priorities associated with each command, the priorities having been determined based in part on an application currently active on the device;

determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and

performing a mapped command of the user interface, on the device, wherein the mapped command is determined based on the selected region.

2. The method of claim 1, wherein the processing the image to identify the interactive surface area is based on a pose of the interactive surface area.

3. The method of claim 1, wherein the interactive surface area is a hand and the interacting object is a thumb attached to the hand.

4. The method of claim 1, wherein the interactive surface area is a hand and the interacting object is a second hand.

5. The method of claim 1, wherein the interactive surface area may be one or more objects.

6. The method of claim 1, wherein the interacting object may be one or more objects.

7. The method of claim 1, wherein determining the at least two regions of the interactive surface area is based on physical features of the interactive surface area.

8. The method of claim 1, wherein the proximity comprises direct contact between the interacting object and the interactive surface area.

9. The method of claim 1, further comprising tracking the interacting object.

10. The method of claim 2, wherein the processing the image to identify the interactive surface area is further based on a number of objects in the image, a size of an object, a number of regions on the object, visibility of regions on an object, mobility of the object, the interacting object or any combination thereof.

11. The method of claim 1, wherein the interacting object is detected based on the number of objects in the image, size of an object, visibility of an object, mobility of an object, the interactive surface area or any combination thereof.

12. The method of claim 1, further comprising:

determining a motion of the interacting object and whether the motion corresponds to a gesture;

determining boundaries of each region of the interactive surface area that allows gestures;

determining a command correlated to the gesture; and

performing the command if the motion was performed within the boundaries.

13. The method of claim 12, wherein the motion was at least partially performed within the boundaries.

14. The method of claim 1, wherein the processing the image to identify the interactive surface area comprises identifying a predetermined movement in the image data that identifies the interactive surface area.

15. A method of a device, comprising:

processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

determining at least two regions of the interactive surface area;

tracking each of the at least two regions of the interactive surface area;

mapping commands of a user interface to the at least two regions of the interactive surface area;

determining visibility of each tracked region of the interactive surface area;

determining priority of commands mapped on a hidden tracked region and on a visible tracked region responsive to the hidden tracked region being hidden;

remapping commands from the hidden tracked region to the visible tracked region based on the priority of commands, wherein remapping the commands comprises comparing a first priority associated with a first command mapped to a hidden tracked region and a second priority associated with a second command mapped to a visible tracked region, and remapping the first command to the second tracked region responsive to the first priority exceeding the second priority;

determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and

performing a mapped command of the user interface, on the device, wherein the mapped command is determined based on the selected region.

16. The method of claim 15, wherein the priority of commands are based on a command'"'"'s likelihood of use, number of commands required, organization of commands, and regions previously associated with the command.

17. An apparatus, comprising:

at least one processor configured to;

process image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

determine at least two regions of the interactive surface area;

track each of the at least two regions of the interactive surface area;

map commands of a user interface to the at least two regions of the interactive surface area, wherein the at least one processor configured to map commands of a user interface to the at least two regions of the interactive surface area further configured to;

select a number of commands of the user interface from a plurality of user commands based on how many tracked regions comprise the tracked regions of the interactive surface area, wherein a number of commands of the plurality of commands exceeds how many tracked regions comprise the tracked regions; and

correlate each tracked region with a different command based in part on priorities associated with each command, the priorities having been determined based in part on an application currently active on the apparatus;

determine a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area;

perform a mapped command, wherein the mapped command is determined based on the selected region; and

a memory coupled to the at least one processor.

18. The apparatus of claim 17, wherein the at least one processor configured to determine the at least two regions of the interactive surface area is based on physical features of the interactive surface area.

19. The apparatus of claim 17, wherein the at least one processor further configured to:

determine a motion of the interacting object and if the motion corresponds to a gesture;

determine boundaries of each region of the interactive surface area that allows gestures;

determine a command correlated to the gesture; and

perform the command if the motion was at performed within the boundaries.

20. An apparatus, comprising:

at least one processor configured to;

process image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

determine at least two regions of the interactive surface area;

map commands of a user interface to the at least two regions of the interactive surface area;

determine visibility of each tracked region of the interactive surface area;

determine priority of commands mapped on a hidden tracked region and on a visible tracked region responsive to the hidden tracked region being hidden;

remap commands from the hidden tracked region to the visible tracked region based on the priority of commands, wherein the at least one processor is configured to compare a first priority associated with a first command mapped to a hidden tracked region and a second priority associated with a second command mapped to a visible tracked region and to remap the first command to the second tracked region responsive to the first priority exceeding the second priority

determine a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area;

perform a mapped command, wherein the mapped command is determined based on the selected region; and

a memory coupled to the at least one processor.

21. An apparatus, comprising:

means for processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

means for determining at least two regions of the interactive surface area;

means for mapping commands of a user interface to the at least two regions of the interactive surface area, wherein the means for mapping commands of a user interface to regions of the interactive surface area further comprises;

means for tracking each of the at least two regions of the interactive surface area;

means for selecting a number of commands of the user interface from a plurality of user commands based on how many tracked regions comprise the tracked regions of the interactive surface area, wherein a number of commands of the plurality of commands exceeds how many tracked regions comprise the tracked regions; and

means for correlating each tracked region with a different command based in part on priorities associated with each command, the priorities having been determined based in part on an application currently active on the apparatus;

means for determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and

means for performing a mapped command of the user interface, wherein the mapped command is determined based on the selected region.

22. The apparatus of claim 21, wherein the means for determining the at least two regions of the interactive surface area is based on physical features of the interactive surface area.

23. The apparatus of claim 21, wherein the means for identifying the interactive surface area is further based on a number of objects in the image, a size of an object, visibility of areas on the object, a number of regions on the object, mobility of the object, the interacting object or any combination thereof.

24. The apparatus of claim 21, wherein the means for identifying the interacting object is further based on the number of objects in the image, size of an object, visibility of an object, mobility of an object, the interactive surface area or any combination thereof.

25. A computer program product, comprising:

a non-transitory computer-readable storage medium comprising;

code for processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

code for determining at least two regions of the interactive surface area;

code for mapping commands of a user interface to the at least two regions of the interactive surface area, wherein the code for mapping commands further comprises

code for tracking each of the at least two regions of the interactive surface area,

code for selecting a number of commands of the user interface from a plurality of user commands based on how many tracked regions comprise the tracked regions of the interactive surface area, wherein a number of commands of the plurality of commands exceeds how many tracked regions comprise the tracked regions, and

code for correlating each tracked region with a different command based in part on priorities associated with each command, the priorities having been determined based in part on an active application;

code for determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and

code for performing a mapped command of the user interface, wherein the mapped command is determined based on the selected region.

26. An apparatus, comprising:

means for processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

means for determining at least two regions of the interactive surface area;

means for tracking each of the at least two regions of the interactive surface area;

means for mapping commands of a user interface to the at least two regions of the interactive surface area;

means for determining visibility of each tracked region of the interactive surface area;

means for determining priority of commands mapped on a hidden tracked region and on a visible tracked region responsive to the hidden tracked region being hidden;

means for remapping commands from the hidden tracked region to the visible tracked region based on the priority of commands, wherein remapping the commands comprises comparing a first priority associated with a first command mapped to a hidden tracked region and a second priority associated with a second command mapped to a visible tracked region, and remapping the first command to the second tracked region responsive to the first priority exceeding the second priority; and

means for determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and

means for performing a mapped command of the user interface, on the device, wherein the mapped command is determined based on the selected region.

27. A computer program product, comprising:

a non-transitory computer-readable storage medium comprising;

code for processing image data to identify an interactive surface area and an interacting object using an object identification technique that differentiates between objects in the image data based at least in part on relative positions of the interactive surface area and the interacting object to one another without first recognizing an object comprising the interactive surface area or the interacting object;

code for determining at least two regions of the interactive surface area;

code for tracking each of the at least two regions of the interactive surface area;

code for mapping commands of a user interface to the at least two regions of the interactive surface area;

code for determining visibility of each tracked region of the interactive surface area;

code for determining priority of commands mapped on a hidden tracked region and on a visible tracked region responsive to the hidden tracked region being hidden;

code for remapping commands from the hidden tracked region to the visible tracked region based on the priority of commands, wherein remapping the commands comprises comparing a first priority associated with a first command mapped to a hidden tracked region and a second priority associated with a second command mapped to a visible tracked region, and remapping the first command to the second tracked region responsive to the first priority exceeding the second priority; and

code for determining a selected region of the interactive surface area based on a proximity of the interacting object to the interactive surface area; and

code for performing a mapped command of the user interface, on the device, wherein the mapped command is determined based on the selected region.