System and method of real-time interactive operation of user interface

• US 9,703,387 B2
• Filed: 08/31/2015
• Issued: 07/11/2017
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Original Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Inventors: Nayak, Nandita M., Tofighi, Ghassem, Gu, Haisong
• Primary Examiner(s): OLSON, JASON C
• Application Number:
  US14/840,596
• Publication Number:
  US 20170060250A1
• Time in Patent Office: 680 Days
• Field of Search: 345156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method, a system, and a non-transitory computer readable medium are disclosed for real-time interaction with a user interface recognizing a gesture. The method including capturing three-dimensional (3D) data on a subject; detecting a pointing action by the subject from the 3D data; computing an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point; and tracking the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action.

Claims

1. A method of recognizing gestures for real-time interaction, comprising:

capturing three-dimensional (3D) data on a subject;

detecting a pointing action by the subject from the 3D data;

computing an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point;

tracking the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action; and

providing feedback to the subject by highlighting a portion around the initial estimate and continuously modifying the highlighted portion until the highlighted portion shrinks to a single point indicating that a desired location has been reached.

2. The method of claim 1, comprising:

reducing the defined radius of the target region if the pointing action is within the defined radius; and

expanding the defined radius if the pointing action by the subject is outside the defined radius.

3. The method of claim 1, comprising:

shifting the center point of the defined radius upon detecting a movement of the pointing action of the user is within the defined radius and a detection of a change in the pointing action relative to the center point of the defined radius.

4. The method of claim 1, comprising:

reducing and expanding the defined radius at a fixed rate at a rate proportional to speed of motion of the pointing action of the subject.

5. The method of claim 1, comprising:

selecting a target within the target region, the selection of the target comprising;

determining if one or more targets are within the defined radius; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

21. The method of claim 1, wherein the continuous modifying of the highlighted portion until the highlighted portion shrinks to the single point indicating that the desired location has been reached further comprises:

expanding or shrinking the highlighted region by a fixed factor based on the detected pointing action of the subject; and

canceling the feedback to the subject upon a lowering of the subject'"'"'s hand.

6. A system for recognizing gestures for real-time interaction, comprising:

a motion and depth sensor configured to capture three-dimensional (3D) data on a subject; and

a processor configured to;

capture three-dimensional (3D) data on a subject;

detect a pointing action by the subject from the 3D data;

compute an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point;

track the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action; and

provide feedback to the subject by highlighting a portion around the initial estimate and continuously modifying the highlighted portion until the highlighted portion shrinks to a single point indicating that a desired location has been reached.

7. The system of claim 6, wherein the processor is configured to:

reduce the defined radius of the target region if the pointing action is within the defined radius; and

expand the defined radius if the pointing action by the subject is outside the defined radius.

8. The system of claim 6, wherein the processor is configured to:

shift the center point of the defined radius upon detecting a movement of the pointing action of the user is within the defined radius and a detection of a change in the pointing action relative to the center point of the defined radius.

9. The system of claim 6, wherein the processor is configured to:

reduce and expand the defined radius at a fixed rate at a rate proportional to speed of motion of the pointing action of the subject.

10. The system of claim 6, wherein the processor is configured to:

select the target within the target region, the selection of the target comprising;

determining if one or more targets are within the defined radius; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

22. The system of claim 6, wherein the continuous modifying of the highlighted portion until the highlighted portion shrinks to the single point indicating that the desired location has been reached further comprises:

expanding or shrinking the highlighted region by a fixed factor based on the detected pointing action of the subject; and

canceling the feedback to the subject upon a lowering of the subject'"'"'s hand.

11. A non-transitory computer readable medium containing a computer program storing computer readable code for recognizing gestures for real-time interaction, the program being executable by a computer to cause the computer to perform a process comprising:

capturing three-dimensional (3D) data on a subject;

detecting a pointing action by the subject from the 3D data;

computing an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point;

tracking the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action; and

providing feedback to the subject by highlighting a portion around the initial estimate and continuously modifying the highlighted portion until the highlighted portion shrinks to a single point indicating that a desired location has been reached.

12. The computer readable storage medium of claim 11, comprising:

reducing the defined radius of the target region if the pointing action is within the defined radius; and

expanding the defined radius if the pointing action by the subject is outside the defined radius.

13. The computer readable storage medium of claim 11, comprising:

shifting the center point of the defined radius upon detecting a movement of the pointing action of the user is within the defined radius and a detection of a change in the pointing action relative to the center point of the defined radius.

14. The computer readable storage medium of claim 11, comprising:

reducing and expanding the defined radius at a fixed rate at a rate proportional to speed of motion of the pointing action of the subject.

15. The computer readable storage medium of claim 11, comprising:

selecting a target within the target region, the selection of the target comprising;

determining if one or more targets are within the defined radius; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

23. The computer readable storage medium of claim 11, wherein the continuous modifying of the highlighted portion until the highlighted portion shrinks to the single point indicating that the desired location has been reached further comprises:

expanding or shrinking the highlighted region by a fixed factor based on the detected pointing action of the subject; and

canceling the feedback to the subject upon a lowering of the subject'"'"'s hand.

16. A method of recognizing gestures for real-time interaction, comprising:

capturing three-dimensional (3D) data on a subject;

detecting a pointing action by the subject from the 3D data to begin a pointing operation;

determining a point of intersection of the pointing action on an actual screen;

determining if one or more targets on the actual screen are within a defined radius around a computed point on the actual screen;

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains; and

providing feedback to the subject by highlighting a portion of the actual screen around the initial estimate and continuously modifying the highlighted portion until the highlighted portion shrinks to a single point indicating that a desired location has been reached.

17. The method of claim 16, comprising:

if the point of intersection of the pointing action by the subject on the virtual screen changes, shifting a center of the defined radius; and

updating the number of targets within the defined radius by excluding targets outside the defined radius and including targets which are within the defined radius.

18. The method of claim 16, comprising:

expanding the defined radius if the pointing action by the subject is outside the defined radius;

updating the number of targets within the defined radius to include targets within the expanded defined radius; and

reducing the expanded defined radius and expanding the defined radius until only one target is identified within the defined radius.

19. The method of claim 16, comprising:

capturing the 3D data with a motion and depth sensor; and

wherein the actual screen is a projection screen or video screen.

20. The method of claim 19, wherein the actual screen is a plurality of selection indicators, each of the plurality of selection indicators corresponding to a consumer good.