SYSTEM AND METHOD OF REAL-TIME INTERACTIVE OPERATION OF USER INTERFACE

• US 20170060250A1
• Filed: 08/31/2015
• Published: 03/02/2017
• Est. Priority Date: 08/31/2015
• Status: Active Grant

• Current Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Original Assignee: Konica Minolta Laboratory U.S.A., Inc. (Konica Minolta Inc.)
• Inventors: NAYAK, Nandita M., TOFIGHI, Ghassem, GU, Haisong
• Granted Patent: US 9,703,387 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06V 40/23 Recognition of whole body movements, e.g. for sport training
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

A method, a system, and a non-transitory computer readable medium are disclosed for real-time interaction with a user interface recognizing a gesture. The method including capturing three-dimensional (3D) data on a subject; detecting a pointing action by the subject from the 3D data; computing an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point; and tracking the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action

Claims

1. A method of recognizing gestures for real-time interaction with a user interface, comprising:

capturing three-dimensional (3D) data on a subject;

detecting a pointing action by the subject from the 3D data;

computing an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point; and

tracking the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action.

2. The method of claim 1, comprising:

reducing the defined radius of the target region if the pointing action is within the defined radius; and

expanding the defined radius if the pointing action by the subject is outside the defined radius.

3. The method of claim 1, comprising:

shifting the center point of the defined radius upon detecting a movement of the pointing action of the user is within the defined radius and a detection of the a change in the pointing action relative to the center point of the defined radius.

4. The method of claim 1, comprising:

reducing and expanding the defined radius at a fixed rate at a rate proportional to speed of motion of the pointing action of the subject.

5. The method of claim 1, comprising:

selecting a target within the target region, the selection of the target comprising;

determining if one or more targets are within the defined radius; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

6. A system for recognizing gestures for real-time interaction with a user interface, comprising:

a motion and depth sensor for capturing three-dimensional (3D) data on a subject; and

a processor configured to;

capture three-dimensional (3D) data on a subject;

detect a pointing action by the subject from the 3D data;

compute an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point; and

track the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action.

7. The system of claim 6, wherein the processor is configured to:

reduce the defined radius of the target region if the pointing action is within the defined radius; and

expand the defined radius if the pointing action by the subject is outside the defined radius.

8. The system of claim 6, wherein the processor is configured to:

shift the center point of the defined radius upon detecting a movement of the pointing action of the user is within the defined radius and a detection of the a change in the pointing action relative to the center point of the defined radius.

9. The system of claim 6, wherein the processor is configured to:

reduce and expand the defined radius at a fixed rate at a rate proportional to speed of motion of the pointing action of the subject.

10. The system of claim 6, wherein the processor is configured to:

selecting a target within the target region, the selection of the target comprising;

determining if one or more targets are within the defined radius; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

11. A non-transitory computer readable medium containing a computer program storing computer readable code for recognizing gestures for real-time interaction with a user interface, the program being executable by a computer to cause the computer to perform a process comprising:

capturing three-dimensional (3D) data on a subject;

detecting a pointing action by the subject from the 3D data;

computing an initial estimate of a target region from the pointing action, the initial estimate of the target region having a defined radius around a center point; and

tracking the pointing action of the subject and performing a series of iterations wherein the defined radius of the target region changes based on the detected pointing action.

12. The computer readable storage medium of claim 11, comprising:

reducing the defined radius of the target region if the pointing action is within the defined radius; and

expanding the defined radius if the pointing action by the subject is outside the defined radius.

13. The computer readable storage medium of claim 11, comprising:

shifting the center point of the defined radius upon detecting a movement of the pointing action of the user is within the defined radius and a detection of the a change in the pointing action relative to the center point of the defined radius.

14. The computer readable storage medium of claim 11, comprising:

reducing and expanding the defined radius at a fixed rate at a rate proportional to speed of motion of the pointing action of the subject.

15. The computer readable storage medium of claim 11, comprising:

selecting a target within the target region, the selection of the target comprising;

determining if one or more targets are within the defined radius; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

16. A method of recognizing gestures for real-time interaction with a user interface, comprising:

capturing three-dimensional (3D) data on a subject;

detecting a pointing action by the subject from the 3D data to begin a pointing operation;

determining a point of intersection of the pointing action on an actual screen;

determining if one or more targets on the actual screen are within a defined radius around a computed point on the actual screen; and

if at least one target is present, determining if a number of targets is equal to one or greater than one, and wherein if the number of targets is equal to one, selecting a target, and if the number of targets is greater than one, reducing the defined radius to reduce the number of targets within the defined radius until a single target remains.

17. The method of claim 16, comprising:

if the point of intersection of the pointing action by the subject on the virtual screen changes, shifting a center of the defined radius; and

updating the number of targets within the defined radius by excluding targets outside the defined radius and including targets which are within the defined radius.

18. The method of claim 16, comprising:

expanding the defined radius if the pointing action by the subject is outside the defined radius;

updating the number of targets within the defined radius to include targets within the expanded defined radius; and

reducing the expanded defined radius and expanding the defined radius until only one target is identified within the defined radius.

19. The method of claim 16, comprising:

capturing the 3D data with a motion and depth sensor; and

wherein the actual screen is a projection screen or video screen.

20. The method of claim 19, wherein the actual screen is a plurality of selection indicators, each of the plurality of selection indicators corresponding to a consumer good.