Systems and methods for implementing and using gesture based user interface widgets with camera input

• US 9,317,171 B2
• Filed: 04/18/2013
• Issued: 04/19/2016
• Est. Priority Date: 04/18/2013
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Chiu, Patrick, Liu, Qiong, Back, Maribeth Joy, Kratz, Sven
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Summers, Geoffrey E
• Application Number:
  US13/865,990
• Publication Number:
  US 20140313363A1
• Time in Patent Office: 1,097 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0425 using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected tracking a projected light spot to determine a position on a display surface G06F3/0386
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures

Abstract

Described is an approach to gesture interaction that is based on user interface widgets. In order to detect user gestures, the widgets are provided with hotspots that are monitored using a camera for predetermined patterns of occlusion. A hotspot is a region where a user interacts with the widget by making a gesture over it. The user'"'"'s gesture may be detected without the user physically touching the surface displaying the widget. The aforesaid hotspots are designed to be visually salient and suggestive of the type of gestures that can be received from the user. Described techniques are advantageous in relation to conventional systems, such as systems utilizing finger tracking, in that they can better support complex tasks with repeated user actions. In addition, they provide better perceived affordance than conventional systems that attempt to use widgets that are not designed for gesture input, or in-the-air gesture detection techniques that lack any visual cues.

Claims

1. A computer-implemented method for detecting a gesture performed by a user in connection with a user interface widget comprising a hotspot, the method being performed in a computerized system comprising a central processing unit, a camera and a memory, the computer-implemented method comprising:

a. acquiring a plurality of images of the hotspot using the camera;

b. using the central processing unit to identify occlusion of at least a portion of the hotspot by the user in the acquired plurality of images of the hotspot, wherein the occlusion of the portion of the hotspot is identified by detecting changes in color values of each of a plurality of non-contiguous image sensor pixels within the acquired plurality of images of the hotspot based on a recorded history of values of the plurality of non-contiguous image sensor pixels; and

c. using the central processing unit to detect the gesture performed by the user based on the identified occlusion of the at least the portion of the hotspot in at least two of the acquired plurality of images of the hotspot.

2. The computer-implemented method of claim 1, wherein the user interface widget is a button widget associated with a button press event and wherein the computer-implemented method further comprises generating the button press event if the detected gesture is a swipe gesture over the hotspot.

3. The computer-implemented method of claim 1, further comprising storing a history of the identified occlusion of the hotspot in the memory.

4. The computer-implemented method of claim 1, further comprising projecting the widget onto a display surface using a projector.

5. The computer-implemented method of claim 1, wherein the computerized system further comprises a liquid crystal display panel, the computer-implemented method further comprising displaying the widget on the liquid crystal display panel.

6. The computer-implemented method of claim 1, wherein in c. the gesture is detected based on a condition that a second portion of the hotspot is not occluded.

7. The computer-implemented method of claim 6, wherein the hotspot comprises a stripe and wherein the gesture performed by the user comprises a swipe over the stripe.

8. The computer-implemented method of claim 6, wherein the hotspot comprises a stripe and wherein the gesture performed by the user comprises a pinch gesture.

9. The computer-implemented method of claim 6, wherein the hotspot comprises a stripe with an empty slot and wherein the gesture performed by the user comprises covering the empty slot.

10. The computer-implemented method of claim 6, wherein the second portion of the hotspot is discontiguous.

11. The computer-implemented method of claim 1, further comprising, responsive to the detection of the gesture performed by the user, issuing a command associated with the gesture.

12. The computer-implemented method of claim 11, further comprising detecting a direction of the gesture performed by the user, wherein the issued command is based, at least in part, on the detected direction of the gesture.

13. The computer-implemented method of claim 11, further comprising detecting a speed of the gesture performed by the user, wherein the issued command is based, at least in part, on the detected speed of the gesture.

14. The computer-implemented method of claim 1, wherein the gesture is performed by the user without physical contact with the surface carrying or displaying the widget.

15. The computer-implemented method of claim 1, further comprising performing a calibration operation by comparing acquired images corresponding to a plurality of appearances of the widget.

16. The computer-implemented method of claim 3, wherein in c. the detecting comprises comparing recorded values associated with a plurality of sensor pixels with a predetermined pattern.

17. The computer-implemented method of claim 3, wherein in c. the detecting comprises using a finite state machine state transition table.

18. The computer-implemented method of claim 3, further comprising repeating a. through c. if it is determined in c. that a predetermined pattern associated with a gesture has not occurred.

19. The computer-implemented method of claim 3, further comprising performing sub-sampling of a plurality of sensor pixels corresponding to the hotspot.

20. A non-transitory computer-readable medium embodying a set of computer-executable instructions, which, when executed in a computerized system comprising a central processing unit, a camera and a memory, cause the computerized system to perform a method for detecting a gesture performed by a user in connection with a widget comprising a hotspot, the method comprising:

a. acquiring a plurality of images of the hotspot using the camera;

b. using the central processing unit to identify occlusion of at least a portion of the hotspot by the user in the acquired plurality of images of the hotspot, wherein the occlusion of the portion of the hotspot is identified by detecting changes in color values of each of a plurality of non-contiguous image sensor pixels within the acquired plurality of images of the hotspot based on a recorded history of values of the plurality of non contiguous image sensor pixels; and

c. using the central processing unit to detect the gesture performed by the user based on the identified occlusion of the at least the portion of the hotspot in at least two of the acquired plurality of images of the hotspot.

21. A computerized system comprising one or more processing units and a memory, the computerized system further comprising:

a. a user interface generation module executed by the one or more processing units for causing a user interface to be displayed to a user, the user interface comprising a widget;

b. a camera for acquiring an image of the user interface;

c. an occlusion identification module executed by the one or more processing units for identifying occlusion of at least a portion of the widget in the acquired image of the user interface, wherein the occlusion of the portion of the widget is identified by the occlusion identification module by detecting changes in color values of each of a plurality of non-contiguous image sensor pixels within the acquired image of the user interface based on a recorded history of values of the plurality of non-contiguous image sensor pixels; and

d. a gesture detection module executed by the one or more processing units for detecting a gesture performed by the user based on the identified occlusion of the at least the portion of the widget.