3-D MOTION CONTROL FOR DOCUMENT DISCOVERY AND RETRIEVAL

• US 20150370472A1
• Filed: 06/19/2014
• Published: 12/24/2015
• Est. Priority Date: 06/19/2014
• Status: Abandoned Application

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Inventors: Privault, Caroline, Guillot, Fabien, Legras, Christophe
• Application Number:
  US14/309,300
• Publication Number:
  US 20150370472A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2203/0381 Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 3/0486 Drag-and-drop
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus

Abstract

A processing method includes, associating each of a plurality of hand gestures that are detectable with a three-dimensional sensor with a respective one of a plurality of item processing tasks in memory. A plurality of graphic objects is displayed on a touch-sensitive display device, each graphic object being associated with a respective item. With the three-dimensional sensor, a hand gesture is detected. The respective one of the item processing tasks that is associated with the detected hand gesture is identified and the identified one of the item processing tasks is implemented on the displayed graphic objects, comprising causing at least a subset of the displayed graphic objects to respond on the display device based on attributes of the respective items. Item processing tasks may also be implemented through predefined touch gestures.

Claims

1. A document processing method comprising:

in memory, associating each of a plurality of hand gestures that are detectable with a three-dimensional sensor with a respective one of a plurality of item processing tasks, the three-dimensional sensor being associated with a touch sensitive display device, at least one touch gesture that is detectable with the touch sensitive display device being associated with at least one of the plurality of item processing tasks;

displaying a set of graphic objects on the touch sensitive display device, each graphic object being associated with a respective item;

with the three-dimensional sensor, detecting a hand gesture;

identifying the respective one of the item processing tasks that is associated with the detected hand gesture; and

implementing the identified one of the item processing tasks on the displayed graphic objects, comprising causing at least a subset of the displayed graphic objects to respond on the display device based on attributes of the respective items.

2. The method of claim 1, wherein at least one of the associating, displaying, detecting, identifying and implementing is performed with a computer processor.

3. The method of claim 1, wherein the plurality of item processing tasks are selected from the group consisting of classification tasks and clustering tasks.

4. The method of claim 3, wherein at least one of the item processing tasks is a classification task and wherein a first of the plurality of hand gestures is associated with filtering items that are positive for the class and a second of the plurality of hand gestures is associated with filtering items that are negative for the class.

5. The method of claim 4, wherein one of the first and second hand gestures corresponds to a palm of a user'"'"'s hand facing upward and the other of the first and second hand gestures corresponds to a palm of the user'"'"'s hand facing downward.

6. The method of claim 5, further comprising after detecting a hand gesture selected from the first and second hand gestures, causing a subset of the displayed graphic objects to move across the display device to follow the user'"'"'s hand movement over the display device, the displayed graphic objects representing items that are respectively positive or negative for the class.

7. The method of claim 1, wherein the associating of at least some of the plurality of hand gestures comprises:

in response to the touch gesture, displaying a contextual menu on the touch-sensitive display device adjacent the user'"'"'s hand, the contextual menu including a plurality of icons, each icon representing a respective one of the item processing tasks;

for each of at least some of the icons, associating a respective one of the hand gestures with the respective icon.

8. The method of claim 7, wherein the method includes detecting a finger touch on one of the icons of the menu and associating a gesture of the corresponding finger with the item processing task associated with the one of the icons of the menu.

9. The method of claim 1, wherein the detecting of the hand gesture comprises detecting movement of a user'"'"'s hand over a group of the graphic objects displayed on the tactile user interface.

10. The method of claim 1, wherein the causing of the at least a subset of the displayed graphic objects to respond on the display device comprises causing the subset of graphic objects to change in at least one of a visible property of the subset of graphic objects and a position of the subset of graphic objects on the display device.

11. The method of claim 10, wherein the subset graphic objects on the display device change position in response to a detected movement of the user'"'"'s hand relative to the display device.

12. The method of claim 11, wherein the method further comprises providing for detecting a hand gesture with the three-dimensional sensor that is associated in memory with releasing the subset of graphic objects and when the hand gesture associated with releasing the graphic objects is detected, causing the subset of graphic objects to become stationary at a location of the user'"'"'s hand.

13. The method of claim 3, wherein the processing task includes a classification task and the method further comprising providing for detecting each of a plurality of predefined heights of a hand gesture, relative to the display device, with the three-dimensional sensor, each of the predefined heights being associated in memory with a respective classification threshold for a same class, and wherein when the user'"'"'s hand is detected at one of the predefined heights, the implementing includes implementing a classification task at the respective classification threshold.

14. The method of claim 13, wherein the classification task based on the user'"'"'s hand height is applied to a single graphic object.

15. The method of claim 3, wherein the processing task includes a classification task and the method further comprises:

activation of a classification mode in response to detection of a touch gesture on the touch-sensitive display device, including displaying a contextual menu of classification tasks;

providing for user selection of one of the classification tasks through interaction of the user'"'"'s hand with the contextual menu;

associating the selected classification task with a hand gesture; and

with the sensor, detecting the hand gesture over the set of graphic objects, wherein graphic objects representing documents that are responsive to the classification task exhibit a response to the hand gesture.

16. The method of claim 3, wherein the processing task includes a clustering task and wherein the associating each of the plurality of hand gestures with a respective one of the plurality of item processing tasks comprises associating each of a number of clusters with a respective number of visible fingers in the hand gesture and wherein the implementing of the clustering tasks comprises partitioning the graphic objects into a number of clusters corresponding to a number of visible fingers detected by the sensor.

17. The method of claim 1 wherein at least one of:

the items comprise text documents and the attributes are computed based on words in the text documents;

the items comprise images and the attributes are computed based on visual features extracted from the images.

18. The method of claim 1, further comprising:

with the three-dimensional sensor, detecting an orientation of the user'"'"'s wrist with respect to the user'"'"'s hand; and

repositioning graphic objects on the display device, based on the detected wrist orientation such that the graphic objects are viewable by the user in a viewing orientation.

19. A computer program product comprising non-transitory memory storing instructions which when executed by a processor, perform the method of claim 1

20. An interactive user interface for processing items comprising:

a touch-sensitive display device;

a three-dimensional sensor for detection of hand gestures adjacent the display device;

instructions stored in memory for;

associating each of a plurality of hand gestures that are detectable with a three-dimensional sensor with a respective one of a plurality of item processing tasks, at least one of the item processing tasks being associated with a touch gesture detectable by the touch-sensitive display device,

displaying a set of graphic objects on the display, each graphic object representing a respective item,

with the three-dimensional sensor, detecting a hand gesture,

identifying the respective one of the item processing tasks that is associated with the detected hand gesture, and

implementing the identified one of the item processing tasks on the displayed graphic objects, comprising causing at least a subset of the displayed graphic objects to respond on the display device based on attributes of the respective items; and

a processor in communication with the memory and display for executing the instructions.

21. The user interface of claim 20, wherein the touch-sensitive display device serves as a tactile user interface for detection of two-dimensional hand motions.

22. A method for using 2D and 3D motion control on a common user interface comprising:

providing for receiving a 2D gesture on a graphic object displayed on a tactile user interface from a user'"'"'s hand;

using a 3D sensor, capturing an orientation of the user'"'"'s hand;

with a processor, calculating a location of the user from the hand orientation; and

performing at least one of;

repositioning the graphic object on the tactile user interface, based on the detected hand orientation, such that the graphic objects are viewable by the user in a correct orientation to the user'"'"'s location, and

creating a workspace boundary around the graphic objects of each of a plurality of users such that each user'"'"'s hand gestures are used for implementing an item processing task on the displayed graphic objects within the respective boundary.