Method and system for ergonomic touch-free interface

• US 9,857,868 B2
• Filed: 03/19/2011
• Issued: 01/02/2018
• Est. Priority Date: 03/19/2011
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Original Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford University)
• Inventors: Plagemann, Christian, Dahlkamp, Hendrik, Ganapathi, Hariraam Varun, Thrun, Sebastian
• Primary Examiner(s): Sharifi-Tafreshi, Koosha
• Application Number:
  US13/052,050
• Publication Number:
  US 20120235904A1
• Time in Patent Office: 2,481 Days
• Field of Search: 345156, 345158, 345173-175, 715863, 382103, 382173, 382181, 382190
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object

Abstract

With the advent of touch-free interfaces such as described in the present disclosure, it is no longer necessary for computer interfaces to be in predefined locations (e.g., desktops) or configuration (e.g., rectangular keyboard). The present invention makes use of touch-free interfaces to encourage users to interface with a computer in an ergonomically sound manner. Among other things, the present invention implements a system for localizing human body parts such as hands, arms, shoulders, or even the fully body, with a processing device such as a computer along with a computer display to provide visual feedback on the display that encourages a user to maintain an ergonomically preferred position with ergonomically preferred motions. For example, the present invention encourages a user to maintain his motions within an ergonomically preferred range without have to reach out excessively or repetitively.

Claims

1. A human interface device, comprising:

a stereo camera configured to receive image information for user action;

a processor;

a display; and

a memory directed to store a software application;

wherein the software application directs the processor to;

generate depth maps from the image information received by the stereo camera;

utilize a color-based tracking algorithm to provide real-time skin segmentation;

localize at least one active part from the image information, wherein;

the active part is a body part of a user; and

the body part of the user includes fingertips;

utilize a model-based tracking algorithm to develop a three-dimensional model of the active parts of a user'"'"'s body;

localize the fingertips of the user using a plurality of methods including;

an optical flow algorithm;

shape and appearance descriptor identification;

a sliding window detector;

or

a voting-based identification algorithm;

monitor the active part for a predetermined initiation gesture wherein the predetermined initiation gesture is the simultaneous touching of the user'"'"'s left thumb to left index finger and the user'"'"'s right thumb to right index finger;

activate additional options upon the use of the predetermined initiation gesture;

receive a first predetermined range of motion for the active part;

determine a second range of motion for the at least one active part, wherein the second range of motion is substantially less than the first predetermined range of motion of the at least one active part;

allowing for a manner by which to input the second range of motion by the user, and the second range of motion is chosen to allow use of the at least one active part to provide input to the device; and

generate a three-dimensional virtual workspace on the display wherein the three-dimensional virtual workspace;

represents a space including the second range of motion;

includes at least one virtual surface;

provides visual cues defining the location of the at least one virtual surface;

maps the active part onto the virtual workspace as a three-dimensional cursor containing anthropomorphic detail; and

provides a first type of response if the cursor is within the second range of motion and second type of response if the cursor is outside the second range of motion wherein the second type of response is less responsive than the first response.

2. The device of claim 1, wherein the image information includes image information from a three-dimensional sensor.

3. The device of claim 1, wherein the visual cues include textual information.

4. The device of claim 1, wherein the processor is further configured to actively learn the second range of motion.

5. The device of claim 1, wherein the processor is further configured to establish a reference frame from which the position of the at least one active part is measured.

6. The device of claim 1, wherein the processor is configured to operate in real time.

7. The device of claim 1, wherein movement of the at least one active part is mapped to a virtual space.

8. The device of claim 1, wherein the software application utilizes a template matching algorithm to localize the at least one active part.

9. The device of claim 1, wherein the software utilizes a background subtraction algorithm to localize the at least one active part.

10. The device of claim 1, wherein the software application utilizes an iterative closest point algorithm to localize the at least one active part.

11. The device of claim 1, wherein the software application utilizes a multi-hypothesis algorithm to localize the at least one active part wherein the image information is partitioned into tracks.

12. The device of claim 1, wherein the software application utilizes a second order Markov model in the color-based tracking algorithm to predict the evolution of a skin color histogram over time.

13. The device of claim 11, wherein the tracks are confined to allow for classification of tracked objects.