Gesture pre-processing of video stream with hold-off period to reduce platform power

• US 9,104,240 B2
• Filed: 01/09/2013
• Issued: 08/11/2015
• Est. Priority Date: 01/09/2013
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Burr, Jeremy
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US13/737,413
• Publication Number:
  US 20140193030A1
• Time in Patent Office: 944 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Techniques are disclosed for processing a video stream to reduce platform power by employing a stepped and distributed pipeline process, wherein CPU-intensive processing is selectively performed. In one example case, the techniques are implemented in a user'"'"'s computer system wherein initial threshold detection (image disturbance) and optionally user presence (e.g., hand image) processing components are proximate to or within the system'"'"'s camera, and the camera is located in or proximate to the system'"'"'s primary display. The threshold detection and/or target presence stages can be selectively disabled for a hold-off period. The hold-off period may be, for example, in the range of 50 to 1000 mSec and triggered in response to an indication that a user of the system is unlikely to be making navigational gestures or that the system is not ready to process video, thereby conserving power by avoiding processing of video frames free of navigation gestures.

Claims

1. A system for processing a raw video stream, comprising:

a threshold detection stage comprising integrated circuitry configured to determine if a current frame of the video stream is sufficiently different from a previous frame of the video stream, and in response to the current frame not being sufficiently different than the previous frame, the threshold detection stage is further configured to stop further processing of the current frame by a subsequent stage; and

a target presence stage comprising integrated circuitry configured to determine if the current frame includes skin tone colored pixels, and in response to the current frame not including skin tone colored pixels, the target presence stage is further configured to stop further processing of the current frame by a subsequent stage;

wherein at least one of the threshold detection and/or target presence stages can be selectively disabled for a hold-off period.

2. The system of claim 1 wherein the threshold detection stage is configured to sum and compare a luma channel of the current frame with a luma channel of the previous frame, and if the delta between the current and previous frames is above a given threshold, then data of the current frame is passed to the target presence stage for further processing.

3. The system of claim 1 wherein the target presence stage is configured to convert data of the current frame into one or more HSV pixels, and if a sufficient number of skin tone colored pixels are found, then data of the current frame is passed to a next processing stage.

4. The system of claim 3 wherein the next processing stage is a segmentation stage configured to separate pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs.

5. The system of claim 4 wherein the segmentation stage is configured to at least one of eliminate disperse pixels due to camera noise and/or lighting, and/or process imaged data using erode and/or dilate morphing methods.

6. The system of claim 4 wherein the segmentation stage is implemented in a graphics processing unit (GPU).

7. The system of claim 4 further comprising:

a parser stage that is configured to perform spatial interpretation of blob data received from the segmentation stage.

8. The system of claim 7 wherein the parser stage performs spatial interpretation by mapping the blob data to a model of target human anatomical geometry.

9. The system of claim 8 wherein the target human anatomical geometry is a hand.

10. The system of claim 7 further comprising:

a gesture recognition stage that is configured to perform temporal interpretation of blob data received from the parser stage.

11. The system of claim 10 wherein at least one of the segmentation, parser, and/or gesture recognition stages are configured to send feedback to at least one of the threshold detection and/or target presence stages.

12. An imaging device comprising the system of claim 1 and a camera operatively coupled to the system.

13. The imaging device of claim 12 wherein at least one of the threshold detection stage and the target presence stage are implemented within sensor circuitry of the camera.

14. A display comprising the system of claim 1.

15. The system of claim 1 wherein the hold-off period is in the range of 50 to 1000 mSec, and is triggered in response to an indication that a user of the system is unlikely to be making navigational gestures, or that the system is not ready to process video.

16. A system for processing a raw video stream, comprising:

a threshold detection stage comprising integrated circuitry configured to determine if a current frame of the video stream is sufficiently different from a previous frame of the video stream, and in response to the current frame not being sufficiently different than the previous frame, the threshold detection stage is further configured to stop further processing of the current frame by a subsequent stage;

a target presence stage comprising integrated circuitry configured to determine if the current frame includes skin tone colored pixels, and in response to the current frame not including skin tone colored pixels, the target presence stage is further configured to stop further processing of the current frame by a subsequent stage;

a segmentation stage configured to receive data from the current frame, and separate pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs;

a hand parser stage that is configured to perform spatial interpretation of blob data received from the segmentation stage; and

a gesture recognition stage that is configured to perform temporal interpretation of blob data received from the hand parser stage and to identify user hand-based navigation gestures;

wherein the threshold detection, target presence, segmentation, parser, and gesture recognition stages are configured in a stepped and distributed arrangement, and at least one of the threshold detection and/or target presence stages can be selectively disabled for a hold-off period.

17. The system of claim 16 wherein at least one of the threshold detection and target presence stages are implemented within or proximate to a camera deployable at a first location and having a field of view that includes a user input area, and the segmentation, parser and gesture recognition stages are at one or more locations different from the first location.

18. The system of claim 17 wherein the threshold detection and target presence stages are implemented within a display.

19. The system of claim 16 wherein the segmentation stage is implemented in a graphics processing unit, and the parser and gesture recognition stages are implemented by application software.

20. The system of claim 16 further comprising an application programming interface (API) for operatively coupling one or more of the stages to a computing platform in which the system is configured to operate.

21. The system of claim 20 wherein the API is presented as an extended universal serial bus human interface (USB HID) class device.

22. The system of claim 16 wherein the hold-off period is in the range of 50 to 1000 mSec, and is triggered in response to an indication that a user of the system is unlikely to be making navigational gestures, or that the system is not ready to process video.

23. A media processing system comprising the system of claim 16.

24. A mobile computing system comprising the system of claim 16.

25. A computer navigation method using a raw video stream, comprising:

determining, at a first processing stage, if a current frame of the video stream is sufficiently different from a previous frame of the video stream, and in response to the current frame not being sufficiently different than the previous frame, stopping further processing of the current frame by a subsequent stage; and

determining, at a second processing stage, if the current frame includes skin tone colored pixels, and in response to the current frame not including skin tone colored pixels, stopping further processing of the current frame by a subsequent stage;

wherein at least one of the first and second processing stages can be selectively disabled for a hold-off period, the hold-off period being in the range of 50 to 1000 mSec and triggered in response to an indication that no navigational gestures are being made, thereby conserving power by avoiding processing of video frames free of navigation gestures.