Gesture pre-processing of video stream using a markered region

• US 8,761,448 B1
• Filed: 03/13/2013
• Issued: 06/24/2014
• Est. Priority Date: 12/13/2012
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Intel Corporation
• Inventors: Burr, Jeremy
• Primary Examiner(s): Seth, Manav
• Application Number:
  US13/800,910
• Publication Number:
  US 20140168084A1
• Time in Patent Office: 468 Days
• Field of Search: None
• US Class Current: 382/103
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Techniques are disclosed for processing a video stream to reduce platform power by employing a stepped and distributed pipeline process, wherein CPU-intensive processing is selectively performed. The techniques are particularly well-suited for hand-based navigational gesture processing. In one example case, for instance, the techniques are implemented in a computer system wherein initial threshold detection (image disturbance) and optionally user presence (hand image) processing components are proximate to or within the system'"'"'s camera, and the camera is located in or proximate to the system'"'"'s primary display. In some cases, image processing and communication of pixel information between various processing stages which lies outside a markered region is suppressed. In some embodiments, the markered region is aligned with, a mouse pad or designated desk area or a user input device such as a keyboard. Pixels evaluated by the system can be limited to a subset of the markered region.

Claims

1. A system for processing a raw video stream, comprising:

a threshold detection stage comprising integrated circuitry configured to determine if a current frame of the video stream is sufficiently different from a previous frame of the video stream, and in response to the current frame not being sufficiently different than the previous frame, the threshold detection stage is further configured to stop further processing of the current frame by a subsequent stage;

a target presence stage comprising integrated circuitry configured to determine if the current frame includes skintone colored pixels, and in response to the current frame not including skintone colored pixels, the target presence stage is further configured to stop further processing of the current frame by a subsequent stage; and

a gesture processing stage configured to perform at least one of spatial and temporal interpretation of image data within frames passed by the threshold detection and target presence stages.

2. The system of claim 1 wherein image processing and communication of pixel information which lies outside a markered region is eliminated, and the markered region is a subset of a given field of view.

3. The system of claim 2 wherein the markered region is aligned with a user input device, or a mouse pad, or a desk area.

4. The system of claim 2 wherein the markered region is aligned with a keyboard and is defined by a perimeter of the keyboard.

5. The system of claim 2 wherein the markered region is aligned with a mouse pad and is defined by a perimeter of the mouse pad.

6. The system of claim 2 wherein a number of pixels evaluated by the system is a subset of the markered region.

7. The system of claim 6 wherein the subset is one of a pixel-scanline, a partial-pixel-scanline, and a single pixel.

8. The system of claim 1 wherein image processing and communication of pixel information which lies outside a markered region is eliminated, and the markered region is a user-configurable subset of a given field of view.

9. The system of claim 1 wherein the gesture processing stage is further configured to perform at least one of spatial and temporal interpretation only on image data within a markered region of a given field of view.

10. The system of claim 1 wherein the target presence stage is configured to convert data of the current frame into one or more HSV pixels, and if a sufficient number of skintone colored pixels are found, then data of the current frame is passed to a segmentation stage configured to separate pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs.

11. The system of claim 10 wherein the segmentation stage is implemented in a graphics processing unit (GPU).

12. The system of any one of claim 10 wherein the gesture processing stage includes a parser stage that is configured to perform spatial interpretation of blob data received from the segmentation stage.

13. The system of claim 12 wherein the parser stage is configured to perform spatial interpretation of the blob data to identify the presence of an ovoid-like area of non-skintone within a skintone region, and wherein the ovoid-like area of non-skintone within a skintone region is one of fully closed or partially closed.

14. The system of claim 12 wherein the gesture processing stage further includes a gesture recognition stage that is configured to perform temporal interpretation of blob data received from the parser stage.

15. The system of claim 14 wherein at least one of the threshold detection and target presence stages are implemented within or proximate to a camera deployable at a first location and having a field of view that includes a user input area, and the segmentation, parser and gesture recognition stages are at one or more locations different from the first location.

16. The system of claim 14 wherein the segmentation stage is implemented in a graphics processing unit, and the parser and gesture recognition stages are implemented by application software.

17. The system of claim 1 further comprising an application programming interface (API) for operatively coupling one or more of the stages to a computing platform in which the system is configured to operate.

18. An imaging device comprising the system of claim 1 and a camera operatively coupled to the system, wherein at least one of the threshold detection stage and the target presence stage are implemented within sensor circuitry of the camera.

19. A display comprising the system of claim 1 wherein at least one of the threshold detection stage and the target presence stage are implemented within sensor circuitry of a camera integrated within the display.

20. A computer navigation method using a raw video stream, comprising:

determining, at a first processing stage, if a current frame of the video stream is sufficiently different from a previous frame of the video stream, and in response to the current frame not being sufficiently different than the previous frame, stopping further processing of the current frame by a subsequent stage;

determining, at a second processing stage, if the current frame includes skintone colored pixels, and in response to the current frame not including skintone colored pixels, stopping further processing of the current frame by a subsequent stage;

receiving, at a third processing stage, data from the current frame and separating pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs;

performing, at a fourth processing stage, spatial interpretation of blob data received from the third processing stage by mapping the blob data to human hand geometry;

performing, at a fifth processing stage, temporal interpretation of blob data received from the fourth processing stage to identify a user navigation gesture; and

directing a computing system based on the user navigation gesture;

wherein the processing stages are configured in a stepped and distributed arrangement and image processing and communication of pixel information which lies outside a markered region is eliminated.

21. The method of claim 20 wherein the markered region is a subset of a given field of view and is aligned with a keyboard or a mouse pad or a desk area, and a number of pixels evaluated by at least one the processing stages is a subset of the markered region.

22. A non-transient computer program product encoded with instructions that when executed by one or more processors causes a navigation method to be carried out using a raw video stream, the method comprising:

determining, at a first processing stage, if a current frame of the video stream is sufficiently different from a previous frame of the video stream, and in response to the current frame not being sufficiently different than the previous frame, stopping further processing of the current frame by a subsequent stage;

determining, at a second processing stage, if the current frame includes skintone colored pixels, and in response to the current frame not including skintone colored pixels, stopping further processing of the current frame by a subsequent stage; and

performing, at a stage subsequent to the first and second processing stages, at least one of spatial and temporal interpretation of image data within frames passed by the threshold detection and target presence stages;

wherein image processing and communication of pixel information which lies outside a markered region is eliminated.

23. The computer readable medium of claim 22 wherein the markered region is a subset of a given field of view and is aligned with a keyboard and defined by a perimeter of that keyboard.

24. The computer readable medium of claim 23 wherein a number of pixels evaluated by at least one the processing stages is a subset of the markered region.

25. The computer readable medium of claim 22 wherein the markered region is a user-configurable subset of a given field of view.

26. The computer readable medium of claim 22 wherein the performing comprises at least one of:

receiving, at a third processing stage, data from the current frame and separating pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs;

performing, at a fourth processing stage, spatial interpretation of blob data received from the third processing stage by mapping the blob data to human hand geometry; and

performing, at a fifth processing stage, temporal interpretation of blob data received from the fourth processing stage to identify a user navigation gesture.

27. The computer readable medium of claim 26 further comprising:

directing a computing system based on the user navigation gesture.