GESTURE PRE-PROCESSING OF VIDEO STREAM USING A MARKERED REGION

• US 20150015480A1
• Filed: 05/22/2014
• Published: 01/15/2015
• Est. Priority Date: 12/13/2012
• Status: Active Grant

• Current Assignee: Tahoe Research Limited (f/k/a Learndale Limited) (Vector Capital Corporation)
• Original Assignee: Jeremy Burr
• Inventors: Burr, Jeremy
• Granted Patent: US 9,720,507 B2
• Time in Patent Office: Days
• US Class Current: 345/156
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/161 Detection; Localisation; Normalisation
• CPC Class Codes: G06V 40/162 using pixel segmentation or colour matching
• CPC Class Codes: G06V 40/28 Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113

Abstract

Techniques are disclosed for processing a video stream to reduce platform power by employing a stepped and distributed pipeline process, wherein CPU-intensive processing is selectively performed. The techniques are particularly well-suited for hand-based navigational gesture processing. In one example case, for instance, the techniques are implemented in a computer system wherein initial threshold detection (image disturbance) and optionally user presence (hand image) processing components are proximate to or within the system'"'"'s camera, and the camera is located in or proximate to the system'"'"'s primary display. In some cases, image processing and communication of pixel information between various processing stages which lies outside a markered region is suppressed. In some embodiments, the markered region is aligned with, a mouse pad or designated desk area or a user input device such as a keyboard. Pixels evaluated by the system can be limited to a subset of the markered region.

Claims

1. A system for processing a raw video stream, comprising:

a threshold detection stage comprising integrated circuitry configured to identify presence of activity in a given image frame, and to stop processing of that image frame by a subsequent processing stage if insufficient activity is identified; and

a target presence stage configured to identify presence of a user in an image frame in which activity has been identified.

2. The system of claim 1 wherein at least one of the threshold detection stage and the target presence stage is further configured to process only image frame data corresponding to a markered region within a given field of view, the markered region being a subset of the field of view.

3. The system of claim 1 wherein at least one of the threshold detection stage and the target presence stage is implemented in at least one of a camera and a display.

4. The system of claim 3 wherein the target presence stage is implemented in a graphics processing unit.

10. The system of claim 2 wherein a number of pixels evaluated by the system is a subset of the markered region.

11. The system of claim 10 wherein the subset is one of a pixel-scanline, a partial-pixel-scanline, and a single pixel.

13. The system of claim 1 further comprising a gesture processing stage configured to perform at least one of spatial and temporal interpretation of image data within frames passed by the threshold detection and target presence stages, wherein the at least one of spatial and temporal interpretation is performed only on image data within a markered region of a given field of view.

14. The system of claim 1 wherein the target presence stage is configured to identify presence of a user in an image frame in which activity has been identified by determining if the current frame includes skintone colored pixels, and in response to the current frame not including skintone colored pixels, the target presence stage is further configured to stop further processing of the current frame by a subsequent segmentation stage implemented in a graphics processing unit.

15. The system of claim 1 wherein the target presence stage is configured to identify presence of a user in an image frame in which activity has been identified by converting data of the current frame into one or more HSV pixels, and if a sufficient number of skintone colored pixels are found, then data of the current frame is passed to a subsequent segmentation stage implemented in a graphics processing unit.

16. The system of claim 15 wherein the segmentation stage is configured to separate pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs.

18. The system of claim 16 further comprising:

a parser stage that is configured to perform spatial interpretation of blob data received from the segmentation stage, wherein the parser stage is configured to perform spatial interpretation of the blob data to identify the presence of an ovoid-like area of non-skintone within a skintone region, and wherein the ovoid-like area of non-skintone within a skintone region is one of fully closed or partially closed; and

a gesture recognition stage that is configured to perform temporal interpretation of blob data received from the parser stage;

wherein at least one of the threshold detection and target presence stages are implemented within or proximate to a camera deployable at a first location and having a field of view that includes a user input area, and the segmentation, parser and gesture recognition stages are at one or more locations different from the first location; and

wherein the segmentation stage is implemented in a graphics processing unit, and the parser and gesture recognition stages are implemented by application software.

21. An imaging device comprising the system of claim 1 and a camera, wherein at least one of the threshold detection stage and the target presence stage are implemented within sensor circuitry of the camera.

22. A display comprising the system of claim 1 wherein at least one of the threshold detection stage and the target presence stage are implemented within sensor circuitry of a camera integrated within the display.

5. (canceled)

6. (canceled)

7. (canceled)

8. (canceled)

9. (canceled)

12. (canceled)

17. (canceled)

19. (canceled)

20. (canceled)

23. A non-transient computer readable medium encoded with instructions that when executed by one or more processors causes a method for processing a raw video stream to be carried out, the method comprising:

identify presence of activity in a given image frame, and stop processing of that image frame if insufficient activity is identified; and

identify presence of a user in an image frame in which activity has been identified.

24. The computer readable medium of claim 23 wherein only image frame data corresponding to a markered region is processed, the markered region being a subset within a given field of view.

25. The computer readable medium of claim 24 wherein a number of pixels evaluated is a subset of the markered region.

27. The computer readable medium of claim 23, the method further comprising:

separate, subsequent to identifying at least one of activity or the presence of a user, pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs;

perform spatial interpretation of the blob data to identify the presence of an ovoid-like area of non-skintone within a skintone region, wherein the ovoid-like area of non-skintone within a skintone region is one of fully closed or partially closed; and

perform temporal interpretation of blob data.

26. (canceled)

28. A computer navigation method using a raw video stream, comprising:

identifying, at a first processing stage, presence of activity in a given image frame, and stopping processing of that image frame if insufficient activity is identified;

identifying, at a second processing stage, presence of a user in an image frame in which activity has been identified;

identifying, at a subsequent processing stage, a user navigation gesture captured in an image frame in which presence of a user has been identified; and

directing a computing system based on the user navigation gesture.

29. The method of claim 28 further comprising:

receiving, at a third processing stage, data from an image frame in which presence of a user has been identified and separating pixels of interest from pixels that are not of interest, wherein pixels of interest are represented as blobs and pixels not of interest are represented by non-blobs; and

performing, at a fourth processing stage, spatial interpretation of blob data received from the third processing stage by mapping the blob data to human anatomical geometry;

wherein identifying the user navigation gesture comprises performing, at a fifth processing stage, temporal interpretation of blob data received from the fourth processing stage to identify the user navigation gesture.

30. The method of claim 28 wherein only image frame data corresponding to a markered region is processed, the markered region being a subset within a given field of view.