COMPUTER VISION APPLICATION PROCESSING

• US 20140368688A1
• Filed: 06/12/2014
• Published: 12/18/2014
• Est. Priority Date: 06/14/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Rabii, Khosro Mohammad, Jarosinski, Tadeusz, Swaminathan, Ashwin, John Archibald, Fitzgerald, Damecharla, Hima Bindu
• Granted Patent: US 10,091,419 B2
• Time in Patent Office: Days
• US Class Current: 348/222.1
• CPC Class Codes: G06F 1/3215 Monitoring of peripheral devices
• CPC Class Codes: G06F 1/325 Power saving in peripheral device
• CPC Class Codes: G06F 18/23 Clustering techniques
• CPC Class Codes: G06V 10/96 Management of image or video recognition tasks
• CPC Class Codes: G06V 40/167 using comparisons between temporally consecutive images
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

Methods, systems, and techniques to enhance computer vision application processing are disclosed. In particular, the methods, systems, and techniques may reduce power consumption for computer vision applications and improve processing efficiency for computer vision applications.

Claims

1. A method comprising:

receiving, at a mobile device, sensory data from an image sensor, wherein the sensory data is related to an image;

extracting a first subset of features of the image from the sensory data;

extracting a second subset of features of the image from the sensory data based on the first subset of features satisfying an analysis criterion; and

generating context data indicating a context of the image based at least in part on the second subset of features.

2. The method of claim 1, wherein the first subset of features is extracted based on an extraction criterion being satisfied, wherein the extraction criterion is based on the sensory data, and wherein determining whether the extraction criterion is satisfied uses fewer resources of the mobile device than extracting the first subset of features.

3. The method of claim 2, further comprising satisfying the extraction criterion based on a comparison of first image statistics of the image and second image statistics of a second image.

4. The method of claim 2, wherein the extraction criterion is satisfied when the sensory data indicates a scene change.

5. The method of claim 1, further comprising periodically receiving the sensory data at a data sampling interval.

6. The method of claim 1, wherein the features of the image include a moving picture experts group (MPEG)-7 visual descriptor, a generalized search tree (GIST) descriptor, a census transform histogram (CENTRIST) visual descriptor, a bag-of-words descriptor, a visually locally aggregated descriptor, a spatial pyramid matching descriptor, or a combination thereof, and wherein the MPEG-7 visual descriptor includes a scalable color descriptor (SCD), a color structure descriptor (CSD), a group of frame (GOF) descriptor, a group of pictures (GOP) descriptor, a dominant color descriptor (DCD), a color layout descriptor (CLD), a texture browsing descriptor (TBD), a homogeneous texture descriptor (HTD), an edge histogram descriptor (EHD), a contour-based shape descriptor, a region-based shape descriptor, a camera motion descriptor (CMD), a motion trajectory descriptor (MTD), a parametric motion descriptor (PMD), or a motion activity descriptor (MAD).

7. The method of claim 1, wherein the context of the image includes a location, a person, an object, an activity, an emotion, an eye-gaze, a physical condition, or any combination thereof.

8. The method of claim 1, wherein extracting the first subset of features uses fewer resources of the mobile device than extracting a subsequent subset of features, and wherein the resources of the mobile device include memory, power, processor execution cycles, or any combination thereof.

9. The method of claim 1, wherein the analysis criterion is based on a confidence level associated with extracting the first subset of features.

10. The method of claim 9, further comprising:

generating a first clustered subset of features based on a first timestamp being within a threshold duration of a second timestamp,

wherein the first timestamp is associated with the image,

wherein the second timestamp is associated with a second image, and

wherein the first clustered subset of features includes the first subset of features and a third subset of features of the second image; and

generating a classified subset of features by classifying the first clustered subset of features based on a first classification model,

wherein the classified subset of features indicates that the first clustered subset of features corresponds to a location, a person, an object, an activity, or any combination thereof.

11. The method of claim 10, further comprising:

determining that the confidence level associated with extracting the first subset of features does not satisfy a threshold confidence level based on determining that a confidence level of intermediate context data does not satisfy the threshold confidence level,

wherein the intermediate context data corresponds to the first subset of features, the first clustered subset of features, or the classified subset of features, and

wherein the intermediate context data indicates a partial context of the image; and

generating a second clustered subset of features by clustering the second subset of features based on the first timestamp and the second timestamp, wherein the context data is generated by classifying the first clustered subset of features, the second clustered subset of features, or both, based on a second classification model.

12. The method of claim 1, wherein the sensory data is generated by the image sensor based on a determination that a particular threshold duration has elapsed, based on a determination that an available bandwidth satisfies a threshold bandwidth, based on a determination that an available battery power satisfies a threshold battery power, based on a user preference, based on user input, or any combination thereof.

13. The method of claim 1, wherein the context data is output to an application, wherein the application includes a life tagging application, a macro geo-fencing application, a micro geo-fencing application, a motion activity detection application, a text detection application, an object detection application, a gesture detection application, or any combination thereof.

14. An apparatus comprising:

a processor configured to;

receive sensory data from an image sensor, wherein the sensory data is related to an image;

extract a first subset of features from the sensory data, the first subset of features selected based on a selection criterion; and

generate context data indicating a context of the image based at least in part on the first subset of features.

15. The apparatus of claim 14, wherein the first subset of features is extracted based on an extraction criterion being satisfied, wherein the extraction criterion is based on the sensory data, wherein the sensory data is stored in a memory, wherein the memory is external to the processor, and wherein extracting the first subset of features includes accessing the memory.

16. The apparatus of claim 15, wherein the processor is further configured to generate first image statistics of the image based on the sensory data, wherein the extraction criterion is based at least in part on the first image statistics, and wherein the first image statistics are generated based on application-specific processing that includes at least one of activity recognition, person recognition, object recognition, location recognition, or gesture recognition.

17. The apparatus of claim 15, wherein the processor is further configured to activate a first power domain based on determining that the extraction criterion is satisfied.

18. The apparatus of claim 17, wherein the processor is further configured to activate a second power domain based on determining that the sensory data indicates that a sharpness of the image satisfies a threshold sharpness level.

19. The apparatus of claim 14, wherein the selection criterion includes a resource budget, a use case, a region of interest, or any combination thereof, wherein the region of interest includes a portion of the image, wherein the resource budget includes a memory budget, a power budget, a number of instructions per second budget, or any combination thereof, wherein the resource budget is based on user preferences, resource availability, or both, and wherein the use case includes activity recognition, person recognition, object recognition, location recognition, gesture recognition, or any combination thereof.

20. The apparatus of claim 14, wherein the first subset of features is extracted based on determining that the sensory data indicates that a sharpness of the image satisfies a threshold sharpness level.

21. An apparatus comprising:

means for receiving sensory data from an image sensor, wherein the sensory data is related to an image;

means for extracting a first subset of features from the sensory data, the first subset of features selected based on a selection criterion; and

means for generating context data indicating a context of the image based at least in part on the first subset of features.

22. The apparatus of claim 21, further comprising:

means for determining whether an extraction criterion is satisfied based on the sensory data, wherein the first subset of features is extracted based on the extraction criterion being satisfied;

means for activating a first power domain based on determining that the sensory data indicates a scene change; and

means for activating a second power domain based on determining that the sensory data indicates that a sharpness of the image satisfies a threshold sharpness level.

23. The apparatus of claim 22, wherein the means for receiving the sensory data, the means for extracting the first subset of features, the means for generating the context data, the means for determining whether the extraction criterion is satisfied, the means for activating the first power domain, and the means for activating the second power domain are integrated into one of a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, or a navigation device.

24. A computer-readable storage medium comprising instructions that, when executed by a computer, cause the computer to perform operations comprising:

receiving first sensory data from a first image sensor, wherein the first sensory data is related to a first image;

extracting a first subset of features of the first image from the first sensory data;

requesting second sensory data from a second image sensor based on the first subset of features satisfying an analysis criterion, wherein the second sensory data is related to a second image;

receiving the second sensory data from the second image sensor;

extracting a second subset of features of the second image from the second sensory data; and

generating context data indicating a context of the first image based at least in part on the second subset of features.

25. The computer-readable storage medium of claim 24, wherein the operations further comprise generating first image statistics of the first image based on the sensory data, wherein the second subset of features is extracted based on a change amount between the first image statistics and second image statistics of the second image satisfying a change threshold, and wherein the first image statistics and the second image statistics are generated based on application-specific processing.

26. The computer-readable storage medium of claim 25, wherein the operations further comprise:

generating a first color histogram of the first image, wherein the first image statistics include a first count of pixels of a particular color in the first color histogram; and

generating a second color histogram of the second image, wherein the second image statistics include a second count of pixels of the particular color in the second color histogram.

27. The computer-readable storage medium of claim 26, wherein the application-specific processing to be performed on the sensory data includes an identification of a particular object having the particular color.

28. The computer-readable storage medium of claim 25, wherein the operations further comprise generating a clustered subset of features by clustering the first subset of features, the second subset of features, or both, wherein the context data is generated by classifying the clustered subset of features based on a classification model.

29. The computer-readable storage medium of claim 25, wherein the first image corresponds to a first setting of the first image sensor, wherein the second image corresponds to a second setting of the second image sensor, wherein the first setting is distinct from the second setting, and wherein a setting of the first image sensor includes a zoom setting, a resolution setting, an angle setting, a location, or any combination thereof.

30. The computer-readable storage medium of claim 25, wherein the first image sensor is integrated into a first device, wherein the second image sensor is integrated into a second device, and wherein the first device and the second device are distinct.