System and method for parallel processing of images from a large number of cameras

• US 8,295,649 B2
• Filed: 11/08/2007
• Issued: 10/23/2012
• Est. Priority Date: 11/08/2006
• Status: Active Grant

• Current Assignee: Nextgenid, Inc. (Zeva, Inc.)
• Original Assignee: Nextgenid, Inc. (Zeva, Inc.)
• Inventors: Berini, Dario, Moica, Simion Adrian, Srivastava, Prateek, Van Beek, Gary, Dadrassan, Hooman, Fevens, Bryon
• Primary Examiner(s): ZARKA, DAVID PETER
• Application Number:
  US11/937,312
• Publication Number:
  US 20080123967A1
• Time in Patent Office: 1,811 Days
• Field of Search: 382/304
• US Class Current: 382/304
• CPC Class Codes: G06V 10/94 Hardware or software architectures specially adapted for image or video understanding
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G07C 9/37 using biometric data, e.g. fingerprints, iris scans or voice recognition
• CPC Class Codes: G08B 13/1961 Movement detection not involving frame subtraction, e.g. motion detection on the basis of luminance changes in the image
• CPC Class Codes: G08B 13/19656 Network used to communicate with a camera, e.g. WAN, LAN, Internet

Abstract

A system and method for processing images includes a plurality of image providers configured to transmit images. A plurality of destination processors receives the transmitted images and transforms the transmitted images to internally useable image data. A plurality of feature object engines find and identify in the internally useable image data a plurality of objects. A plurality of object classifier engines index and classify the plurality of objects found by the feature object engines.

Claims

1. An autonomous system for processing video streams or single images, comprising:

a plurality of image providers configured to transmit images in a plurality of image streams;

a plurality of destination processors configured to receive the transmitted images and to transform the transmitted images to internally useable image data;

a plurality of feature object engines configured to find and identify in the internally useable image data a plurality of objects;

a plurality of object classifier engines configured to index and classify the plurality of objects found by the feature object engines;

a plurality of metadata inserters configured to embed information into the plurality of image streams; and

a plurality of intelligent video multiplexers or routers that use the metadata embedded in the plurality of image streams to route the video data to one or more downstream destination processors,

wherein the information includes data relating to a face comprising one or more of;

facial biometrics, a position of the face, a size of the face, landmarks of the face, an orientation of the face, and a quality of the face image, and

the intelligent video multiplexers or routers use the metadata to dynamically prioritize the plurality of image streams to ensure that higher priority streams are assigned to operational downstream destination processors before assigning lower priority streams.

2. The system for processing images of claim 1, further comprising a plurality of metadata inserters configured to embed information comprising metadata into the plurality of image streams, wherein the information is at least one type of information selected from the list:

static information, dynamic information.

3. The system for processing images of claim 2, further comprising a plurality of alarm engines that generate actionable activities based on the output of the destination processors and the metadata provided by the metadata inserters.

4. The system for processing images of claim 2, wherein the information includes static data selected from the group consisting of:

name of the camera location, serial number of the camera, image source or identification file.

5. The system for processing images of claim 2, wherein the information includes dynamic data related to an environmental monitoring system.

6. The system for processing images of claim 5, wherein the environmental monitoring system comprises at least one system selected from a set consisting of:

fire alarms in the surveillance area, smoke alarms in the surveillance area, noise detector in surveillance area, motion detector in the surveillance area, current physical location of the camera using devices such as GPS system or beacons, current compass bearing where the camera is pointing, timestamp, auxiliary alarm system, lojack in a stolen car, access and proximity sensors, or control systems.

7. The system for processing images of claim 2, wherein the information includes dynamic data generated at the request of a person selected from the list consisting of:

officer in distress, person at a front door, first responders, panic buttons, voice commands and recognition systems.

8. The system for processing images of claim 2, further comprising one or more decision engines which generate alarms in response to the classification of the objects found by one of the feature object engines.

9. The system for processing images of claim 8, wherein the alarm is accompanied by metadata related to a triggering event.

10. The system for processing images of claim 9, wherein the information includes data related to a target object, selected from the list consisting of:

position of the object, orientation of the object, position and orientation of sub-feature of the object, quality of the image, blur factor, or brightness or contrast factors.

11. The system for processing images of claim 10, wherein the information includes data relating to a face selected from the group consisting of:

a position of eyes, a position of tip of the nose or corners of mouth, and orientation of the face including pitch, roll, and yaw angles.

12. The system for processing images of claim 2, wherein the information and additional metadata generated after image processing, is used to generate an actionable activity selected from the set consisting of:

decision to send the data and image for long term storage, send image data to video displays being monitored by operators, perform object recognition, send pertinent information to field dispatch systems, or raise visual or audio alarms.

13. The system for processing images of claim 1, wherein the plurality of image providers includes at least one image provider selected from the list consisting of:

an analog or digital video camera, an analog or digital video recorder, a static image acquisition device, or a static image storage device.

14. The system for processing images of claim 1, wherein the destination processors include at least one image provider selected from the list consisting of:

a computer-based client application, a visual transducer, a video transducer, or an audio transducer.

15. The system for processing images of claim 1, wherein the system is configured such that the lower priority stream is not processed further in the case of downstream destination process unavailability.

16. The system for processing images of claim 1, wherein the intelligent video multiplexers or routers use the information to assign more than one of the plurality of image streams to one downstream destination processor to maximize resource usage.

17. The system for processing images of claim 1, wherein the system is configured to utilize destination processors based on one or more of the following:

application specific hardware architecture, reconfigurable hardware architecture, or embedded processors architecture.

18. The system for processing images of claim 17, wherein destination processors are configured to alter their behavior, and be targeted to find a different class of object.

19. The system for processing images of claim 1, wherein the system is configured for face recognition and outputs a standardized format video stream.

20. The system for processing images of claim 19, wherein the standardized format video stream frames are processed and normalized to International Civil Aviation Organization (ICAO) static image format, type 2 or type 3, International Organization for Standardization format (ISO) or International Committee for Information Technology Standards (INCITS) or any other required image standards.

21. The system for processing images of claim 1, wherein at least one of the image providers is an image enhancement device comprising:

an image provider configured to transmit an image to an image enhancement device, or an image enhancement device configured to add additional information to the image and to retransmit the image.

22. The system for processing images of claim 21, wherein at least one image provider is a video camera and the additional data is embedded in the camera information on a Vertical Blanking Interval.

23. The system for processing images of claim 1, further comprising a system management engine which manages the system and performs system management functions.

24. The system for processing images of claim 23, wherein the system management engine performs one or more operations selected from the list consisting of:

status report generation, reconfiguration, code updates and code upgrades.

25. The system for processing images of claim 1, wherein the system is configured to utilize a flexible and scalable modular deployment configuration tailored to an application'"'"'s need where destination processors are added as required to deploy additional functionality.

26. The system for processing images of claim 1, wherein each of the plurality of image providers and the plurality of destination processors is hot swappable.

27. An autonomous system for processing video streams or single images, comprising:

a plurality of image providers configured to transmit images in a plurality of image streams;

a plurality of destination processors configured to receive the transmitted images and to transform the transmitted images to internally useable image data;

a plurality of feature object engines configured to find and identify in the internally useable image data a plurality of objects;

a plurality of object classifier engines configured to index and classify the plurality of objects found by the feature object engines;

a plurality of metadata inserters configured to embed information into the plurality of image streams; and

a plurality of intelligent video multiplexers or routers that use the metadata embedded in the plurality of image streams to route the video data to one or more downstream destination processors,

wherein the information includes data relating to a face comprising one or more of;

facial biometrics, a position of the face, a size of the face, landmarks of the face, an orientation of the face, and a quality of the face image, and

the intelligent video multiplexers or routers use the metadata to assign one of the plurality of image streams to more than one downstream destination processor to allow simultaneous identification of two or more different kinds of objects in the video stream.

28. The system for processing images of claim 27, further comprising a plurality of metadata inserters configured to embed information comprising metadata into the plurality of image streams, wherein the information is at least one type of information selected from the list:

static information, dynamic information.

29. The system for processing images of claim 28, further comprising a plurality of alarm engines that generate actionable activities based on the output of the destination processors and the metadata provided by the metadata inserters.

30. The system for processing images of claim 28, wherein the information includes static data selected from the group consisting of:

name of the camera location, serial number of the camera, image source or identification file.

31. The system for processing images of claim 28, wherein the information includes dynamic data related to an environmental monitoring system.

32. The system for processing images of claim 31, wherein the environmental monitoring system comprises at least one system selected from a set consisting of:

fire alarms in the surveillance area, smoke alarms in the surveillance area, noise detector in surveillance area, motion detector in the surveillance area, current physical location of the camera using devices such as GPS system or beacons, current compass bearing where the camera is pointing, timestamp, auxiliary alarm system, lojack in a stolen car, access and proximity sensors, or control systems.

33. The system for processing images of claim 28, wherein the information includes dynamic data generated at the request of a person selected from the list consisting of:

officer in distress, person at a front door, first responders, panic buttons, voice commands and recognition systems.

34. The system for processing images of claim 28, further comprising one or more decision engines which generate alarms in response to the classification of the objects found by one of the feature object engines.

35. The system for processing images of claim 34, wherein the alarm is accompanied by metadata related to a triggering event.

36. The system for processing images of claim 35, wherein the information includes data related to a target object, selected from the list consisting of:

position of the object, orientation of the object, position and orientation of sub-feature of the object, quality of the image, blur factor, or brightness or contrast factors.

37. The system for processing images of claim 36, wherein the information includes data relating to a face selected from the group consisting of:

a position of eyes, a position of tip of the nose or corners of mouth, and orientation of the face including pitch, roll, and yaw angles.

38. The system for processing images of claim 28, wherein the information and additional metadata generated after image processing, is used to generate an actionable activity selected from the set consisting of:

decision to send the data and image for long term storage, send image data to video displays being monitored by operators, perform object recognition, send pertinent information to field dispatch systems, or raise visual or audio alarms.

39. The system for processing images of claim 27, wherein the plurality of image providers includes at least one image provider selected from the list consisting of:

an analog or digital video camera, an analog or digital video recorder, a static image acquisition device, or a static image storage device.

40. The system for processing images of claim 27, wherein the destination processors include at least one image provider selected from the list consisting of:

a computer-based client application, a visual transducer, a video transducer, or an audio transducer.

41. The system for processing images of claim 27, wherein the intelligent video multiplexers or routers use the information to assign more than one of the plurality of image streams to one downstream destination processor to maximize resource usage.

42. The system for processing images of claim 27, wherein the system is configured to utilize destination processors based on one or more of the following:

application specific hardware architecture, reconfigurable hardware architecture, or embedded processors architecture.

43. The system for processing images of claim 42, wherein destination processors are configured to alter their behavior, and be targeted to find a different class of object.

44. The system for processing images of claim 27, wherein the system is configured for face recognition and outputs a standardized format video stream.

45. The system for processing images of claim 44, wherein the standardized format video stream frames are processed and normalized to International Civil Aviation Organization (ICAO) static image format, type 2 or type 3, International Organization for Standardization format (ISO) or International Committee for Information Technology Standards (INCITS) or any other required image standards.

46. The system for processing images of claim 27, wherein at least one of the image providers is an image enhancement device comprising:

an image provider configured to transmit an image to an image enhancement device, or an image enhancement device configured to add additional information to the image and to retransmit the image.

47. The system for processing images of claim 46, wherein at least one image provider is a video camera and the additional data is embedded in the camera information on a Vertical Blanking Interval.

48. The system for processing images of claim 47, further comprising a system management engine which manages the system and performs system management functions.

49. The system for processing images of claim 48, wherein the system management engine performs one or more operations selected from the list consisting of:

status report generation, reconfiguration, code updates and code upgrades.

50. The system for processing images of claim 27, wherein the system is configured to utilize a flexible and scalable modular deployment configuration tailored to an application'"'"'s need where destination processors are added as required to deploy additional functionality.

51. The system for processing images of claim 27, wherein each of the plurality of image providers and the plurality of destination processors is hot swappable.

52. An autonomous system for processing data streams, comprising:

a plurality of data providers configured to transmit data in a plurality of data streams;

a plurality of destination processors configured to receive the transmitted data and to transform the transmitted data to internally useable data;

a plurality of feature object engines configured to find and identify in the internally useable data a plurality of objects;

a plurality of object classifier engines configured to index and classify the plurality of objects found by the feature object engines;

a plurality of metadata inserters configured to embed information into the plurality of data streams; and

a plurality of intelligent multiplexers or routers that use the metadata embedded in the plurality of data streams to route the video data to one or more downstream destination processors,

wherein the information relates to a face comprising one or more of;

facial biometrics, a position of the face, a size of the face, landmarks of the face, an orientation of the face, and a quality of the face image, and

the intelligent multiplexers or routers use the metadata to dynamically prioritize the plurality of data streams to ensure that higher priority streams are assigned to operational downstream destination processors before assigning lower priority streams.

53. An autonomous system for processing data streams, comprising:

a plurality of data providers configured to transmit data in a plurality of data streams;

a plurality of destination processors configured to receive the transmitted data and to transform the transmitted data to internally useable data;

a plurality of feature object engines configured to find and identify in the internally useable data a plurality of objects;

a plurality of object classifier engines configured to index and classify the plurality of objects found by the feature object engines;

a plurality of metadata inserters configured to embed information into the plurality of data streams; and

a plurality of intelligent multiplexers or routers that use the metadata embedded in the plurality of image streams to route the data to one or more downstream destination processors,

wherein the information relates to a face comprising one or more of;

facial biometrics, a position of the face, a size of the face, landmarks of the face, an orientation of the face, and a quality of the face image, and

the intelligent multiplexers or routers use the metadata to assign one of the plurality of data streams to more than one downstream destination processor to allow simultaneous identification of two or more different kinds of objects in the data stream.