Video processing system

• US 9,565,403 B1
• Filed: 05/05/2011
• Issued: 02/07/2017
• Est. Priority Date: 05/05/2011
• Status: Active Grant

• Current Assignee: The Boeing Co.
• Original Assignee: The Boeing Co.
• Inventors: Higgins, Robert Paul
• Primary Examiner(s): Findley, Christopher
• Application Number:
  US13/101,906
• Time in Patent Office: 2,105 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/17 taken from planes or by drones
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 1/00042 Monitoring, i.e. observation H04N1/00029 - H04N1/00039 take precedence
• CPC Class Codes: H04N 5/77 between a recording apparatus and a television camera
• CPC Class Codes: H04N 5/917 for bandwidth reduction
• CPC Class Codes: H04N 7/188 Capturing isolated or intermittent images triggered by the occurrence of a predetermined event, e.g. an object reaching a predetermined position signal generation from motion picture films H04N5/253
• CPC Class Codes: H04N 9/8042 involving data reduction

Abstract

A method and apparatus for processing images. A sequence of images is received from a sensor system. A number of objects is present in the sequence of images. Information about the number of objects is identified using the sequence of images and a selection of a level of reduction of data from different levels of reduction of data. A set of images from the sequence of images is identified using the selection of the level of reduction of data. The set of images and the information about the number of objects are represented in data. An amount of the data for the sequence of images is based on the selection of the level of reduction of data.

Claims

1. A method for processing images, the method comprising:

receiving, by a sensor platform comprising a sensor system, a sequence of images from the sensor system;

recognizing a number of objects in the sequence of images;

identifying, by the sensor platform, object information about the number of objects in the sequence of images;

identifying a number of threats from within the number of objects;

receiving a selection of a level of reduction of data from different levels of reduction of data; and

generating data, by the sensor platform, using the selection of the level of reduction of data,

wherein, an amount of the data is based on the selection of the level of reduction of data;

wherein, the different levels of reduction of data comprise a first level of reduction, a second level of reduction, and a third level of reduction;

wherein, when the first level of reduction is selected, the data comprises a set of images selected from a number of images in the sequence of images that is less than the number of images in the sequence of images, such that the set of images are selected based on an overlap between images in the set of images and a reference image;

wherein, when the second level of reduction is selected, the data comprises no images from the sequence of images and the object information; and

wherein, when the third level of reduction is selected, the data comprises no images from the sequence of images and object information only for the number of threats identified within the number of objects.

2. The method of claim 1 further comprising:

sending the data for the sequence of images from the sensor platform to another platform using a wireless communications link, wherein a bandwidth used in the wireless communications link for sending the data representing the information about the number of objects and the set of images is less than the bandwidth needed for sending the data representing all of the sequence of images.

3. The method of claim 1, wherein the step of identifying the information about the number of objects comprises:

identifying track information for the number of objects using the sequence of images, wherein the track information describes movement of the number of objects in a scene present in the sequence of images.

4. The method of claim 3, wherein the step of identifying the information about the number of objects further comprises:

identifying object information for the number of objects using at least one of the track information and the sequence of images, wherein the object information describes the number of objects present in the scene in the sequence of images.

5. The method of claim 4, wherein the step of identifying the information about the number of objects further comprises:

identifying a number of events that occur in the scene using the sequence of images; and

identifying event information for the number of events using at least one of the track information, the object information, and the sequence of images, wherein the event information is for a number of events of interest in the number of events.

6. The method of claim 5, wherein the step of identifying the information about the number of objects further comprises:

determining whether a number of threats are predicted based on the number of events identified; and

responsive to a determination that the number of threats is predicted based on the number of events identified, identifying threat information for the number of threats.

7. The method of claim 1, wherein the step of identifying the set of images from the sequence of images comprises:

determining whether the overlap between a current image in the sequence of images and the reference image in the sequence of images is less than a selected threshold; and

responsive to a determination that the overlap is less than the selected threshold, adding the current image to the set of images.

8. The method of claim 1 further comprising:

receiving user input selecting the level of reduction of data from the different levels of reduction of data.

9. The method of claim 1 further comprising:

sending the data to a platform using a wireless communications link, wherein an average data rate used for sending the data for the sequence of images is reduced when a higher level of reduction of data is selected from the different levels of reduction of data as compared to when a lower level of reduction of data is selected from the different levels of reduction of data.

10. The method of claim 1 further comprising:

receiving the data;

identifying the information and the set of images represented in the data; and

displaying the information and the set of images on a display on a display device.

11. The method of claim 10, wherein the step of displaying the information and the set of images on the display on the display device comprises:

displaying a current image in the set of images; and

displaying a number of tracks for the number of objects, wherein each track in the number of tracks indicates a current position and a direction of movement for an object in the number of objects.

12. The method of claim 10, wherein the set of images is a null set wherein the step of displaying the information and the set of images on the display on the display device comprises:

displaying a scene on the display on the display device; and

displaying a number of object graphics on the scene using object information in the information.

13. The method of claim 10, wherein the set of images is a null set wherein the step of displaying the information and the set of images on the display on the display device comprises:

displaying at least one of event information, threat information, a scene, and a number of object graphics on the display on the display device using the information.

14. The method of claim 2, further comprising:

sending, by the sensor platform, the information about the number of objects instead of sending the sequence of images to a platform;

wherein the sensor platform is one of an unmanned aerial vehicle and an unmanned ground vehicle; and

wherein the platform is a ground station configured to receive and process the data from the sensor platform.

15. The method of claim 14,

wherein when the first level of reduction is selected, the data comprises representations of the set of images and the information about the number of objects;

wherein when a third level of reduction is selected, the set of images represented in the data is a null set and the information about the number of objects relates to objects involved in one or more of number of events of interest described in event information and number of threats described in threat information; and

wherein an amount of data related to the third level of reduction is less than an amount of data related to the second level of reduction.

16. The method of claim 7, further comprising:

measuring the overlap by aligning the current image and the reference image such that one or more features in the current image and the reference image substantially match.

17. The method of claim 1, wherein the data further includes platform meta-data, wherein the platform meta-data comprises a route for the sensor platform.

18. An apparatus comprising:

an image processing module of a sensor platform comprising a sensor system, the image processing module configured to;

receive a sequence of images from the sensor system;

recognize a number of objects in the sequence of images;

identify object information about the number of objects in the sequence of images;

identify a number of threats from within the number of objects;

receive a selection of a level of reduction of data from different levels of reduction of data; and

generate data using the selection of the level of reduction of data,

wherein an amount of the data is based on the selection of the level of reduction of data;

wherein the different levels of reduction of data comprise a first level of reduction, a second level of reduction, and a third level of reduction;

wherein, when the first level of reduction is selected, the data comprises a set of images selected from a number of images in the sequence of images that is less than the number of images in the sequence of images, such that the set of images are selected based on an overlap between images in the set of images and a reference image;

wherein, when the second level of reduction is selected, the data comprises no images from the sequence of images and the object information; and

wherein, when the third level of reduction is selected, the data comprises no images from the sequence of images and object information only for the number of threats identified with the number of objects.

19. The apparatus of claim 18 further comprising:

a communications system of the sensor platform, the communications system configured to send the data for the sequence of images from the sensor platform to another platform using a wireless communications link, wherein a bandwidth used in the wireless communications link for sending the data representing the information about the number of objects and the set of images is less than the bandwidth needed for sending the data representing all of the sequence of images and wherein the communications system, the sensor system, and the image processing module are in a sensor platform.

20. The apparatus of claim 18, wherein in being configured to identify the information about the number of objects, the image processing module is configured to identify at least one of track information for the number of objects, object information for the number of objects, event information for a number of events of interest identified using the sequence of images, and threat information for a number of threats predicted based on a number of events identified using the sequence of images.

21. The apparatus of claim 19, wherein in being configured to identify the set of images, the image processing module is configured to determine whether the overlap between a current image in the sequence of images and the reference image in the sequence of images is less than a selected threshold;

and add the current image to the set of images in response to a determination that the overlap is less than the selected threshold.

22. A method for processing images, the method comprising:

receiving, by a sensor platform comprising a sensor system, a sequence of images from the sensor system;

forming compressed data from the sequence of images based on a selection of one of a first level of data reduction, a second level of data reduction, and a third level of data reduction;

sending the compressed data by the sensor platform to a platform;

wherein a number of objects is present in the sequence of images;

wherein when the first level of data reduction is selected, the compressed data comprises track information and one or more images;

wherein when the second level of data reduction is selected, the compressed data comprises a null set of images, the track information, and object information;

wherein when the third level of data reduction is selected, the compressed data comprises the null set of images, the track information, the object information, event information, and threat information;

wherein the track information;

is identified from the sequence of images,

describes movement of the number of objects, and

includes, for each object of the number of objects, a previous location, a current location, a direction of travel, a speed, a size, and a color;

wherein the object information;

is identified from the sequence of images and the track information,

describes the number of objects, and

includes for each object of the number of objects, an identification of the object, a type of the object, a color the object, dimensions for the object, and a shape of the object;

wherein the event information;

is identified from the sequence of images, the track information, and the object information,

describes a number of events of interest, and

includes, for each event of interest of the number of events of interest, an identification of the event of interest, a list of objects in the number of objects involved in the event of interest, a location of the event of interest, a start time for the event of interest, a duration of the event of interest, and information about events that lead up to the event of interest;

wherein the threat information;

includes a number of threats predicted based on the number of events,

is identified from the sequence of images, the track information, the object information, and the event information, and

includes for each threat of the number of threats, an identification of the threat, a list of events in the number of events that contribute to a prediction of the threat, event information identified for events contributing to prediction of the threat, a list of objects in the number of objects involved in the threat;

wherein the sensor platform is an unmanned aerial vehicle; and

wherein the platform is a ground station.

23. A method for processing images, the method comprising:

receiving, by a sensor platform comprising a sensor system, a sequence of images from the sensor system;

receiving a selection of a level of reduction of data selected from one of;

a first level of reduction, a second level of reduction, and a third level of reduction;

determining, by the sensor platform, whether an overlap between a current image in the sequence of images and a reference image in the sequence of images is less than a selected threshold, and responsive to a determination that a determination that the overlap is less than the selected threshold, adding the current to a set of images such that the set of images selected is less than a number of images in the sequence of images, when the first level of reduction is selected;

identifying, by the sensor platform, object information about a number of objects in the sequence when the second of level of reduction is selected;

identifying, by the sensor platform, threat information in the number of objects in the sequence of images when the third level of reduction is selected; and

sending one or more of the set of images, the object information, or the threat information to another platform.

24. The method of claim 23, further comprising:

displaying, on a display device, the set of images a number of tracks for the number of objects, wherein each track in the number of tracks indicates a current position and a direction of movement for an object in the number of objects.