Method and apparatus for rapidly determining whether a digitized image frame contains an object of interest

• US 6,449,384 B2
• Filed: 01/29/2001
• Issued: 09/10/2002
• Est. Priority Date: 10/23/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Facet Technology Corp.
• Inventors: Retterath, James Eugene, Laumeyer, Robert Anthony
• Primary Examiner(s): Johns, Andrew W.
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US09/772,191
• Publication Number:
  US 20010043717A1
• Time in Patent Office: 589 Days
• Field of Search: 382/105, 382/103, 382/104, 382/106, 382/156, 382/159, 382/164, 358/453, 358/456, 358/462, 701/117
• US Class Current: 382/104
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 20/582 of traffic signs

Abstract

The present invention relates to an apparatus for rapidly analyzing frame(s) of digitized video data which may include objects of interest randomly distributed throughout the video data and wherein said objects are susceptible to detection, classification, and ultimately identification by filtering said video data for certain differentiable characteristics of said objects. The present invention may be practiced on pre-existing sequences of image data or may be integrated into an imaging device for real time, dynamic, object identification, classification, logging/counting, cataloging, retention (with links to stored bitmaps of said object), retrieval, and the like. The present invention readily lends itself to the problem of automatic and semi-automatic cataloging of vast numbers of objects such as traffic control signs and utility poles disposed in myriad settings. When used in conjunction with navigational or positional inputs, such as GPS, an output from the inventive system indicates the identity of each object, calculates object location, classifies each object by type, extracts legible text appearing on a surface of the object (if any), and stores a visual representation of the object in a form dictated by the end user/operator of the system. The output lends itself to examination and extraction of scene detail which cannot practically be successfully accomplished with just human viewers operating video equipment, although human intervention can still be used to help judge and confirm a variety of classifications of certain instances and for types of identified objects.

Claims

1. A method of rapidly identifying whether a digitized image frame may contain objects of interest that are depicted in at least one frame of a digital image comprising the steps of:

for each digitized image frame of said digital image;

applying at least two separate filters to said frame to generate a discrete output value from each other, wherein each of at least two of said filters screens for a differentiable characteristic associated with the objects of interests that is unique to that filter;

comparing the discrete output values for each of said at least two filters to at least one reference; and

identifying said frame as potentially having an object of interest present if at least one of the discrete output values indicates a differentiable characteristic is present for said frame when the discrete output values are compared to the at least one reference.

2. The method of claim 1, wherein the step of identifying said frame further comprises the step of saving at least a portion of said digitized image frame.

3. The method of claim 1, wherein said digital image is recorded by at least one camera that is in motion during the recording of said at least one digitized image frames and wherein the step of identifying said digitized image frame further comprises the step of saving a location data metric corresponding to a location of the camera when said digitized image frame was originally recorded.

4. The method of claim 3, wherein the step of saving the location data metric comprises saving at least an orientation of the camera while recording, a focal length of the camera, and the location of the camera.

5. The method of claim 1, further comprising the steps of:

for each digitized image frame identified as having an object of interest;

segmenting said frame into at least one region of interest;

applying at least one filter to each of said regions of interest which differentiates between various types of objects of interest so as to classify each of said regions of interest by type; and

creating a record in a database corresponding to each type of object of interest that stores at least a portion of a bitmap corresponding to each region of interest in said frame of classified as containing that type of object of interest.

6. The method of claim 1, wherein each of said at least two filters is selected from the set of the following filters:

an edge filter, a color-pair filter, a color filter operating in the L*u*v color space, an edge filter combined with a line extender, or a color filter operating in the LCH color space.

7. The method of claim 1, further comprising the step of discarding said digitized image frames not identified as potentially having an object of interest present.

8. The method of claim 1, wherein the differentiable characteristics of each of said at least two filters is selected from the set comprising:

color characteristics, edge characteristics, texture characteristics, symmetry, convexity, lack of three dimensional volume, number and orientation of side edges, characteristic corner angles, frequency, and luminescence.

9. The method of claim 1, wherein said digital image is captured as said at least one digitized image frames in a frame buffer.

10. The method of claim 1, wherein said digitized image frames in said frame buffer are operably generated by a digital image capture device that is recording said digital image as an image selected from the set comprising:

live images, a pre-recorded set of images, a series of still images or a digitized version of an original analog image sequence.

11. The method of claim 1, further comprising the steps of:

for each digitized image frame identified as having an object of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics.

12. The method of claim 11, wherein the step of expanding further comprises the step of:

utilizing morphology techniques to grow and erode said search space by adding or subtracting segments adjacent said search space until either said search space meets or fails to meet uniform criteria for a differentiable characteristic.

13. The method of claim 1, wherein said digitized image frames comprise a large number of frames of image data and the method is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of image data.

14. In a computer system, a computer-readable storage media storing:

at least one computer program that operates to rapidly identify whether a digitized image frame may contain objects of interest that are depicted in at least one frame of a digital image by performing for each digitized image frame of said digital image the steps of;

applying at least two separate filters to said frame to generate a discrete output value from each filter, wherein each of at least two of said filters screens for a differentiable characteristic associated with the objects of interest that is unique to that filter;

comparing the discrete output values for each of said at least two filters to at least one reference; and

identifying said frame as potentially having an object of interest present if at least one of the discrete output values indicates a differentiable characteristics is present for said frame when the discrete output values are compared to the at least one reference.

15. The storage media of claim 14, wherein the step of identifying said frame further comprises the step of saving at least a portion of said digitized image frame.

16. The storage media of claim 14, wherein said digital image is recorded by at least one camera that is in motion during the recording of said at least one digitized image frames and wherein the step of identifying said digitized image frame further comprises the step of saving a location data metric corresponding to a location of the camera when said digitized image frame was originally recorded.

17. The storage media of claim 16, wherein the step of saving the location data metric comprises saving at least an orientation of the camera while recording, a focal length of the camera, and the location of the camera.

18. The storage media of claim 14, further comprising the steps of:

for each digitized image frame identified as having an object of interest;

segmenting said frame into at least one region of interest;

applying at least one filter to each of said regions of interest which differentiates between various types of objects of interests so as to classify each of said regions of interest by type; and

creating a record in a database corresponding to each type of object of interest that stores at least a portion of a bitmap corresponding to each region of interest in said frame of classified as containing that type of object of interest.

19. The storage media of claim 14, wherein each of said at least two filters is selected from the set of the following filters;

an edge filter, a color-pair filter, a color filter operating in the L*u*v color space, an edge filter combined with a line extender, or a color filter operating in the LCH color space.

20. The storage media of claim 14, further comprising the step of discarding said digitized image frames not identified as potentially having an object of interest present.

21. The storage media of claim 14, wherein the differentiable characteristics, edge characteristics, texture characteristics, symmetry, convexity, lack of three dimensional volume, number and orientation of side edges, characteristic corner angles, frequency, and luminescence.

22. The storage media of claim 14, wherein said digital image is captured as said at least one digitized image frames in a frame buffer.

23. The storage media of claim 14, wherein said digitized image frames in said frame buffer are operably generated by a digital image capture device that is recording said digital image as an image selected from the set comprising:

live images, a pre-recorded set of images, a series of still images or a digitized version of an original analog image sequence.

24. The storage media of claim 14, further comprising the steps of:

for each digitized image frame identified as having an object of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics.

25. The storage media of claim 24, wherein the step of expanding further comprises the step of:

utilizing morphology techniques to grow and erode said search space by adding or subtracting segments adjacent said search space until either said search space meets or fails to meet uniform criteria for a differentiable characteristic.

26. The storage media of claim 14, wherein said digitized image frames comprise a large number of frames of image data and the method is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of image data.

27. A computer system for rapidly identifying whether a digitized image frame may contain objects of interest:

a frame buffer that stores at least one frame of a digital image;

at least two separate filters operably connected to said frame buffer to generate a discrete output from each filter in response to each filter to each frame of said digital image in said frame buffer, wherein each of at least two of said filters screens for a differentiable characteristic associated with the objects of interest that is unique to that filter; and

a neural network operably connected to said discrete outputs of said at least two separate filters to identify a frame of said digital image as potentially having an object of interest present in response to said discrete outputs of said at least two separate filters.

28. The computer system of claim 27, further comprising:

a database that stores at least a portion of only said frames of said digital image that are identified by said neural network.

29. The computer system of claim 27, further comprising:

at least one camera that is in motion during the recording of said at least one digitized image frames; and

a database that stores at least a portion of only said frames of said digital image that are identified by said neural network along with a location data metric corresponding to a location of the camera when said digitized image frame was originally recorded.

30. The computer system of claim 29, wherein the location data metric stored in said database includes at least an orientation of the camera while recording, a focal length of the camera, and the location of the camera.

31. The computer system of claim 27, wherein each of said at least two filters is selected from the set of the following filters;

an edge filter, a color-pair filter, a color filter operating in the L*u*v color space, an edge filter combined with a line extender, or a color filter operating in the LCH color space.

32. The computer system of claim 27, wherein the differentiable characteristics of each of said at least two filters is selected from the set comprising:

color characteristics, edge characteristics, texture characteristics, symmetry, convexity, lack of three dimensional volume, number and orientation of side edges, characteristics corner angles, frequency, and luminescence.

33. The computer system of claim 27, further comprising:

a digital image capture device that records said digital image for storage in said frame buffer as an image selected from the set comprising;

live images, a pre-recorded set of images, a series of still images or a digitized version of an original analog image sequence.

34. The computer system of claim 27, wherein said digitized image frames comprise a large number of frames of image data and said computer system is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of image data.

35. A computer system for rapidly identifying whether a digitized image frame may contain objects of interest:

a frame buffer that stores at least one frame of a digital image;

at least two separate filters operably connected to said frame buffer to generate a discrete output from each filter in response to each frame of said digital image in said frame buffer, wherein each of at least two of said filters screens for a differentiable characteristic associated with the objects of interest that is unique to that filter; and

means operably connected to said discrete outputs of said at least two separate filters for identifying a frame of said digital image as potentially having an object of interest present in response to said discrete outputs of said at least two separate filters.

36. The computer system of claim 35, further comprising:

a database that stores at least a portion of only said frames of said digital image that are identified by said means for identifying.

37. The computer system of claim 35, further comprising:

at least one camera that is in motion during the recording of said at least one digitized image frames; and

a database that stores at least a portion of only said frames of said digital image that are identified by said neural network along with a location data metric corresponding to a location of the camera when said digitized image frame was originally recorded.

38. The computer system of claim 37, wherein the location data metric stored in said database includes at least an orientation of the camera while recording, a focal length of the camera, and the location of the camera.

39. The computer system of claim 35, wherein each of said at least two filters is selected from the set of the following filters:

an edge filter, a color-pair filter, a color filter operating in the L*u*v color space, an edge filter combined with a line extender, or a color filter operating in the LCH color space.

40. The computer system of claim 35, wherein the differentiable characteristics of each of said at least two filters is selected from the set comprising:

color characteristics, edge characteristics, texture characteristics, symmetry, convexity, lack of three dimensional volume, number and orientation of side edges, characteristics corner angles, frequency, and luminescence.

41. The computer system of claim 35, further comprising:

a digital image capture device that records said digital image for storage in said frame buffer as an image selected from the set comprising;

live images, a pre-recorded set of images, a series of still images or a digitized version of an original analog image sequence.

42. The computer system of claim 35, wherein said digitized image frames comprise a large number of frames of image data and said computer system is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of image data.