Method and apparatus for identifying objects depicted in a videostream

• US 7,092,548 B2
• Filed: 08/05/2003
• Issued: 08/15/2006
• Est. Priority Date: 10/23/1998
• Status: Expired due to Term

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Facet Technology Corp.
• Inventors: Retterath, James Eugene, Laumeyer, Robert Anthony
• Primary Examiner(s): Wu, Jingge
• Assistant Examiner(s): AZARIAN, SEYED H
• Application Number:
  US10/634,630
• Publication Number:
  US 20040062442A1
• Time in Patent Office: 1,106 Days
• Field of Search: 382103-107, 382/113, 382/122, 382/123, 382/154, 382/156, 382/162, 382/170, 382/178, 382/181, 382/189, 382/194, 382/199, 382232-254, 382/260, 382/274, 382285-295, 382/305, 382/319, 359/630, 701/209, 348/169, 240/907, 240/933
• US Class Current: 382/104
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/443 by matching or filtering
• CPC Class Codes: G06V 20/582 of traffic signs

Abstract

The present invention relates to an apparatus for rapidly analyzing frame(s) of digitized video data which may include objects of interest randomly distributed throughout the video data and wherein said objects are susceptible to detection, classification, and ultimately identification by filtering said video data for certain differentiable characteristics of said objects. The present invention may be practiced on pre-existing sequences of image data or may be integrated into an imaging device for real-time, dynamic, object identification, classification, logging/counting, cataloging, retention (with links to stored bitmaps of said object), retrieval, and the like. The present invention readily lends itself to the problem of automatic and semi-automatic cataloging of vast numbers of objects such as traffic control signs and utility poles disposed in myriad settings. When used in conjunction with navigational or positional inputs, such as GPS, an output from the inventative systems indicates the identity of each object, calculates object location, classifies each object by type, extracts legible text appearing on a surface of the object (if any), and stores a visual representation of the object in a form dictated by the end user/operator of the system. The output lends itself to examination and extraction of scene detail, which cannot practically be successfully accomplished with just human viewers operating video equipment, although human intervention can still be used to help judge and confirm a variety of classifications of certain instances and for types of identified objects.

Claims

1. A method of identifying whether a plurality of image frames may contain a common object of interest comprising the steps of:

receiving a plurality of image frames in an image processing apparatus wherein said plurality of image frames are generated by at least one imaging device;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter; and

using said image processing apparatus to identify image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest.

2. The method of claim 1 wherein image frames not having at least a potential region of interest are discarded from further processing.

3. The method of claim 1 wherein said common object of interest is a two-dimensional object.

4. The method of claim 1 wherein said at least one imaging device generates images in a spectral band other than the visible spectrum.

5. The method of claim 1 wherein said plurality of image frames are generated by at least two imaging devices.

6. The method of claim 5 wherein at least one of said imaging device generates images in a spectral band other than the visible spectrum.

7. The method of claim 3 wherein said at least two imaging devices have overlapping fields of view.

8. The method of claim 3 wherein said at least two imaging devices have different fields of view selected from the set consisting of:

a side view, a forward view, a downward view and a rearward view.

9. The method of claim 1 wherein said plurality of image frames are operably generated by a digital image capture device that is recording said digital image as images selected from the set consisting of:

live images, pre-recorded images, a series of still images or a digitized version of an original analog image sequence.

12. The method of claim 1 wherein said image portion having a region of interest is filtered after said identifying step.

13. The method of claim 1 further comprising the step of:

relating a plurality of regions of interest from said plurality of image frames in a three-dimensional representation based on a location of said region of interest within said image frame.

14. The method of claim 13 wherein said plurality of regions of interest are dynamically sized in said three-dimensional representation.

15. The method of claim 1 further comprising the step of:

relating each image portion with a location data metric associated with that image portion.

16. The method of claim 1 wherein the step of using said image processing apparatus to identify image portions comprises saving to a separate memory each of said separate image portions.

17. The method of claim 1 wherein the step of using said image processing apparatus to identify image portions comprises creating a record in a database of a pointer to a bitmap image representing each of said separate image portions.

18. The method of claim 1 wherein said image frames comprise a large number of frames of digitized image data and the method is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of digitized image data.

10. A method of identifying whether a plurality of image frames may contain a common object of interest comprising the steps of:

receiving a plurality of image frames in an image processing apparatus wherein said plurality of image frames are generated by at least one imaging device;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter;

using said image processing apparatus to identify image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest;

for each digitized image frame identified as having a region of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics.

11. A method of identifying whether a plurality of image frames may contain a common object of interest comprising the steps of:

receiving a plurality of image frames in an image processing apparatus wherein said plurality of image frames are generated by at least one imaging device;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter;

using said image processing apparatus to identify image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest;

for each digitized image frame identified as having a region of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics; and

utilizing morphology techniques to grow and erode said search space by adding or subtracting segments adjacent said search space until either said search space meets or fails to meet uniform criteria for a differentiable characteristic.

19. An image processing apparatus for determining whether a plurality of image frames may contain a common object of interest, said apparatus comprising:

a frame buffer that stores digitized pixels of said plurality of digitized image frames wherein said plurality of image frames are generated by at least one imaging device;

at least one filter operably connected to said frame buffer that generates a discrete output value in response to each pixel of each image frame, wherein each filter screens for a

differentiable characteristic associated with an object of interest that is unique to that filter; and

an image processor that identifies image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest.

20. The apparatus of claim 19 wherein image frames not having at least a potential region of interest are discarded from said frame buffer.

21. The apparatus of claim 19, wherein said common object of interest is a two-dimensional object.

22. The apparatus of claim 19 wherein said at least one imaging device generates images in a spectral band other than the visible spectrum.

23. The apparatus of claim 19 wherein said plurality of image frames are generated by at least two imaging devices.

24. The apparatus of claim 23 wherein at least one of said imaging device generates images in a spectral band other than the visible spectrum.

25. The apparatus of claim 23 wherein said at least two imaging devices have overlapping fields of view.

26. The apparatus of claim 23 wherein said at least two imaging devices have different fields of view selected from the set consisting of:

a side view, a forward view, a downward view and a rearward view.

27. The apparatus of claim 23 wherein said plurality of image frames are operably generated by a digital image capture device that is recording said digital image as images selected from the set consisting of:

live images, pre-recorded images, a series of still images or a digitized version of an original analog image sequence.

30. The apparatus of claim 19 further comprising at least a second filter operably connected to said image processor that filters said image portions which are identified as having a region of interest.

31. The apparatus of claim 19 wherein said image processor relates a plurality of regions of interest from said plurality of image frames in a three-dimensional representation based on a location of said region of interest within said image frame.

32. The apparatus of claim 31 wherein said plurality of regions of interest are dynamically sized in said three-dimensional representation.

33. The apparatus of claim 19 wherein said image processor includes a separate memory into which each of said separate image portions are stored once identified.

34. The apparatus of claim 19 wherein said image processor includes a database and creates a record in said database of a pointer to a bitmap image representing each of said separate image portions that are identified.

35. The apparatus of claim 19 wherein said image frames comprise a large number of frames of digitized image data and the apparatus is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of digitized image data.

28. An image processing apparatus for determining whether a plurality of image frames may contain a common object of interest, said apparatus comprising:

a frame buffer tat stores digitized pixels of said plurality of digitized image frames wherein said plurality of image frames are generated by at least one imaging device;

at least one filter operably connected to said frame buffer that generates a discrete output value in response to each pixel of each image frame, wherein each filter screens for a

differentiable characteristic associated with an object of interest that is unique to that filter; and

an image processor that identifies image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest

wherein the image processor further identifies segments of interest within each image frame identified as having a region of interest by segmenting said image frame into a plurality of non-overlapping image segments, searching through a search space of said segments to identify a segment that exhibits a differentiable characteristic as a search space, and expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics.

29. An image processing apparatus for determining whether a plurality of image frames may contain a common object of interest, said apparatus comprising:

a frame buffer tat stores digitized pixels of said plurality of digitized image frames wherein said plurality of image frames are generated by at least one imaging device;

at least one filter operably connected to said frame buffer that generates a discrete output value in response to each pixel of each image frame, wherein each filter screens for a

differentiable characteristic associated with an object of interest tat is unique to that filter; and

an image processor that identifies image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest wherein the image processor further identifies segments of interest within each image frame identified as having a region of interest by segmenting said image frame into a plurality of non-overlapping image segments, searching through a search space of said segments to identify a segment that exhibits a differentiable characteristic as a search space, and expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics and

wherein the image processor utilizes morphology techniques to grow and erode said search space by adding or subtracting segments adjacent said search space until either said search space meets or fails to meet uniform criteria for a differentiable characteristic.

36. A computer-readable storage media storing:

at least one computer program that operates to identify whether a plurality of image frames may contain a common object of interest by;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter; and

identifying image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest.

37. The storage media of claim 36 wherein image frames not having at least a potential region of interest are discarded from further processing.

38. The storage media of claim 36 wherein said common object of interest is a two-dimensional object.

39. The storage media of claim 36 wherein said plurality of images are generated by at least one imaging device that generates images in a spectral band other than the visible spectrum.

40. The storage media of claim 36 wherein said plurality of image frames are generated by at least two imaging devices.

41. The storage media of claim 40 wherein at least one of said imaging devices generates images in a spectral band other than the visible spectrum.

42. The storage media of claim 40 wherein said at least two imaging devices have overlapping fields of view.

43. The storage media of claim 40 wherein said at least two imaging devices have different fields of view selected from the set consisting of:

a side view, a forward view, a downward view and a rearward view.

44. The storage media of claim 40 wherein said plurality of image frames are operably generated by a digital image capture device that is recording said digital image as images selected from the set consisting of:

live images, pre-recorded images, a series of still images or a digitized version of an original analog image sequence.

48. The storage media of claim 36 wherein the computer program further operates by:

relating a plurality of regions of interest from said plurality of image frames in a three-dimensional representation based on a location of said region of interest within said image frame.

49. The storage media of claim 48 wherein said plurality of regions of interest are dynamically sized in said three-dimensional representation.

50. The storage media of claim 36 wherein the computer program further operates by

relating each image portion with a location data metric associated with that image portion.

51. The storage media of claim 36 wherein the computer program saves to a separate memory each of said separate image portions.

52. The storage media of claim 36 wherein the computer program creates a record in a database of a pointer to a bitmap image representing each of said separate image portions.

53. The storage media of claim 36 wherein said image frames comprise a large number of frames of digitized image data and the storage media is used as part of a graphic-based search engine to recognize a desired single object within said large number of frames of digitized image data.

45. A computer-readable storage media storing:

at least one computer program that operates to identify whether a plurality of image frames may contain a common object of interest by;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter; and

identifying image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest

wherein the computer program further operates by;

for each digitized image frame identified as having a region of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics.

46. A computer-readable storage media storing:

at least one computer program that operates to identify whether a plurality of image frames may contain a common object of interest by;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter; and

identifying image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest wherein the computer program further operates by;

for each digitized image frame identified as having a region of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics

wherein the expanding of said search space is accomplished by utilizing morphology techniques to grow and erode said search space by adding or subtracting segments adjacent said search space until either said search space meets or fails to meet uniform criteria for a differentiable characteristic.

47. A computer-readable storage media storing:

at least one computer program that operates to identify whether a plurality of image frames may contain a common object of interest by;

applying at least one filter to each of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable characteristic associated with an object of interest that is unique to that filter; and

identifying image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said plurality of image frames as potentially having a region of interest representing the common object of interest wherein the computer program further operates by;

for each digitized image frame identified as having a region of interest;

segmenting said frame into a plurality of non-overlapping image segments;

identifying a segment that exhibits a differentiable characteristic as a search space; and

expanding said search space to include segments adjacent to said segment that exhibits the differentiable characteristics

wherein said image portions having a region of interest are filtered after said identifying step.

54. A method of identifying whether a plurality of image frames may contain a common object of interest comprising the steps of:

receiving a plurality of image frames in an image processing apparatus wherein said plurality of image frames are generated by at least one imaging device;

separating each image frame into at least two channels of data;

applying at least one filter to each of said two channels of data of said plurality of image frames to generate a discrete output value, wherein each filter screens for a differentiable

characteristic associated with an object of interest that is unique to that filter; and

using said image processing apparatus to identify image portions from said plurality of image frames that exhibit said differentiable characteristic in response to a logical combination of said discrete output values for each of said at least one filter for said at least two channels of data as potentially having a region of interest representing the common object of interest.