Location based image classification with map segmentation

• US 7,663,671 B2
• Filed: 11/22/2005
• Issued: 02/16/2010
• Est. Priority Date: 11/22/2005
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Eastman Kodak Company
• Inventors: Loui, Alexander C., Gallagher, Andrew C., Kraus, Bryan D.
• Primary Examiner(s): Tran; Nhan T
• Assistant Examiner(s): Berhan; Ahmed A
• Application Number:
  US11/284,927
• Publication Number:
  US 20070115373A1
• Time in Patent Office: 1,547 Days
• Field of Search: 348/231.3, 348/158, 348/46, 348/231.99, 348/231.2, 348/231.5, 348/231.9, 348/333.05, 707/100, 386/46
• US Class Current: 348/231.2
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06V 20/00 Scenes; Scene-specific elements control of digital cameras H04N23/60

Abstract

In methods and systems for classifying capture records, such as images. A collection of capture records is provided. Each capture record has metadata defining a map location. This metadata can be earlier determined from a stream of data transmissions, even if there are gaps in transmission. The provided capture records are clustered into groups based on capture locations. A map, inclusive of the capture locations, is segmented into a plurality of regions based on relative positions of the capture locations associated with each group. The regions are associated with the capture records of respective groups.

Claims

1. A computerized method for classifying capture records recorded on a capture device, said method comprising the steps of:

providing a collection of capture records to a microprocessor-based unit, each said capture record having metadata defining a capture location;

clustering said capture records into a plurality of groups based on said capture locations;

defining each of said plurality of groups of capture records within an event threshold;

providing a map inclusive of said capture locations;

segmenting the map inclusive of said capture locations, into a plurality of regions based on relative positions of said capture locations associated with each group; and

identifying map locations beyond the event threshold as region boundaries separating the regions and associating the capture records with said groups based upon the region boundaries.

2. The method of claim 1 wherein said segmenting further comprises:

determining distances between said capture locations; and

segmenting said map into different ranges of geographical area based on said determined distances.

3. The method of claim 1 wherein said segmenting further comprises:

determining a core location of each of said groups, said core locations each being a function of the capture locations of the respective said group; and

segmenting said map based on relative separations of said capture locations from each of said core locations.

4. The method of claim 3 wherein said core locations are each one of:

one or more of said capture locations of the respective said group, a combination of one or more of said capture locations of the respective said group, and a map location defined by the centroid of the respective said group.

5. The method of claim 3 wherein said capture locations in each of said regions are closer to the core location of the respective said region than to any other of said core locations.

6. The method of claim 1 wherein said clustering is also based upon a predetermined geographic feature of said map.

7. The method of claim 6 wherein said geographic feature is selected from the group consisting of:

geographic areas, political subdivisions, and terrain types.

8. The method of claim 6 wherein said geographic feature defines a mapping of said capture locations into different semantic categories.

9. The method of claim 8 wherein said geographic feature defines a binary mapping of said capture locations into land and water.

10. The method of claim 1 wherein said metadata of each of said capture records defines a probability distribution and said clustering is in accordance with said probability distributions.

11. The method of claim 1 wherein said clustering places each of said capture records exclusively in one of said groups.

12. The method of claim 1 wherein said providing further comprises:

supplying a plurality of additional capture records lacking said metadata;

estimating capture locations of each of said additional capture records from said metadata of one or more of said capture records of said collection of capture records to provide estimated capture locations; and

assigning respective said estimated capture locations to each of said additional capture records.

13. The method of claim 1 wherein said providing further comprises:

receiving, with a camera, a stream of periodic data transmissions, each said data transmission defining a map location;

capturing a plurality of located capture records captured during said periodic data transmissions

associating with each of said located capture records the map location defined by the concurrent said data transmission to provide respective said metadata defining a capture location.

14. The method of claim 13 wherein said geopositioning data further comprise one or more of:

locally and remotely transmitted geographic coordinates, identifications of cell sites, wired and wireless network addresses, and transmitted and user input place names.

15. The method of claim 1 wherein said providing further comprises:

receiving, with a camera, a stream of periodic data transmissions interrupted by one or more gaps in transmission , each said data transmission defining a map location;

capturing a sequence of capture records with the camera, said sequence including one or more located capture records captured during said periodic data transmissions and one or more non-located capture records captured during said gaps in transmission;

associating with each of said located capture records the map location defined by the concurrent said data transmission to provide respective said metadata defining a capture location;

associating with each of said non-located capture records a nominal map location within a geographic area delimited by the map locations of the data transmissions bounding the gap in transmission concurrent with the respective said non-located capture record to provide respective said metadata defining a capture location.

16. The method of claim 15 further comprising:

determining temporal separations between each of said non-located capture records and each of said data transmissions bounding a respective said gap in transmission; and

interpolating each of said nominal map locations between the capture locations of ones of said located capture records concurrent with said data transmissions bounding the respective said gap in transmission, said interpolating being proportional to the respective said temporal separation.

17. The method of claim 1 wherein each of said capture records has associated supplemental metadata additional to said metadata defining a capture location and said clustering is also based on said supplemental metadata.

18. The method of claim 17 wherein said supplemental metadata is data-time information.

19. The method of claim 18 further comprising scaling said supplemental metadata prior to said clustering.

20. The method of claim 1 wherein said capture records each include one or more of:

still images, video sequences, audio sequences, and combinations thereof.

21. The method of claim 1 further comprising associating a common annotation with each of said images of one or more of said groups.

22. The method of claim 21 wherein said annotation is a place name on said map.

23. A computer program product for capture record classification, the computer program product comprising computer readable storage medium having a computer program stored thereon for performing the steps of claim 1.

38. The method of claim 1 further comprising determining a geographic feature of interest, and said clustering is also based upon relative distances from said geographic feature.

39. The method of claim 1 wherein said geopositioning data further comprise one or more of:

locally and remotely transmitted geographic coordinates, identifications of cell sites, wired and wireless network addresses, and transmitted and user input place names.

24. A computerized method for classifying capture records, said method comprising the steps of:

providing a plurality of capture records recorded on a capture device to a microprocessor-based unit, each said capture record having metadata defining a capture location;

clustering said capture records into a two or more groups based on said capture locations;

determining distances between said capture locations;

defining each of said two or more groups of capture records within an event threshold;

providing a map inclusive of said capture locations;

segmenting the map inclusive of said capture locations, into a plurality of regions based on said determined distances; and

identifying map locations beyond the event threshold as region boundaries separating the regions into different ranges of geographic area and associating said regions with the capture records of respective with said groups based upon the region boundaries so that groups of capture records are encompassed within different ranges of geographic area.

25. The method of claim 24 wherein said clustering places each of said capture records exclusively in one of said groups.

26. The method of claim 25 wherein said regions each include a cluster center of a respective one of said groups and said regions are everywhere closer to the respective said cluster center than to the other said cluster centers.

27. A computerized method for classifying capture records, said method comprising the steps of:

providing a collection of capture records to a microprocessor-based unit, each said capture record having metadata defining a capture location;

clustering said capture records into a plurality of groups based on said capture locations;

determining a core location of each of said groups, said core locations each being a function of the capture locations of the respective said group;

defining each of said groups of capture records within an event threshold;

segmenting a map inclusive of said capture locations, into a plurality of regions based on relative separations of said capture locations from each of said core locations; and

identifying map locations beyond the event threshold as region boundaries separating the regions into different ranges of geographic area and associating said regions with the capture records of respective said groups based upon the region boundaries so that groups of capture records are encompassed within different ranges of geographic area.

28. A classification method for use with a camera, said method comprising the steps of:

receiving, with the camera, a stream of data transmissions, each said data transmission defining a map location;

capturing a sequence of capture records with the camera, said sequence including one or more located capture records captured concurrent with said data transmissions;

assigning to each of said located capture records the map location of a concurrent one of said data transmissions to provide respective capture locations;

clustering all of said capture records into groups based upon said capture locations;

segmenting a map inclusive of said capture locations, into a plurality of regions based on relative positions of said capture locations associated with each group; and

identifying map locations beyond the map location of said concurrent one of said data transmissions as region boundaries separating each region into different ranges of area and separating said groups based upon the region boundaries so that the groups of capture records are assigned locations within different ranges of area.

29. The method of claim 28 wherein said stream is periodic and is interrupted by one or more gaps in transmission;

and said method further comprises;

capturing one or more non-located capture records during said gaps in transmission;

assigning to each of said non-located capture records a nominal map location within a geographic area of said map delimited by the capture locations of the data transmissions bounding the gap in transmission concurrent with the respective said non-located capture record;

treating said nominal map locations as capture locations during said clustering.

30. The method of claim 29 wherein said providing further comprises ascertaining date-times of capture of said located capture records and date-times of receipt of said data transmissions, and wherein said assigning steps use said date-times.

31. The method of claim 30 further comprising the steps of:

determining temporal separations between each of said non-located capture records and each of said data transmissions bounding a respective said gap in transmission; and

interpolating each of said nominal map locations between the capture locations of respective said located capture records corresponding to said data transmissions bounding a respective said gap in transmission, said interpolating being proportional to respective said temporal separations.

32. The method of claim 28 wherein said capture records include at least one of:

still images and video sequences.

33. The method of claim 28 further comprising the steps of:

receiving a plurality of external capture records having preassigned map locations; and

treating said preassigned map locations as capture locations during said clustering.

34. The method of claim 33 further comprising:

prior to said receiving, loading said external capture records into storage remote from said camera; and

assigning a default map location defined by said loading, to each of said external capture records to provide said preassigned locations.

35. The method of claim 28 wherein each of said capture records has associated supplemental metadata additional to said geopositioning data and said clustering is also based upon said supplemental metadata.

36. The method of claim 35 wherein said supplemental metadata is data-time information.

37. The method of claim 35 further comprising scaling said supplemental metadata prior to said clustering.

40. A capture system comprising:

a classification engine clustering all capture records into groups based upon a capture location associated with each of said capture records;

defining each of said groups of capture records within an event threshold;

segmenting a map inclusive of said capture locations, into a plurality of regions based on relative positions of said capture locations associated with each group;

identifying map locations beyond the event threshold as region boundaries to define the regions as different ranges of area and associating said regions with the capture records of respective said groups.

41. The system of claim 40 wherein said segmenting further comprises:

determining distances between said capture locations; and

partitioning said map based on said determined distances.

42. The system of claim 40 wherein said segmenting further comprises:

determining a core location of each of said groups, said core locations each being a function of the capture locations of the respective said group; and

partitioning said map based on relative separations of said capture locations from each of said core locations.

43. The system of claim 42 wherein said core locations are each one of:

one or more of said capture locations of the respective said group, a combination of one or more of said capture locations of the respective said group, and a map location defined by the centroid of the respective said group.

44. The method of claim 42 wherein said capture locations in each of said regions are closer to the core location of the respective said region than to any other of said core locations.

45. The method of claim 40 wherein said clustering is also based upon a predetermined geographic feature of said map.

46. The system of claim 40 further comprising a capture device including:

a geolocation information receiver operable to receive a stream of data transmissions defining locations;

a capture unit joined to said geopositioning system, said capture unit being operable to capture a sequence of capture records;

a memory storing said capture records;

a controller operatively connected to said memory and said receiver, said controller associating concurrent ones of said data transmissions and said capture records.

47. The system of claim 46 further comprising a location determiner capable of determining respective said map locations from said data transmissions, said location determiner being physically associated with one of said classification engine and said capture device and physically remote from the other of said classification engine and said capture device.

48. The system of claim 40 wherein said associated data transmissions each define a capture location;

said stream of data transmissions is interrupted by gaps;

said camera includes an estimator estimating nominal map locations of said capture records concurrent with said gaps, and said classification engine treats said nominal map locations as capture locations.

49. A classification method for use with a camera, said method comprising the steps of:

receiving, with the camera, a stream of periodic data transmissions interrupted by one or more gaps in transmission, each said data transmission defining a map location;

capturing a sequence of capture records with the camera, said sequence including one or more located capture records captured during said periodic data transmissions and one or more non-located capture records captured during said gaps in transmission;

assigning to each of said located capture records the map location of a concurrent one of said data transmissions to provide capture locations;

assigning to each of said non-located capture records a nominal location within a geographic area of said map delimited by the capture locations of the data transmissions bounding the concurrent said gap in transmission;

clustering all of said capture records into groups based upon said assigned locations;

treating said nominal locations as capture locations during said clustering; and

identifying map locations beyond the map location of said concurrent one of said data transmissions as region boundaries separating the regions into different ranges of area and separating said groups based upon the region boundaries so that the groups of capture records are assigned locations within different ranges of area.