Navigation with contextual color, texture, and structure cues

• US 8,046,167 B2
• Filed: 05/08/2008
• Issued: 10/25/2011
• Est. Priority Date: 05/08/2008
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mishra, Pragyana K, Kolluru, Nagendra V
• Primary Examiner(s): ARTHUR JEANGLAUD, GERTRUDE
• Application Number:
  US12/117,734
• Publication Number:
  US 20090281728A1
• Time in Patent Office: 1,265 Days
• Field of Search: 701/201, 701/206, 701/208, 701210-212, 340/995.2, 340/995.14, 340/995.23, 340/995.24, 340/995.27
• US Class Current: 701/210
• CPC Class Codes: G01C 21/3644 Landmark guidance, e.g. using POIs or conspicuous other objects

Abstract

A method and a processing device may provide navigational information including non-photographic cues, such as, contextual color, texture, and structure cues, corresponding to physical entities visible to a party traveling along a determined route. In one embodiment, the non-photographic cues may be previously stored in a database. In another embodiment, non-photographic cues may be extracted from at least one provided image of a geographic area. In a third embodiment, a sequence of non-photographic cues may be automatically extracted from at least one image of a geographic area and, if the sequence of non-photographic cues was not previously saved, the sequence of non-photographic cues may be saved, along with related information, for later use when the sequence of non-photographic cues corresponds to a desired location. The provided navigational information may be presented in a number of ways including a simplified map, textual instructions, or generated speech.

Claims

1. A machine-implemented method for providing navigational information based, at least partly, on non-photographic cues, the machine-implemented method comprising:

automatically extracting a sequence of the non-photographic cues from at least one image of a geographic area, the automatically extracting comprising;

segmenting physical entities appearing in at least one image of a geographic area based on at least one three-dimensional model, including a representation of the geographic area, to produce segmentation information,

deriving color information with respect to the physical entities appearing in the at least one image based on the produced segmentation information, and

producing and storing color cues based on the color information, the extracted sequence of non-photographic queues including the color cues;

providing navigational information for traveling along a route from an origin to a destination, the navigational information including information based, at least partly, on the extracted sequence of non-photographic cues.

2. The machine-implemented method of claim 1, comprising:

determining which of the physical entities are ones of the physical entities that are observable by a party traveling along the route, and

deriving a second sequence of non-photographic cues, with respect to the ones of the physical entities, the second sequence of non-photographic cues including at least some of the extracted sequence of non-photographic cues.

3. The machine-implemented method of claim 2, wherein:

the extracted non-photographic cues further include cues with respect to texture and structure of the physical entities within the geographic area, and

the deriving of the second sequence of non-photographic cues, with respect to the ones of the physical entities, include the cues regarding the texture and the structure of the physical entities.

4. The machine-implemented method of claim 1, wherein the providing of navigational information for traveling along a route from an origin to a destination, further comprises:

providing the navigational information as text or generated speech.

5. The machine-implemented method of claim 1, wherein the providing of navigational instructions for traveling along a route from an origin to a destination, further comprises:

providing a map having the sequence of non-photographic cues encoded into ones of the physical entities represented on the map, the non-photographic cues including cues regarding a plurality of visual attributes of the physical entities.

6. The machine-implemented method of claim 5, wherein:

the map comprises at least one drawing including streets along the route and the ones of the physical entities, and

the cues regarding the plurality of visual attributes of the physical entities include information with respect to color, texture, and structure.

7. The machine-implemented method of claim 2, wherein:

the provided navigational instructions include driving directions, and

the second sequence of non-photographic cues include color, texture, and structure cues corresponding to the driving directions.

8. A processing device comprising:

at least one processor; and

a memory connected to the at least one processor, the memory comprising;

instructions for automatically extracting non-photographic cues from at least one image of a geographic area, the extracted non-photographic cues being representative of a plurality of visual attributes of physical entities, the instructions for automatically extracting non-photographic cues from at least one image further comprising;

instructions for segmenting the physical entities appearing in the at least one image based on at least one three-dimensional model, including a representation of the geographic area, to produce segmentation information, and

instructions for deriving color information, or texture information, or both the color information and the texture information based, at least partly, on the segmentation information; and

instructions for providing navigational information based, at least partly, on the extracted non-photographic cues.

9. The processing device of claim 8, wherein the instructions for automatically extracting non-photographic cues from at least one image of a geographic area further comprise:

instructions for deriving the color information with respect to the physical entities appearing in the at least one image based on the produced segmentation information and the at least one image.

10. The processing device of claim 9, wherein the instructions for automatically extracting non-photographic cues from at least one image of a geographic area further comprise:

instructions for deriving the texture information with respect to the physical entities appearing in the at least one image based, at least partly, on the segmentation information and the at least one image.

11. The processing device of claim 8, wherein:

the instructions for automatically extracting non-photographic cues from at least one image of a geographic area further comprise;

instructions for deriving the color information with respect to the physical entities appearing in the at least one image based, at least partly, on the segmentation information and the at least one image,

instruction for deriving structure information based, at least partly, on the at least one three dimensional model of the geographic area, and

instructions for producing color and structure cues based, at least partly, on the segmentation information, the color information, and the structure information, and the instructions for providing navigational information based, at least partly, on the extracted cues further comprise;

instructions for using the produced color and structure cues to provide information about a location and to provide distance information.

12. The processing device of claim 8, wherein:

the instructions for automatically extracting non-photographic cues from at least one image of a geographic area further comprise;

instructions for deriving the color information with respect to the physical entities appearing in the at least one image based, at least partly, on the segmentation information and the at least one image,

instruction for deriving structure information based, at least partly, on the at least one three dimensional model of the geographic area,

instructions for deriving the texture information based, at least partly, on the at least one image and the segmentation information, and

instructions for producing color, structure and texture cues based, at least partly, on the segmentation information, the color information, the structure information, and the texture information;

the memory further comprising;

instructions for saving information with respect to a sequence of the color, structure and texture cues for a location, and

instructions for using the saved information at a later time, with respect to a sequence of the color, structure and texture cues for the location.

13. The processing device of claim 8, wherein:

the instructions for automatically extracting non-photographic cues from at least one image of a geographic area further comprise;

instructions for deriving the color information with respect to the physical entities appearing in the at least one image based, at least partly, on the segmentation information and the at least one image,

instruction for deriving structure information based, at least partly, on the at least one three dimensional model of the geographic area,

instructions for deriving the texture information based, at least partly, on the at least one image and the segmentation information, and

instructions for producing color, structure and texture cues based, at least partly, on the segmentation information, the color information, and the structure information, and the texture information, and

the instructions for providing navigational information further comprise;

instructions for presenting a simplified drawing as a map, the map having the color, structure and texture cues encoded therein.

14. A tangible machine-readable medium having instructions recorded thereon for at least one processor, the instructions comprising:

instructions for extracting color and texture cues from a plurality of images of a geographic area based, at least partly, on projecting a mesh from at least one three-dimensional model onto the plurality of images, the instructions for extracting color and texture cues from a plurality of images of a geographic area further comprising;

instructions for projecting meshes from three-dimensional models onto physical entities appearing in the plurality of images using calibration parameters of the plurality of images to produce segmented image regions,

instructions for deriving color information with respect to physical entities appearing in the plurality of images based, at least partly, on the produced segmented image regions, and

instructions for deriving texture information with respect to the physical entities appearing in the plurality of images based, at least partly, on the produced segmented image regions;

instructions for forming a sequence of visual cues with respect to ones of the physical entities along a determined route based, at least partly, on the extracted color and texture cues; and

instructions for providing driving directions to a destination, the driving directions including a reference to the sequence of visual cues.

15. The tangible machine-readable medium of claim 14, wherein the instructions for providing driving directions to the destination further comprise:

instructions for providing a map having the sequence of visual cues encoded therein.

16. The tangible machine-readable medium of claim 14, wherein the instructions for providing driving directions to a destination further comprise:

instructions for producing a map having the sequence of visual cues encoded therein, and

instructions for adding pushpins or hotspots to the map.

17. The tangible machine-readable medium of claim 14 wherein the instructions for extracting color and texture cues for a plurality of images of a geographic area further comprise:

instructions for forming histograms of image pixels over image areas corresponding to building facades, and

instructions for determining a dominant color of the histograms of image pixels based on a median value of hues within a dominant peak of the histogram.

18. The tangible machine-readable medium of claim 14, wherein the instructions for extracting color and texture cues for a plurality of images of a geographic area further comprise:

instructions for determining texture descriptors based on a gradient of intensities of image pixels within the segmented regions, and

instructions for determining textures by comparing the texture descriptors to a predefined set of descriptors.

19. The tangible machine-readable medium of claim 14, wherein:

the instructions for extracting color and texture cues from a plurality of images of a geographic area further comprise instructions for extracting structure cues from a plurality of images of a geographic area, and

the instructions recorded on the tangible medium further comprise instructions for providing navigational information based, at least partly, on the extracted color cues, the extracted texture cues, and the extracted structure cues from the plurality of images of the geographic area.

20. The tangible machine readable medium of claim 14, wherein the instructions further comprise:

instructions for producing the color and texture cues based on the derived color information and the derived texture information.