Semantic image navigation experiences

• US 9,471,695 B1
• Filed: 12/02/2014
• Issued: 10/18/2016
• Est. Priority Date: 12/02/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Brualla, Ricardo Martin, Gallup, David Robert, Seitz, Steven Maxwell
• Primary Examiner(s): Wiener, Eric
• Application Number:
  US14/558,092
• Time in Patent Office: 686 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/26 specially adapted for navigation in a road network
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/434 using image data, e.g. images, photos, pictures taken by a user
• CPC Class Codes: G06F 16/4393 Multimedia presentations, e.g. slide shows, multimedia albums
• CPC Class Codes: G06F 16/44 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/444 Spatial browsing, e.g. 2D maps, 3D or virtual spaces
• CPC Class Codes: G06F 16/447 Temporal browsing, e.g. timeline
• CPC Class Codes: G06F 16/487 using geographical or spatial information, e.g. location
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/70 of video data
• CPC Class Codes: G06F 16/9537 Spatial or temporal dependent retrieval, e.g. spatiotemporal queries
• CPC Class Codes: G06F 16/954 Navigation, e.g. using categorised browsing
• CPC Class Codes: G06F 2216/07 Guided tours
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2200/32 involving image mosaicing
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G09B 29/008 : Touring maps or guides to public transport networks
• CPC Class Codes: Y10S 707/918 : Location

Abstract

Aspects of the disclosure relate to generating a sequence of images or other visual representations associated with an entity, otherwise known as a semantic image navigation experience. After an entity is selected, a set of sub-entities may be identified. Each sub-entity in the set has a containment relationship with the selected entity as well as at least one associated landmark and one associated pre-stored navigation experience. Then, a ranking order of the sub-entities in the set may be determined based on characteristics of each entity. Based on the determined ranking order, a subset of sub-entities may be selected. A semantic image navigation experience for the selected entity may then be generated using the pre-stored navigation experiences associated with the subset of sub-entities.

Claims

1. A method for generating semantic image navigation experiences, the method comprising:

selecting, by one or more computing devices, a first geographic entity of a plurality of distinct geographic entities;

identifying, by the one or more computing devices, a plurality of distinct geographic sub-entities, each distinct geographic sub-entity having a geographic containment relationship with the first geographic entity, the plurality of distinct geographic sub-entities including;

a first distinct geographic sub-entity that is both (i) visually represented by at least one landmark corresponding to the first distinct geographic sub-entity and (ii) associated with a pre-stored navigation experience of a plurality of pre-stored navigation experiences that is unique to the first distinct geographic sub-entity, and

a second distinct geographic sub-entity that is not at least one of (i) visually represented by at least one landmark corresponding to the second distinct geographic sub-entity and (ii) associated with a pre-stored navigation experience of the plurality of pre-stored navigation experiences that is unique to the second distinct geographic sub-entity,

wherein each pre-stored navigation experience in the plurality of pre-stored navigation experiences corresponds to a given geographic entity and comprises a sequence of images and transitions between the images that produces a tour of at least one landmark associated with the given geographic entity;

filtering, by the one or more computing devices, the plurality of distinct geographic sub-entities to remove at least one distinct geographic sub-entity that is not at least one of (i) visually represented by at least one landmark and (ii) associated with a pre-stored navigation experience of the plurality of pre-stored navigation experiences that is unique to the at least one distinct geographic sub-entity, the at least one distinct geographic sub-entity including the second geographic sub-entity;

determining, by the one or more computing devices, a ranking order of the filtered plurality of distinct geographic sub-entities based at least in part on one or more characteristics of each distinct geographic sub-entity in the filtered plurality of distinct geographic sub-entities;

selecting, by one or more computing devices, a subset of at least two distinct geographic sub-entities based on the ranking order; and

generating, by the one or more computing devices, a semantic image navigation experience for the first geographic entity based on at least the pre-stored navigation experiences associated with the subset of at least two distinct geographic sub-entities by;

automatically selecting a plurality of images from each pre-stored navigation experience of the subset of at least two distinct geographic sub-entities, and

including the plurality of selected images in the semantic image navigation experience as a sequence of images based on the ranking order.

2. The method of claim 1, further comprising:

receiving, by the one or more computing devices, a request from a client computing device; and

sending, in response to the request, by the one or more computing devices, the semantic image navigation experience for the first geographic entity to the client computing device.

3. The method of claim 1, wherein generating the semantic image navigation experience further comprises:

selecting, by the one or more computing devices, a set of images from a pre-stored navigation experience for the first geographic entity; and

inserting, by the one or more computing devices, the set of images from the pre-stored navigation experience for the first geographic entity at a beginning of the semantic image navigation experience for the first geographic entity.

4. The method of claim 1, wherein determining the ranking order is based on at least a landmark popularity metric.

5. The method of claim 1, wherein identifying the plurality of distinct geographic sub-entities further comprises:

clustering, by the one or more computing devices, a set of images of the first geographic entity into subsets of images based on at least common features found in the images in the set of images; and

defining, by the one or more computing devices, each subset of images as landmarks of a geographic sub-entity of the first geographic entity.

6. The method of claim 1, wherein generating the semantic image navigation experience further comprises including information about each distinct geographic sub-entity in the subset of at least two distinct geographic sub-entities.

7. The method of claim 1, further comprising:

clustering, by the one or more computing devices, at least two landmarks that visually represent the first geographic entity into a group;

defining, by the one or more computing devices, the group of the at least two landmarks as a third distinct geographic sub-entity of the plurality of distinct geographic sub-entities.

8. A system comprising one or more computing devices configured to:

select a first geographic entity of a plurality of distinct geographic entities;

identify a plurality of distinct geographic sub-entities, each distinct geographic sub-entity having a geographic containment relationship with the first geographic entity, the plurality of distinct geographic sub-entities including;

a first distinct geographic sub-entity that is both (i) visually represented by at least one landmark corresponding to the first distinct geographic sub-entity and (ii) associated with a pre-stored navigation experience of a plurality of pre-stored navigation experiences that is unique to the first distinct geographic sub-entity, and

a second distinct geographic sub-entity that is not at least one of (i) visually represented by at least one landmark corresponding to the second distinct geographic sub-entity and (ii) associated with a pre-stored navigation experience of the plurality of pre-stored navigation experiences that is unique to the second distinct geographic sub-entity,

wherein each pre-stored navigation experience in the plurality of pre-stored navigation experiences corresponds to a given geographic entity and comprises a sequence of images and transitions between the images that produces a tour of at least one landmark associated with the given geographic entity;

filter the plurality of distinct geographic sub-entities to remove at least one distinct geographic sub-entity that is not at least one of (i) visually represented by at least one landmark and (ii) associated with a pre-stored navigation experience of the plurality of pre-stored navigation experiences that is unique to the at least one distinct geographic sub-entity, the at least one distinct geographic sub-entity including the second geographic sub-entity;

determine a ranking order of the filtered plurality of distinct geographic sub-entities based at least in part on one or more characteristics of each distinct geographic sub-entity in the filtered plurality of distinct geographic sub-entities;

select a subset of at least two distinct geographic sub-entities based on the ranking order; and

generate a semantic image navigation experience for the first geographic entity based on at least the pre-stored navigation experiences associated with the subset of at least two distinct geographic sub-entities by;

automatically selecting a plurality of images from each pre-stored navigation experience of the subset of at least two distinct geographic sub-entities, and

including the plurality of selected images in the semantic image navigation experience as a sequence of images based on the ranking order.

9. The system of claim 8, wherein the one or more computing devices are further configured to:

receive a request from a client computing device; and

send, in response to the request, the semantic image navigation experience for the first geographic entity to the client computing device.

10. The system of claim 8, wherein the one or more computing devices are further configured to generate the semantic image navigation experience by:

selecting a set of images from a pre-stored navigation experience for the first geographic entity; and

inserting the set of images from the pre-stored navigation experience for the first geographic entity at a beginning of the semantic image navigation experience for the first geographic entity.

11. The system of claim 8, wherein the ranking order is determined based on at least a landmark popularity metric.

12. The system of claim 8, wherein the one or more computing devices are further configured to generate the semantic image navigation experience by including information about each distinct geographic sub-entity in the subset of at least two distinct geographic sub-entities.

13. The system of claim 8, wherein the one or more computing devices are further configured to identify the plurality of distinct geographic sub-entities by:

clustering a set of images of the first geographic entity into subsets of images based on at least common features found in the images in the set of images; and

defining each subset of images as landmarks of a geographic sub-entity of the first geographic entity.

14. The system of claim 8, wherein the one or more computing devices are further configured to:

cluster at least two landmarks that visually represent the first geographic entity into a group;

define the group of the at least two landmarks as a third distinct geographic sub-entity of the plurality of distinct geographic sub-entities.

15. A non-transitory, tangible machine readable medium on which instructions are stored, the instructions, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

selecting a first geographic entity of a plurality of distinct geographic entities;

identifying a plurality of distinct geographic sub-entities, each distinct geographic sub-entity having a geographic containment relationship with the first geographic entity, the plurality of distinct geographic sub-entities including;

a first distinct geographic sub-entity that is both (i) visually represented by at least one landmark corresponding to the first distinct geographic sub-entity and (ii) associated with a pre-stored navigation experience of a plurality of pre-stored navigation experiences that is unique to the first distinct geographic sub-entity, and

a second distinct geographic sub-entity that is not at least one of (i) visually represented by at least one landmark corresponding to the second distinct geographic sub-entity and (ii) associated with a pre-stored navigation experience of the plurality of pre-stored navigation experiences that is unique to the second distinct geographic sub-entity,

wherein each pre-stored navigation experience in the plurality of pre-stored navigation experiences corresponds to a given geographic entity and comprises a sequence of images and transitions between the images that produces a tour of at least one landmark associated with the given geographic entity;

filtering the plurality of distinct geographic sub-entities to remove at least one distinct geographic sub-entity that is not at least one of (i) visually represented by at least one landmark and (ii) associated with a pre-stored navigation experience of the plurality of pre-stored navigation experiences that is unique to the at least one distinct geographic sub-entity, the at least one distinct geographic sub-entity including the second geographic sub-entity;

determining a ranking order of the filtered plurality of distinct geographic sub-entities based at least in part on one or more characteristics of each distinct geographic sub-entity in the filtered plurality of distinct geographic sub-entities;

selecting a subset of at least two distinct geographic sub-entities based on the ranking order; and

generating a semantic image navigation experience for the first geographic entity based on at least the pre-stored navigation experiences associated with the subset of at least two distinct geographic sub-entities by;

automatically selecting a plurality of images from each pre-stored navigation experience of the subset of at least two distinct geographic sub-entities, and

including the plurality of selected images in the semantic image navigation experience as a sequence of images based on the ranking order.

16. The medium of claim 15, wherein the method further comprises:

receiving a request from a client computing device; and

sending, in response to the request, the semantic image navigation experience for the first geographic entity to the client computing device.

17. The medium of claim 15, wherein the method further comprises generating the semantic image navigation experience by:

selecting a set of images from a pre-stored navigation experience for the first geographic entity; and

inserting the set of images from the pre-stored navigation experience for the first geographic entity at a beginning of the semantic image navigation experience for the first geographic entity.

18. The medium of claim 15, wherein the method further comprises generating the semantic image navigation experience by including information about each distinct geographic sub-entity in the subset of at least two distinct geographic sub-entities.

19. The medium of claim 15, wherein the method further comprises identifying the plurality of distinct geographic sub-entities by:

clustering a set of images of the first geographic entity into subsets of images based on at least common features found in the images in the set of images; and

defining each subset of images as landmarks of a geographic sub-entity of the geographic first entity.

20. The medium of claim 15, wherein the method further comprises:

clustering at least two landmarks that visually represent the first geographic entity into a group;

defining the group of the at least two landmarks as a third distinct geographic sub-entity of the plurality of distinct geographic sub-entities.