Using pose data and positioning information to locate online photos of a user

• US 9,501,832 B1
• Filed: 12/28/2015
• Issued: 11/22/2016
• Est. Priority Date: 11/22/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Seitz, Steven Maxwell
• Primary Examiner(s): Ahmed, Samir
• Application Number:
  US14/979,823
• Time in Patent Office: 330 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2218/08 Feature extraction
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/16 Human faces, e.g. facial parts, sketches or expressions
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation

Abstract

The aspects described herein include receiving a request for available images depicting a user. One or more time and location indicators indicating one or more locations visited by the user are determined. Based on at least in part the one or more time and location indicators, a set of candidate images may be identified. The set of candidate images depict one or more locations at a time corresponding to at least one of the time indicators. Pose data related to the user may be obtained based on the location indicators. The pose data indicates a position and orientation of the user during a visit at a given location depicted in the set of candidate images. One or more images from the set of candidate images may be selected based on the pose data and the 3D reconstruction. The selected images include at least a partial view of the user.

Claims

1. A computer-implemented method, comprising:

receiving, by one or more computing devices, a request for available images depicting a specific subject;

determining, by the one or more computing devices, one or more time and location indicators indicating one or more locations associated with the user specific subject;

identifying, by the one or more computing devices, a set of candidate images based on the one or more time and location indicators, the set of candidate images depicting one or more locations at a time corresponding to at least one of the time indicators;

obtaining, by the one or more computing devices, pose data related to the specific subject based on the location indicators, the pose data indicating at least one of a position and orientation of the specific subject at a given location depicted in the set of candidate images;

determining, by the one or more computing devices, a 3D geometry of the given location based on the set of candidate images; and

selecting, by the one or more computing devices, one or more images from the set of candidate images based on the pose data and the 3D geometry, the selected one or more images including at least a partial view of the specific subject.

2. The computer-implemented method of claim 1, wherein the one or more location indicators include at least one of:

position information from a mobile computing device, coordinates associated a certain position at a particular location, and user-specified location.

3. The computer-implemented method of claim 1, wherein the specific subject is a user, and further comprising:

comparing the pose data for the specific subject with data representing a focal length of each image associated with the 3D geometry; and

determining whether a certain image of the one or more selected images for the particular location includes a view of the user'"'"'s face based at least in part on the comparison.

4. The computer-implemented method of claim 3, wherein position information associated with the certain image overlaps with position information associated with the pose data related to the specific subject.

5. The computer-implemented method of claim 1, further comprising determining whether the specific subject is visible in the set of candidate images by estimating a distance from a given image of the set of candidate images to the specific subject based on the representation 3D geometry.

6. The computer-implemented method of claim 1, wherein the specific subject is a user, and wherein selecting images from the set of candidate images includes:

receiving one or more facial images depicting facial features of the user;

comparing the one or more facial images to objects depicted in a given image from the set of candidate images; and

detecting whether a percentage of the user'"'"'s face is in the given image based on the comparison.

7. The computer-implemented method of claim 6, wherein the percentage of the user'"'"'s face in the given image meets a facial recognition threshold.

10. The computer-implemented method of claim 1, wherein the specific subject is a user, and further comprising:

comparing the pose data for the user with data representing a focal length of each image associated with the 3D geometry; and

determining whether a certain image of the one or more selected images for the particular location includes a view of the user'"'"'s face based at least in part on the comparison.

11. The computer-implemented method of claim 10, wherein position information associated with the certain image overlaps with position information associated with the pose data related to the specific subject.

8. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform a method, the method comprising:

receiving, by one or more processors, a request for available images depicting a specific subject;

determining, by the one or more computing devices, one or more time and location indicators indicating one or more locations associated with the specific subject;

identifying, by the one or more computing devices, a set of candidate images based on the one or more time and location indicators, the set of candidate images depicting one or more locations at a time corresponding to at least one of the time indicators;

obtaining, by the one or more computing devices, pose data related to the specific subject based on the location indicators, the pose data indicating at least one of a position and orientation of the specific subject at a given location depicted in the set of candidate images;

determining, by the one or more computing devices, a 3D geometry of the given location based on the set of candidate images; and

selecting, by the one or more computing devices, one or more images from the set of candidate images based on the pose data and the 3D geometry, the selected one or more images including at least a partial view of the specific subject.

9. The non-transitory computer readable medium of claim 8, wherein the one or more time and location indicators include at least one of:

position information from a mobile computing device, coordinates associated with a certain position at a location, and user-specified location.

12. The non-transitory computer readable medium of claim 8, wherein the specific subject is a user, and wherein selecting images from the set of candidate images includes:

receiving one or more facial images depicting facial features of the user;

comparing the one or more facial images to objects depicted in a given image from the set of candidate images; and

detecting whether a percentage of the user'"'"'s face is in the given image based on the comparison.

13. The non-transitory computer readable medium of claim 12, wherein the percentage of the user'"'"'s face in the given image meets a facial recognition threshold.

14. A system, comprising:

memory; and

one or more processors coupled to the memory, the one or more processors being configured to;

receive a request for available images depicting a specific subject;

determine one or more time and location indicators indicating one or more locations associated with the specific subject;

identify a set of candidate images based on the one or more time and location indicators, the set of candidate images depicting one or more locations at a time corresponding to at least one of the time indicators;

obtain pose data related to the specific subject based on the location indicators, the pose data indicating at least one of a position and orientation of the specific subject at a given location depicted in the set of candidate images;

determine a 3D geometry of the given location based on the set of candidate images; and

select one or more images from the set of candidate images based on the pose data and the 3D geometry, the selected one or more images including at least a partial view of the specific subject.

15. The system of claim 14, wherein the one or more time and location indicators include at least one of:

position information from a mobile computing device, coordinates associated with a certain position at a location and user-specified location.

16. The computer system of claim 14, wherein the specific subject is a user, and wherein the one or more processors are further configured to:

compare the pose data for the user with data representing a focal length of each image associated with the 3D geometry; and

determine whether a certain image of the one or more selected images for the particular location includes a view of the user'"'"'s face based at least in part on the comparison.

17. The computer system of claim 16, wherein position information associated with the certain image overlaps with position information associated with the pose data related to the user.

18. The system of claim 14, wherein the one or more processors are further configured to determine whether the specific subject is visible in the set of candidate images by estimating a distance from a given image of the set of candidate images to the specific subject based on the 3D geometry.

19. The system of claim 14, wherein the specific subject is a user, and wherein to select images from the set of candidate images the one or more processors a further configured to:

receive one or more facial images depicting facial features of the user;

compare the one or more facial images to objects depicted in a given image from the set of candidate images; and

detect whether a percentage of the user'"'"'s face is in the given image based on the comparison.

20. The system of claim 19, wherein the percentage of the user'"'"'s face in the given image meets a facial recognition threshold.