Optimization of label placements in street level images

• US 9,583,074 B2
• Filed: 12/06/2013
• Issued: 02/28/2017
• Est. Priority Date: 07/20/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Chau, Stephen
• Primary Examiner(s): McDowell, Jr., Maurice L
• Application Number:
  US14/099,338
• Publication Number:
  US 20150022555A1
• Time in Patent Office: 1,180 Days
• Field of Search: 345/419, 345/632, 715/815
• US Class Current: 1/1
• CPC Class Codes: G01C 21/3673 Labelling using text of road map data items, e.g. road names, POI names
• CPC Class Codes: G09G 2360/10 Display system comprising arrangements, such as a coprocessor, specific for motion video images
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 5/377 for mixing or overlaying two or more graphic patterns G09G5/02, G09G5/397 take precedence

Abstract

A system, computer-implemented method and computer-readable medium for labeling an image. A two-dimensional street-level image and a three-dimensional model representing at least a portion of the image'"'"'s content are received. Data representing a label to annotate at least a portion of a building in the two dimensional image is also received. A determination is made using the three dimensional mode as to where position the label for presentation within the two-dimensional model such that the label does not obscure presentation of the building in the two-dimensional image.

Claims

1. A computer-implemented method for labeling an image, comprising:

receiving, with one or more computing devices, a two-dimensional, street-level image and a three-dimensional model representing at least a portion of the image'"'"'s content;

identifying, with the one or more computing devices, one or more objects in the two-dimensional, street level image;

determining, with the one or more computing devices, object data for each of the one or more identified objects based at least in part on the three-dimensional model, wherein the object data includes an object type chosen from a set of three or more predetermined types, an object location, and an area occupied by the object;

receiving, with the one or more computing devices, label data representing a label to annotate at least a portion of a given one of the one or more objects in the two-dimensional, street-level image; and

using the determined object data to determine, with the one or more computing devices, a label position for presentation of the label within the two-dimensional, street-level image, such that the label does not obscure presentation of the given object in the two-dimensional, street-level image, wherein the object type is used to determine whether the label can be associated with the given object without obscuring the presentation.

2. The computer-implemented method of claim 1, further comprising:

determining, using the three-dimensional model, an area of the two-dimensional, street-level image representing a road or sky,

wherein the determined position for presentation of the label is within the determined area representing the road or sky.

3. The computer-implemented method of claim 1, further comprising:

sending the determined label position to a client device, wherein the client device presents the two-dimensional, street-level image and the label within the two-dimensional image at the determined label position.

4. The computer-implemented method of claim 1, further comprising:

generating a pointer for presentation within the two-dimensional, street-level image, the pointer illustrating that the label annotates the given object in the two-dimensional, street-level image.

5. The computer-implemented method of claim 1, wherein the label includes an input module.

6. The method of claim 1, wherein the object type is used to determine whether the given object requires the label and whether the given object can include the label without interfering with a viewing experience of a user.

7. The computer-implemented method of claim 1, further comprising:

determining, using the three-dimensional model, an area of the two-dimensional, street-level image representing a sky by identifying portions of the three-dimensional model with an infinite “

z”

coordinate, wherein the determined label position is within the determined area representing the sky.

8. The computer-implemented method of claim 1, further comprising:

determining, using the three-dimensional model, an area of the two-dimensional, street-level image representing a road by identifying portions of the three-dimensional model below a predetermined elevation, wherein the determined label position is within the determined area representing the road.

9. The computer-implemented method of claim 8, wherein the object type of at least one object in the two-dimensional, street level image is determined to be a building by detecting one or more vertical or substantially vertical facades adjacent to the determined area representing the road.

10. The computer-implemented method of claim 1, wherein the set of three or more predetermined types includes “

building”

, “

bridge”

, “

street sign”

, “

road”

, and “

sky”

.

11. A system for labeling an image, comprising:

one or more memories storing a two-dimensional, street-level image and a three-dimensional model representing at least a portion of the image'"'"'s content;

one or more processors in communication with the one or more memories and configured to;

identify one or more objects in the two-dimensional, street-level image;

determine object data for each of the one or more identified objects based at least in part on the three-dimensional model, wherein the object data includes an object type chosen from a set of three or more predetermined types, an object location, and an area occupied by the object;

receive label data representing a label to annotate at least a portion of a given one of the identified one or more objects in the two-dimensional, street-level image; and

using the determined object data to determine a label position for presentation of the label within the two-dimensional, street-level image, such that the label does not obscure presentation of the given object in the two-dimensional, street-level image, wherein the object type is used to determine whether the label can be associated with the given object without obscuring the presentation.

12. The system of claim 11, wherein the one or more processors are further configured to provide for display the two-dimensional image and the label within the two-dimensional, street-level image at the determined position.

13. The system of claim 11, further comprising a communication interface configured to send the determined position to a client device, wherein the client device presents the two-dimensional, street-level image and the label within the two-dimensional image at the determined position.

14. The system of claim 11, wherein the one or more processors are further configured to generate a pointer for presentation within the two-dimensional, street-level image, the pointer illustrating that the label annotates the given object in the two-dimensional, street-level image.

15. The system of claim 11, wherein the one or more processors are further configured to:

determine, using the three-dimensional model, an area of the two-dimensional, street-level image representing a road by identifying portions of the three-dimensional model below a predetermined elevation, wherein the determined label position is within the determined area representing the road.

16. The system of claim 15, wherein the object type of at least one object in the two-dimensional, street level image is determined to be a building by detecting one or more vertical or substantially vertical facades adjacent to the determined area representing the road.

17. The system of claim 11, wherein the set of three or more predetermined types includes “

building”

, “

bridge”

, “

street sign”

, “

road”

, and “

sky”

.

18. An article of manufacture including a non-transitory computer-readable medium having instructions retrieved thereon that, when executed by one or more processors, cause the one or more processors to perform operations for labeling an image, comprising:

receiving a two-dimensional, street-level image and a three-dimensional model representing at least a portion of the image'"'"'s content;

identifying one or more objects in the two-dimensional, street-level image;

determining object data for each of the one or more identified objects based at least in part on the three-dimensional model, wherein the object data includes an object type chosen from a set of three or more predetermined types, an object location, and an area occupied by the object;

receiving label data representing a label to annotate at least a portion of a given one of the one or more objects in the two-dimensional, street-level image; and

using the determined object data to determine a label position for presentation of the label within the two-dimensional, street-level image, such that the label does not obscure presentation of the given object in the two-dimensional, street-level image, wherein the object type is used to determine whether the label can be associated with the given object without obscuring the presentation.

19. The non-transitory computer-readable medium of claim 18, wherein the instructions that cause the one or more processors to perform operations for labeling the image, further comprise instructions to perform operations comprising:

sending the determined label position to a client device, wherein the client device presents the two-dimensional, street-level image and the label within the two-dimensional, street-level image at the determined label position.

20. The non-transitory computer-readable medium of claim 18, wherein the instructions that cause the one or more processors to perform operations for labeling the image, further comprise instructions to perform operations comprising:

generating a pointer for presentation within the two-dimensional, street-level image, the pointer illustrating that the label annotates the given object in the two-dimensional, street-level image.

21. The non-transitory computer-readable medium of claim 18, wherein the label includes a link to information about the given object.

22. A computer-implemented method for labeling an image, comprising:

receiving, with one or more computing devices, a two-dimensional, street-level image and a three-dimensional model representing at least a portion of the image'"'"'s content, wherein the three-dimensional model is generated, in part, by using a search and optimization technique comprising;

selecting a shape from a plurality of primitive shapes; and

defining geometric parameters of the shape to fit a portion of a point cloud representation of dimensions of one or more objects in the two-dimensional, street level image;

identifying, with the one or more computing devices, one or more objects in the two-dimensional, street level image;

determining, with the one or more computing devices, object data for each of the one or more identified objects based at least in part on the three-dimensional model, wherein the object data includes an object type, an object location, and an area occupied by the object;

receiving, with the one or more computing devices, label data representing a label to annotate at least a portion of a given one of the one or more objects in the two-dimensional, street-level image; and

determining, with the one or more computing devices and based on the determined object data, a label position for presentation of the label within the two-dimensional, street-level image, such that the label does not obscure presentation of the given object in the two-dimensional, street-level image, wherein the object type is used to determine whether the label can be associated with the given object without obscuring the presentation.

23. The computer-implemented method of claim 22, wherein the object type is chosen from a set of three or more predetermined types and the set of three or more predetermined types includes “

building”

, “

bridge”

, “

street sign”

, “

road”

, and “

sky”

.