Annotating or editing three dimensional space

• US 8,941,641 B2
• Filed: 03/31/2009
• Issued: 01/27/2015
• Est. Priority Date: 03/31/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chen, Billy, Ofek, Eyal
• Primary Examiner(s): Martello, Edward
• Application Number:
  US12/416,134
• Publication Number:
  US 20100245344A1
• Time in Patent Office: 2,128 Days
• Field of Search: None
• US Class Current: 345/419
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 2203/04802 3D-info-object: information is displayed on the internal or external surface of a three dimensional manipulable object, e.g. on the faces of a cube that can be rotated by the user
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: H04N 21/8146 involving graphical data, e.g. 3D object, 2D graphics

Abstract

In one example, images may be used to create a model of a three-dimensional space, and the three-dimensional space may be annotated and/or edited. When a three-dimensional model of a space has been created, a user may associate various items with points in the three-dimensional space. For example, the user may create a note or a hyperlink, and may associate the note or hyperlink with a specific point in the space. Additionally, a user may experiment with the space by adding images to, or deleting images from, the space. Annotating and editing the space, rather than the underlying images, allows annotations and edits to be associated with the underlying objects depicted in the images, rather than with the images themselves.

Claims

1. One or more computer-readable disks or memories that store executable instructions that, when executed by a computer, cause the computer to perform acts comprising:

receiving a plurality of images;

determining what regions, in three-dimensional space, said images represent;

constructing a three-dimensional point cloud based on said images and further based on what regions said images represent;

receiving an item, said item comprising an image of an object to be inserted into said three-dimensional point cloud, said image of said object being obtained from a location other than said three-dimensional space that said plurality of images represent, said image of said object being a three-dimensional image created by a user from whom said item is received;

receiving a location, in said three-dimensional point cloud, at which said item is to be placed;

associating said item with said location; and

displaying modified versions of said plurality of images, with said item, in a tangible form, said modified images having said object inserted into said images at points in said images that correspond to said location.

2. The one or more computer-readable disks or memories of claim 1, wherein said receiving of said location comprises:

receiving, from said user through a physical input device, an indication of a polygon that defines a region, said location being, or being within, said region.

3. The one or more computer-readable disks or memories of claim 1, wherein said item comprises a note, and wherein said displaying comprises:

displaying said note with a portion of said images that are associated with said location.

4. The one or more computer-readable disks or memories of claim 1, wherein said item comprises a hyperlink, and wherein said acts further comprise:

receiving a request to access content associated with said location; and

based on said request, accessing said hyperlink.

5. The one or more computer-readable disks or memories of claim 1, wherein said item comprises a graphic, and wherein said displaying comprises:

displaying said graphic as occurring in said three-dimensional point cloud in said location.

6. The one or more computer-readable disks or memories of claim 5, wherein said graphic comprises a two-dimensional image, and wherein said acts further comprise:

receiving an indication of a rotation of said two-dimensional image around one or more axes.

7. The one or more computer-readable disks or memories of claim 1, wherein said item comprises a video, and wherein said displaying comprises:

displaying said video with a portion of said images that are associated with said location.

8. The one or more computer-readable disks or memories of claim 1, wherein said item comprises audio, and wherein said acts further comprise:

rendering said audio.

9. The one or more computer-readable disks or memories of claim 1, further comprising:

receiving an editing instruction to remove content that occurs in a region of said three-dimensional point cloud.

10. The one or more computer-readable disks or memories of claim 1, said object being a piece of furniture, said image of said furniture being obtained from a published source.

11. A system for interacting with images in three-dimensional space, the system comprising:

a data remembrance component;

a processor;

one or more components that are stored in said data remembrance component and that execute on said processor, said one or more components comprising;

a three-dimensional point cloud generator that receives a plurality of images and that generates a point cloud that associates said images with points in three-dimensional space; and

an editor component that receives an item, and that further receives an indication of a location in said three-dimensional space with which said item is to be associated, said item comprising an image of an object to be inserted into said three-dimensional point cloud, said image of said object being obtained from a location other than said three-dimensional space that said plurality of images represent, said image of said object being a three-dimensional image created by a user from whom said item is received by said editor component;

wherein said system further comprises;

a display that shows modified versions of a set of said images in said three-dimensional space that are associated with said location, and that shows said item in association with images in said set, said modified images having said object inserted into said images at points in said images that correspond to said location.

12. The system of claim 11, wherein said editor component receives, from said user through a physical input device, an indication of a polygon that defines a region that coincides with, or contains, said location.

13. The system of claim 11, wherein said item comprises a note, and wherein said display shows said note with images in said set, or portions of images in said set.

14. The system of claim 11, wherein said item comprises a hyperlink, and wherein the system receives a request to access content associated with said location and, based on said request, accesses said hyperlink.

15. The system of claim 11, wherein said item comprises a graphic, and wherein said display shows said graphic as occurring in said three-dimensional space at said location.

16. The system of claim 15, wherein said graphic comprises a two-dimensional image, and wherein said editor component receives an indication of a rotation of said two-dimensional image around one or more axes.

17. The system of claim 15, wherein said graphic comprises a three-dimensional image, and wherein said editor component places said graphic at a non-planar set of points in said three-dimensional space.

18. A method of interacting with visual content in three-dimensional space, the method comprising:

using a processor to perform acts comprising;

receiving a plurality of images;

constructing a three-dimensional space from said images, in which said images are placed, in said three-dimensional space, at locations that correspond to locations of objects that the images depict;

receiving, from a user, a specification of a region;

receiving, from said user, an indication of an action to be performed with respect to said region;

performing said action; and

displaying, to said user, a first representation of said three-dimensional space that includes a second representation of said action at said region, said action comprising inserting an image of an object into said region of said three-dimensional space, said image of said object being obtained from a location other than said three-dimensional space that said plurality of images represent, said image of said object being a three-dimensional image created by said user, said action comprising deletion of a portion of said three-dimensional space, said performing of said action comprising storing data that indicates deletion of content in said region, and said displaying comprising showing a modified version of said first representation of said three-dimensional space that comprises said object inserted at points in said plurality of images that correspond to said region without showing images that are associated with said region.

19. The method of claim 18, wherein said action comprises addition of a graphic to said three-dimensional space, wherein said performing of said action comprises storing data that represents an association between said region and said graphic, and wherein said displaying comprises showing said graphic at said region in said three-dimensional space.

20. The method of claim 18, wherein said action comprises associating a note or a hyperlink with said region, and wherein said displaying comprises showing said note or hyperlink at said region in said three-dimensional space.