Augmented reality navigation for repeat photography and difference extraction

• US 8,803,992 B2
• Filed: 05/12/2010
• Issued: 08/12/2014
• Est. Priority Date: 05/12/2010
• Status: Active Grant

• Current Assignee: Fujifilm Business Innovation Corp. (Fujifilm Holdings Corporation)
• Original Assignee: Fuji Xerox Company Limited (Xerox Holdings Corp.)
• Inventors: Shingu, Jun, Kimber, Donald, Rieffel, Eleanor, Vaughan, James, Tuite, Kathleen
• Primary Examiner(s): KHAN, USMAN A
• Application Number:
  US12/779,021
• Publication Number:
  US 20110279697A1
• Time in Patent Office: 1,553 Days
• Field of Search: 382/222, 348/222.1
• US Class Current: 348/222.1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: H04N 5/2621 Cameras specially adapted for the electronic generation of special effects during image pickup, e.g. digital cameras, camcorders, video cameras having integrated special effects capability

Abstract

Systems and methods for repeat photography and difference extraction that help users take pictures from the same position and camera angle as earlier photos. The system automatically extracts differences between the photos. Camera poses are estimated and then indicators are rendered to show the desired camera angle, which guide the user to the same camera angle for repeat photography. Using 3D rendering techniques, photos are virtually projected onto a 3D model to adjust them and improve the match between the photos, and the difference between the two photos are detected and highlighted. Highlighting the detected differences helps users to notice the differences.

Claims

1. A system, comprising:

a camera receiving live feed;

a database of pictures;

a determining unit determining an angle and a position associated with an object to be captured, wherein the determining unit determines a plurality of angles and a position associated with the live feed relative to the object to be captured;

a guiding unit providing a first indication being overlaid onto the live feed and displayed on a display device to indicate the angle associated with the object to be captured relative to the live feed, and providing a second indication being overlaid onto the live feed and displayed on the display device to indicate the position of the object to be captured relative to the live feed, and wherein a third indication is provided if at least one angle in the plurality of angles and the position associated with the live feed is within a specified tolerance to the angle and position of the object to be captured;

wherein said first and second indications are simultaneously overlaid onto the live feed,

wherein the first indication is represented by a cone,

wherein a vertex of the cone is located at the object and an axis of the cone is extended in a camera-to-object direction, and

a vertex angle formed between the vertex and the axis indicates an amount of difference between the live feed and the object that is acceptable.

2. The system of claim 1, further comprising a virtual 3D model.

3. The system of claim 2, wherein the guiding unit utilizes the first and second indications to indicate at least one angle and position of the virtual 3D model relative to the live feed.

4. The system of claim 2, further comprising a comparator unit, the comparator unit analyzing and highlighting visual differences between a snapshot from the live feed and a selected picture from the database of pictures.

5. The system of claim 4, wherein the comparator unit uses geometry of the virtual 3D model to aid in conducting difference detection and wherein the snapshot is within the specified tolerance of the object to be captured.

6. The system of claim 1, wherein the third indication is a color change of the first indication and the second indication when the position associated with the live feed is within a specified tolerance to the angle and position of the object to be captured.

7. The system of claim 1, wherein a vector associated with the object to be captured and a width representing the specified tolerance is calculated.

8. The system of claim 4, wherein the comparator unit analyzes the snapshot with respect to the virtual 3D model and determines uncaptured regions on the surface of the virtual 3D model.

9. The system of claim 8, wherein the uncaptured regions are disregarded during the highlighting of visual differences.

10. The system of claim 4, wherein image regions are extracted from the snapshot and wherein the extracted image regions are projected onto the virtual 3D model and wherein the projected images are used as part of the difference extraction.

11. The system of claim 1, further comprising a voice navigation unit generating voice guidance for navigating the camera to line up with a previous image.

12. The system of claim 2, wherein additional indications are overlaid onto the live feed to indicate at least one of a distance from the object to be captured, a pan and tilt angle of the camera, and an up angle of the camera.

13. The system of claim 1, wherein the second indication is represented by a sphere.

14. A system, comprising:

a camera receiving live feed;

a database of pictures;

a determining unit determining an angle and a position associated with an object to be captured, wherein the determining unit determines a plurality of angles and a position associated with the live feed relative to the object to be captured;

a guiding unit providing a first indication indicating the angle associated with the object to be captured relative to the live feed, and providing a second indication indicating the position of the object to be captured relative to the live feed, and wherein a third indication is provided if at least one angle in the plurality of angles and the position associated with the live feed is within a specified tolerance to the angle and position of the object to be captured;

wherein said first and second indications are overlaid onto the live feed,

wherein a vector associated with the object to be captured and a width representing the specified tolerance is calculated,

wherein the first indication is represented by a cylinder, the radius of the cylinder representing the specified tolerance and an axis of the cylinder represents the calculated vector associated with the camera angle of the object to be captured.

15. A method for navigating a camera receiving live feed to take a snapshot at an angle and position similar to a previous snapshot, the method comprising:

determining the angles and position of the previous snapshot based on a virtual 3D model;

determining a plurality of camera angles and position of the live feed with respect to the virtual 3D model; and

simultaneously overlaying indications onto the live feed of the camera based on the angles and the position of the previous snapshot relative to the plurality of camera angles and the position of the live feed,

wherein said indications are overlaid onto the live feed and displayed on a display device to indicate the angles and the position of the previous snapshot relative to the plurality of camera angles and the position of the live feed,

wherein a first indication is represented by a cone,

wherein a vertex of the cone is located at the object and an axis of the cone is extended in a camera-to-object direction, and

a vertex angle formed between the vertex and the axis indicates an amount of difference between the live feed and the object that is acceptable.

16. The method of claim 15, further comprising another indication when an angle and position of the live feed is within a specified tolerance of an angle and position of the previous snapshot.

17. A method for navigating a camera receiving live feed to take a snapshot at a camera angle and position similar to a previous snapshot, the method comprising:

receiving an input;

interpolating the camera angle and the position of the previous snapshot based on the input;

determining an angle and position of the live feed; and

simultaneously overlaying indications onto the live feed of the camera based on the interpolated angle and position of the previous snapshot relative to the angle and position of the live feed,

wherein said indications are overlaid onto the live feed and displayed on a display device to indicate the interpolated angle and position of the previous snapshot relative to the angle and position of the live feed,

wherein a first indication is represented by a cone,

wherein a vertex of the cone is located at the object and an axis of the cone is extended in a camera-to-object direction, and

a vertex angle formed between the vertex and the axis indicates an amount of difference between the live feed and the object that is acceptable.

18. The method of claim 17, further comprising another indication when the angle and position of the live feed is within a specified tolerance of the angle and position of the previous snapshot.

19. A non-transitory computer readable medium provided with instructions being executed by a processor for receiving live feed to take a snapshot at an angle and position similar to a previous snapshot, the instructions comprising:

determining the angles and position of the previous snapshot based on a virtual 3D model;

determining a plurality of camera angles and position of the live feed with respect to the virtual 3D model; and

simultaneously overlaying indications onto the live feed of the camera based on the angles and the position of the previous snapshot relative to the plurality of camera angles and the position of the live feed,

wherein said indications are overlaid onto the live feed and displayed on a display device to indicate the angles and the position of the previous snapshot relative to the plurality of camera angles and the position of the live feed,

wherein a first indication is represented by a cone,

wherein a vertex of the cone is located at the object and an axis of the cone is extended in a camera-to-object direction, and

a vertex angle formed between the vertex and the axis indicates an amount of difference between the live feed and the object that is acceptable.

20. The non-transitory computer readable medium of claim 19, wherein the instructions further comprise providing an indication when an angle and position of the live feed is within a specified tolerance of an angle and position of the previous snapshot.

21. A non-transitory computer readable medium provided with instructions being executed by a processor for navigating a camera receiving live feed to take a snapshot at a camera angle and position similar to a previous snapshot, the instructions comprising:

receiving an input;

interpolating the camera angle and the position of the previous snapshot based on the input;

determining an angle and position of the live feed; and

simultaneously overlaying indications onto the live feed of the camera based on the interpolated angle and position of the previous snapshot relative to the angle and position of the live feed,

wherein said indications are overlaid onto the live feed and displayed on a display device to indicate the angles and the position of the previous snapshot relative to the plurality of camera angles and the position of the live feed,

wherein a first indication is represented by a cone,

wherein a vertex of the cone is located at the object and an axis of the cone is extended in a camera-to-object direction, and

a vertex angle formed between the vertex and the axis indicates an amount of difference between the live feed and the object that is acceptable.

22. The non-transitory computer readable medium of claim 21, wherein the instructions further comprise providing an indication when the angle and position of the live feed is within a specified tolerance of the angle and position of the previous snapshot.