Visual annotation using tagging sessions

• US 10,382,739 B1
• Filed: 04/26/2018
• Issued: 08/13/2019
• Est. Priority Date: 04/26/2018
• Status: Active Grant

• Current Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Original Assignee: Fyusion, Inc. (Cox Enterprises Incorporated)
• Inventors: Rusu, Radu Bogdan, Morrison, Dave, Martin, Keith, Miller, Stephen David, Kalogiros, Pantelis, Penz, Mike, Wawro, Martin Markus Hubert, Dumeljic, Bojana, Chaudhry, Jai, Parham, Luke, Santiago, Julius, Holzer, Stefan Johannes Josef
• Primary Examiner(s): Kim, Hee-Yong
• Application Number:
  US15/963,896
• Time in Patent Office: 474 Days
• Field of Search: 348 48
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: H04N 13/183 On-screen display [OSD] information, e.g. subtitles or menus
• CPC Class Codes: H04N 13/207 using a single 2D image sensor
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Various embodiments of the present invention relate generally to systems and methods for analyzing and manipulating images and video. In particular, a multi-view interactive digital media representation (MVIDMR) of an object can be generated from live images of an object captured from a camera. After the MVIDMR of the object is generated, a tag can be placed at a location on the object in the MVIDMR. The locations of the tag in the frames of the MVIDMR can vary from frame to frame as the view of the object changes. When the tag is selected, media content can be output which shows details of the object at location where the tag is placed. In one embodiment, the object can be car and tags can be used to link to media content showing details of the car at the locations where the tags are placed.

Claims

1. A method comprising:

on a mobile device including a processor, a memory, a camera, a plurality of sensors, a microphone and a display and a touch screen sensor, receiving via an input interface on the mobile device a request to generate a multi-view interactive digital media representation (MVIDMR) of an object;

recording a first plurality of frames from the camera on the mobile device from a live video stream as the mobile device moves along a trajectory such that different views of the object are captured in the first plurality of frames;

generating the MVIDMR of the object including a second plurality of frames from the first plurality of frames wherein the different views of the object are included in each of the second plurality of frames;

outputting a first frame from the MVIDMR including a selector rendered over the first frame to the display;

receiving, via the touch screen sensor and the selector, a selection of a location on the object in the first frame;

removing the selector and rendering a first selectable tag at the location selected in the first frame;

outputting the first frame including the first selectable tag to the display;

for each remaining frame in the second plurality of frames of the MVIDMR, determining a first location where the location on the object appears in the each remaining frame including determining whether the location on the object appears in the each remaining frame;

for each remaining frame where the location on the object appears, rendering the first selectable tag into each remaining frame at the first location to generate a third plurality of frames to form a tagged MVIDMR;

outputting to the display the tagged MVIDMR;

receiving media content associated with the first selectable tag;

outputting a first frame from the third plurality of frames of the tagged MVIDMR that includes the first selectable tag;

receiving input from the touch screen sensor indicating the first selectable tag is selected in the first frame from the tagged MVIDMR; and

in response outputting the media content associated with the first selectable tag to the display.

2. The method of claim 1, further comprising outputting to the display the third plurality of frames associated with the tagged MVIDMR wherein the object appears in each of the third plurality of frames and wherein the first selectable tag appears in only portion of the third plurality of frames.

3. The method of claim 1, further comprising generating a prompt to save the tagged MVIDMR and in response to receiving a selection of the prompt saving the third plurality of frames associated with the tagged MVIDMR.

4. The method of claim 1, further comprising generating a prompt to move a current location of the first selectable tag, receiving an input to move the current location of the first selectable tag to a new location on the object, outputting a second frame including the first selectable tag at the new location on the object to the display, for each remaining frame in the second plurality of frames of the MVIDMR, determining a second location where the new location on the object appears in the each remaining frame including determining whether the new location on the object appears in the each remaining frame;

and for each remaining frame where the new location on the object appears, rendering the first selectable tag into each remaining frame at the second location to generate a fourth plurality of frames for a second tagged MVIDMR.

5. The method of claim 1, based upon the first location where the first selectable tag is rendered in each of the third plurality of frames of the tagged MVIDMR where the first selectable tag appears and an area of the first selectable tag, determining a mapping between the first selectable tag and the touch screen sensor wherein the mapping is used to determine whether an input on the touch screen sensor indicates a selection of the first selectable tag.

6. The method of claim 1, further comprising outputting a first frame from the third plurality of frames of the tagged MVIDMR including the selector rendered over the first frame to the display;

receiving, via the touch screen sensor and the selector, a selection of a second location on the object in the first frame;

removing the selector and rendering a second selectable tag at the second location selected in the first frame of the tagged MVIDMR;

outputting the first frame including the second selectable tag from the tagged MVIDMR to the display;

for each remaining frame in the third plurality of frames of the tagged MVIDMR, determining a third location where the second location on the object appears in the each remaining frame including determining whether the second location on the object appears in the each remaining frame;

for each remaining frame where the second location on the object appears, rendering the second selectable tag into each remaining frame at the third location to generate a fourth plurality of frames for a second tagged MVIDMR; and

outputting the second tagged MVIDMR, including the first selectable tag and the second selectable tag, to the display.

7. The method of claim 6, wherein the first selectable tag and the second selectable tag both appear in a portion of the fourth plurality of frames of the second tagged MVIDMR.

8. The method of claim 6, wherein only the first selectable tag appears in a first portion of the fourth plurality of frames of the second tagged MVIDMR and only the second selectable tag appears in a second portion of the fourth plurality of frames of the second tagged MVIDMR.

9. The method of claim 6, wherein neither the first selectable tag nor the second selectable tag appear in a portion of the fourth plurality of frames of the second tagged MVIDMR.

10. The method of claim 6, further comprising receiving second media content associated with the second selectable tag;

outputting a first frame from the fourth plurality of frames of the second tagged MVIDMR that includes the second selectable tag;

receiving input from the touch screen sensor indicating the second selectable tag is selected in the first frame; and

in response, outputting the second media content associated with the second selectable tag to the display.

11. The method of claim 1, wherein the media content shows one or more close-up views of the location on the object.

12. The method of claim 1, wherein the media content is one of a photo showing a close-up view of the location on the object or a second MVIDMR showing close-up views of the location on the object.

13. The method of claim 1, further comprising, generating a prompt to capture the media content associated with the first selectable tag.

14. The method of claim 1, wherein the object is a car.

15. The method of claim 14, wherein the first selectable tag is associated with a damaged location on the car and wherein the media content shows one or more close-up views of the damaged location.

16. The method of claim 14, wherein the first selectable tag is associated with a component or a region of the car and wherein the media content shows one or more close-up views of the component or the region of the car.

17. The method of claim 1, wherein the object includes an exterior and an interior and wherein the tagged MVIDMR shows the exterior of the object further comprising generating a second tagged MVIDMR of the interior of the object, wherein the tagged MVIDMR of the exterior of the object includes a second selectable tag that when selected causes the second tagged MVIDMR of the interior of the object to be output to the display.

18. The method of claim 17, wherein the second tagged MVIDMR of the interior of the object includes a third selectable tag that when selected causes first media content showing one or more close up views of an interior location to be output to the display.

19. The method of claim 1, further comprising generating a plan view of the object, determining where the location on the object associated with the first selectable tag is located on the plan view, rendering a second selectable tag corresponding to the first selectable tag onto the plan view and outputting the plan view including the second selectable tag onto to the display.

20. The method of claim 19, further comprising:

receiving a selection of the second selectable tag on the plan view, outputting to the display a second frame selected from among the third plurality of frames of the tagged MVIDMR which includes the first selectable tag.

21. The method of claim 19, further comprising:

receiving a selection of the second selectable tag on the plan view and outputting to the display the media content associated with the first selectable tag.

22. The method of claim 1, wherein the location selected on the object is a component of the object, further comprising determining a plurality of key points associated with the component, tracking the key points in each of the remaining frames of the second plurality of frames to determine the first location in each of the remaining frames where the location on the object appears.

23. The method of claim 1, further comprising outputting to the display a textual description of the location on the object in the first frame that is to be selected and tagged.

24. The method of claim 1, further comprising, prior to recording first plurality of frames including the object, receiving an input indicating a selection of the object.

25. The method of claim 1, further comprising applying stabilization and smoothing to the first plurality of frames to generate the second plurality of frames.