Method and apparatus for 3-D auto tagging
First Claim
1. A method comprising:
- on a mobile device including a processor, a memory, a camera, a plurality of sensors, a microphone and a display and a touch screen sensor, receiving via an input interface on the mobile device a request to generate a multi-view interactive digital media representation (MVIDMR) of an object;
recording a first plurality of frames from the camera on the mobile device from a live video stream as the mobile device moves along a trajectory such that different views of the object are captured in the first plurality of frames;
generating the MVIDMR of the object including a second plurality of frames from the first plurality of frames wherein the different views of the object are included in each of the second plurality of frames;
using a machine learning algorithm on the second plurality of frames to generate heatmaps and part affinity fields associated with possible 2-D pixel locations of a plurality of landmarks on the object wherein the machine learning algorithm is trained to recognize the plurality of landmarks;
based upon the heatmaps and part affinity fields, determining a skeleton for the object wherein the plurality of landmarks form joints of the skeleton and wherein determining the skeleton includes determining the 2-D pixel locations of the joints;
rendering a first selectable tag into the second plurality of frames to form a third plurality of frames associated with a tagged MVIDMR wherein the first selectable tag is associated with a first landmark positioned at a first joint within the skeleton and wherein the first selectable tag is rendered into the second plurality frames relative to first 2-D pixel locations determined for the first joint in the second plurality of frames;
receiving media content associated with the first selectable tag;
outputting a first frame from the third plurality of frames of the tagged MVIDMR that includes the first selectable tag;
receiving input from the touch screen sensor indicating the first selectable tag is selected in the first frame from the tagged MVIDMR; and
in response, outputting the media content associated with the first selectable tag to the display.
2 Assignments
0 Petitions
Accused Products
Abstract
A multi-view interactive digital media representation (MVIDMR) of an object can be generated from live images of an object captured from a camera. Selectable tags can be placed at locations on the object in the MVIDMR. When the selectable tags are selected, media content can be output which shows details of the object at location where the selectable tag is placed. A machine learning algorithm can be used to automatically recognize landmarks on the object in the frames of the MVIDMR and a structure from motion calculation can be used to determine 3-D positions associated with the landmarks. A 3-D skeleton associated with the object can be assembled from the 3-D positions and projected into the frames associated with the MVIDMR. The 3-D skeleton can be used to determine the selectable tag locations in the frames of the MVIDMR of the object.
-
Citations
26 Claims
-
1. A method comprising:
-
on a mobile device including a processor, a memory, a camera, a plurality of sensors, a microphone and a display and a touch screen sensor, receiving via an input interface on the mobile device a request to generate a multi-view interactive digital media representation (MVIDMR) of an object; recording a first plurality of frames from the camera on the mobile device from a live video stream as the mobile device moves along a trajectory such that different views of the object are captured in the first plurality of frames; generating the MVIDMR of the object including a second plurality of frames from the first plurality of frames wherein the different views of the object are included in each of the second plurality of frames; using a machine learning algorithm on the second plurality of frames to generate heatmaps and part affinity fields associated with possible 2-D pixel locations of a plurality of landmarks on the object wherein the machine learning algorithm is trained to recognize the plurality of landmarks; based upon the heatmaps and part affinity fields, determining a skeleton for the object wherein the plurality of landmarks form joints of the skeleton and wherein determining the skeleton includes determining the 2-D pixel locations of the joints; rendering a first selectable tag into the second plurality of frames to form a third plurality of frames associated with a tagged MVIDMR wherein the first selectable tag is associated with a first landmark positioned at a first joint within the skeleton and wherein the first selectable tag is rendered into the second plurality frames relative to first 2-D pixel locations determined for the first joint in the second plurality of frames; receiving media content associated with the first selectable tag; outputting a first frame from the third plurality of frames of the tagged MVIDMR that includes the first selectable tag; receiving input from the touch screen sensor indicating the first selectable tag is selected in the first frame from the tagged MVIDMR; and in response, outputting the media content associated with the first selectable tag to the display. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21)
-
-
22. A method comprising:
-
on a mobile device including a processor, a memory, a camera, a plurality of sensors, a microphone and a display and a touch screen sensor, receiving via an input interface on the mobile device a request to generate a multi-view interactive digital media representation (MVIDMR) of car; recording a first plurality of frames from the camera on the mobile device from a live video stream as the mobile device moves along a trajectory such that different views of the car are captured in the first plurality of frames; generating the MVIDMR of the car including a second plurality of frames from the first plurality of frames wherein the different views of the car are included in each of the second plurality of frames; using a machine learning algorithm on the second plurality of frames to generate heatmaps and part affinity fields associated with possible 2-D pixel locations of a plurality of landmarks on the car wherein the machine learning algorithm is trained to recognize the plurality of landmarks; based upon the heatmaps and part affinity fields, determining a 3-D skeleton for the car wherein the plurality of landmarks form joints of the 3-D skeleton and wherein determining the 3-D skeleton includes determining the 3-D positions of the joints; rendering a first selectable tag into the second plurality of frames to form a third plurality of frames associated with a tagged MVIDMR wherein the first selectable tag is associated with a first landmark positioned at a first joint in the 3-D skeleton and wherein the first selectable tag is rendered into the second plurality frames relative to first 2-D pixel locations determined for the first joint in the second plurality of frames from first 3-D positions associated with the first joint; receiving media content associated with the first selectable tag; outputting a first frame from the third plurality of frames of the tagged MVIDMR of the car that includes the first selectable tag; receiving input from the touch screen sensor indicating the first selectable tag is selected in the first frame from the tagged MVIDMR; and in response outputting the media content associated with the first selectable tag to the display. - View Dependent Claims (23, 24, 25, 26)
-
Specification