Apparatus, systems and methods for generating 3D model data from a media content event

• US 10,715,684 B2
• Filed: 04/22/2019
• Issued: 07/14/2020
• Est. Priority Date: 11/29/2016
• Status: Active Grant

• Current Assignee: Dish Technologies LLC (Echostar Corporation)
• Original Assignee: Dish Technologies LLC (Echostar Corporation)
• Inventors: Newell, Nick, Tilaye, Swapnil
• Primary Examiner(s): Broome, Said
• Application Number:
  US16/390,705
• Publication Number:
  US 20190253567A1
• Time in Patent Office: 449 Days
• Field of Search: None
• CPC Class Codes: B24D 7/16 Bushings; Mountings
• CPC Class Codes: B29C 64/386 Data acquisition or data processing for additive manufacturing
• CPC Class Codes: B33Y 50/00 Data acquisition or data processing for additive manufacturing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20104 Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04M 1/00 Substation equipment, e.g. for use by subscribers subscriber services or facilities provided at exchanges H04M3/00; prepayment telephone coin boxes H04M17/00; current supply arrangements H04M19/08
• CPC Class Codes: H04N 1/00137 Transmission
• CPC Class Codes: H04N 1/00164 at a remote location
• CPC Class Codes: H04N 21/4333 Processing operations in response to a pause request
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/4722 for requesting additional data associated with the content
• CPC Class Codes: H04N 21/4728 for selecting a Region Of Interest [ROI], e.g. for requesting a higher resolution version of a selected region
• CPC Class Codes: H04N 5/00 Details of television systems scanning details or combination thereof with generation of supply voltages H04N3/00

Abstract

Three dimension (3D) model generation methods are operable to generate 3D model data corresponding to images of a physical object of interest that are viewed in a presenting media content event. An exemplary embodiment receives a user request that is associated with an interest by the user to obtain a 3D model of a physical object of interest that is being shown in a scene of a currently presenting media content event. A plurality of video image frames are selected from the scene. Then, 3D model data of the physical object of interest is generated based on at least the selected video image frames of the scene.

Claims

1. A method that generates three dimensional (3D) models based on images in a media content event, the method comprising:

receiving, at a media device, a user request that is associated with an interest by a user to obtain a 3D model of a physical object of interest that is being shown in a scene of a media content event that the user is currently viewing;

accessing a first video image frame of the scene of the media content event, wherein the first video image frame corresponds to a video image presented in the scene of the media content event at a time that the user request is generated;

identifying a candidate physical object that is shown in the accessed first video image frame;

generating an image of the candidate physical object using the accessed first video image frame;

generating a user confirmation image that is presentable to the user, wherein the user confirmation image includes the generated image of the candidate physical object;

receiving a user response,

wherein the user response indicates that the candidate physical object corresponds to the physical object of interest that the user is interested in, or

wherein the user response indicates that the candidate physical object does not correspond to the physical object of interest that the user is interested in;

defining the candidate physical object as the physical object of interest when the user response indicates that the candidate physical object corresponds to the physical object of interest that the user is interested in;

accessing a plurality of second video image frames, wherein the plurality of second video image frames are accessed from the scene of the media content event that was presented to the user at the time that the user request is generated;

generating an image of the physical object of interest from each one of the plurality of second video image frames when the physical object of interest is identified in the plurality of second video image frames; and

generating 3D model data based on at least the images of the physical object of interest that are generated from the first video image frame and the plurality of second video image frames that were accessed from the scene of the media content event that was presented to the user at the time that the user request is generated.

2. The method of claim 1, wherein identifying the candidate physical object that is shown in the accessed first video image frame is performed at the media device.

3. The method of claim 1, wherein accessing the first video image frame of the scene of the media content event and identifying the candidate physical object comprises:

accessing the first video image frame from a buffer of the media device;

communicating the first video image frame from the media device to a remote 3D model site,

wherein the candidate physical object that is shown in the accessed first video image frame is identified at the remote 3D model site,

wherein the image of the candidate physical object in generated using the first video image frame at the remote 3D model site, and

wherein the image of the candidate physical object is communicated from the remote 3D model site to the media device;

generating the user confirmation image that includes the image of the candidate physical object at the remote 3D model site; and

communicating the image of the candidate physical object to the media device from the remote 3D model site,

wherein the user confirmation image is generated at the media device based on the image of the candidate physical object.

4. The method of claim 1, wherein accessing the first video image frame of the scene of the media content event and identifying the candidate physical object that is shown in the accessed first video image frame comprises:

determining, at the media device, information that identifies at least one of an image frame that was being presented at the time when the user request was generated, image frame identification information that identifies the image frame that was being presented at the time when the user request was generated, and a presentation time of the media content event corresponding to the time that the user request was generated;

communicating the information from the media device to a remote 3D model site;

accessing the first video image frame from a stored media content event that is the same as the media content event that was being viewed by the user at the time that the user request was generated,

wherein the first video image frame is accessed based on the information,

wherein the candidate physical object that is shown in the accessed first video image frame is identified at the remote 3D model site,

wherein the image of the candidate physical object in generated using the first video image frame at the remote 3D model site, and

wherein the user confirmation image with the image of the candidate physical object is communicated from the remote 3D model site to the media device; and

receiving the user confirmation image at the media device from the remote 3D model site,

wherein the user confirmation image is generated at the media device based on received information identifying the candidate physical object.

5. The method of claim 1, wherein identifying the candidate physical object that is shown in the accessed first video image frame comprises:

identifying a plurality of different objects that are shown in the first video image frame;

determining, for each identified object of the identified plurality of different objects, whether the identified object is suitable for printing using a 3D printer; and

selecting the candidate physical object from those ones of the identified plurality of different objects that are identified as being suitable for printing using the 3D printer.

6. The method of claim 5,

wherein a plurality of candidate objects are selected that are identified as being suitable for printing using the 3D printer,

wherein an image of each of the selected plurality of candidate objects are generated,

wherein the user confirmation image includes the generated image of each of the selected plurality of candidate objects, and

wherein the user response includes a selection of one of the selected plurality of candidate objects that have been identified as being suitable for printing using the 3D printer.

7. The method of claim 1, wherein identifying the candidate physical object that is shown in the accessed first video image frame comprises:

identifying a plurality of different objects that are shown in the first video image frame; and

determining, for each one of the identified plurality of different objects, a size of the different object as shown in the first video image frame,

wherein the size for the candidate physical object is largest out of the sizes of the identified plurality of different objects.

8. The method of claim 7,

wherein a plurality of candidate objects are selected that are identified as each having the size that is larger than a threshold size,

wherein an image of each of the selected plurality of candidate objects are generated,

wherein the user confirmation image includes the generated image of each of the selected plurality of candidate objects, and wherein the user response includes a selection of one of the selected plurality of candidate objects that have been identified as being suitable for printing using a 3D printer.

9. The method of claim 1, wherein identifying the candidate physical object that is shown in the accessed first video image frame comprises:

identifying a plurality of different objects that are shown in the first video image frame;

for each object of the identified plurality of different objects;

determining an area corresponding to a size of the different object as shown in the first video image frame;

determining a ratio between the area corresponding to the size of the different object and an area of the first video image frame; and

comparing the ratio with a threshold;

wherein the candidate physical object has a ratio that is larger than the threshold.

10. The method of claim 9,

wherein a plurality of candidate objects are selected that are identified as having a ratio that is larger than the threshold,

wherein an image of each of the selected plurality of candidate objects are generated,

wherein the user confirmation image includes the generated image of each of the selected plurality of candidate objects, and

wherein the user response includes a selection of one of the selected plurality of candidate objects that have been identified as being suitable for printing using a 3D printer.

11. The method of claim 1, wherein identifying the candidate physical object that is shown in the accessed first video image frame comprises:

identifying a plurality of different objects that are shown in the first video image frame;

determining, for each object of the identified plurality of different objects, an outline of the object; and

comparing the outline of the object with a plurality of stored object outlines,

wherein the candidate physical object has its outline that correlates with one of the stored object outlines.

12. The method of claim 1, wherein the candidate physical object is a first candidate object, wherein the user confirmation image is a first user confirmation image, wherein the user response is a first user response, and wherein when the user response indicates that the candidate physical object does not correspond to the physical object of interest that the user is interested in, the method further comprises:

identifying a plurality of different objects that are shown in the first video image frame;

generating an image of at least two of the identified plurality of different objects using the first video image frame;

generating a second user confirmation image that is presentable to the user, wherein the second user confirmation image includes the generated image of the at least two of the identified plurality of different objects; and

receiving a second user response, wherein the user response indicates that one of the at least two of the identified plurality of different objects corresponds to the physical object of interest that the user is interested in.

13. A method that generates three dimensional (3D) models based on images in a media content event, the method comprising:

receiving, at a remote 3D model site from a media device, a user request that is associated with an interest by a user to obtain a 3D model of a physical object of interest that is being shown in a scene of the media content event that the user was viewing at a time that the user request was generated, and wherein the user request includes information that can be used to identify a first image frame that is in the scene of the media content event being presented at the time that the user request was generated by the user;

accessing the first image frame from that is identifiable based on the information of the user request;

identifying a candidate physical object that is shown in the first image frame;

generating an image of the candidate physical object using the first image frame;

communicating one or both of a user confirmation image and the generated image of the candidate physical object from the remote 3D model site to the media device that originated the user request, wherein the media device presents the user confirmation image to the user, and wherein the user confirmation image includes the generated image of the candidate physical object;

receiving a user response at the remote 3D model site from the media device that initially generated the user request,

wherein the user response indicates that the candidate physical object corresponds to the physical object of interest that the user is interested in, or

wherein the user response indicates that the candidate physical object does not correspond to the physical object of interest that the user is interested in;

defining the candidate physical object as the physical object of interest when the user response indicates that the candidate physical object corresponds to the physical object of interest that the user is interested in;

accessing a plurality of second video image frames from a stored media content event that is the same as the media content event that was viewed by the user at the time that the user request was generated, wherein the plurality of second video image frames are from the scene of the media content event that was being presented to the user at the time that the user request was generated;

generating an image of the physical object of interest from each one of the plurality of second video image frames when the physical object of interest is identified in the plurality of second video image frames; and

generating 3D model data based on at least the images of the physical object of interest that are generated from the first image frame and the plurality of second video image frames.

14. The method of claim 13, wherein the information in the user request that can be used to identify a first image frame includes the first image frame that is in the scene of the media content event.

15. The method of claim 13, wherein the information in the user request includes information that identifies a scene of the media content event that was presented to the user at a time that the user request is generated, the method further comprising:

accessing the first image frame from the scene from the stored media content event that is the same as the media content event that was presented to the user at the time that the user request is generated, the method further comprising.

16. The method of claim 13, wherein the information in the user request includes information that identifies the first image frame that is in the scene of the media content event that was presented to the user at the time that the user request is generated, the method further comprising, the method further comprising:

accessing the first image frame from the scene from a stored media content event that is the same as the media content event that was presented to the user at the time that the user request is generated.

17. A method that generates three dimensional (3D) models of an object of interest to a user, wherein the object of interest has been viewed by the user in a scene of a media content event that was presented to the user at a time that a user request is generated, wherein a 3D model of the object of interest is generated based on images in the media content event, the method comprising:

identifying a scene in the media content event that was being viewed by the user at the time that the user request was generated for 3D printing of the object of interest using a 3D printer;

accessing a plurality of video image frames from the identified scene, wherein the plurality of video image frames are accessed from a stored media content event that is the same as the media content event that was viewed by the user when the user request was generated by the user, wherein each of the plurality of video image frames each show an image of the object of interest from a different viewpoint;

identifying the object of interest in each of the plurality of video image frames;

identifying, on each object of interest identified in each of the plurality of video image frames, at least one vertices that is visible on the identified object of interest, wherein each of the vertices correspond to a same point on the identified object of interest;

comparing each of the vertices to determine a depth attribute of the identified object of interest shown in each one of the plurality of video image frames; and

generating 3D model data based on the depth attributes of the identified object of interest shown in each one of the plurality of video image frames.

18. The method of claim 17, wherein plurality of video image frames from the scene define a filmed side of the identified object of interest, and wherein the plurality of video image frames do not include a sufficient number of viewpoints of the identified object of interest such that a hidden side of the identified object of interest occurs, and wherein the hidden side of the identified object of interest was not filmed in the scene, the method further comprising:

determining at least one characteristic associated with the filmed side of the identified object of interest;

determining that the hidden side is a mirror image of a corresponding filmed portion of the identified object of interest, wherein determining that the hidden side is a mirror image of a corresponding filmed portion is made in response to determining the at least one characteristic associated with the filmed side of the identified object of interest;

defining a plane that bisects the identified object of interest into a first mirror image half that is visible in the plurality of video image frames and a second mirror half that includes at least the hidden side of the identified object of interest; and

completing generation of the 3D model data based on defining depth attributes for the second mirror half, wherein the depth attributes for the second mirror half correspond to the depth attributes of the first mirror image half of the identified object of interest shown in each one of the plurality of video image frames.

19. The method of claim 17, wherein the scene is a first scene, wherein the plurality of video image frames are a plurality of first video image frames, wherein plurality of first video image frames from the first scene define a filmed side of the identified object of interest, and wherein the plurality of first video image frames do not include a sufficient number of viewpoints of the identified object of interest such that a hidden side of the identified object of interest occurs, and wherein the hidden side of the identified object of interest was not filmed in the first scene, the method further comprising:

identifying a second scene that includes images of the object of interest;

accessing a plurality of second video image frames from the identified second scene, wherein the plurality of second video image frames are accessed from a stored media content event, wherein each of the plurality of second video image frames each show an image of the hidden side of the identified object of interest from a different viewpoint;

identifying the hidden side of the object of interest in each of the plurality of second video image frames;

identifying, on each one of object of interest identified in each of the plurality of video image frames, at least one second vertices that is visible on the hidden side of the identified object of interest, wherein each of the second vertices correspond to the same point on the hidden side of the identified object of interest;

comparing each of the second vertices to determine a depth attribute of the hidden side of the identified object of interest shown in each one of the plurality of second video image frames; and

completing generation of the 3D model data based on defining depth attributes for the hidden side, wherein the depth attributes for the hidden side correspond to the depth attributes of the first mirror image half of the identified object of interest shown in each one of the plurality of video image frames.

20. The method of claim 19, wherein the second scene is in a media content event that is not the same as the media content event that was being presented to the user at a time that the user generated a request for 3D printing of the object of interest using a 3D printer, the method further comprising:

identifying a difference in color between a portion of the object of interest identified in the plurality of second video image frames and a corresponding portion of the object of interest identified in the plurality of first video image frames,

wherein during generation of the 3D model data, the color of the portion of the object of interest identified in the plurality of second video image frames is changed to match a color of the corresponding portion of the object of interest identified in the plurality of first video image frames, and

wherein the portion of the object of interest identified in the plurality of second video image frames and the corresponding portion of the object of interest identified in the plurality of first video image frames have the same color when the 3D model data is generated.

21. The method of claim 19, wherein the second scene is in a media content event that is not the same as the media content event that was being presented to the user at a time that the user generated a request for 3D printing of the object of interest using a 3D printer, the method further comprising:

identifying a difference in size of the object of interest identified in the plurality of second video image frames and a corresponding size of the object of interest identified in the plurality of first video image frames,

wherein during generation of the 3D model data, a size of a portion of the object of interest identified in the plurality of second video image frames is changed to match a size of the portion of the object of interest identified in the plurality of first video image frames, and

wherein the portion of the object of interest identified in the plurality of second video image frames and the portion of the object of interest identified in the plurality of first video image frames have a same size when the 3D model data is generated.