Methods and systems for detecting and analyzing a region of interest from multiple points of view

• US 10,586,124 B2
• Filed: 08/03/2017
• Issued: 03/10/2020
• Est. Priority Date: 08/03/2017
• Status: Active Grant

• Current Assignee: Streaming Global, Inc.
• Original Assignee: Streaming Global, Inc.
• Inventors: Oesterreicher, Richard, Hessing, Jonathan, Ukani, Zunair, Schmidt, Austin
• Primary Examiner(s): Tabatabai, Abolfazl
• Application Number:
  US15/668,510
• Publication Number:
  US 20190042868A1
• Time in Patent Office: 950 Days
• Field of Search: 382103
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/255 Detecting or recognising potential candidate objects based on visual cues, e.g. shapes
• CPC Class Codes: G06V 20/41 Higher-level, semantic clustering, classification or understanding of video scenes, e.g. detection, labelling or Markovian modelling of sport events or news items segmenting video sequences G06V20/49

Abstract

Disclosed are devices, systems, methods, techniques, and computer program products for estimating a Region Of Interest (ROI) corresponding to a plurality of content streams. The method may include receiving, by a system having at least one processor, a plurality of sensor data associated with a plurality of mobile devices. The plurality of sensor data corresponds to a plurality of content streams captured by the plurality of mobile devices. Further, each of the plurality of mobile devices may include at least one recorder configured to capture a corresponding content steam. Further, a sensor data associated with a mobile device may include a location data and an orientation data of the mobile device during capturing of a content stream. The method may further include, analyzing, by the system, the plurality of sensor data and determining, by the system, a ROI based on at least the plurality of sensor data, thereby generating an estimation of the ROI.

Claims

1. A method of estimating a Region of Interest (ROI) corresponding to a plurality of content streams, the method comprising:

receiving, by a system having at least one processor, a plurality of sensor data associated with a plurality of mobile devices, wherein the plurality of sensor data corresponds to the plurality of content streams captured by the plurality of mobile devices, wherein each of the plurality of mobile devices comprises at least one recorder configured to capture a corresponding content steam, wherein a sensor data associated with a mobile device comprises a location data and an orientation data of the mobile device during capturing of a content stream;

analyzing, by the system, the plurality of sensor data;

determining, by the system, a ROI based on at least the plurality of sensor data, thereby generating an estimation of the ROI; and

receiving, by the system, a plurality of optical characteristics corresponding to the plurality of mobile devices, wherein determining the ROI is based at least on the plurality of optical characteristics.

2. The method of claim 1, wherein an optical characteristic of the mobile device comprises a Field of View (FOV) of a lens comprised in the mobile device.

3. The method of claim 1, further comprising receiving a plurality of device identifiers from the plurality of mobile devices, wherein the plurality of optical characteristics is identified based on at least the plurality of device identifiers.

4. The method of claim 1, wherein analyzing the plurality of sensor data comprises:

determining a plurality of origin points based at least on a plurality of location data corresponding to the plurality of mobile devices; and

determining a plurality of vectors associated with the plurality of origin points based at least on a plurality of orientation data corresponding to the plurality of mobile devices, a vector associated with an origin point originating at the origin point.

5. The method of claim 4, wherein determining the ROI is based at least on the plurality of origin points and the plurality of vectors.

6. The method of claim 5, wherein the plurality of origin points is configured to be used as coordinates in a mapping coordinate system.

7. The method of claim 6, further comprising determining a plurality of vector intersections of the plurality of vectors;

and wherein determining the ROI is based at least on the plurality of vector intersections.

8. The method of claim 1, further comprising:

receiving a plurality of Field of Views (FOVs) associated with the plurality of mobile devices, a Field of View (FOV) associated with the mobile device being associated with a lens comprised in the mobile device;

determining a plurality of origin points based at least on a plurality of location data corresponding to the plurality of mobile devices, the plurality of origin points configured as coordinates in a mapping coordinate system;

determining a plurality of vectors associated with the plurality of origin points based at least on a plurality of orientation data corresponding to the plurality of mobile devices, a vector associated with an origin point originating at the origin point;

generating a plurality of viewing frustums based at least on the plurality of FOVs, wherein an apex of a viewing frustum associated with the mobile device is coincident with the origin point associated with the mobile device, wherein a view direction of the viewing frustum is aligned with the vector associated with the mobile device; and

determining a plurality of intersections of the plurality of viewing frustums, wherein determining the ROI is based at least on the plurality of intersections of the plurality of viewing frustums.

9. The method of claim 8, wherein determining the plurality of origin points comprises:

converting the plurality of location data corresponding to the plurality of mobile devices into the plurality of origin points in the mapping coordinate system; and

wherein determining the plurality of vectors comprises converting the plurality of orientation data corresponding to the plurality of mobile devices into a plurality of vectors associated with the plurality of origin points, wherein the vector associated with an origin point originates at the origin point.

10. The method of claim 1, further comprising:

receiving the plurality of content streams from the plurality of mobile devices;

analyzing the plurality of content streams;

determining at least one object associated with the ROI; and

determining at least one spatial dimension associated with the at least one object;

wherein determining the ROI is based at least on the at least one spatial dimension.

11. The method of claim 1 further comprising:

receiving the plurality of content streams from the plurality of mobile devices;

analyzing the plurality of content streams;

determining at least one object associated with the ROI based at least on the plurality of content streams;

determining at least one spatial dimension associated with the at least one object based at least on a predetermined association between a plurality of objects and a plurality of spatial dimensions; and

determining a plurality of distances corresponding to the plurality of mobile devices and the ROI, wherein a distance between the mobile device and the ROI is estimated based on a spatial dimension of the at least one object determined in the content stream captured by the mobile device;

wherein determining the ROI is based at least on the plurality of distances.

12. The method of claim 11, wherein determining the plurality of distances comprises determining the plurality of distances corresponding to the plurality of mobile devices, wherein the distance between the mobile device and the ROI is determined based at least on the spatial dimension of the at least one object determined in the content stream captured by the mobile device.

13. The method of claim 1, wherein the sensor data further comprises a distance data representing a distance of the mobile device from at least one object comprised in the ROI, wherein the mobile device comprises a ranging sensor configured to determine the distance.

14. The method of claim 13, wherein the ranging sensor comprises at least one of a LIDAR sensor and an image sensor.

15. The method of claim 1, further comprising:

executing a search on a venue database based on the location data;

determining a venue from the venue database based at least on the location data; and

retrieving at least one spatial characteristic of the venue;

wherein determining the ROI is based at least on the at least one spatial characteristic.

16. The method of claim 15, further comprising determining at least one trajectory associated with at least one mobile device based at least on analysis of the plurality of sensor data;

wherein determining the ROI is based at least on the at least one trajectory.

17. The method of claim 1, wherein the plurality of mobile devices comprises a drone configured for flight.

18. The method of claim 1, further comprising:

generating a visualization of the ROI and the plurality of mobile devices; and

transmitting the visualization to at least one mobile device of the plurality of mobile devices.

19. The method of claim 18, further comprising:

receiving a selection associated with the visualization;

identifying one or more mobile devices associated with the selection; and

transmitting one or more content streams captured by the plurality of mobile devices.

20. A method of estimating a Region of Interest (ROI) corresponding to a plurality of content streams, the method comprising:

receiving, by a system having at least one processor, a plurality of sensor data associated with a plurality of mobile devices, wherein the plurality of sensor data corresponds to the plurality of content streams captured by the plurality of mobile devices, wherein each of the plurality of mobile devices comprises at least one recorder configured to capture a corresponding content steam, wherein a sensor data associated with a mobile device comprises a location data and an orientation data of the mobile device during capturing of a content stream;

analyzing, by the system, the plurality of sensor data;

determining, by the system, a ROI based on at least the plurality of sensor data, thereby generating an estimation of the ROI;

receiving the plurality of content streams from the plurality of mobile devices;

analyzing the plurality of content streams;

determining at least one object associated with the ROI; and

determining at least one spatial dimension associated with the at least one object.

21. A method of estimating a Region of Interest (ROI) corresponding to a plurality of content streams, the method comprising:

receiving, by a system having at least one processor, a plurality of sensor data associated with a plurality of mobile devices, wherein the plurality of sensor data corresponds to a plurality of content streams captured by the plurality of mobile devices, wherein each of the plurality of mobile devices comprises at least one recorder configured to capture a corresponding content steam, wherein a sensor data associated with a mobile device comprises a location data and an orientation data of the mobile device during capturing of a content stream;

analyzing, by the system, the plurality of sensor data, wherein analyzing the plurality of sensor data comprises;

determining a plurality of origin points based at least on a plurality of location data corresponding to the plurality of mobile devices; and

determining a plurality of vectors associated with the plurality of origin points based at least on a plurality of orientation data corresponding to the plurality of mobile devices, a vector associated with an origin point originating at the origin point; and

determining, by the system, a ROI based on at least the plurality of sensor data, thereby generating an estimation of the ROI.