Broadband passive tracking for augmented reality

• US 9,390,503 B2
• Filed: 12/02/2013
• Issued: 07/12/2016
• Est. Priority Date: 03/08/2010
• Status: Expired due to Fees

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Leung, Henry, Liu, Xiaoxiang
• Primary Examiner(s): RAHAMAN, MOHAMMED S
• Application Number:
  US14/094,623
• Publication Number:
  US 20140085483A1
• Time in Patent Office: 953 Days
• US Class Current: 1/1
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Technologies are generally described for a broadband passive sensing and tracking system that may employ a number of passive receivers that each have the capability of sensing electromagnetic waves (e.g., Radio Frequency “RF” signals) from surrounding broadcast sources. Each passive receiver may be adapted to sense through one or more antennas. Multiple receivers at different positions may be utilized to form a broadband sensing network adapted to perform collaborative tracking of a scene of interest. According to some examples, a beam-forming algorithm may be applied over the broadband sensing network utilizing an antenna array formed by the passive receivers to localize and track objects.

Claims

1. A method to track one or more objects of interest in an augmented reality (AR) system, the method comprising:

receiving signals at each one of a plurality of tracking sensors, wherein;

each of the plurality of tracking sensors form part of a passive broadband sensing network and utilize passive sensing to analyze the received signals,

the received signals are scattered by the one or more objects of interest and received through passive receivers,

the received signals correspond to one or more of a direct signal from an existing illumination source and/or an echo signal that is scattered by the one or more objects of interest as a result of the direct signal, and

the existing illumination source is not part of the passive broadband sensing network;

processing the received signals to generate data associated with the object of interest through one or more of determining a location of one or more feature points in an image, computation of affine projections, tracking of edges, and filtering, and wherein the data corresponds to one or more of a position parameter and/or a motion parameter associated with the object of interest;

developing a point-to-point communication layer to enable communication and sharing of information between any two sensors in the passive broadband sensing network; and

augmenting a real scene that includes the one or more objects of interest, by employing one or more of the position parameter and/or the motion parameter to augment the one or more objects of interest in the real scene with at least one virtual object.

2. The method according to claim 1, wherein processing the received signals includes one or more of:

pre-processing the received signals;

identifying each of the received signals as one or more of the direct signal and/or the echo signal through the filtering, which includes adaptive filtering;

deriving Doppler spectra of the direct signal and the echo signal; and

/or

phase compensating the Doppler spectra of the echo signals.

3. The method according to claim 2, wherein pre-processing the received signals includes one or more of signal correction, bandwidth correction, signal averaging, amplification, down-conversion, and/or digitization.

4. The method according to claim 2, wherein processing the received signals to generate data associated with the object of interest comprises employing a beam forming process on the phase compensated signals to determine one or more of position and/or motion parameters associated with the object of interest.

5. The method according to claim 4, wherein employing the beam forming process comprises:

adaptively selecting a number and a location of the tracking sensors, the tracking sensors integrated into a plurality of wireless communication devices communicating with a plurality of transmission sources.

6. The method according to claim 5, further comprising:

dynamically configuring a wireless network of the tracking sensors to mitigate multipath fading and reduce interference through diversity of antennas of the wireless communication devices.

7. An apparatus to track one or more objects of interest in an augmented reality (AR) system, the apparatus comprising:

a wireless communication device; and

an antenna and a tracking sensor included in the wireless communication device, wherein the wireless communication device is adapted to;

receive signals with the antenna, wherein;

the tracking sensor is a part of a passive broadband sensing network and utilizes passive sensing to analyze the received signals,

a point-to-point communication layer is developed to enable communication and sharing of information between the tracking sensor and another tracking sensor in the passive broadband sensing network,

the received signals are scattered by the one or more objects of interest and received through passive receivers,

the received signals correspond to one or more of direct signals from an existing illumination source or echo signals that are scattered from the one or more of the objects of interest as a result of one or more of the direct signals, and

the existing illumination source is not part of the passive broadband sensing network;

process the received signals to generate data corresponding to one or more of a position parameter and/or a motion parameter associated with the one or more objects of interest by;

pre-process of the received signals to obtain one or more frequency domain signals;

cross-correlation of each direct signal and each echo signal to generate a Doppler spectrum signal;

performance of range compression on the cross-correlated signals in a range compression filter;

feeding of the cross-correlated signals through a beam forming process to obtain a parametric space to derive location and motion information;

phase compensation of the one or more frequency domain signals to generate phase compensated signals; and

derivation of the location and motion information associated with one or more of the objects of interest from the phase compensated signals; and

augment a real scene that includes the one or more objects of interest, by use of one or more of the position parameter and/or the motion parameter to augment the one or more objects of interest in the real scene with at least one virtual object.

8. The apparatus according to claim 7, wherein the wireless communication device is adapted to pre-process the received signals by application of one or more of signal correction, bandwidth correction, signal averaging, amplification, down-conversion, and/or digitization.

9. A method to process signals of one or more of objects of interest in an augmented reality (AR) system, the method comprising:

receiving signals with at least one of a plurality of tracking sensors, wherein;

each of the plurality of tracking sensors form part of a broadband sensing network and utilize passive sensing to analyze the received signals,

a point-to-point communication layer is developed to enable communication and sharing of information between the plurality of tracking sensors in the passive broadband sensing network,

the received signals are scattered by the one or more objects of interest and received through passive receivers,

the received signals correspond to one or more of direct signals from an existing illumination source or echo signals that are scattered by the one or more objects of interest as a result of the direct signals, and

the existing illumination source is not part of the passive broadband sensing network;

applying a beam forming process to the received signals to estimate location and motion parameters for the object of interest; and

augmenting a real scene that includes the one or more objects of interest, by use of one or more of the location and motion parameters to augment the one or more objects of interest in the real scene with at least one virtual object.

10. The method according to claim 9, wherein the beam forming process comprises selecting tracking sensors associated with one or more of transmission sources having distinct frequencies, transmission sources having distinct bandwidths, or transmission sources having distinct antenna patterns, so that diversity gain is increased.

11. The method according to claim 9, wherein the signals are received with tracking sensors that are communicatively coupled through a wireless network distinct from a communication network of corresponding wireless communication devices.

12. The method according to claim 11, further comprising dynamically configuring the wireless network of the tracking sensors to mitigate multipath fading and reduce interference through diversity of antennas of the wireless communication devices.

13. An apparatus to process signals of one or more of objects of interest in an augmented reality (AR) system, the apparatus comprising:

a memory having an AR engine stored herein;

a processor coupled to the memory, the processor configured in cooperation with the memory to execute the AR engine such that the processor is adapted to;

identify signals received with at least one of a plurality of tracking sensors and which are received from an existing illumination source, wherein;

each of the plurality of tracking sensors form part of a passive broadband sensing network and utilize passive sensing to analyze the received signals,

a point-to-point communication layer is developed to enable communication and sharing of information between the plurality of tracking sensors in the passive broadband sensing network,

the received signals are scattered by the one or more objects of interest and received through passive receivers,

the received signals correspond to one or more of direct signals from the existing illumination source or echo signals that are scattered by the one or more objects of interest as a result of the direct signals, and

the existing illumination source is not part of the passive broadband sensing network; and

apply a beam forming process to the received signals to estimate location and motion parameters for the object of interest; and

augment a real scene that includes the one or more objects of interest, by use of one or more of the location and motion parameters to augment the one or more objects of interest in the real scene with at least one virtual object.

14. The apparatus according to claim 13, wherein the beam forming process comprises selecting tracking sensors associated with one or more of transmission sources having distinct frequencies, transmission sources having distinct bandwidths, or transmission sources having distinct antenna patterns, so that diversity gain is increased.

15. The apparatus according to claim 13, wherein the signals are received with tracking sensors that are communicatively coupled through a wireless network distinct from a communication network of corresponding wireless communication devices.

16. The apparatus according to claim 15, wherein the processor is further adapted to dynamically configure the wireless network of the tracking sensors to mitigate multipath fading and reduce interference through diversity of antennas of the wireless communication devices.