Broadband passive tracking for augmented reality

• US 8,610,771 B2
• Filed: 03/08/2010
• Issued: 12/17/2013
• Est. Priority Date: 03/08/2010
• Status: Expired due to Fees

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Leung, Henry, Liu, Xiaoxiang
• Primary Examiner(s): Dastouri, Mehrdad
• Assistant Examiner(s): RAHAMAN, MOHAMMED S
• Application Number:
  US12/719,797
• Publication Number:
  US 20110216192A1
• Time in Patent Office: 1,380 Days
• Field of Search: 348/143, 348/E5.024, 348/E7.085, 348/169
• US Class Current: 348/143
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Technologies are generally described for a broadband passive sensing and tracking system that may employ a number of passive receivers that each have the capability of sensing electromagnetic waves (e.g., Radio Frequency “RF” signals) from surrounding broadcast sources. Each passive receiver may be adapted to sense through one or more antennas. Multiple receivers at different positions may be utilized to form a broadband sensing network adapted to perform collaborative tracking of a scene of interest. According to some examples, a beam-forming algorithm may be applied over the broadband sensing network utilizing an antenna array formed by the passive receivers to localize and track objects.

Claims

1. A method to track one or more objects of interest in an Augmented Reality (AR) system, the method comprising:

receiving signals at each one of a plurality of tracking sensors, wherein

each of the plurality of tracking sensors form part of a passive broadband sensing network,

the passive broadband sensing network includes a plurality of passive receivers arranged at different positions in a scene and adapted to perform collaborative tracking of the scene by sensing signals from a plurality of transmission sources,

the received signals correspond to one or more of a direct signal and/or an echo signal,

each echo signal is associated with a corresponding one of the direct signals that is scattered by an object of interest, and

the direct signals are transmitted from transmission sources that include one or more of;

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points;

processing the received signals to generate data associated with the object of interest through one or more of determining a location of one or more feature points in an image, computation of affine projections, tracking of edges, and filtering, and wherein the data corresponds to one or more of a position parameter and/or a motion parameter associated with the object of interest;

developing a point-to-point communication layer to enable communication and sharing of information between any two sensors in the passive broadband sensing network; and

augmenting the scene that includes the object of interest, by employing one or more of the position parameter and/or the motion parameter to augment the object of interest in the scene with at least one virtual object.

2. The method according to claim 1, wherein the signals are received at the plurality of tracking sensors integrated into a plurality of wireless communication devices communicating with the plurality of transmission sources.

3. The method according to claim 2, wherein the received signals correspond to one or more of:

a TV signal, a digital TV signal, a GSM signal, a CDMA signal, an AM/FM signal, a DAB signal, a DVB-T signal, a WLAN signal, a WAN signal, a MAN signal, and/or a PAN signal.

4. The method according to claim 1, wherein processing the received signals includes one or more of

pre-processing the received signals;

identifying each of the received signals as one or more of the direct signal and/or the echo signal through the filtering, which includes adaptive filtering;

deriving Doppler spectra of the direct signal and the echo signal; and

/or

phase compensating the Doppler spectra of the echo signals.

5. The method according to claim 4, wherein pre-processing the received signals includes one or more of signal correction, bandwidth correction, signal averaging, amplification, down-conversion, and/or digitization.

6. The method according to claim 4, wherein processing the received signals to generate data associated with the object of interest comprises employing a beam forming process on the phase compensated signals to determine one or more of position and/or motion parameters associated with the object of interest.

7. The method according to claim 6, wherein employing the beam forming process further comprises

adaptively selecting a number and a location of the tracking sensors.

8. The method according to claim 1, wherein the signals are received from the tracking sensors that are communicatively coupled through a wireless network distinct from a communication network of corresponding wireless communication devices.

9. The method according to claim 8, further comprising

dynamically configuring the wireless network of the tracking sensors to mitigate multipath fading and reduce interference through diversity of antennas of the wireless communication devices.

10. A system to track an object of interest in a real scene in Augmented Reality (AR) systems, the system comprising:

a plurality of tracking sensors integrated into a plurality of handheld or head-worn wireless communication devices in communication with a plurality of transmission sources as part of a passive broadband sensing network that includes a plurality of passive receivers arranged at different positions adapted to perform collaborative tracking of the scene by sensing signals from the plurality of transmission sources, each tracking sensor adapted to;

receive, through downlink communication channels, signals corresponding to one or more of a direct signal and/or an echo signal, wherein the direct signals are associated with one or more corresponding transmission sources that include one or more of;

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points, and wherein the echo signals are associated with the one or more direct signals scattered from the object of interest; and

pre-process the received signals at one or more of the wireless communication devices with active and passive bandwidth filters;

an image processing server adapted to derive image information based on a captured 2D image associated with the object of interest;

a reality server adapted to;

enable communication and information sharing between any two tracking sensors in the passive broadband sensing network through a point-to-point communication layer;

communicate with the plurality of tracking sensors through a wireless network;

receive the pre-processed signals;

track the object of interest based on the pre-processed signals; and

augment the real scene based on the tracked object of interest and the image information.

11. The system according to claim 10, wherein each tracking sensor comprises:

a blind channel equalizer adapted to process received direct signals to mitigate multipath interference;

an adaptive filter adapted to process received echo signals to suppress clutter; and

a correlator adapted to cross-correlate each direct signal and each echo signal to generate a Doppler spectrum signal.

12. The system according to claim 11, wherein one or more functions of the blind channel equalizer, the adaptive filter, and/or the correlator are provided by a digital signal processor.

13. The system according to claim 10, wherein the reality server comprises:

a phase compensation module adapted to phase compensate the pre-processed signals received from the tracking sensor;

a tracking/localization module adapted to determine one or more of-position and/or motion parameters for the object of interest based on the phase compensated signals;

a reality engine adapted to generate one or more virtual objects based on position/motion information of the object of interest and the image information; and

an image generator adapted to augment the real scene with the one or more virtual objects.

14. The system according to claim 10, further comprising a visualization device adapted to generate a visualization of the augmented real scene.

15. The system according to claim 14, wherein the visualization device comprises one or more of a head-mounted display, a virtual display, and/or a monitor.

16. The system according to claim 10, wherein a portion of the tracking sensors include homogeneous sensors adapted to receive a same type transmission and another portion of the tracking sensors include heterogeneous sensors adapted to receive different types of transmissions.

17. The system according to claim 10, wherein the reality server is further adapted to calibrate a synthesized array of antennas, the synthesized array of antennas comprising antennas from each of the tracking sensors, each of the tracking sensors being adapted to employ a location based service.

18. An apparatus to track one or more objects of interest in an Augmented Reality (AR) system, the apparatus comprising:

a wireless communication device; and

an antenna and a tracking sensor included in the wireless communication device, wherein the wireless communication device is adapted to;

receive signals with the antenna, wherein the received signals correspond to one or more of direct signals from one or more transmission sources or echo signals that are scattered from one or more of the objects of interest as a result of one or more of the direct signals, the direct signals transmitted from the one or more transmission sources that include one or more of;

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points;

pre-process the received signals to obtain one or more frequency domain signals;

cross-correlate each direct signal and each echo signal to generate a Doppler spectrum signal;

perform range compression on the cross-correlated signals in a range compression filter;

feed the cross-correlated signals through a beam forming process to obtain a parametric space to derive location and motion information;

phase compensate the one or more frequency domain signals to generate phase compensated signals;

enable communication and information sharing between any two sensors in the passive broadband sensing network through a point-to-point communication layer, wherein the tracking sensor included in the wireless communication device forms the passive broadband sensing network with other tracking sensors included in other wireless communication devices;

derive the location and motion information associated with one or more of the objects of interest from the phase compensated signals; and

augment a real scene that includes the one or more objects of interest, by employing the location and motion parameters to augment the one or more objects of interest in the real scene with at least one virtual object.

19. The apparatus according to claim 18, wherein the wireless communication device is adapted to pre-process the received signals by application of one or more of signal correction, bandwidth correction, signal averaging, amplification, down-conversion, and/or digitization.

20. The apparatus according to claim 18, wherein the wireless communication device includes one or more of:

a TV receiver, a digital TV receiver, a GSM device, a CDMA device, an AM/FM receiver, a DAB device, a DVB-T device, a WLAN device, a WAN device, a MAN device, and/or a PAN device.

21. An apparatus to track one or more objects of interest in Augmented Reality (AR) systems, the apparatus comprising:

a memory;

a processor coupled to the memory, the processor configured in cooperation with the memory to execute an AR engine such that the processor is adapted to perform or cause to be performed;

communicate with a plurality of tracking sensors integrated into handheld and head-worn AR enabled wireless devices through a wireless network that is distinct from communication networks of corresponding AR enabled wireless devices;

adaptively select a number and a location of the plurality of tracking sensors, wherein the plurality of tracking sensors form a passive broadband sensing network that includes a plurality of passive receivers arranged at different positions and is adapted to perform collaborative tracking of a real scene by sensing signals from a plurality of transmission sources;

receive, through downlink communication channels, pre-processed signals that are detected by the plurality of tracking sensors, wherein the pre-processed signals correspond to one or more of direct signals and/or echo signals that result from the direct signals being reflected by the one or more objects of interest, and wherein the direct signals are transmitted from the plurality of transmission sources that include one or more of;

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points;

cross-correlate each direct signal and each echo signal to generate a Doppler spectrum signal;

perform range compression on the cross-correlated signals in a range compression filter;

apply a beam forming process to the received cross-correlated signals to obtain a parametric space to estimate location and motion parameters for one or more of the objects of interest based on the parametric space;

develop a point-to-point communication layer to enable communication and information sharing between any two sensors in the passive broadband sensing network; and

augment the real scene that includes the one or more objects of interest, by employing the location and motion parameters to augment the one or more objects of interest in the real scene with at least one virtual object.

22. The apparatus according to claim 21, wherein the processor is further configured by the beam forming process to select tracking sensors associated with one or more of the plurality of transmission sources having distinct frequencies, transmission sources having distinct bandwidths, and/or transmission sources having distinct antenna patterns, so that diversity gain is increased.

23. The apparatus according to claim 21, wherein the processor is further adapted by the AR engine to provide the estimated location and motion parameters for the one or more objects of interest to an AR application, wherein the AR application is configured to generate an augmented real scene by superimposition of virtual objects generated based on the estimated location and motion parameters for at least one of the objects of interest and a digitized image of at least one of the objects of interest.

24. A computer-readable memory device having instructions stored thereon to track one or more objects of interest in Augmented Reality (AR) systems, the instructions comprising:

at a tracking sensor included in a handheld or head-worn wireless device, detecting one or more of direct signals received from one or more transmission sources or echo signals that are scattered from one or more of the objects of interest as a result of one or more of the direct signals, the direct signals transmitted from the one or more transmission sources that include one or more of;

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points;

pre-processing the received signals at the tracking sensor, wherein pre-processing the received signals includes determining a location of one or more feature points in an image, computation of affine projections, tracking of edges, and active and passive bandwidth filtering;

enabling communication and information sharing between any two sensors in the passive broadband sensing network through a point-to-point communication layer, wherein the tracking sensor included in the wireless device forms the passive broadband sensing network with other tracking sensors included in other wireless devices;

deriving a frequency spectrum signal based on cross-correlating each pre-processed direct and echo signal;

performing range compression on the cross-correlated signals in a range compression filter;

applying a beam forming process to the received cross-correlated signals to obtain a parametric space to estimate location and motion parameters for one or more of the objects of interest;

phase compensating pairs of frequency spectrum signals received from a plurality of tracking sensors at a phase-compensation module;

deriving location and/or motion information associated with one or more of the objects of interest from the phase compensated signals at a tracking/localization module by adaptively selecting a number and a location of the plurality of tracking sensors; and

augmenting a real scene that includes the one or more objects of interest, by employing the derived location and motion information to augment the one or more objects of interest in the real scene with at least one virtual object.

25. The computer-readable memory device of claim 24, wherein the instructions further comprise:

applying blind channel equalization to the received direct signals to mitigate multipath interference; and

adaptively filtering the received echo signals to suppress clutter.

26. The computer-readable memory device of claim 24, wherein the instructions further comprise:

transmitting the location and/or motion information to a wireless device executing an AR client application.

27. The computer-readable memory device of claim 24, wherein the instructions further comprise:

dynamically configuring a network of the wireless devices to mitigate multipath fading and reduce interference through diversity of antennas of the wireless devices.