BROADBAND PASSIVE TRACKING FOR AUGMENTED REALITY

• US 20110216192A1
• Filed: 03/08/2010
• Published: 09/08/2011
• Est. Priority Date: 03/08/2010
• Status: Active Grant

• Current Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Original Assignee: Empire Technology Development LLC (Allied Inventors Management, LLC)
• Inventors: Liu, Xiaoxiang, Leung, Henry
• Granted Patent: US 8,610,771 B2
• Time in Patent Office: Days
• US Class Current: 348/143
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: H04N 23/00 Cameras or camera modules comprising electronic image sensors; Control thereof
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 7/181 for receiving images from a plurality of remote sources

Abstract

Technologies are generally described for a broadband passive sensing and tracking system that may employ a number of passive receivers that each have the capability of sensing electromagnetic waves (e.g., Radio Frequency “RF” signals) from surrounding broadcast sources. Each passive receiver may be adapted to sense through one or more antennas. Multiple receivers at different positions may be utilized to form a broadband sensing network adapted to perform collaborative tracking of a scene of interest. According to some examples, a beam-forming algorithm may be applied over the broadband sensing network utilizing an antenna array formed by the passive receivers to localize and track objects.

Claims

1. A method for tracking one or more objects of interest in an Augmented Reality (AR) systems the method comprising:

receiving signals at each one of a plurality of tracking sensors, wherein each of the plurality of tracking sensors form part of a broadband sensing network, the received signals correspond to one or more of a direct signal and/or an echo signal, and each echo signal is associated with a corresponding one of the direct signals that is scattered by the object of interest;

processing the received signals to generate data associated with the object of interest, wherein the data corresponds to one or more of a position parameter and/or a motion parameter associated with an object of interest; and

augmenting a real scene with employing one or more of the position parameter and/or the motion parameter to augment a real scene including the object of interest with at least one virtual object.

2. The method according to claim 1, wherein the signals are received at the plurality of tracking sensors integrated into a plurality of wireless communication devices communicating with a plurality of transmission sources.

3. The method according to claim 2, wherein the direct signals are transmitted from transmission sources that include one or more of:

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points.

4. The method according to claim 2, wherein the received signals correspond to one or more of:

a TV signal, a digital TV signal, a GSM signal, a CDMA signal, an AM/FM signal, a DAB signal, a DVB-T signal, a WLAN signal, a WAN signal, a MAN signal, and/or a PAN signal.

5. The method according to claim 1, wherein processing the received signals includes one or more of

pre-processing the received signals;

identifying each of the received signals as one or more of the direct signal and/or the echo signal through adaptive filtering;

deriving Doppler spectra of the direct signal and the echo signal; and

/or

phase compensating the Doppler spectra of the echo signals.

6. The method according to claim 5, wherein pre-processing the received signals includes one or more of signal correction, bandwidth correction, signal averaging, amplification, down-conversion, and/or digitization.

7. The method according to claim 5, wherein processing the received signals to generate data associated with the object of interest comprises employing a beam forming process on the phase compensated signals to determine one or more of position and/or motion parameters associated with the object of interest.

8. The method according to claim 7, wherein employing the beam forming process further comprises

adaptively selecting a number and a location of the tracking sensors.

9. The method according to claim 1, wherein the signals are received from the tracking sensors that are communicatively coupled through a wireless network distinct from a communication network of corresponding wireless communication devices.

10. The method according to claim 9, further comprising

dynamically configuring the wireless network of the tracking sensors to mitigate multipath fading and reduce interference through diversity of antennas of the wireless communication devices.

11. A system for tracking an object of interest in a real scene in Augmented Reality (AR) systems, the system comprising:

a plurality of tracking sensors, each tracking sensor adapted to;

receive signals corresponding to one or more of a direct signal and/or an echo signal, wherein the direct signals are associated with one or more corresponding transmission sources, and wherein the echo signals are associated with the one or more direct signals scattered from the object of interest; and

pre-process the received signals;

an image processing server adapted to derive image information based on a captured 2D image associated with the object of interest;

a reality server adapted to;

communicate with the plurality of tracking sensors through a wireless network;

receive the pre-processed signals;

track the object of interest based on the pre-processed signals; and

augment the real scene based on the tracking of the object of interest and the image information.

12. The system according to claim 11, wherein each tracking sensor comprises:

a blind channel equalizer adapted to process received direct signals to mitigate multipath interference;

an adaptive filter adapted to process received echo signals to generate suppress clutter; and

a correlator adapted to cross-correlate each direct signal and each echo signal to generate a Doppler spectrum signal.

13. The system according to claim 12, wherein one or more functions of the blind channel equalizer, the adaptive filter, and/or the correlator are provided by a digital signal processor.

14. The system according to claim 11, wherein the reality server comprises:

a phase compensation module adapted to phase compensate the pre-processed signals received from the tracking sensor;

a tracking/localization module adapted to determine one or more of a position and/or a motion parameters for the object of interest based on the phase compensated signals;

a reality engine adapted to generate one or more virtual objects based on position/motion information of the object of interest and the image information; and

an image generator adapted to augment the real scene with the one or more virtual objects.

15. The system according to claim 11, further comprising a visualization device adapted to generate a visualization of the augmented real scene.

16. The system according to claim 15, wherein the visualization device comprises one or more of a head-mounted display, a virtual retinal display, and/or a monitor.

17. The system according to claim 11, wherein a portion of the tracking sensors are homogeneous sensors adapted to receive the same type transmission and another portion of the tracking sensors are heterogeneous sensors adapted to receive different types of transmissions.

18. The system according to claim 11, wherein the reality server is further adapted to calibrate a synthesized array of antennas, the synthesized array of antennas comprising antennas from each of the tracking sensors, each of the tracking sensors being adapted to employ a location based service.

19. An apparatus for tracking one or more objects of interest in an Augmented Reality (AR) system, the apparatus comprising a wireless communication device including an antenna and a tracking sensor wherein the wireless communication device is adapted to:

receive signals with the antenna, wherein the received signals correspond to one or more of direct signals from one or more transmission sources or echo signals that are scattered from one or more of the objects of interest as a result of one or more of the direct signals;

pre-process the received signals to obtain one or more frequency domain signals;

phase compensate the one or more frequency domain signals to generate phase compensated signals; and

derive location and/or motion information associated with one or more of the objects of interest from the phase compensated signals.

20. The apparatus according to claim 19, wherein the wireless communication device is adapted to pre-process the received signals by applying one or more of signal correction, bandwidth correction, signal averaging, amplification, down-conversion, and/or digitization.

21. The apparatus according to claim 19, wherein the transmission sources comprise one or more of:

TV broadcast towers, Global System for Mobile communications (GSM) towers, Code Division Multiple Access (CDMA) cellular communication towers, Amplitude Modulation (AM) or Frequency Modulation (FM) broadcast towers, Digital Audio Broadcasting (DAB) sources, Digital Video Broadcasting-Terrestrial (DVB-T) sources, Wireless Local Area Network (WLAN) access points, Wide Area Network (WAN) access points, Metropolitan Area Network (MAN) access points, and/or Personal Area Network (PAN) access points; and

wherein the wireless communication device includes one or more of;

a TV receiver, a digital TV receiver, a GSM device, a CDMA device, an AM/FM receiver, a DAB device, a DVB-T device, a WLAN device, a WAN device, a MAN device, and/or a PAN device.

22. An apparatus for tracking one or more objects of interest in Augmented Reality (AR) systems, the apparatus comprising:

a memory;

a processor coupled to the memory, the processor configured in cooperation with the memory to execute an AR engine such that the processor is adapted to;

communicate with a plurality of tracking sensors integrated into AR enabled wireless devices through a wireless network that is distinct from communication networks of corresponding AR enabled wireless devices;

adaptively select a number and a location of the tracking sensors;

receive pre-processed signals that are detected by the tracking sensors wherein the pre-processed signals correspond to one or more of direct signals and/or echo signals that result from the direct signals being reflected by the one or more objects of interest; and

apply a beam forming process to the received signals to estimate location and motion parameters for one or more of the objects of interest.

23. The apparatus according to claim 22, wherein the processor is further configured by the beam forming process to select tracking sensors associated with one or more of transmission sources having distinct frequencies, transmission sources having distinct bandwidths, and/or transmission sources having distinct antenna patterns, whereby diversity gain may be increased.

24. The apparatus according to claim 22, wherein the processor is further adapted by the AR engine to provide the estimated location and motion parameters for the one or more objects of interest to an AR application, wherein the AR application is configured to generate and augmented reality scene by superimposing virtual objects generated based on the estimated location and motion parameters for the object of interest and a digitized image of the object of interest.

25. A computer-readable storage medium having instructions stored thereon for tracking one or more objects of interest in Augmented Reality (AR) systems, the instructions comprising:

at a tracking sensor receiving from a transmission source associated with a wireless device that includes the tracking sensor, one or more of direct signals from one or more transmission sources or echo signals that are scattered from one or more of the objects of interest as a result of one or more of the direct signals;

pre-processing the received signals at the tracking sensor;

deriving a frequency spectrum signal based on cross-correlating each pre-processed direct and echo signal;

phase compensating pairs of frequency spectrum signals received from a plurality of tracking sensors at a phase-compensation module; and

deriving location and/or motion information associated with one or more of the objects of interest from the phase compensated signals at a tracking/localization module by adaptively selecting a number and a location of the plurality of tracking sensors.

26. The computer-readable storage medium of claim 25, wherein the instructions further comprise:

applying blind channel equalization to the received direct signals to mitigate multipath interference; and

adaptively filtering the received echo signals to generate suppress clutter.

27. The computer-readable storage medium of claim 25, wherein the instructions further comprise:

transmitting the tracking information to a wireless device executing an AR client application.

28. The computer-readable storage medium of claim 25, wherein the instructions further comprise:

dynamically configuring a network of the wireless devices to mitigate multipath fading and reduce interference through diversity of antennas of the wireless devices.