Radar imaging on a mobile computing device

• US 10,823,841 B1
• Filed: 04/29/2016
• Issued: 11/03/2020
• Est. Priority Date: 10/06/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lien, Jaime, Poupyrev, Ivan
• Primary Examiner(s): Heard, Erin F
• Assistant Examiner(s): Braswell, Donald H B
• Application Number:
  US15/142,471
• Time in Patent Office: 1,649 Days
• Field of Search: None
• CPC Class Codes: A63F 13/21 characterised by their sensors, purposes or types
• CPC Class Codes: A63F 13/24 Constructional details thereof, e.g. game controllers with detachable joystick handles
• CPC Class Codes: A63F 2300/8082 Virtual reality
• CPC Class Codes: G01S 13/56 for presence detection presence detection using near field arrangements G01V3/00, e.g. G01V3/08, G01V3/12; burglar, theft or intruder alarms with electrical actuation G08B13/22 - G08B13/26
• CPC Class Codes: G01S 13/66 Radar-tracking systems; Analogous systems
• CPC Class Codes: G01S 13/86 Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
• CPC Class Codes: G01S 13/865 Combination of radar systems with lidar systems
• CPC Class Codes: G01S 13/867 Combination of radar systems with cameras
• CPC Class Codes: G01S 13/88 Radar or analogous systems specially adapted for specific applications electromagnetic prospecting or detecting of objects, e.g. near-field detection, G01V3/00
• CPC Class Codes: G01S 13/888 through wall detection
• CPC Class Codes: G01S 13/90 using synthetic aperture techniques , e.g. synthetic aperture radar [SAR] techniques
• CPC Class Codes: G01S 13/904 SAR modes
• CPC Class Codes: G01S 13/931 of land vehicles
• CPC Class Codes: G01S 19/42 Determining position
• CPC Class Codes: G01S 2013/9322 using additional data, e.g. driver condition, road state or weather data
• CPC Class Codes: G01S 7/4004 of parts of a radar system
• CPC Class Codes: G01S 7/41 using analysis of echo signal for target characterisation; Target signature; Target cross-section
• CPC Class Codes: G01S 7/412 based on a comparison between measured values and known or stored values
• CPC Class Codes: G01S 7/415 Identification of targets based on measurements of movement associated with the target
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 16/245 : Query processing
• CPC Class Codes: G06F 18/217 : Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/25 : Fusion techniques
• CPC Class Codes: G06F 18/253 : of extracted features
• CPC Class Codes: G06F 18/28 : Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06F 18/41 : Interactive pattern learning with a human teacher
• CPC Class Codes: G06F 21/32 : using biometric data, e.g. fingerprints, iris scans or voiceprints
• CPC Class Codes: G06F 21/6245 : Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 2203/0384 : Wireless input, i.e. hardware and software details of wireless interface arrangements for pointing devices
• CPC Class Codes: G06F 2221/2105 : Dual mode as a secondary aspect
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 : with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06F 3/04815 : Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 : for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/165 : Management of the audio stream, e.g. setting of volume, audio stream path
• CPC Class Codes: G06N 20/00 : Machine learning
• CPC Class Codes: G06T 7/75 : involving models
• CPC Class Codes: G06V 10/806 : of extracted features
• CPC Class Codes: G06V 20/64 : Three-dimensional objects
• CPC Class Codes: G06V 40/28 : Recognition of hand or arm movements, e.g. recognition of deaf sign language static hand signs G06V40/113
• CPC Class Codes: G08C 17/02 : using a radio link
• CPC Class Codes: G08C 2201/93 : Remote control using other portable devices, e.g. mobile phone, PDA, laptop
• CPC Class Codes: H04Q 2209/883 : where the sensing device enters an active or inactive mode
• CPC Class Codes: H04Q 9/00 : Arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station, in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom
• CPC Class Codes: H04W 16/28 : using beam steering
• CPC Class Codes: H04W 4/80 : Services using short range communication, e.g. near-field communication [NFC], radio-frequency identification [RFID] or low energy communication

Abstract

Systems and methods of capturing images are disclosed. For instance, a plurality of position signals associated with a mobile computing device can be received, the plurality of position signals can be obtained at least in part using one or more sensors implemented within the mobile computing device. A relative motion between the mobile computing device and a scattering point associated with a target can be determined. A plurality of return signals reflected from the scattering point can be received. Each return signal can correspond to a pulse transmitted by the mobile computing device. A target response associated with the scattering point can be determined based at least in part on the relative motion between the mobile computing device and the scattering point.

Claims

1. A method performed by a computing device for creating radar-based images, the computing device comprising a smartphone, tablet computing device, wearable computing device, or laptop computing device the method comprising:

at a first location of the computing device relative to a fixed target;

transmitting, by the computing device, a first radar signal;

receiving, by the computing device, a first return signal caused by a reflection of the first radar signal off the fixed target, the first return signal comprising a plurality of channels of first radar data from a plurality of receive antennas; and

determining, by the computing device, a first target response associated with the fixed target based on the first return signal, the first target response determined by combining the channels of first radar data in a digital domain using a beamforming algorithm to enable spatial discrimination; and

at a second location of the computing device relative to the fixed target, the second location not corresponding to a pre-determined trajectory of the computing device;

transmitting, by the computing device, a second radar signal;

receiving, by the computing device, a second return signal caused by a reflection of the second radar signal off the fixed target, the second return signal comprising a plurality of channels of second radar data from the receive antennas;

determining, by the computing device, a second target response associated with the fixed target based on the second return signal, the second target response determined by combining the channels of second radar data in the digital domain using the beamforming algorithm;

determining, by the computing device, a movement of the computing device relative to the fixed target between the first and second locations based on at least one of a difference between the first and second target responses or sensor data from at least one sensor implemented within the computing device, the movement comprising a change in range to the fixed target and a velocity of the computing device relative to the fixed target;

adjusting, by the computing device, the second target response based on the movement; and

creating, by the computing device, a radar-based image of the fixed target based on the first and adjusted second target responses.

2. The method of claim 1, wherein the millimeter wave radar signals have frequencies between 50 gigahertz (GHz) and 70 GHz.

3. The method of claim 1, wherein the sensor comprises an accelerometer, gyroscope, depth camera, optical camera, or ranging device.

4. The method of claim 1, wherein:

the second radar signal is transmitted via a plurality of transmit antennas; and

the second radar signal is steered towards the fixed target using beamforming techniques.

5. The method of claim 4, wherein the second radar signal is steered based on the movement.

6. The method of claim 4:

further comprising, identifying, by the computing device, a location of the fixed target relative to the computing device based on the first target response; and

wherein the second radar signal is steered based on the identified location of the fixed target and the movement.

7. The method of claim 1, wherein the movement further comprises a change in orientation of the computing device relative to the fixed target.

8. The method of claim 1, wherein the radar-based image is a see-through-the-wall image.

9. The method of claim 1, wherein the radar-based image is a three-dimensional image.

10. The method of claim 1, wherein:

the movement is used to determine a migration of an individual scattering point of the fixed target relative to the computing device; and

the second target response is adjusted based on the migration of the individual scattering point.

11. A computing system comprised by a smartphone, tablet computing device, wearable computing device, or laptop computing device, the computing system comprising:

at least one processor;

at least one radar transmitter;

at least one radar receiver;

a plurality of receive antennas;

at least one sensor; and

at least one memory device, the memory device storing computer-readable instructions that when executed by the processor cause the processor to;

at a first location of the computing system relative to a fixed target;

cause the radar transmitter to transmit a first radar signal;

receive, via the radar receiver, a first return signal caused by a reflection of the first radar signal off the fixed target, the first return signal comprising channels of first radar data from each of the receive antennas; and

determine a first target response associated with the fixed target based on the first return signal, the first target response determined by combining the channels of first radar data in a digital domain using a beamforming algorithm to enable spatial discrimination; and

at a second location of the computing system relative to the fixed target, the second location not corresponding to a pre-determined trajectory of the computing system;

cause the radar transmitter to transmit a second radar signal;

receive, via the radar receiver, a second return signal caused by a reflection of the second radar signal off the fixed target, the second return signal comprising channels of second radar data from each of the receive antennas;

determine a second target response associated with the fixed target based on the second return signal, the second target response determined by combining the channels of second radar data in the digital domain using the beamforming algorithm;

determine a movement of the computing system relative to the fixed target between the first and second locations based on at least one of a difference between the first and second target responses or sensor data from the at least one sensor, the movement comprising a change in range to the fixed target and a velocity of the computing device relative to the fixed target;

adjust the second target response based on the movement; and

create a radar-based image of the fixed target based on the first and adjusted second target responses.

12. The computing system of claim 11, wherein the millimeter wave radar signals have frequencies between 50 gigahertz (GHz) and 70 GHz.

13. The computing system of claim 11, wherein the sensor comprises an accelerometer, gyroscope, depth camera, optical camera, or ranging device.

14. The computing system of claim 11, wherein:

the computing system further comprises a plurality of transmit antennas;

the second radar signal is transmitted via the plurality of transmit antennas; and

the second radar signal is steered using beamforming techniques.

15. The computing system of claim 14, wherein the second radar signal is steered towards the fixed target.

16. The computing system of claim 15, wherein the second radar signal is steered based on the movement.

17. The computing system of claim 15, wherein:

the instructions further cause the processor to identify a location of the fixed target relative to the computing system based on the first target response; and

the second radar signal is steered based on the identified location of the fixed target and the relative movement.

18. The computing system of claim 11, wherein the movement further comprises a change in orientation of the computing device relative to the fixed target.

19. The computing system of claim 11, wherein the radar-based image is one or more of a three-dimensional image or a see-through-the-wall image.

20. The computing system of claim 11, wherein:

the movement is used to determine a migration of an individual scattering point of the fixed target relative to the computing device; and

the second target response is adjusted based on the migration of the individual scattering point.