Smartphone-based power-efficient radar processing and memory provisioning for detecting gestures

• US 10,794,997 B2
• Filed: 08/21/2018
• Issued: 10/06/2020
• Est. Priority Date: 08/21/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Amihood, Patrick M., Shah, Abhijit Aroon, Bors, Emanoil Felician
• Primary Examiner(s): Gregory, Bernarr E
• Application Number:
  US16/106,315
• Publication Number:
  US 20200064445A1
• Time in Patent Office: 777 Days
• CPC Class Codes: G01S 13/58 Velocity or trajectory determination systems; Sense-of-movement determination systems
• CPC Class Codes: G01S 7/414 Discriminating targets with respect to background clutter
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1684 Constructional details or arrangements related to integrated I/O peripherals not covered by groups G06F1/1635 - G06F1/1675
• CPC Class Codes: G06F 1/1686 the I/O peripheral being an integrated camera
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3275 Power saving in memory, e.g. RAM, cache
• CPC Class Codes: G06F 1/3293 by switching to a less power-consuming processor, e.g. sub-CPU
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: H04M 1/0202 Portable telephone sets, e.g. cordless phones, mobile phones or bar type handsets constructional features of telephone transmitters or receivers, e.g. of speakers or microphones H04M1/03
• CPC Class Codes: H04W 52/0261 managing power supply demand, e.g. depending on battery level
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management
• CPC Class Codes: Y02D 30/70 in wireless communication networks

Abstract

Techniques and apparatuses are described that implement smartphone-based power-efficient radar processing and memory provisioning for detecting gestures. The described techniques map different situations that occur with a user to different memory states. A radar processing system'"'"'s memory management module allocates at least one memory pool according to an active memory state. As the radar system detects different situations that occur with the user, the memory management module reallocates the memory pool for the appropriate memory state, which can adjust a power mode of the radar system. In some cases, physically separate memories may be allocated for different memory states, which respectively include one or more sequences that are executed by different processors. The memory management module enables efficient use of available power and available memory for radar processing such that a total amount of power and a total size of memory used may be significantly reduced for detecting gestures.

Claims

1. A smartphone comprising:

a radar system configured to;

transmit and receive multiple radar signals, at least one radar signal of the multiple radar signals reflected by a user of the smartphone; and

detect, based on the multiple radar signals, different situations associated with the user, the different situations including;

a first situation in which the user is outside a detectable range of the radar system; and

a second situation in which the user performs a gesture, the radar system including;

at least one processor; and

at least one computer-readable storage medium comprising computer-executable instructions that, responsive to execution by the at least one processor, is configured to;

create a memory pool within the at least one computer-readable storage medium; and

adjust an allocation of memory within the memory pool according to different memory states that are respectively activated at different times according to the different situations that are detected by the radar system, the different memory states including;

a first memory state that is activated responsive to detection of the first situation; and

a second memory state that is activated responsive to detection of the second situation.

2. The smartphone of claim 1, wherein:

the at least one computer-readable storage medium is configured to;

allocate at least a portion of the memory pool for a first sequence of the first memory state while the first memory state is active for the first situation; and

allocate at least another portion of the memory pool for a second sequence of the second memory state while the second memory state is active for the second situation; and

the at least one processor is configured to;

execute the first sequence while the first memory state is active to cause the radar system to conserve power; and

execute the second sequence while the second memory state is active to recognize the gesture performed by the user.

3. The smartphone of claim 2, wherein the radar system is configured to:

use a high duty cycle responsive to the second sequence being executed by the at least one processor; and

use a low duty cycle responsive to the first sequence being executed by the at least one processor, the low duty cycle reducing power consumption of the radar system relative to the use of the high duty cycle.

4. The smartphone of claim 2, wherein:

the second sequence comprises a swipe gesture detection sequence and a reach gesture detection sequence;

the memory management module is configured to allocate at least the other portion of the memory pool for the swipe gesture detection sequence and the reach gesture detection sequence while the second memory state is active for the second situation; and

the at least one processor is configured to concurrently execute both the swipe gesture detection sequence and the reach gesture detection sequence while the second memory state is active to determine whether the gesture performed by the user comprises a swipe gesture or a reach gesture.

5. The smartphone of claim 4, wherein the at least one computer-readable storage medium is configured to:

allocate a static memory block within the memory pool to enable the static memory block to be shared by the swipe gesture detection sequence and the reach gesture detection sequence; and

allocate a dynamic memory block within the memory pool to enable data within the dynamic memory block to be passed between at least two compute blocks of the swipe gesture detection sequence and to be passed between at least two other compute blocks of the reach gesture detection sequence.

6. The smartphone of claim 5, wherein the at least one computer-readable storage medium is further configured to:

allocate the static memory block to store a clutter map that is shared by the swipe gesture detection sequence and the reach gesture detection sequence; and

allocate the dynamic memory block to enable a first range-Doppler map to be passed between the at least two compute blocks of the swipe gesture detection sequence and a second range-Doppler map to be passed between the at least two other compute blocks of the reach gesture detection sequence.

7. The smartphone of claim 1, wherein the at least one computer-readable storage medium is configured to allocate a global memory block within the at least one computer-readable storage medium and outside of the memory pool to enable the global memory block to be shared by the different memory states.

8. The smartphone of claim 7, wherein the at least one computer-readable storage medium is further configured to allocate the global memory block to store a clutter map that is shared by the first memory state and the second memory state.

9. The smartphone of claim 1, further comprising another computer-readable storage medium, wherein:

the at least one computer-readable storage medium is configured to;

instantiate another memory pool within the other computer-readable storage medium; and

adjust an allocation of memory within the other memory pool according to other memory states that are respectively activated according to other situations that occur at other times.

10. The smartphone of claim 9, wherein the other computer-readable storage medium and the at least one computer-readable storage medium are physically separate memories.

11. The smartphone of claim 9, wherein:

the at least one computer-readable storage medium is configured to;

activate a third memory state of the other memory states responsive to a third situation of the other situations; and

allocate at least a portion of the other memory pool for one or more sequences of the third memory state while the third memory state is active for the third situation; and

the at least one processor is configured to execute the one or more sequences of the third memory state while the third memory state is active.

12. The smartphone of claim 11, wherein the radar system includes another processor configured to execute, while the first memory state is active, a first sequence that detects a transition from the first situation to the third situation and causes the at least one computer-readable storage medium to activate the third memory state.

13. The smartphone of claim 1, wherein:

the at least one computer-readable storage medium is configured to allocate at least a portion of the memory pool for multiple sequences of the first memory state that process radar data generated by the radar system; and

the at least one processor is configured to execute the multiple sequences in series or in parallel.

14. A computer-readable storage medium comprising computer-executable instructions that, responsive to execution by a processor, is configured to:

create a memory pool within the computer-readable storage medium to allocate a portion of the computer-readable storage medium for processing radar data;

allocate memory within the memory pool according to a first memory state, the first memory state including one or more first sequences for processing the radar data;

accept the radar data from a transceiver of a radar system, the radar data associated with a first reflected radar signal;

process, using the first memory state, the radar data associated with the first reflected radar signal by executing the one or more first sequences;

determine, based on the executing of the one or more first sequences, that a user is present within a detectable range of the radar system; and

adjust the allocation of memory within the memory pool by allocating the memory within the memory pool according to a second memory state, the second memory state including one or more second sequences for monitoring the user based on subsequent radar data provided by the radar system.

15. The computer-readable storage medium of claim 14, configured to selectively activate:

an idle state of the different memory states responsive to a first situation of the different situations that includes the user being outside a detectable range of the radar system;

a presence state of the different memory states responsive to a second situation of the different situations that includes the user being within the detectable range of the radar system and located at a range that is beyond a range threshold;

an awareness state of the different memory states responsive to a third situation of the different situations that includes the user being within the detectable range of the radar system and located at another range that is within the range threshold; and

an engagement state of the different memory states responsive to a fourth situation of the different situations that includes the user being within the detectable range of the radar system and performing a gesture.

16. The computer-readable storage medium of claim 15, wherein the memory management module is configured to allocate, while the idle state is active, at least a portion of the memory pool for a presence detection sequence of the idle state that, responsive to execution by the processor, is configured to:

cause the radar system to operate in a low-power mode to collect the radar data;

analyze the radar data to detect a presence of the user responsive to the user entering the detectable range of the radar system; and

cause the memory management module to activate the presence state responsive to the user entering the detectable range.

17. The computer-readable storage medium of claim 15, wherein the memory management module is configured to allocate, while the presence state is active, at least a portion of the memory pool for a presence monitoring sequence of the presence state that, responsive to execution by the processor, is configured to:

cause the radar system to operate in a medium-low power mode to collect the radar data;

analyze the radar data to determine the range to the user or to detect the user exiting the detectable range of the radar system;

cause the memory management module to activate the awareness state responsive to the range to the user being less than the range threshold; and

cause the memory management module to activate the idle state responsive to the user exiting the detectable range.

18. The computer-readable storage medium of claim 15, wherein the memory management module is configured to allocate, while the awareness state is active, at least a portion of the memory pool for a low-power gesture detection sequence of the awareness state that, responsive to execution by the processor, is configured to:

cause the radar system to operate in a medium-high power mode to collect the radar data;

analyze the radar data to determine the user is in the process of preparing to perform the gesture; and

cause the memory management module to activate the engagement state responsive to determination.

19. The computer-readable storage medium of claim 15, wherein the memory management module is configured to allocate, while the engagement state is active, at least a portion of the memory pool for at least one high-power gesture detection sequence of the engagement state that, responsive to execution by the processor, is configured to:

cause the radar system to operate in a high-power mode to collect the radar data;

analyze the radar data to recognize the gesture performed by the user; and

cause the memory management module to activate the awareness state responsive to recognizing the gesture.

20. A method comprising:

transmitting and receiving multiple radar signals using a radar system, at least one radar signal of the multiple radar signals reflected by a user;

creating a memory pool within at least one computer-readable storage medium;

detecting, by the radar system and based on at least a portion of the multiple radar signals, a first situation in which the user is outside a detectable range of the radar system;

activating a first memory state responsive to the detecting of the first situation, the activating of the first memory state comprising allocating memory within the memory pool according to the first memory state;

processing, based on the first memory state, radar data generated by the radar system, the processing utilizing the memory pool;

detecting, by the radar system and based on at least another portion of the multiple radar signals, a second situation in which the user is inside the detectable range of the radar system;

activating a second memory state responsive to the detection of the second situation, the activating of the second memory state comprising reallocating the memory within the memory pool according to the second memory state; and

processing, based on the second memory state, the radar data generated by the radar system, the processing reusing at least a portion of the memory pool.