Sensor based component activation

• US 10,484,530 B2
• Filed: 11/07/2017
• Issued: 11/19/2019
• Est. Priority Date: 11/07/2017
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Wantland, Tim
• Primary Examiner(s): Gonzales, April G
• Application Number:
  US15/805,886
• Publication Number:
  US 20190141181A1
• Time in Patent Office: 742 Days
• Field of Search: 4555561
• CPC Class Codes: G06F 1/1626 with a single-body enclosure integrating a flat display, e.g. Personal Digital Assistants [PDAs]
• CPC Class Codes: G06F 1/1694 the I/O peripheral being a single or a set of motion sensors for pointer control or gesture input obtained by sensing movements of the portable computer
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 1/3287 by switching off individual functional units in the computer system
• CPC Class Codes: G06F 2200/1637 Sensing arrangement for detection of housing movement or orientation, e.g. for controlling scrolling or cursor movement on the display of an handheld computer
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: H04M 1/72454 according to context-related or environment-related conditions
• CPC Class Codes: H04M 2250/12 including a sensor for measuring a physical value, e.g. temperature or motion
• CPC Class Codes: H04M 2250/22 including a touch pad, a touch sensor or a touch detector
• CPC Class Codes: H04M 2250/52 including functional features of a camera
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Provided are methods, systems, and devices for activating components of a computing device based on a mode associated with the state of objects including a computing device. Sensor data based sensor outputs from one or more sensors that detect one or more objects can be received. Based in part on the sensor data, the state of the one or more objects can be determined. The state of the one or more objects can include one or more spatial relations of the one or more objects with respect to a computing system. When the state of the one or more objects satisfies one or more operational criteria, a mode associated with activating a component of the computing system can be activated. The component can include a microphone component, a camera component, or a text input component. The component can be associated with performing including input detection.

Claims

1. A computer-implemented method of mode determination, the method comprising:

receiving, by a computing system comprising one or more computing devices, object detection data based in part on one or more sensor outputs from a plurality of sensors that detect one or more portions of a user of the computing system;

receiving, by the computing system, motion data based in part on one or more sensor outputs from the plurality of sensors, wherein the motion data describes a motion of the computing system relative to the user of the computing system;

determining, by the computing system, based in part on the object detection data, a state of the one or more portions of the user of the computing system comprising one or more spatial relations of the one or more portions of the user of the computing system with respect to the computing system;

determining, by the computing system, based in part on the object detection data and the motion data, when the state of the one or more portions of the user relative to the computing system matches one or more falsing profiles associated with a state of the one or more portions of the user relative to the computing system that does not activate the one or more components, wherein the satisfying the one or more operational criteria comprises the state of the one or more portions of the user relative to the computing system not matching any of the one or more falsing profiles; and

responsive to the state of the one or more portions of the user of the computing system satisfying one or more operational criteria, activating, by the computing system, an input mode of a plurality of input modes associated with activating one or more components of the computing system, the one or more components comprising one or more microphone components, one or more camera components, or one or more text input components, wherein the one or more components are associated with performing one or more operations comprising detection of one or more inputs.

2. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the one or more spatial relations of the one or more portions of the user, a distance or an angle between the computing system and the one or more portions of the user; and

determining, by the computing system, when the distance or the angle between the computing system and the one or more portions of the user is within a distance range or angle range respectively, wherein the satisfying the one or more operational criteria comprises the determining that the distance or the angle between the computing system and the one or more portions of the user is within the distance range or the angle range respectively.

3. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the object detection data, when the one or more portions of the user comprise a face;

determining, by the computing system, a distance between the face and the computing system; and

determining, by the computing system, when the distance between the computing system and the face is within a distance range, wherein the satisfying the one or more operational criteria comprises the determining that the distance between the computing system and the face is within the distance range.

4. The computer-implemented method of claim 3, further comprising:

determining, by the computing system, based in part on the object detection data, a position of at least one eye of the face with respect to the computing system; and

determining, by the computing system, based in part on the position of the at least one eye of the face with respect to the computing system, when the computing system is being gazed at for a predetermined period of time, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being gazed at for the predetermined period of time.

5. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the motion data, an acceleration of the computing system in one or more directions relative to the one or more portions of the user; and

determining, by the computing system, when the acceleration of the computing system in a predetermined direction of the one or more directions exceeds an acceleration threshold, wherein the satisfying the one or more operational criteria comprises the acceleration of the computing system in the predetermined direction exceeding the acceleration threshold.

6. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the motion data, an acceleration and a duration of the acceleration of the computing system; and

determining, by the computing system, based in part on the motion data, when the computing system undergoes acceleration within a predetermined acceleration range for a predetermined duration, wherein the satisfying the one or more operational criteria comprises the computing system undergoing acceleration within the predetermined acceleration range for the predetermined duration.

7. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the object detection data, that the computing system is being held in two hands when two of the one or more portions of the user are a predetermined distance range apart and in contact with one or more tactile sensors of the computing system, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being held in two hands.

8. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the object detection data, that the computing system is being held in two hands when two of the one or more portions of the user are within a predetermined distance and a field of view associated with a display output component of the computing system, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being held in two hands.

9. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, based in part on the object detection data, that the computing system is being held in two hands when at least two of the one or more portions of the user are detected to be in contact with two or more portions of one or more tactile sensors behind a front portion of a display output component of the computing system and two of the one or more portions of the user are detected within a predetermined distance in front of the front portion of the display output component, wherein the satisfying the one or more operational criteria comprises the determining that the computing system is being held in two hands.

10. The computer-implemented method of claim 1, further comprising:

determining, by the computing system, when the one or more inputs are not received within a predetermined period of time after the activating the one or more components;

generating, by the computing system, a falsing profile based in part on the object detection data and the motion data received within a falsing time interval comprising a time when the one or more components are activated and the one or more inputs are not received; and

adding, by the computing system, the falsing profile to the one or more falsing profiles.

11. The computer-implemented method of claim 1, wherein the one or more sensors comprise one or more image sensors, one or more audio sensors, one or more tactile sensors, one or more gyroscopic sensors, one or more accelerometers, one or more electromagnetic sensors, one or more thermal sensors, or one or more radar devices.

12. The computer-implemented method of claim 1, wherein determining, by the computing system, based in part on the object detection data and the motion data, the state of the one or more portions of the user, the state of the one or more portions of the user comprising one or more spatial relations of the one or more portions of the user with respect to the computing comprises:

inputting, by the computing system, the object detection data and the motion data to a machine-learned mode determination model, wherein the machine-learned mode determination model has been trained to receive object detection data and motion data and, in response to receipt of the object detection data and motion data, output one or more predicted modes for operating the computing system; and

receiving, by the computing system as an output of the machine-learned mode determination model, one or more predicted modes for operating the computing system.

13. A mode determination system, comprising:

one or more processors;

a machine-learned mode determination model, wherein the machine-learned mode determination model has been trained to receive object detection data and, in response to receipt of the object detection data, output mode predictions for a computing device; and

at least one tangible, non-transitory computer readable medium that stores instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations comprising;

obtaining sensor data from a plurality of sensors of a computing device, wherein the sensor comprises at least object detection data and motion data, the object detection data associated with the state of one or more portions of a user of a computing device, wherein the motion data describes a motion of the computing device relative to the user of the computing device;

inputting the object detection data into the machine-learned mode determination model, wherein the machine-learned mode determination model determines, based in part on the object detection data and the motion data, when the state of the one or more portions of the user relative to the computing device matches one or more falsing profiles associated with a state of the one or more portions of the user relative to the computing device that does not activate the one or more components, wherein the satisfying the one or more operational criteria comprises the state of the one or more portions of the user relative to the computing device not matching any of the one or more falsing profiles; and

receiving, as output of the machine-learned mode determination model, a mode prediction for the computing device.

14. The mode determination system of claim 13, wherein the operations further comprise activating a component of the computing device that correlates with the mode prediction.

15. The mode determination system of claim 13, wherein the machine-learned mode determination model comprises a recurrent neural network.

16. A computing device that automatically initiates an input mode, comprising:

a plurality of sensors;

at least one processor; and

at least one tangible, non-transitory computer-readable medium that stores instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, the operations comprising;

receiving object detection data based in part on one or more sensor outputs from the plurality of sensors that detect one or more portions of a user of the computing device;

receiving motion data based in part on one or more sensor outputs from the plurality of sensors, the motion data describing a motion of the computing device relative to the user of the computing device;

determining, based in part on the object detection data, a state of the one or more portions of the user of the computing device, the state of the one or more portions of the user of the computing device comprising one or more spatial relations of the one or more portions of the user of the computing device with respect to the computing device;

determining, based in part on the object detection data and the motion data, when the state of the one or more portions of the user relative to the computing system matches one or more falsing profiles associated with a state of the one or more portions of the user relative to the computing system that does not activate the one or more components, wherein the satisfying the one or more operational criteria comprises the state of the one or more portions of the user relative to the computing system not matching any of the one or more falsing profiles;

responsive to the state of the one or more portions of the user of the computing device satisfying one or more operational criteria, activating an input mode of a plurality of input modes associated with activating one or more components of the computing device, the one or more components comprising one or more microphone components, one or more camera components, or one or more text input components, wherein the one or more components are associated with performing one or more operations comprising detection of one or more inputs.

17. The computing device of claim 16, wherein the plurality of sensors include one or more accelerometers, one or more radar sensors, one or more tactile sensors, or one or more image sensors.

18. The computing device of claim 16, further comprising:

determining, based in part on the one or more spatial relations of the one or more portions of the user, a distance or an angle between the computing device and the one or more portions of the user; and

determining when the distance or the angle between the computing device and the one or more portions of the user is within a distance range or angle range respectively, wherein the satisfying the one or more operational criteria comprises the determining that the distance or the angle between the computing device and the one or more portions of the user is within the distance range or the angle range respectively.

19. The computing device of claim 16, further comprising:

determining, based in part on the object detection data, when the one or more portions of the user comprise a face;

determining a distance between the face and the computing device; and

determining when the distance between the computing device and the face is within a distance range, wherein the satisfying the one or more operational criteria comprises the determining that the distance between the computing device and the face is within the distance range.

20. The computing device of claim 16, further comprising:

determining when the one or more inputs are not received within a predetermined period of time after the activating the one or more components;

generating a falsing profile based in part on the object detection data and the motion data received within a falsing time interval comprising a time when the one or more components are activated and the one or more inputs are not received; and

adding the falsing profile to the one or more falsing profiles.