Augmented video processing

• US 9,807,291 B1
• Filed: 11/06/2014
• Issued: 10/31/2017
• Est. Priority Date: 01/29/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tome, Basheer, Raffle, Hayes Solos
• Primary Examiner(s): Ye, Lin
• Assistant Examiner(s): Yoder, III, Chriss
• Application Number:
  US14/534,980
• Time in Patent Office: 1,090 Days
• Field of Search: 3482082, 348211114, 3482313-2316, 348239, 34833302, 34833312
• CPC Class Codes: A61B 3/113 for determining or recording eye movement
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 2027/0178 Eyeglass type eyeglass details G02C
• CPC Class Codes: G02B 2027/0187 slaved to motion of at least a part of the body of the user, e.g. head, eye
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G11B 27/031 Electronic editing of digitised analogue information signals, e.g. audio or video signals
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: H04N 1/00204 with a digital computer or a digital computer system, e.g. an internet server programmed control between transmitter and receiver or between image input and image output device H04N1/32561
• CPC Class Codes: H04N 2101/00 Still video cameras
• CPC Class Codes: H04N 23/66 Remote control of cameras or camera parts, e.g. by remote control devices
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 5/772 the recording apparatus and the television camera being placed in the same enclosure
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

Embodiments described herein may allow for dynamic image processing based on biometric data. An example device may include: an interface configured to receive video data that is generated by an image capture device; an interface configured to receive biometric data of a user of the image capture device from one or more sensors generated synchronously with the video data; and an image processing system configured to apply image processing to the video data to generate edited video data. The image processing may be based, at least in part, on the biometric data.

Claims

1. A computing device, comprising:

an interface configured to receive video data that is generated by an image capture device;

an interface configured to receive, from one or more sensors, biometric data of a user of the device generated during capture of the received video data, wherein the biometric data is generated synchronously with the video data, wherein a first portion of the biometric data indicates a faster than normal heartrate, and a second portion of the biometric data indicates a normal or slower than normal heartrate, and wherein the first portion of the biometric data corresponds to a first portion of the video data and the second portion of the biometric data corresponds to a second portion of the video data;

an image processing system configured to apply image processing to the video data to generate edited video data, wherein the image processing system is configured to generate edited video data comprising the first portion of the video data at a first frame rate and the second portion of the video data at a second frame rate, wherein the second frame rate is less than the first frame rate; and

generate a video file comprising the edited video data for subsequent playback at a playback frame rate equal to the second frame rate, such that the first portion of the video data is played back in slow motion.

2. The computing device of claim 1, further comprising:

an interface configured to receive time-synchronization data that indicates a timing relationship between the biometric data and the video data, wherein the image processing of the video data is further based on the time-synchronization data.

3. The computing device of claim 1, wherein the computing device is implemented as part of or takes the form of a head-mountable device (HMD).

4. The computing device of claim 1, wherein the computing device is implemented as or takes the form of a cloud server.

5. The computing device of claim 1, wherein the image processing system is further configured to:

apply image processing based, at least in part, on biometric data comprising one or more of;

(a) heart rate, (b) respiration rate, (c) body temperature, (d) level of perspiration, (e) muscle movement, (f) eye movement, (g) blinking, and (h) speech.

6. The computing device of claim 5, wherein the one or more sensors comprise one or more biometric sensors selected from the group consisting of:

(a) a heart rate monitor, (b) a respiration rate sensor, (c) a thermometer, (e) a perspiration sensor, (f) a microphone, (g) a decibel meter, (h) a bone conducting transducer, (i) an electromyograph, (j) a strain gauge, and (k) an eye-detection sensor.

7. The computing device of claim 1, wherein the one or more sensors comprise one or more movement or positioning sensors selected from the group consisting of:

(a) an accelerometer, (b) an inertial measurement unit, (c) a proximity sensor;

(d) a microphone;

(e) a gyroscope, (f) a magnetometer, (g) an optical sensor, (h) an ultrasonic sensor, (i) an odometer, and (j) a pedometer.

8. The computing device of claim 7, wherein the one or more sensors comprise one or more contextual sensors selected from the group consisting of:

(a) a location-tracking sensor, (b) light intensity sensor, (c) a clock, and (d) a sensor configured to receive an input from a user.

9. The computing device of claim 1, wherein the image processing system is further configured to:

apply image processing based, at least in part, on biometric data comprising one or more of;

(a) location, (b) ambient light intensity, (c) time of day, (d) a user'"'"'s mode of travel, (e) a type of activity a user is participating in, (f) height of a user, (g) weight of a user, and (h) age of a user.

10. The computing device of claim 1, wherein the image processing comprises at least one of:

(a) fast motion effects, (b) slow motion effects, (c) blur effects, (d) color calibration, (e) light intensity calibration, (f) fade-in or fade-out effects, (g) frame cutting, (h) frame rate conversion, (i) white point conversion, (j) color space conversion, (k) noise reduction, (l) detail enhancement, (m) edge enhancement, or (n) motion compensation.

11. The computing device of claim 10, wherein the image processing system is configured to increase a speed of the video data to generate edited video data based, at least in part, on receiving biometric data indicative of a normal or slower than normal heart rate.

12. The computing device of claim 10, wherein the image processing system is configured to decrease a speed of the video data to generate edited video data based, at least in part, on receiving biometric data indicative of a higher than normal heart rate.

13. The computing device of claim 10, wherein the image processing system is configured to decrease a speed of the video data to generate edited video data based, at least in part, on receiving biometric data indicative of falling.

14. The computing device of claim 10, wherein the image processing system is configured to decrease a speed of the video data to generate edited video data based, at least in part, on receiving biometric data indicative of a higher than normal respiration rate.

15. The computing device of claim 10, wherein the image processing system is configured to adjust a color saturation of the video data to generate edited video data based, at least in part, on receiving biometric data indicative of speed of travel.

16. The computing device of claim 1, wherein the image processing comprises one or more audio effects including:

(a) audio overlay, (b) noise removal, (c) noise amplification, (d) noise suppression, (e) audio volume adjustment, (f) pitch shift, and (g) time stretching.

17. The computing device of claim 1, further comprising a display configured to display at least one of:

an indication of the biometric data and the edited video data.

18. The computing device of claim 1, wherein the generated video file specifies that the playback frame rate is the second frame rate, such that a portion of the generated video file corresponding to the first portion appears in slow motion when played back at the second frame rate.

19. A method comprising:

receiving, by a computing device, video data generated by an image capture device;

receiving, by the computing device, from one or more sensors, biometric data of a user of the image capture device that is generated during capture of the received video data, wherein the biometric data is generated synchronously with the video data, wherein a first portion of the biometric data indicates a faster than normal heartrate, and a second portion of the biometric data indicates a normal or slower than normal heartrate, and wherein the first portion of the biometric data corresponds to a first portion of the video data and the second portion of the biometric data corresponds to a second portion of the video data; and

applying image processing to the video data to generate edited video data, wherein the image processing system is configured to generate edited video data comprising the first portion of the video data at a first frame rate and the second portion of the video data at a second frame rate, wherein the second frame rate is less than the first frame rate; and

generating a video file comprising the edited video data for subsequent playback at a playback frame rate equal to the second frame rate, such that the first portion of the video data is played back in slow motion.

20. The method of claim 19, further comprising:

receiving time-synchronization data that indicates a timing relationship between the biometric data and the video data, wherein the image processing of the video data is further based on the time-synchronization data.

21. The method of claim 19, further comprising:

displaying, on a display in communication with the computing device, one or more of;

an indication of the biometric data and the edited video data.

22. A non-transitory computer readable medium having stored therein instructions executable by a processor to cause a computing device to perform functions comprising:

receiving video data generated by an image capture device;

receiving, from one or more sensors, biometric data of a user of the image capture device that is generated during capture of the received video data, wherein the biometric data is generated synchronously with the video data, wherein a first portion of the biometric data indicates a faster than normal heartrate, and a second portion of the biometric data indicates a normal or slower than normal heartrate, and wherein the first portion of the biometric data corresponds to a first portion of the video data and the second portion of the biometric data corresponds to a second portion of the video data; and

applying image processing to the video data to generate edited video data wherein the image processing system is configured to generate edited video data comprising the first portion of the video data at a first frame rate and the second portion of the video data at a second frame rate, wherein the second frame rate is less than the first frame rate; and

generating a video file comprising the edited video data for subsequent playback at a playback frame rate equal to the second frame rate, such that the first portion of the video data is played back in slow motion.

23. The non-transitory computer readable medium of claim 22, wherein the functions further comprise:

receiving time-synchronization data that indicates a timing relationship between the biometric data and the video data, wherein the image processing of the video data is further based on the time-synchronization data.