Visual and audio recognition for scene change events

• US 9,536,161 B1
• Filed: 06/17/2014
• Issued: 01/03/2017
• Est. Priority Date: 06/17/2014
• Status: Active Grant

• Current Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Original Assignee: Amazon Technologies, Inc. (Amazon.com, Inc.)
• Inventors: Cooper, Neil David, Lish, Christopher John, Rybakov, Oleg, Jahagirdar, Sonjeev, Jia, Junxiong, Sikka, Avnish
• Primary Examiner(s): Prabhakher, Pritham
• Application Number:
  US14/307,090
• Time in Patent Office: 931 Days
• Field of Search: 348/222.1
• US Class Current: 1/1
• CPC Class Codes: G01S 3/00 Direction-finders for determining the direction from which infrasonic, sonic, ultrasonic, or electromagnetic waves, or particle emission, not having a directional significance, are being received position-fixing by co-ordinating a plurality of determinations of direction or position lines G01S5/00
• CPC Class Codes: G06V 10/62 relating to a temporal dimension, e.g. time-based feature extraction; Pattern tracking
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/61 based on recognised objects
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/634 Warning indications
• CPC Class Codes: H04N 23/661 Transmitting camera control signals through networks, e.g. control via the Internet
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

Various embodiments describe systems and methods for utilizing a reduced amount of processing capacity for incoming data over time, and, in response to detecting a scene-change-event, notify one or more data processors that a scene-change-event has occurred, and cause incoming data to be processed as new data. In some embodiments, an incoming frame can be compared with a reference frame to determine a difference between the reference frame and the incoming frame. The reference frame may be correlated to a latest scene-change-event. In response to a determination that the difference does not meet one or more difference criteria, a user interface or at least one processor of the computing device can be notified to reduce processing of incoming data over time. In response to a determination that the difference meets the one or more difference criteria, the user interface or the at least one processor can be notified that a scene-change-event has occurred. Incoming data to the computing device is then treated as new and processed as those under an active condition. The current incoming frame can be selected as a new reference frame for detecting next scene-change-event.

Claims

1. A computing device, comprising:

a camera;

at least one processor; and

memory including instructions that, when executed by the at least one processor, cause the computing device to;

acquire a series of frames of content using the camera;

select a reference frame from the series of frames of content;

acquire a subsequent frame of content using the camera;

determine a first feature point that is included in a set of feature points represented in the reference frame;

determine a first location associated with the first feature point represented in the reference frame;

determine a second location associated with the first feature point represented in the subsequent frame;

determine a change between the first location and the second location;

determine that the change is below a motion threshold; and

cause the at least one processor to reduce processing of incoming frames of content.

2. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

calculate the change using one or more homography algorithms.

3. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

determine that no object in the subsequent frame is identified by the computing device; and

skip the incoming frames of content until the computing device is stabilized on a scene for at least a predefined period of time.

4. The computing device of claim 1, wherein the instructions when executed further cause the computing device to:

track movement of the computing device by analyzing data from at least one sensor of the computing device; and

determine whether a scene-change-event has occurred based at least upon the movement of the computing device.

5. A computer-implemented method, comprising:

under the control of a computer system configured with executable instructions,

selecting a reference frame from a series of frames of content;

acquiring an incoming frame of content using the computer system;

determining a first feature point that is included in a set of feature points represented in the reference frame;

determining a first location associated with the first feature point represented in the reference frame;

determining a second location associated with the first feature point represented in the incoming frame;

determining a difference between the first location and the second location;

determining that the difference does not meet one or more difference criteria; and

causing at least one processor of the computing system to reduce processing of incoming frames of content.

6. The computer-implemented method of claim 5, further comprising:

determining that the difference meets the one or more difference criteria, indicating that a scene change event has occurred;

selecting the incoming frame as a new reference frame; and

processing the incoming frame using an amount of processing capacity under an active condition.

7. The computer-implemented method of claim 6, wherein the indicating includes a type of texts, features or objects of the incoming frame.

8. The computer-implemented method of claim 7, wherein the difference is determined by measuring a percentage of a length of sides of the incoming frame with respect to the reference frame.

9. The computer-implemented method of claim 5, further comprising:

performing a sound identification to determine an audio reference for the computer system;

comparing an incoming audio data with the audio reference;

determining an audio change between the incoming audio data and the audio reference;

determining that the audio change is below an audio threshold; and

causing the at least one processor to reduce processing of the incoming frames of content.

10. The computer-implemented method of claim 9, wherein the audio change is determined by analyzing at least frequency, pattern, or volume differences between the incoming audio data and the reference audio.

11. The computer-implemented method of claim 9, wherein the performing the sound identification includes analyzing diatonic and chromatic scales, frequency, or harmony of audio data to the computer system.

12. The computer-implemented method of claim 5, further comprising:

displaying a status of the at least one processor of the computing system on a user interface of the computer system.

13. The computer-implemented method of claim 5, further comprising:

tracking movement of the computer system by analyzing data from at least one sensor of the computer system or network connection data of the computer system; and

determining whether a scene-change-event has occurred based at least upon the movement of the computer system, or the network connection data.

14. The computer-implemented method of claim 13 wherein the at least one sensor of the computer system includes at least one of accelerometers, inertial sensors, magnetometers, electronic gyroscopes, ambient light sensors, proximity sensors, or electronic compasses.

15. The computer-implemented method of claim 5, further comprising:

determining that no object in the incoming frame is identified by the computer system; and

skipping the incoming frames of content until the computer system is stabilized on a scene for at least a predefined period of time.

16. The computer-implemented method of claim 5, further comprising:

down-sampling subsequent incoming frames in determining subsequent differences between the subsequent incoming frames and the reference frame until a scene-change-event is detected.

17. A non-transitory computer-readable storage medium including instructions that, when executed by at least one processor of a computing system, cause the computing system to:

select a reference frame from a series of frames of content;

acquire an incoming frame of content using the computer system;

determine a first feature point that is included in a set of feature points represented in the reference frame;

determine a first location associated with the first feature point represented in the reference frame;

determine a second location associated with the first feature point represented in the incoming frame;

determine a difference between the first location and the second location;

determine that the difference does not meet one or more difference criteria; and

cause the at least one processor to reduce processing of incoming frames of content.

18. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

determine that the difference meets the one or more difference criteria, indicating that a scene-change-event has occurred;

select the incoming frame as a new reference frame; and

processing the incoming frame using an amount of processing capacity under an active condition.

19. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

perform a sound identification to determine an audio reference for the computer system;

compare an incoming audio data with the audio reference;

determine an audio change between the incoming audio data and the audio reference;

determine that the audio change is below an audio threshold; and

cause the at least one processor to reduce processing of the incoming frames of content.

20. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

track movement of the computer system by analyzing data from at least one sensor of the computer system or network connection data of the computing system; and

determine whether a scene-change-event has occurred based at least upon the movement of the computer system, or the network connection data.

21. The non-transitory computer-readable storage medium of claim 17, wherein the instructions when executed further cause the system to:

aggregate the incoming frames that are determined within a scene; and

provide an option for a user of the computing system to customize historical review of the incoming frames within the scene by grouping together one or more identified features or objects from the incoming frames.