Multi-level video processing within a vehicular communication network

• US 10,469,408 B2
• Filed: 08/21/2015
• Issued: 11/05/2019
• Est. Priority Date: 11/03/2010
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Inventors: Yousefi, Nariman, Kim, Yongbum, Walley, John, Chen, Sherman (Xuemin), Diab, Wael W., Ilyadis, Nicholas
• Primary Examiner(s): Senfi, Behrooz M
• Application Number:
  US14/832,837
• Publication Number:
  US 20150365664A1
• Time in Patent Office: 1,537 Days
• Field of Search: 37524002
• CPC Class Codes: B60L 50/00 Electric propulsion with power supplied within the vehicle with power supply from force of nature, e.g. sun or wind, B60L8/00; for monorail vehicles, suspension vehicles or rack railways B60L13/00
• CPC Class Codes: B60R 16/023 for transmission of signals between vehicle parts or subsystems
• CPC Class Codes: B60R 16/03 for supply of electrical power to vehicle subsystems or for circuit arrangements for charging batteries H02J7/00
• CPC Class Codes: B60R 16/0315 using multiplexing techniques multiplexing for transmission of signals G08C15/00; multiplexing for electrical communication in general H04J
• CPC Class Codes: B60R 19/03 characterised by material, e.g. composite B60R19/18 takes precedence
• CPC Class Codes: G06F 1/26 Power supply means, e.g. regulation thereof for memories G11C
• CPC Class Codes: G06F 1/266 Arrangements to supply power to external peripherals either directly from the computer or under computer control, e.g. supply of power through the communication port, computer controlled power-strips
• CPC Class Codes: G06F 13/4022 using switching circuits, e.g. switching matrix, connection or expansion network G06F13/4009 takes precedence
• CPC Class Codes: G06F 7/76 Arrangements for rearranging, permuting or selecting data according to predetermined rules, independently of the content of the data
• CPC Class Codes: G07C 5/008 communicating information to a remotely located station transmission systems for measured values G08C
• CPC Class Codes: G07C 5/085 using electronic data carriers
• CPC Class Codes: G07C 5/0866 the electronic data carrier being a digital video recorder in combination with video camera
• CPC Class Codes: G08G 1/16 Anti-collision systems road vehicle drive control systems for predicting or avoiding probable or impending collision otherwise than by control of a particular sub-unit B60W30/08
• CPC Class Codes: H04B 1/3822 specially adapted for use in vehicles H04B1/3827 takes precedence
• CPC Class Codes: H04L 12/10 Current supply arrangements
• CPC Class Codes: H04L 12/46 Interconnection of networks
• CPC Class Codes: H04L 12/4625 Single bridge functionality, e.g. connection of two networks over a single bridge
• CPC Class Codes: H04L 12/56 Packet switching systems
• CPC Class Codes: H04L 45/74 Address processing for routing
• CPC Class Codes: H04L 47/2433 Allocation of priorities to traffic types
• CPC Class Codes: H04L 47/6275 : based on priority
• CPC Class Codes: H04L 47/76 : using dynamic resource allocation, e.g. in-call renegotiation requested by the user or requested by the network in response to changing network conditions
• CPC Class Codes: H04L 47/821 : Prioritising resource allocation or reservation requests
• CPC Class Codes: H04L 49/25 : Routing or path finding in a switch fabric
• CPC Class Codes: H04L 67/12 : specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
• CPC Class Codes: H04L 69/08 : Protocols for interworking; Protocol conversion
• CPC Class Codes: H04L 69/22 : Parsing or analysis of headers
• CPC Class Codes: H04N 19/102 : characterised by the element, parameter or selection affected or controlled by the adaptive coding
• CPC Class Codes: H04N 7/181 : for receiving images from a plurality of remote sources
• CPC Class Codes: H04N 7/183 : for receiving images from a single remote source
• CPC Class Codes: H04W 24/08 : Testing, supervising or monitoring using real traffic
• CPC Class Codes: H04W 28/08 : Load balancing or load distribution transferring a connection for handling the traffic H04W36/22; wireless traffic scheduling H04W72/12
• CPC Class Codes: H04W 4/40 : for vehicles, e.g. vehicle-to-pedestrians [V2P]
• CPC Class Codes: H04W 4/44 : for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
• CPC Class Codes: H04W 4/46 : for vehicle-to-vehicle communication [V2V]
• CPC Class Codes: H04W 72/02 : Selection of wireless resources by user or terminal
• CPC Class Codes: H04W 72/541 : using the level of interference
• CPC Class Codes: H04W 72/56 : based on priority criteria
• CPC Class Codes: H04W 88/02 : Terminal devices
• CPC Class Codes: Y02D 30/70 : in wireless communication networks
• CPC Class Codes: Y04S 40/18 : Network protocols supporting networked applications, e.g. including control of end-device applications over a network

Abstract

A system for performing multi-level video processing within a vehicle includes a pre-processing module for determining an encoding mode and enabling one or more levels of encoding based on the encoding mode. The pre-processing module further receives a video stream from a camera attached to the vehicle via a vehicular communication network and encodes the video stream based on the encoding mode to produce a packet stream output. The system further includes a video decoder for receiving the packet stream output and decoding the packet stream output in accordance with the encoding mode to produce a decoded video output.

Claims

1. A vehicle video processor comprising:

a plurality of video encoders configured to;

receive one or more video streams that correspond to one or more perspectives of an environment, wherein the one or more video streams include one or more of;

a safety related video function, a user display related video function, or a driver assist related video function; and

encode the one or more video streams to generate a plurality of levels of encoded video streams adaptively based on an operational mode, wherein the operational mode determines at least a level of quality for video encoding where a first encoded video stream has lower quality than a second encoded video stream;

a plurality of video decoders configured to decode a plurality of levels of encoded packets that are associated with the plurality of levels of encoded video streams adaptively based on handshaking with the plurality of video encoders and based on the operational mode; and

a processor configured to;

determine an availability of the plurality of video encoders and the plurality of video decoders to determine a resource availability status;

select the operational mode from a plurality of operational modes based on the resource availability status; and

direct the plurality of video encoders and the plurality of video decoders to operate adaptively based on the handshaking and based on the operational mode.

2. The vehicle video processor of claim 1 further comprising:

a plurality of cameras configured to generate a plurality of video streams that corresponds to a plurality of perspectives of the environment, wherein the environment is a vehicle environment around at least some of the vehicle; and

the plurality of video encoders configured to;

receive the plurality of video streams that corresponds to the plurality of perspectives of the vehicle environment; and

encode the plurality of video streams to generate the plurality of levels of encoded video streams adaptively based on the operational mode.

3. The vehicle video processor of claim 1 further comprising:

a first video encoder of the plurality of video encoders configured to process a video stream to generate a first level encoded video stream that corresponds to the safety related video function;

a second video encoder of the plurality of video encoders configured to process the video stream to generate a second level encoded video stream that corresponds to the user display related video function;

a third video encoder of the plurality of video encoders configured to process the video stream to generate a third level encoded video stream that corresponds to the driver assist related video function;

a first video decoder of the plurality of video decoders configured to decode the first level encoded video stream that corresponds to the safety related video function to generate safety related data;

a second video decoder of the plurality of video decoders configured to decode the second level encoded video stream that corresponds to the user display related video function to generate user display related data;

a third video decoder of the plurality of video decoders configured to decode the third level encoded video stream that corresponds to the driver assist related video function to generate driver assist related data; and

the processor configured to;

process the safety related data to determine whether criteria for detecting a safety issue has been met and generate a safety command when the safety issue has been met;

process the user display related data to generate a user output; and

process the driver assist related data to generate a driver assist output.

4. The vehicle video processor of claim 3 further comprising:

a first camera configured to generate the video stream, wherein the video stream corresponds to a first perspective of the environment, wherein the environment is a vehicle environment around the vehicle; and

a second camera configured to generate another video stream, wherein the another video stream corresponds to a second perspective of the vehicle environment.

5. The vehicle video processor of claim 3 further comprising:

a vehicle safety system configured automatically to perform at least one of perform an evasive maneuver or trigger an alarm based on the safety command.

6. The vehicle video processor of claim 1 further comprising:

the plurality of video encoders, the plurality of video decoders, and the processor coupled via a communication network within the vehicle; and

the processor also configured to determine at least one;

of vehicle status, availability of network resources of the communication network, bandwidth of the network resources of the communication network, availability of processing modules of the vehicle video processor, availability of memory of the vehicle video processor, availability of video encoding resources of the plurality of video encoders, or availability of video decoding resources of the plurality of video decoders to determine the resource availability status.

7. The vehicle video processor of claim 1 further comprising:

a packetizing module configured to process the plurality of levels of encoded video streams to generate a single video packet stream output, wherein the plurality of levels of encoded video streams includes a first level encoded video stream and a second level encoded video stream; and

a de-packetizing module configured to process the single video packet stream output to generate the plurality of levels of encoded packets including to generate a first level of encoded packets that corresponds to the first level encoded video stream and a second level of encoded packets that corresponds to the second level encoded video stream.

8. The vehicle video processor of claim 1 further configured to:

determine, based on a condition change associated with at least one of the vehicle video processor or the vehicle, another status of the plurality of video encoders and the plurality of video decoders to determine another resource availability status;

select another operational mode from the plurality of operational modes based on the another resource availability status;

direct the plurality of video encoders and the plurality of video decoders to operate adaptively based on the handshaking and based on the another operational mode;

the plurality of video encoders configured to;

receive another one or more video streams that correspond to the one or more perspectives of the environment; and

encode the another one or more video streams to generate another plurality of levels of encoded video streams adaptively based on the another operational mode; and

the plurality of video decoders configured to;

receive another plurality of levels of encoded packets that are associated with the another plurality of levels of encoded video streams; and

decode the plurality of levels of encoded packets adaptively based on the handshaking with the plurality of video encoders and based on the another operational mode.

9. A vehicular video processor comprising:

a plurality of cameras configured to generate a plurality of video streams that corresponds to a plurality of perspectives of a vehicle environment around at least some of a vehicle, wherein the plurality of video streams include one or more of;

a safety related video function, a user display related video function, or a driver assist related video function;

a plurality of video encoders configured to encode the plurality of video streams to generate a plurality of levels of encoded video streams adaptively based on an operational mode, wherein the operational mode indicates at least a level of quality of video encoding where a first encoded video stream has lower quality than a second encoded video stream;

a plurality of video decoders configured to decode a plurality of levels of encoded packets that are associated with the plurality of levels of encoded video streams adaptively based on handshaking with the plurality of video encoders and based on the operational mode to generate at least one of safety related data, user display related data, or driver assist related data; and

a processor configured to;

determine an availability status of the plurality of cameras, the plurality of video encoders, and the plurality of video decoders to determine a resource availability status;

select the operational mode from a plurality of operational modes based on the resource availability status;

direct the plurality of cameras, the plurality of video encoders, and the plurality of video decoders to operate adaptively based on the handshaking and based on the operational mode; and

process the at least one of the safety related data, the user display related data, or the driver assist related data to determine whether criteria for detecting a safety issue has been met and generate a safety command when the safety issue has been met; and

a vehicle safety system configured automatically to perform at least one of perform an evasive maneuver or trigger an alarm based on the safety command.

10. The vehicular video processor of claim 9 further comprising:

a first video encoder of the plurality of video encoders configured to process a video stream to generate a first level encoded video stream that corresponds to the safety related video function;

a second video encoder of the plurality of video encoders configured to process the video stream to generate a second level encoded video stream that corresponds to the user display related video function;

a third video encoder of the plurality of video encoders configured to process the video stream to generate a third level encoded video stream that corresponds to the driver assist related video function;

a first video decoder of the plurality of video decoders configured to decode the first level encoded video stream that corresponds to the safety related video function to generate safety related data;

a second video decoder of the plurality of video decoders configured to decode the second level encoded video stream that corresponds to the user display related video function to generate user display related data;

a third video decoder of the plurality of video decoders configured to decode the third level encoded video stream that corresponds to the driver assist related video function to generate driver assist related data; and

the processor configured to;

process the safety related data to determine whether criteria for detecting a safety issue has been met and generate a safety command when the safety issue has been met;

process the user display related data to generate a user output; and

process the driver assist related data to generate a driver assist output.

11. The vehicular video processor of claim 10 further comprising:

a first camera of the plurality of cameras configured to generate a first video stream of the plurality of video streams, wherein the first video stream corresponds to a first perspective of the vehicle environment; and

a second camera of the plurality of cameras configured to generate a second video stream of the plurality of video streams, wherein the second video stream corresponds to a second perspective of the vehicle environment.

12. The vehicular video processor of claim 9 further comprising:

the plurality of video encoders, the plurality of video decoders, and the processor coupled via a communication network within the vehicle; and

the processor also configured to determine at least one of;

vehicle status, availability of network resources of the communication network, bandwidth of the network resources of the communication network, availability of processing modules of the vehicular video processor, availability of memory of the vehicular video processor, availability of video encoding resources of the plurality of video encoders, or availability of video decoding resources of the plurality of video decoders to determine the resource availability status.

13. The vehicular video processor of claim 9 further comprising:

a packetizing module configured to process the plurality of levels of encoded video streams to generate a single video packet stream output, wherein the plurality of levels of encoded video streams includes a first level encoded video stream and a second level encoded video stream; and

a de-packetizing module configured to process the single video packet stream output to generate the plurality of levels of encoded packets including to generate a first level of encoded packets that corresponds to the first level encoded video stream and a second level of encoded packets that corresponds to the second level encoded video stream.

14. A method for execution by a vehicle video processor, the method comprising:

receiving one or more video streams that correspond to one or more perspectives of an environment, wherein the one or more video streams include one or more of;

a safety related video function, a user display related video function, or a driver assist related video function;

operating a plurality of video encoders to encode the one or more video streams to generate a plurality of levels of encoded video streams adaptively based on an operational mode, wherein the operational mode indicates at least a level of quality of video encoding where a first encoded video stream has lower quality than a second encoded video stream;

operating a plurality of video decoders to decode a plurality of levels of encoded packets that are associated with the plurality of levels of encoded video streams adaptively based on handshaking with the plurality of video encoders and based on the operational mode;

determine an availability status of the plurality of video encoders and the plurality of video decoders to determine a resource availability status;

selecting the operational mode from a plurality of operational modes based on the resource availability status; and

directing the plurality of video encoders and the plurality of video decoders to operate adaptively based on the handshaking and based on the operational mode.

15. The method of claim 14, wherein the vehicle video processor is implemented within a vehicle, and further comprising:

operating a plurality of cameras to generate a plurality of video streams that corresponds to a plurality of perspectives of the environment, wherein the environment is a vehicle environment around at least some of the vehicle; and

operating the plurality of video encoders to;

receive the plurality of video streams that corresponds to the plurality of perspectives of the vehicle environment; and

encode the plurality of video streams to generate the plurality of levels of encoded video streams adaptively based on the operational mode.

16. The method of claim 14 further comprising:

operating a first video encoder of the plurality of video encoders to process a video stream to generate a first level encoded video stream that corresponds to the safety related video function;

operating a second video encoder of the plurality of video encoders to process the video stream to generate a second level encoded video stream that corresponds to the user display related video function;

operating a third video encoder of the plurality of video encoders to process the video stream to generate a third level encoded video stream that corresponds to the driver assist related video function;

operating a first video decoder of the plurality of video decoders to decode the first level encoded video stream that corresponds to the safety related video function to generate safety related data;

operating a second video decoder of the plurality of video decoders to decode the second level encoded video stream that corresponds to the user display related video function to generate user display related data;

operating a third video decoder of the plurality of video decoders to decode the third level encoded video stream that corresponds to the driver assist related video function to generate driver assist related data;

processing the safety related data to determine whether criteria for detecting a safety issue has been met and generating a safety command when the safety issue has been met;

processing the user display related data to generate a user output;

processing the driver assist related data to generate a driver assist output; and

performing at least one of performing an evasive maneuver or trigger an alarm within a vehicle safety system based on the safety command.

17. The method of claim 16 further comprising:

operating a first camera to generate the video stream, wherein the video stream corresponds to a first perspective of the environment, wherein the environment is a vehicle environment around the vehicle; and

operating a second camera to generate another video stream, wherein the another video stream corresponds to a second perspective of the vehicle environment.

18. The method of claim 14, wherein the plurality of video encoders, the plurality of video decoders, and the processor are coupled via a communication network within the vehicle, and further comprising:

determining at least one of;

vehicle status, availability of network resources of the communication network, bandwidth of the network resources of the communication network, availability of processing modules of the vehicular video processor, availability of memory of the vehicular video processor, availability of video encoding resources of the plurality of video encoders, or availability of video decoding resources of the plurality of video decoders to determine the resource availability status.

19. The method of claim 14 further comprising:

operating a packetizing module to process the plurality of levels of encoded video streams to generate a single video packet stream output, wherein the plurality of levels of encoded video streams includes a first level encoded video stream and a second level encoded video stream; and

operating a de-packetizing module to process the single video packet stream output to generate the plurality of levels of encoded packets including to generate a first level of encoded packets that corresponds to the first level encoded video stream and a second level of encoded packets that corresponds to the second level encoded video stream.

20. The method of claim 14 further comprising:

determining, based on a condition change associated with at least one of the vehicular video processor or the vehicle, another status of the plurality of video encoders and the plurality of video decoders to determine another resource availability status;

selecting another operational mode from the plurality of operational modes based on the another resource availability status;

directing the plurality of video encoders and the plurality of video decoders to operate adaptively based on the handshaking and based on the another operational mode;

operating the plurality of video encoders to;

receive another one or more video streams that correspond to the one or more perspectives of the environment; and

encode the another one or more video streams to generate another plurality of levels of encoded video streams adaptively based on the another operational mode; and

operating the plurality of video decoders to;

receive another plurality of levels of encoded packets that are associated with the another plurality of levels of encoded video streams; and

decode the plurality of levels of encoded packets adaptively based on the handshaking with the plurality of video encoders and based on the another operational mode.