事例紹介cases

Toyota Technological Institute: Smart Vehicle Research Center​ ​​ ​

Smart Vehicle Research Center of Toyota Technological Institute is promoting R & D on driving assistance systems that are safe for elderly people and beginner drivers and have less burden on the environment. Specifically, the computer detects various objects such as a pedestrian, a vehicle, a road, a white line and information obtained from a sensor such as a TV camera, a laser scanner, a GPS (global positioning system).We are developing technologies that can recognize the running environment.In this research and development, RoboVision 2 is utilized as a tool for developing nighttime Image Recognition technology with cameras.

Study on development of robust stereo camera technology to changes in driving environment

1. Outline of stereo vision

Mr. Mita's is conducting research on distance calculation algorithms that are resistant to various environmental changes such as rain, snow, and nighttime as a method of constructing a robust recognition system in the running environment using RoboVision 2. Professor Mita of Toyota Technological Institute has been introducing RoboVision 2 from the past and is conducting research on sensor fusion technology during running.
Experimental vehicle and sensor configuration
Stereo vision uses measurement of two cameras to measure distances. There are several methods of distance measurement of stereo vision, but in this research we extract features, extract matching and calculate parallax images.
There is also a problem of calculation load in matching, but it calculates it by taking correlation between right and left camera images.

From professor`s research results, it calculates by taking the correlation with the product of the three kinds of correlation patterns of vertical, horizontal, diagonal of the right and left images.

Also, as a matter of the stereo camera, we will use Gradient's information of gradient to fill in information about how to calculate the distance in a texture (feature) -free region such as white wall, night sky and road surface.

In computation, computation speed of about 60 fps is secured by dynamically computing necessary places.

2. Responding to changes in the driving environment

As a change in the driving environment, which can include sunny, rain, snow, other bad weather conditions.
When the driving environment becomes of snow or rain, the wiper is reflected on one of the cameras, the distance cannot be calculated well, snow and rain are calculated as obstacles, calculation of the distance in front becomes unreliable value.
Also, in terms of sunshine conditions, noise occurs even in spaces and the situation may occur where accurate distance cannot be calculated.
Therefore, we calculate the index of reliability when matching the left and right pictures to the existing stereo matching computation as introduced above as a new method and calculate the distance in the area of ​​high reliability .
In the calculation of reliability, reliability in each area of ​​the acquired image is calculated by incorporating the matching cost into the calculation.
In running such as during night time, the CMOS sensor equipped with RoboVision 2 is highly sensitive and is affected by noise at night, so cases where the distance is not accurately calculated due to the influence of noise or cases where the distance such as the road surface is abnormal can occur.
Therefore, we use a median filter to handle smooth processing results.
Below is the image of the result of adapting these algorithms.

In general SGM (Semi Global Matching) and MPV (Multi Path Viterbi) methods,the distance measurement result value flickers in the forward direction.
On the contrary, Mr.Mita's examination shows that smooth and noisy measurements can be performed.
Measurement result during rainy weather
Measurement result during snowfall
In this way, by improving the algorithm of image processing, it is possible to acquire reliable distance information even during rain or snowfall.

3. On the measurement of nighttime stereo vision

In the lecture, we also introduced the case of the nighttime measurement results.

As noise increases at nighttime images, incorrect distance measurement results.As a countermeasure, we use non-local means (NLM) filter.

If the NLM filter is used as it is, the computation time will be long, so we are trying to shorten the calculation time by improving the algorithm. Computation processing is possible with 20 ms (50 fps) by improvement of algorithm. As a result, computation processing can be performed almost in real time, and studies on application as automotive algorithms are underway.
Measurement result at night
As a result of such examination, we have received comments that distance measurement with nighttime, rain, snow is also possible if there is spec of RoboVision 2 or so.

5. Application example of stereo vision

In the last introduction case, as an application example of stereo vision, we introduced self-positioning of vehicle and 3D representation and distance measurement.
Self positioning of the vehicle
In the self-positioning of the vehicle, we use a high-precision map as used in Autonomous Driving from the information of the point cloud obtained by stereo matching and perform self-position estimation. In the examination of the teacher, in the daytime condition as a result of comparison between the stereo camera and the point cloud map, it is 0.08 m in the longitudinal direction and 0.09 m in the lateral direction.
Comparison of self position estimation results in stereo camera and point cloud
3D representation and distance measurement
As a new effort as a stereo camera, we are studying various detections such as detection of surrounding objects, detection of travelable areas, point cloud display as a bird's eye view.
Peripheral recognition using stereo camera
In the future research, we will need a sensor configuration to cope with fog and backlighting. By utilizing the sensor fusion technique such as thermal sensor, 3D-LiDAR and deep learning technology we can apply it to all weather and various situations.​ ​

About Smart Vehicle Research Center of Toyoda Institute of Technology​ ​

Professor Mita working on this research belongs to the Smart Vehicle Research Center of Toyota Technological Institute and is engaged in research to realize a society free of traffic accidents. Check the page for the latest research efforts.

​ ​Image recognition and Image processing products

ZMP deals with stereo camera unit RoboVision 2s, which is used in this research and stereo camera RoboVision 3 capable of sensing distance and field of view up to 150 m and horizontal 110 ° for Autonomous Driving / ADAS.

For details of each product, please check the following product page.
RoboVision®2s
Super high sensitivity stereo camera system compatible with ranging and object detection software
RoboVision®3
A stereo camera capable of sensing a distance of up to 150 m and horizontal 110 ° field of view​ ​
To TOP