Autobahn Autopilot — Euro Truck Simulator 2

Lane detection ML training and steering modes in Euro Truck Simulator 2 (ETS2).

What do we need?

  1. Grab screen

A repository is offered here: https://github.com/lotdcotw/ets2ap
It is written with Python.

Before we continue; KUDOS!

Zou Q, Jiang H, Dai Q, Yue Y, Chen L and Wang Q
Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks, IEEE Transactions on Vehicular Technology, 2019.

Sentdex
https://pythonprogramming.net/game-frames-open-cv-python-plays-gta-v/
https://www.youtube.com/user/sentdex

1. Grab Screen

Python or not, does not matter. Find a way to grab a screenshot.

Please refer to grab.py .

pip3 install pyscreenshot
img = ImageGrab.grab(bbox=(2, 36, 1280, 720+36), backend="mss", childprocess=False)
# X1, Y1, X2, Y2 # zero 'childprocess' and 'mss' gives the best performance in most cases

You can use pyscreenshot package. https://pypi.org/project/pyscreenshot/

The pyscreenshot module can be used to copy the contents of the screen to a Pillow image memory using various back-ends. Replacement for the ImageGrab Module.

The game should run in windowed mode so you can do your debugging.

2. Lane Detection

Search for deep neural network.
Train or use a pre-trained model.

You need data.

Real Life Data

If you do not have any from the game, please refer to a real life data set by “Machine Vision & Robotics Laboratory, School of Computer Science, Wuhan University”.
http://mvr.whu.edu.cn/codedata/index.htm

ETS2 Data

A little script will help you to create a data set from the game.
Please refer to dataset_creator.pyand dataset_indexer.py .

If you would like to skip “data” part, get the pre-trained model. See below.

You need a model.

Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Network

https://github.com/qinnzou/Robust-Lane-Detection

Train your data with the data set you have collected.
I am afraid you are going to need GPU power or cloud services.

Pre-trained Model

In case of not having enough resources, a pre-trained model from real life data still works good enough with ETS2.

https://drive.google.com/drive/folders/1BHbYU3AoU5i2mduMtfUkFHVuYObA_fWt?usp=sharing

Please cite their paper if you use their code or data in your own work:

@article{zou2019tvt, title={Robust lane detection from continuous driving scenes using deep neural networks}, author={Q. Zou and H. Jiang and Q. Dai and Y. Yue and L. Chen and Q. Wang}, journal={IEEE Transactions on Vehicular Technology}, volume={69}, number={1}, pages={41–54}, year={2020}, }

3. Lanes

  1. Grab screen
    It is too big!

You will be seeing all these steps with comments in the repository.

A random shot from ETS2

The model will predict lanes and give you a tensor.
What is inside the prediction? 0s and 1s in a 256x128 matrix.
The matrix element is 1 if the coordinate is predicted as part of a lane.

Printed tensor[0][0], zoomed out
Printed tensor[0][0], zoomed in

In this big matrix, 1s are visible as lanes.

(While training; scale, crop and set region of interest just like Sentdex’s page)

4. Steering

  1. Simple Geometry (mode=0)

1. Simple Geometry (mode=0)

This section is only for brain gymnastic, NOT a solution.

Let’s say that we do not have such matrix, and only predicted image.

cv2.HoughLinesP is commonly used to to find lines.

Depending on HoughLineP arguments, you will see different results.

Predicted image -> HoughLinesP -> Optimization

After ‘HoughLineP’ is applied, multiple lines appear on the same road lane.
Remove the extra lines on same angles with a threshold.
Only geometry formulas will not going to be a solution of course.
Please refer to live.py

Re-detection of the line and poor geometry functions are make it unacceptable of course.
We have to train the model with steering data.

Tune in for the next development with steering learning!

3. Train Steering Data (mode=2)

Work in progress…

Operates in the middle of a triangle; customer, product, and software.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store