Autobahn Autopilot — Euro Truck Simulator 2

Lane detection ML training and steering modes in Euro Truck Simulator 2 (ETS2).

What do we need?

  1. Grab screen
  2. Lane Detection
  3. Lanes
  4. Steering
  5. (More in progress)

A repository is offered here:
It is written with Python.

Before we continue; KUDOS!

Zou Q, Jiang H, Dai Q, Yue Y, Chen L and Wang Q
Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Networks, IEEE Transactions on Vehicular Technology, 2019.


1. Grab Screen

Python or not, does not matter. Find a way to grab a screenshot.

Please refer to .

pip3 install pyscreenshot
img = ImageGrab.grab(bbox=(2, 36, 1280, 720+36), backend="mss", childprocess=False)
# X1, Y1, X2, Y2 # zero 'childprocess' and 'mss' gives the best performance in most cases

You can use pyscreenshot package.

The pyscreenshot module can be used to copy the contents of the screen to a Pillow image memory using various back-ends. Replacement for the ImageGrab Module.

The game should run in windowed mode so you can do your debugging.

2. Lane Detection

Search for deep neural network.
Train or use a pre-trained model.

You need data.

If you do not have any from the game, please refer to a real life data set by “Machine Vision & Robotics Laboratory, School of Computer Science, Wuhan University”.

A little script will help you to create a data set from the game.
Please refer to dataset_creator.pyand .

If you would like to skip “data” part, get the pre-trained model. See below.

You need a model.

Robust Lane Detection from Continuous Driving Scenes Using Deep Neural Network

Train your data with the data set you have collected.
I am afraid you are going to need GPU power or cloud services.

Pre-trained Model

In case of not having enough resources, a pre-trained model from real life data still works good enough with ETS2.

Please cite their paper if you use their code or data in your own work:

@article{zou2019tvt, title={Robust lane detection from continuous driving scenes using deep neural networks}, author={Q. Zou and H. Jiang and Q. Dai and Y. Yue and L. Chen and Q. Wang}, journal={IEEE Transactions on Vehicular Technology}, volume={69}, number={1}, pages={41–54}, year={2020}, }

3. Lanes

  1. Grab screen
    It is too big!
  2. Resize it to model’s image size
    Must keep aspect ratio!
  3. Crop it to model’s image size
  4. Use the model to predict the lanes

You will be seeing all these steps with comments in the repository.

A random shot from ETS2

The model will predict lanes and give you a tensor.
What is inside the prediction? 0s and 1s in a 256x128 matrix.
The matrix element is 1 if the coordinate is predicted as part of a lane.

Printed tensor[0][0], zoomed out
Printed tensor[0][0], zoomed in

In this big matrix, 1s are visible as lanes.

(While training; scale, crop and set region of interest just like Sentdex’s page)

4. Steering

  1. Simple Geometry (mode=0)
  2. Vectors in a matrix (mode=1)
  3. Training Steering Data (mode=2)

This section is only for brain gymnastic, NOT a solution.

Let’s say that we do not have such matrix, and only predicted image.

cv2.HoughLinesP is commonly used to to find lines.

Depending on HoughLineP arguments, you will see different results.

Predicted image -> HoughLinesP -> Optimization

After ‘HoughLineP’ is applied, multiple lines appear on the same road lane.
Remove the extra lines on same angles with a threshold.
Only geometry formulas will not going to be a solution of course.
Please refer to

Re-detection of the line and poor geometry functions are make it unacceptable of course.
We have to train the model with steering data.

Tune in for the next development with steering learning!

Work in progress…

Operates in the middle of a triangle; customer, product, and software.