Lanes as Vectors — Autobahn Autopilot

After a successful lane detection, it is time to extract valuable vectors from tensors.

In case you missed part 1: Autobahn Autopilot — Euro Truck Simulator 2

This frame is directly taken from the game and size is set to the model’s size which is 256x128.

Here is the result after lane detection (part 1) ran on the frame.

This part shows how to filter important lines and find them as vectors.

Let’s remember what is in one of tensors.

> Grab a frame
// check part 1

> Detect lanes
// check part 1


Let’s take a closer look.

This is a 256x128 binary matrix, and elements are “1” only if they are detected as a piece of a lane.

A line is always displayed as “1”s and ends with “0” from all sides.

Labeling can separate those lines from each other.
Keyword is image processing.

from scipy.ndimage.measurements import label

Old documentation, New documentation

Get the prediction as numpy array to process.

data = pred[0][0].cpu().numpy()

This method helps you to label all neighbor “1”s.

labeled_array, num_features = label(data)

num_features tells you how many labels are created — how many different possible lines are detected.
It will not going to look as you expected until you request with a proper structure.

Default structure is;


However, we are looking for all neighbors.
For this use case, a different structure is needed;


So the correction;

labeled_array, num_features = label(
data, structure=[[1, 1, 1], [1, 1, 1], [1, 1, 1]]


Closer look;

All “1” are labeled with a different number as long as the section can continue as our transform value.

Find Vectors

In this matrix, with the following actions, vectors can be extracted.

  1. Look for labels
  2. Convex hull
  3. Get distance between pairs
  4. Select furthest candidates

A quick look into methods;


argwhere finds the indices of array elements that are non-zero, grouped by element.


In geometry, the convex hull or convex envelope or convex closure of a shape is the smallest convex set that contains it.


Computes the distance matrix and returns the matrix of all pair-wise distances.


unravel_index converts a flat index or array of flat indices into a tuple of coordinate arrays.


vectors = []for i in range(num_features):
index = i + 1
# look for the next group label
indices = np.argwhere(labeled_array == index)
if len(indices) < THRESHOLD:
# not an important line if under threshold (optional)
# convex hull
candidates = indices[spatial.ConvexHull(indices).vertices]
# distance between pairs
dist_mat = spatial.distance_matrix(candidates, candidates)
# furthest candidates
points = np.unravel_index(dist_mat.argmax(), dist_mat.shape)

Now, you have a list of vectors which you can precisely work with.

Tune in for part 3; steering better with precise vectors!

Operates in the middle of a triangle; customer, product, and software.