View on GitHub

temporal-MR-GCN

Temporal MR-GCN

Towards Accurate Vehicle Behaviour Classification
With Multi Relational Graph Convolutional Networks

Link to our paper on arxiv : https://arxiv.org/abs/2002.00786
For any queries mail : saisravan.m@research.iiit.ac.in (or) mahtab.sandhu@gmail.com

Our method detects and classifies objects of interest (vehicles) in a video scene into 6 classes moving away, moving towards us, parked, lane change(L->R), lane change(R->L), overtake. Note that our method is not based on classifying the ego-vehicle.

Base-line Implementation details

Apart from comparisons in the paper, we compare with Structural-RNN, a LSTM based graph network. Since the tasks in their paper confine only to driver-anticipation, we use one of their methods similar to our task. Specifically, we use the detection method of activity-anticipation mentioned in the paper due to the similarity in the architecture and task . We use Vehicles as Humans and Lane Markings as Objects in their architecture for our purpose. Similar to the Human-Object, Human-Human and Object-Object interactions, we observe the Vehicle-Lane, Vehicle-Vehicle and Lane-Lane interactions for all time-steps. We give features similar to the baselines in the paper.

Method	St-RNN	LSTM	LSTM + Multi-head attetnion	Ours
Moving away	76	37	56	85.3
moving towards us	51	35	41	89.5
Parked	83	34	46	94.8
lane-change(L->R)	52	6	8	84.1
lane-change(R->L)	57	24	27	86.4
Overtake	63	15	13	72.3

Dataset

We selected 3 main datasets to perform the experiments.

Apollo
Kitti
Indian

Graphs for all datasets can be downloaded from graphs.
For information on how each graph is stored as a npz file, go through the README file in the same link.

On apollo we have selected sequences from scene-parsing dataset and picked around 70 small sequences(each containing aroud 100 images) manually that include behaviours of our interest. Similarly on Kitti, we use tracking sequences 4,5,10 which are in line with our class requirement.

Installation

Requirements

dgl
pytorch == 1.2.0
pandas
numpy
tqdm

Installing without GPU:

pip3 install requirements.txt

To install and use GPU for dgl, cuda support can be installed from their official website, dgl .
And set use_cuda = 1 in training/testing codes.

Testing and Training on apollo dataset

git clone https://github.com/ma8sa/temporal-MR-GCN.git
cd temporal-MR-GCN
--testing
python3 lstm_rgcn_test_apollo.py
--training
python3 lstm_rgcn_train_apollo.py

NOTE : Make sure to extract the corresponding graphs(lstm_graphs_apollo) and place it in the same folder where you are running the code from.

In training, lstm_rgcn_train_apollo has all the parameters in main function to tune. main_model.py contains the complete model. rgcn_layer.py contains the dgl format of MR-GCN. graphs_preproc_apollo.py conatains all the data preprocessing methods used.

Testing on Indian/Kitti dataset (Transfer Learning)

for indian,
python3 lstm_rgcn_test_ind_kitti.py indian
for kitti,
python3 lstm_rgcn_test_ind_kitti.py kitti

NOTE : Make sure to extract the corresponding graphs (lstm_graphs_kitti for kitti and lstm_graphs_indian for indian) and place it in the same folder where you are running the code from.

RESULTS

0->Move forward
1->Moving towards us
2->Parked
3-> lane-change(L->R)
4-> lane-change(R->L)
5-> Overtake

Results on Apollo

| | 0 | 1 | 2 | 3 | 4 | 5 | | ————- | ————- | ———— | ———— | ———— | ———— | ———— | | class accuracy(train)| 95 | 98 | 97 | 96 | 96 | 97 |
| class counts(train) | 2673 | 685 | 3574 |424 | 452 | 525 | | class accuracy(val) | 85 | 89 | 94 | 84 | 86 | 72 | | class counts(val) | 814 | 237 | 1415 | 162 | 130 | 73 |

Since the number of cars showing overtake behaviour are less, we augmented and added few synthetic-graphs and augmented data to train data only for apollo. With a little more data, the model can clearly learn overtake too above 80%, as number of synthetic graphs added were too low 76.

Results on Kitti tested with weights trained on Apollo

| | 0 | 1 | 2 | | ————- | ————- | ———— | ———— | | class accuracy| 99 | 98 | 98 | | class counts | 504 | 230 | 674 |

Results on Indian tested with weights trained on Apollo

| | 0 | 1 | 2 | | ————- | ————- | ———— | ———— | | class accuracy| 99 | 92 | 99 | | class counts | 324 | 229 | 2547 |