This repo contains code to run feature inference and finetune backbone models on the Soccernet dataset. The Soccernet features are used in down-stream tasks, in particular event spotting and replay grounding. The winning solutions for the CVPR 2021 and 2022 ActivityNet Challange, Temporal Localization track, used features from this Soccernet Baidu Features repo. This repo makes a single model based on TimeSFormer fully opensource with pretrained weights available to run inference or train on any video.
Load the features in as follows:
import numpy as np
a = np.load('features.py')
The whole numpy array could look like this:
array({'cls_score_all': array([[[-1.1469777, 1.2446018, 1.7308334, ..., -3.7974577,
-3.9211693, 9.876837 ]],
[[-1.1502348, 1.2446928, 1.7270398, ..., -3.7986917,
-3.922049 , 9.873158 ]],
[[-1.1553113, 1.2458539, 1.7238224, ..., -3.8000093,
-3.9231768, 9.866276 ]],
...,
[[-1.1867781, 1.2394937, 1.7207626, ..., -3.7993345,
-3.9266846, 9.682111 ]],
[[-1.1717311, 1.3373474, 1.7501842, ..., -3.809371 ,
-3.9297879, 9.226656 ]],
[[-1.0841012, 1.4193138, 1.7844383, ..., -3.8307285,
-3.9134252, 8.695862 ]]], dtype=float32), 'event_times': array([[[0.61532754, 0.48600802, 0.384552 , ..., 0.4478651 ,
0.63936603, 1. ]],
[[0.595665 , 0.4832232 , 0.37086746, ..., 0.45608336,
0.63960594, 1. ]],
[[0.57046086, 0.48085424, 0.35832942, ..., 0.46679202,
0.63987064, 1. ]],
...,
[[0.5881998 , 0.47877887, 0.51381993, ..., 0.42987552,
0.64069575, 1. ]],
[[0.52614266, 0.48890987, 0.49975097, ..., 0.43349805,
0.63012475, 1. ]],
[[0.61633825, 0.48636538, 0.46834537, ..., 0.5310676 ,
0.6127171 , 1. ]]], dtype=float32), 'features': array([[[-0.0132505 , 0.02137352, -0.21291943, ..., -0.07798928,
0.21858051, -0.23259796]],
[[-0.01204887, 0.02450565, -0.21337315, ..., -0.07861489,
0.21864586, -0.23138241]],
[[-0.01019135, 0.02715371, -0.21588442, ..., -0.07936225,
0.2186201 , -0.22969423]],
...,
[[ 0.01318725, -0.02435086, -0.12325487, ..., -0.00325756,
0.21175556, -0.11824775]],
[[-0.00307255, -0.02015326, -0.10546789, ..., 0.02826597,
0.18887816, -0.095611 ]],
[[-0.01093192, -0.06524777, -0.15625732, ..., 0.0292125 ,
0.17296971, -0.11872528]]], dtype=float32)}, dtype=object)
Extract the useful features (other ones were left unimplemented):
data_dict = a.item()
features = data_dict['features']
features.shape = (3489, 1, 768). The old soccernet features have a shape of (number of frames, 8576) being an ensemble of models. In the new feature, the feature dimension is reduced to just 768, 3489 is the number of frames. One still need to train a stage 2 model to use these features (I do not have access to the one I verified the features with any more. Let me know if someone trained one and wants to share. )
This pipeline can be used for extracting features for Soccernet videos or other broadcast soccer videos in general.
Install cuda、cudnn、nccl first. Clone this repo.
pip3 install paddlepaddle-gpu --upgrade
pip3 install --upgrade -r requirements.txt
A config file for the video file in the follow format needs to be constructed and its filename set as VIDEO_CONFIG. It needs to have the path to the video file, fps of the video and the total length in seconds.
{
"filename": "/mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_5fps/england_epl.2014-2015.2015-02-21_-_18-00_Chelsea_1_-_1_Burnley.1_LQ.mkv",
"fps": 5,
"length_secs": 3196
}
Download weight file and set the filename to be WEIGHT_FILE.
Set enviroment variables LOG_DIR to save logs. The test config do not need to be changed. Set INFERENCE_DIR. This is where the features.npy containing the weights will be saved.
python3.7 -B -m paddle.distributed.launch --gpus='0' --log_dir=$LOG_DIR main.py \
--test -c data/soccernet_inference/soccernet_pptimesformer_k400_one_file_inference.yaml \
-w $WEIGHT_FILE -o inference_dir=$INFERENCE_DIR \
-o DATASET.test.file_path=$VIDEO_CONFIG
Inference on a 45 min video at 5fps takes approximately 2-3 hours.
In this section, we will be extracting short 10 seconds clips from the Soccernet videos and the clips will have a lower resolution for training.
Download raw HQ video data from the Soccernet official website. Put it in a folder and set $RAW_VIDEOS_ROOT to that folder.
Notes:
-
Three resolutions of videos are provided 1080p, 720p, 224p. The caveat here is that the scripts assumes 1080p is used. The key difference is that compared to the 720p, 224p videos, which are trippmed from the gamestart to the end of the game. Therefore, a lot of the offsets in the scripts need to be removed.
-
The code as is only lowers the resolution for training clips. Potential resource saving (decoding time and memory) can be achieved using even lower fps. Decoding speed is the bottleneck in training. Lower fps sacrifices data diversity a little bit. This tradeoff was not investigated but is worth considering if resource constrained.
Run the following command. The output is commands to extract the clips. Redirect the output into a file because there are many extraction commands and you will need to split the file to run the commands in parallel. Choose a folder and set the environment variable $CLIPS_FOLDER to save your clips.
python data/soccernet/generate_training_short_clips.py --input_folder $RAW_VIDEOS_ROOT --clips_folder=$CLIPS_FOLDER > data/soccernet/generate_training_short_clips.sh
Make a folder to save the paralle scripts:
mkdir data/soccernet/short_clips_parallel
This is a sample to split into 400 equal parts:
for i in {0..399};
do
sed -n ${i}~400p data/soccernet/generate_training_short_clips.sh > data/soccernet/short_clips_parallel/${i}.sh;
done
Manchester_United_1_-_1_Liverpool.1_HQ
The commands may get stuck on a few videos and render the jobs stuck, so here is another split into 401 parts to run after the above job.
for i in {0..400};
do
sed -n ${i}~401p data/soccernet/generate_training_short_clips.sh > data/soccernet/short_clips_parallel_401/${i}.sh;
done
Set $RAW_VIDEOS_ROOT to the root of the folder where all HQ Soccernet videos are downloaded into. Set $LABELS_ROOT to the folder where the labels are.
python data/soccernet/generate_labels.py \
--extension mkv \
--raw_videos_root $RAW_VIDEOS_ROOT \
--labels_root $LABELS_ROOT \
--clips_folder $CLIPS_FOLDER
Sample code to run them on a slurm based cluster:
for i in {0..399};
do
sbatch -p 1080Ti,2080Ti --gres=gpu:1 --cpus-per-task 4 -n 1 --wrap \
"echo no | bash data/soccernet/short_clips_parallel/${i}.sh" \
--output="data/soccernet/short_clips_parallel/${i}.log"
done
For the seconds split,
for i in {0..400};
do
sbatch -p 1080Ti,2080Ti,M40x8 --gres=gpu:1 --cpus-per-task 4 -n 1 --wrap \
"echo no | bash data/soccernet/short_clips_parallel_401/${i}.sh" \
--output="data/soccernet/short_clips_parallel_401/${i}.log"
done
Generate label_mapping.txt (for category to category index map) and dense.list files.
python data/soccernet_dense_anchors/generate_dense_anchors_labels.py \
--clips_folder /mnt/storage/gait-0/xin/dataset/soccernet_456x256 \
--output_folder ./
Split into train, val, test
python data/soccernet/split_annotation_into_train_val_test.py \
--annotation_file dense.list \
--clips_folder ./ \
--mode json
Then you will have train.list, val.list, test.list in this folder.
python -u -B -m paddle.distributed.launch --gpus="0,1,2,3,4,5,6,7" --log_dir=logs/soccernet_pptimesformer_k400_videos_dense main.py --validate -c data/soccernet/experiments/pptimesformer/soccernet_pptimesformer_k400_videos_dense.yaml -w pretrained_weights/ppTimeSformer_k400_16f_distill.pdparams
sbatch -p V100x8_mlong --gres=gpu:8 --cpus-per-task 40 -n 1 \
--wrap "python -u -B -m paddle.distributed.launch --gpus="0,1,2,3,4,5,6,7" --log_dir=logs/soccernet_pptimesformer_k400_videos_dense main.py --validate -c data/soccernet/experiments/pptimesformer/soccernet_pptimesformer_k400_videos_dense.yaml -w pretrained_weights/ppTimeSformer_k400_16f_distill.pdparams" \
--output="/mnt/storage/gait-0/xin//logs/soccernet_pptimesformer_k400_videos_dense.log"
The feature extractor training log is here. The folder also contains the trained NetVLAD++ action detection result (final mAP by category). Soccernet-features/data/soccernet/archived_logs/
Could be convenient to consider gamestart.
This generates a sample script that converts all of the Soccernet videos. We have to be a little bit careful with if the video starts from gamestart or not. (trim_to_gametime)
python data/soccernet_inference/convert_video_to_lower_resolution_for_inference.py \
--input_folder /mnt/big/multimodal_sports/SoccerNet_HQ/raw_data \
--output_folder /mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference > \
data/soccernet_inference/convert_video_to_lower_resolution_for_inference.sh
In order to run inference on half game videos, besides lower resolution to 456x256, the videos had to be run on a lower fps. (The inference sampler decodes the whole video. Then it extracts clips of time intervals [0,n), [1,n+1) etc.
Each 45 min video files takes about 10 min to convert to lower resolution. So we parallelize to 100 such jobs.
for i in {0..99};
do
sed -n ${i}~100p data/soccernet_inference/convert_video_to_lower_resolution_for_inference.sh > data/soccernet_inference/convert_video_to_lower_resolution_for_inference_parallel/${i}.sh;
done
Run the parallel jobs on a cluster, slurm based for example.
for i in {0..99};
do
sbatch -p 1080Ti,2080Ti,TitanXx8 --gres=gpu:1 --cpus-per-task 4 -n 1 --wrap \
"echo no | bash data/soccernet_inference/convert_video_to_lower_resolution_for_inference_parallel/${i}.sh" \
--output="data/soccernet_inference/convert_video_to_lower_resolution_for_inference_parallel/${i}.log"
done
python data/soccernet_dense_anchors/generate_whole_video_inference_jsons.py \
--videos_folder /mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference \
--output_folder /mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists
INFERENCE_WEIGHT_FILE=output/ppTimeSformer_dense_event_lr_100/ppTimeSformer_dense_event_lr_100_epoch_00007.pdparams
INFERENCE_JSON_CONFIG=/mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists/spain_laliga.2016-2017.2017-05-21_-_21-00_Malaga_0_-_2_Real_Madrid.2_LQ.mkv
INFERENCE_DIR_ROOT=/mnt/storage/gait-0/xin/soccernet_features
SHORTNAME=`basename "$INFERENCE_JSON_CONFIG" .mkv`
INFERENCE_DIR=$INFERENCE_DIR_ROOT/$SHORTNAME
echo $INFERENCE_DIR
mkdir $INFERENCE_DIR
python3.7 -B -m paddle.distributed.launch --gpus="0" --log_dir=log_videoswin_test main.py --test -c data/soccernet_inference/soccernet_pptimesformer_k400_videos_dense_event_lr_50_one_file_inference.yaml -w $INFERENCE_WEIGHT_FILE -o inference_dir=$INFERENCE_DIR -o DATASET.test.file_path=$INFERENCE_JSON_CONFIG
Needed to sample down to 5fps because of our scale to run all Soccernet data
CONFIG_DIR=/mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists_5fps/ INFERENCE_WEIGHT_FILE=output/ppTimeSformer_dense_event_lr_100_fc_lr_100/ppTimeSformer_dense_event_lr_100_fc_lr_100_epoch_00038.pdparams INFERENCE_WEIGHT_FILE=output/ppTimeSformer_dense_event_lr_100_fc_lr_100_balanced/ppTimeSformer_dense_event_lr_100_fc_lr_100_balanced_epoch_00004.pdparams
INFERENCE_DIR_ROOT=/mnt/storage/gait-0/xin/soccernet_features_4_ppTimeSformer INFERENCE_DIR_ROOT=/mnt/storage/gait-0/xin/soccernet_features_5_ppTimeSformer_balanced
for FILE in /mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists_5fps/*;
do
line=basename "$FILE" .mkv
INFERENCE_JSON_CONFIG=$CONFIG_DIR/$line.mkv
INFERENCE_DIR=$INFERENCE_DIR_ROOT/$line
echo "rm /mnt/storage/gait-0/xin//logs/$line.log"
echo "sbatch -p 1080Ti --gres=gpu:1 --cpus-per-task 4 -n 1
--wrap "python3.7 -B -m paddle.distributed.launch --gpus='0' --log_dir=/mnt/storage/gait-0/xin//logs/$line main.py --test -c data/soccernet_inference/soccernet_pptimesformer_k400_videos_dense_event_lr_50_one_file_inference.yaml -w $INFERENCE_WEIGHT_FILE -o inference_dir=$INFERENCE_DIR -o DATASET.test.file_path=$INFERENCE_JSON_CONFIG"
--output="/mnt/storage/gait-0/xin//logs/$line.log" "
done > unfinished_inference.sh
These command help you check the last lines of the jobs and see if any bug arose.
CONFIG_DIR=/mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists_5fps/ INFERENCE_WEIGHT_FILE=output/ppTimeSformer_dense_event_lr_100/ppTimeSformer_dense_event_lr_100_epoch_00028.pdparams INFERENCE_DIR_ROOT=/mnt/storage/gait-0/xin/soccernet_features
for FILE in /mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists_5fps/*;
do
line=basename "$FILE" .mkv
INFERENCE_JSON_CONFIG=$CONFIG_DIR/$line.mkv
INFERENCE_DIR=$INFERENCE_DIR_ROOT/$line
log_file=/mnt/storage/gait-0/xin//logs/$line.log echo $log_file tail -n 1 /mnt/storage/gait-0/xin//logs/$line.log tail -n 1 /mnt/storage/gait-0/xin//logs/$line/workerlog.0 done
python data/soccernet_dense_anchors/check_unfinished_inference.py
--inference_root /mnt/storage/gait-0/xin/soccernet_features_4_ppTimeSformer/ > inference_matches_todo.txt
CONFIG_DIR=/mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists_5fps/ INFERENCE_WEIGHT_FILE=output/ppTimeSformer_dense_event_lr_100/ppTimeSformer_dense_event_lr_100_epoch_00028.pdparams INFERENCE_DIR_ROOT=/mnt/storage/gait-0/xin/soccernet_features
cat inference_matches_todo.txt | while read line do INFERENCE_JSON_CONFIG=$CONFIG_DIR/$line.mkv INFERENCE_DIR=$INFERENCE_DIR_ROOT/$line
echo "sbatch -p 1080Ti --gres=gpu:1 --cpus-per-task 4 -n 1
--wrap "python3.7 -B -m paddle.distributed.launch --gpus='0' --log_dir=/mnt/storage/gait-0/xin//logs/$line main.py --test -c data/soccernet_inference/soccernet_pptimesformer_k400_videos_dense_event_lr_50_one_file_inference.yaml -w $INFERENCE_WEIGHT_FILE -o inference_dir=$INFERENCE_DIR -o DATASET.test.file_path=$INFERENCE_JSON_CONFIG"
--output="/mnt/storage/gait-0/xin//logs/$line.log" "
done > unfinished_inference.sh
CONFIG_DIR=/mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists/ INFERENCE_WEIGHT_FILE=output/ppTimeSformer_dense_event_lr_100/ppTimeSformer_dense_event_lr_100_epoch_00028.pdparams INFERENCE_DIR_ROOT=/mnt/storage/gait-0/xin/soccernet_features
cat inference_matches_todo.txt | while read line do INFERENCE_JSON_CONFIG=$CONFIG_DIR/$line.mkv INFERENCE_DIR=$INFERENCE_DIR_ROOT/$line
echo "sbatch -p 1080Ti,2080Ti,TitanXx8 --gres=gpu:1 --cpus-per-task 4 -n 1
--wrap "python3.7 -B -m paddle.distributed.launch --gpus='0' --log_dir=/mnt/storage/gait-0/xin//logs/$line main.py --test -c data/soccernet_inference/soccernet_pptimesformer_k400_videos_dense_event_lr_50_one_file_inference.yaml -w $INFERENCE_WEIGHT_FILE -o inference_dir=$INFERENCE_DIR -o DATASET.test.file_path=$INFERENCE_JSON_CONFIG"
--output="/mnt/storage/gait-0/xin//logs/$line.log" "
done
Errors:
/mnt/storage/gait-0/xin//logs/spain_laliga.2015-2016.2015-09-12_-_17-00_Espanyol_0_-_6_Real_Madrid.1_LQ.log
Runtime:
/mnt/storage/gait-0/xin//logs/spain_laliga.2015-2016.2015-09-12_-_17-00_Espanyol_0_-_6_Real_Madrid.1_LQ/workerlog.0
Note:
nano /mnt/storage/gait-0/xin/dataset/soccernet_456x256_inference_json_lists_5fps//england_epl.2016-2017.2016-11-06_-19-30_Leicester_1-_2_West_Brom.1_LQ.mkv override time to 3780 secs