I am working on a segmentation project and would like to use this awesome project to stabilize the segment result. I used base_tracker as the tracker of my project. however, it went OOM just by tracking 29 objects, eating up all my GPU memory. I have the policy that will remove the track of objects that does not appear again in 30 frames but I can hardly went pass 10 frames of the video. logs are below.
final text_encoder_type: bert-base-uncased
Hyperparameters read from the model weights: C^k=64, C^v=512, C^h=64
Single object mode: False
Processing frames: 0%| | 0/1326 [00:00<?, ?frame/s]
detecting new objects
found 11 new objects, adding to tracker, current 11 objects to track
tracked one frame
painted one frame
Processing frames: 0%| | 1/1326 [00:06<2:30:35, 6.82s/frame]
tracked one frame
painted one frame
Processing frames: 0%| | 2/1326 [00:07<1:14:02, 3.36s/frame]
tracked one frame
painted one frame
Processing frames: 0%| | 3/1326 [00:08<49:29, 2.24s/frame]
tracked one frame
painted one frame
Processing frames: 0%| | 4/1326 [00:09<37:47, 1.71s/frame]
detecting new objects
found 8 new objects, adding to tracker, current 19 objects to track
tracked one frame
painted one frame
Processing frames: 0%| | 5/1326 [00:16<1:22:54, 3.77s/frame]
tracked one frame
painted one frame
Processing frames: 0%| | 6/1326 [00:18<1:07:18, 3.06s/frame]
tracked one frame
painted one frame
Processing frames: 1%|▏ | 7/1326 [00:20<56:21, 2.56s/frame]
tracked one frame
painted one frame
Processing frames: 1%|▏ | 8/1326 [00:21<49:11, 2.24s/frame]
tracked one frame
painted one frame
Processing frames: 1%|▏ | 9/1326 [00:23<44:23, 2.02s/frame]
detecting new objects
found 10 new objects, adding to tracker, current 29 objects to track
tracked one frame
painted one frame
Processing frames: 1%|▏ | 10/1326 [00:31<1:27:31, 3.99s/frame]Traceback (most recent call last):
File "/home/lnt/Documents/jiachen_project/track_anything/tracking_dino.py", line 457, in <module>
painted_frame, mask, tracked_objects = tracking_dino.track(frame)
File "/home/lnt/Documents/jiachen_project/track_anything/tracking_dino.py", line 365, in track
mask, prob, painted_frame = self.tracker.track(current_frame)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/base_tracker.py", line 82, in track
probs, _ = self.tracker.step(frame_tensor, mask, labels) # logits 2 (bg fg) H W
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/inference/inference_core.py", line 66, in step
hidden, pred_logits_with_bg, pred_prob_with_bg = self.network.segment(multi_scale_features, memory_readout,
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/model/network.py", line 110, in segment
hidden_state, logits = self.decoder(*multi_scale_features, hidden_state, memory_readout, h_out=h_out)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/model/modules.py", line 238, in forward
g4 = self.up_8_4(f4, g8)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/model/modules.py", line 190, in forward
g = self.out_conv(g)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/model/group_modules.py", line 49, in forward
out_g = self.conv1(F.relu(g))
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1739, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1750, in _call_impl
return forward_call(*args, **kwargs)
File "/home/lnt/Documents/jiachen_project/track_anything/tracker/model/group_modules.py", line 32, in forward
g = super().forward(g.flatten(start_dim=0, end_dim=1))
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 554, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/lnt/anaconda3/envs/jiachen_project/lib/python3.10/site-packages/torch/nn/modules/conv.py", line 549, in _conv_forward
return F.conv2d(
torch.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.61 GiB. GPU 0 has a total capacity of 23.68 GiB of which 3.19 GiB is free. Including non-PyTorch memory, this process has 19.76 GiB memory in use. Of the allocated memory 13.54 GiB is allocated by PyTorch, and 5.91 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (https://pytorch.org/docs/stable/notes/cuda.html#environment-variables)
Processing frames: 1%|▏ | 10/1326 [00:32<1:11:44, 3.27s/frame]
my GPU is 3090, and it has 24GB memory. that is not enough, so I am thinking if this is the problem of xmem, because it requires lots of memory to run. or it is something that can be optimized, like, track first 10 objects and then track again on 10 - 20 objects. And how can I do that.
I am working on a segmentation project and would like to use this awesome project to stabilize the segment result. I used base_tracker as the tracker of my project. however, it went OOM just by tracking 29 objects, eating up all my GPU memory. I have the policy that will remove the track of objects that does not appear again in 30 frames but I can hardly went pass 10 frames of the video. logs are below.
my GPU is 3090, and it has 24GB memory. that is not enough, so I am thinking if this is the problem of xmem, because it requires lots of memory to run. or it is something that can be optimized, like, track first 10 objects and then track again on 10 - 20 objects. And how can I do that.