Motion tracking#
For a more detailed description see: https://www.phenopype.org/gallery/projects/motion-tracking/
import phenopype as pp
import os
import pandas as pd
import trackpy as tp ## install with `pip install trackpy`
## my root directory - modify as you see fit
os.chdir(r"D:\science\packages\phenopype\phenopype-gallery_exec")
## my laptop has a small screen, so I use a smaller phenopype window
pp._config.window_max_dim = 800
Initialize motion-tracker#
Video analsyis in phenopype works a bit differently than images. Rather than making a project, videos are analyzed one by one. However. to batch process images, you can recycle some of the code. Here, I show how to analyze a single video though. First we initialize the motion tracker with our example video, and specify the video output.
mt = pp.motion_tracker(r"data_raw\motion-tracking\isopods_fish.mp4")
mt.video_output(save_suffix="v1", dirpath=r"phenopype\motion-tracking")
Then we create a mask to only include the gravel area - this will exclude the walls where reflections and floating particles may result in false positives. A second mask in the center of the area can later be used to compare whether isopods tend to stay more on the side (shy behavior) or whether they move freely (bold behavior). In each captured frame there will be an entry for each isopods in which area it was detected.
masks = pp.preprocessing.create_mask(mt.image, label="full", annotation_id="a")
masks = pp.preprocessing.create_mask(mt.image, label="center", annotations=masks, annotation_id="b")
Now we set up two tracking_methods: the “fish” method is set to "single" and will only capture the largest object, whereas the isopod method will capture all objects smaller than 30 pixels. You can play around with blur and threshold settings to see if you can get better results. The operations argument is used to return specific information of the detected objects. We specify a few parameters to retrieve phenotypic information of the isopods. Later, we can then analyze which phenotypes were foraged on which sediment background.
fish = pp.tracking_method(label="fish", remove_shadows=True, min_length=30,
overlay_colour="red", mode="single",
blur=15, # bigger blurring kernel
threshold=200 #higher sensitivity
)
isopod = pp.tracking_method(label="isopod", remove_shadows=True, max_length=30,
overlay_colour="green", mode="multiple",
blur=9, # smaller blurring kernel
threshold=180, # lower sensitivity
operations=["diameter", # isopod size
"area", # isopod area
"grayscale", # isopod pigmentation
"grayscale_background"] # background darkness
)
Finally, we pass the methods on the detection settings, and also activate consecutive masking (c_mask=True). This option will prohibit repeated detection of objects when several tracking methods are applied. For example, inside the detected fish-area sometimes isopod objects are detected, due to artifacts or incomplete subtraction results. Consecutive masking will “block” an area after the first method has been applied - the order by which methods are applied matters. The following settings will create a rectangle shaped mask around the fish with a 200 pixel border.
mt.detection_settings(methods=[fish, isopod],
c_mask=True,
c_mask_shape="rect",
c_mask_size=200,
masks=masks)
coordinates = mt.run_tracking()
Note that the objects outside the masks were still detected and drawn onto the overlay. If you do not want to include them in the trajectory analysis, or your analysis, simply filter those rows out (i.e., the rows where all masks return False).
After completing the tracking, we end up with a big data frame of all contours
coordinates.to_csv(os.path.join(working_dir, mt.name + "coordinates.csv"), sep=',')
coordinates
Analyzing tracking results with trackpy#
Now we can use the frame-wise coordinates to construct trajectories of isopods and fish. For this we will use the excellent trackpy library (https://soft-matter.github.io/trackpy). Trackpy is a Python package for particle tracking in 2D, 3D, and higher dimensions.
Here you need to find out what works best for your specific case - the larger search_range or memory are, the more challenging it is for the algorithm to find a solution, especially if you have many moving objects in your video. With only one, it should be ok to go to high values. The filtering step is optional, but can be useful to eliminate spurious trajectories.
## matplotlib requires rgb format, but opencv/phenopype use bgr. we need to convert our static background image
import cv2
mt.image = cv2.cvtColor(mt.image, cv2.COLOR_BGR2RGB)
## fish trajectories
fish_df = coordinates[coordinates['label'] == "fish" ] # just use the fish-coordinates
traj_fish = tp.link_df(fish_df,
search_range = 100, #how for to look for the fish in the next frame. can be large if fish swims fast
memory=60, # how long can the fish sit still
neighbor_strategy="KDTree",
link_strategy="nonrecursive")
traj_fish_filter = tp.filtering.filter_stubs(traj_fish,
threshold=20) # filter out particles that were only found 20 times
## plot
plot = tp.plot_traj(traj_fish_filter,
superimpose=mt.image)
fig1 = plot.get_figure()
fig1.savefig(os.path.join(working_dir, mt.name + "_fish_trajectories.png"), dpi=300)
For isopods, we perform one additional step: because we know that we have 20 isopds, we remove frames with more particles.
## isopod trajectories
df_isopod = coordinates[coordinates['label'] == "isopod" ] # just use the isopod-coordinates
df_isopod_filtered = df_isopod.groupby("frame").filter(lambda x: len(x) < 20) # filter out frames with too many points
traj_isopod = tp.link(df_isopod_filtered,
search_range = 50, # isopods are slow, so this can be small. especially important when using "multiple"
memory=200, # isopods can sit still longer
neighbor_strategy="KDTree",
link_strategy="nonrecursive")
traj_isopod_filter = tp.filtering.filter_stubs(traj_isopod,
threshold=10) # each particle needs to be found at least 10 times
##p lot
plot = tp.plot_traj(traj_isopod_filter, superimpose=mt.image, colorby="particle")
fig1 = plot.get_figure()
fig1.savefig(os.path.join(working_dir, mt.name + "_isopod_trajectories.png"), dpi=300)
Finally we save all trajectories into one DataFrame and export them into to the specified directory, so you can do fine tuning in your favorite data analysis program (e.g. R or in Python with Pandas).
# save all
df = traj_isopod_filter.append(pd.DataFrame(data = traj_fish_filter), ignore_index=True)
df.to_csv(os.path.join(working_dir, mt.name + "_trajectories.csv"), sep=',', index=False)