Heatmap Module

Heatmap module.

class landmarker.heatmap.GaussianHeatmapGenerator

Gaussian heatmap generator for generating heatmaps from landmarks.

Parameters:

• nb_landmarks (int) – number of landmarks

• sigmas (float or list[float] or torch.Tensor or np.ndarray) – sigmas of the gaussian heatmap function

• gamma (float or None) – scaling factor of the gaussian heatmap function

• rotation (float or list[float] or torch.Tensor or np.ndarray) – rotation of the gaussian heatmap function

• heatmap_size (tuple[int, int]) – size of the heatmap

• full_map (bool) – whether to return the full heatmap or only the part around the landmark

• learnable (bool) – whether the sigmas and rotation are learnable

• background (bool) – whether to add a background channel to the heatmap

• all_points (bool) – whether to add a channel with the sum of all the landmarks

• continuous (bool) – whether to use continuous or discrete landmarks

• device (str) – device to use for the heatmap generator

__init__(nb_landmarks, sigmas=1.0, gamma=None, rotation=0, heatmap_size=(512, 512), full_map=True, learnable=False, background=False, all_points=False, continuous=True, device='cpu')

Parameters:

• nb_landmarks (int) –

• sigmas (float | list[float] | Tensor | ndarray) –

• gamma (float | None) –

• rotation (float | list[float] | Tensor | ndarray) –

• heatmap_size (tuple[int, int]) –

• full_map (bool) –

• learnable (bool) –

• background (bool) –

• all_points (bool) –

• continuous (bool) –

• device (str | device) –

Return type:

None

heatmap_fun(landmark_t, coords, covariance, gamma)

Gaussian heatmap function

Parameters:

• landmark_t (torch.Tensor) – coordinates of the landmark (y, x)

• coords (torch.Tensor) – coordinates of the pixel (y, x)

• covariance (torch.Tensor) – covariance matrix (y, x)

• gamma (float or None) – scaling factor of the heatmap function

Returns:

value of the gaussian heatmap function for the given pixel

Return type:

torch.Tensor

class landmarker.heatmap.LaplacianHeatmapGenerator

Laplacian heatmap generator for generating heatmaps from landmarks.

Parameters:

• nb_landmarks (int) – number of landmarks

• sigmas (float or list[float] or torch.Tensor or np.ndarray) – sigmas of the Laplacian heatmap function

• gamma (float or None) – scaling factor of the Laplacian heatmap function

• rotation (float or list[float] or torch.Tensor or np.ndarray) – rotation of the laplacian heatmap function

• heatmap_size (tuple[int, int]) – size of the heatmap

• full_map (bool) – whether to return the full heatmap or only the part around the landmark

• learnable (bool) – whether the sigmas and rotation are learnable

• background (bool) – whether to add a background channel to the heatmap

• all_points (bool) – whether to add a channel with the sum of all the landmarks

• continuous (bool) – whether to use continuous or discrete landmarks

• device (str) – device to use for the heatmap generator

__init__(nb_landmarks, sigmas=1.0, gamma=None, rotation=0, heatmap_size=(512, 512), full_map=True, learnable=False, background=False, all_points=False, continuous=True, device='cpu')

Parameters:

• nb_landmarks (int) –

• sigmas (float | list[float] | Tensor | ndarray) –

• gamma (float | None) –

• rotation (float | list[float] | Tensor | ndarray) –

• heatmap_size (tuple[int, int]) –

• full_map (bool) –

• learnable (bool) –

• background (bool) –

• all_points (bool) –

• continuous (bool) –

• device (str | device) –

Return type:

None

heatmap_fun(landmark_t, coords, covariance, gamma=None)

Laplacian heatmap function

Parameters:

• landmark_t (torch.Tensor) – coordinates of the landmark (y, x)

• coords (torch.Tensor) – coordinates of the pixel (y, x)

• covariance (torch.Tensor) – covariance matrix (y, x)

• gamma (float or None) – scaling factor of the heatmap function

Returns:

value of the gaussian heatmap function for the given pixel

Return type:

torch.Tensor

landmarker.heatmap.coord_argmax(heatmap)

Returns the coordinates of the maximum value of the heatmap for each batch and channel (landmark).

Parameters:

heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_cov_from_gaussian_ls(heatmap, gamma=None, ls_library='scipy')

Returns the modal coordinates and covariance matrix from a heatmap through fitting the heatmap on Gaussian distribution with a specicic scaling factor gamma with help of least squares optimization.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• gamma (float) – gamma parameter of the gaussian heatmap generator

• ls_library (str) – library to use for least squares optimization. (scipy or pytorch)

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_cov_windowed_weigthed_sample_cov(heatmap)

Calculate the covariance matrix from a heatmap by calculating the mean of the heatmap values weighted by the heatmap values. The window is determined by the distance to the closest edge.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_local_soft_argmax(heatmap, window=3, t=10.0)

Returns coordiantes through applying the local soft-argmax function on the heatmaps.

Source: “Subpixel Heatmap Regression for Facial Landmark Localization” - Bulat et al. (2021)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• idx_pred (torch.Tensor) – predicted indices of shape (B, C, 2)

• window (int) – local window size

• t (float) – temperature that controls the resulting probability map

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_soft_argmax(heatmap, eps=1e-06, logit_scale=False, require_grad=False)

Returns the spatial mean over the softmax distribution of the heatmap. Source: “2D/3D Pose Estimation and Action Recognition using Multitask

Deep Learning” - Luvizon et al. (2018)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• eps (float) – epsilon to avoid division by zero

• logit_scale (bool) – whether the input is logit scaled

• require_grad (bool) – whether to require gradient for the coordinates

Return type:

Tensor

landmarker.heatmap.coord_soft_argmax_cov(heatmap, eps=1e-06, logit_scale=True, require_grad=True)

Returns the spatial mean over the softmax distribution of the heatmap and the covariance matrix. The covariance matrix is the weigthed sample covariance matrix.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• eps (float) – epsilon to avoid division by zero

• logit_scale (bool) – whether the input is logit scaled

• require_grad (bool) – whether to require gradient for the coordinates

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_weighted_spatial_mean(heatmap, eps=1e-05)

Returns the spatial weighted mean of the possitive elements of the heatmap by the heatmap values. Source: “UGLLI Face Alignment: Estimating Uncertainty with

Gaussian Log-Likelihood Loss” - Kumar et al. (2019)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• eps (float) – epsilon to avoid division by zero

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_weighted_spatial_mean_cov(heatmap, eps=1e-05)

Returns the spatial weighted mean and the weighted sample covariance of the possitive elements of the heatmap by the heatmap values.

source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• eps (float) – epsilon to avoid division by zero

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.cov_from_gaussian_ls(heatmap, coords, gamma=None, ls_library='scipy')

Returns covariance matrix from a heatmap through fitting the heatmap on Gaussian distribution with a specicic scaling factor gamma and specified coordinates with help of least squares optimization.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• gamma (float) – gamma parameter of the gaussian heatmap generator

• ls_library (str) – library to use for least squares optimization. (scipy or pytorch)

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_coord_to_weighted_sample_cov(heatmap, coords, eps=1e-06, apply_softmax=False)

Returns the weighted sample covariance matrix from a heatmap and coordinates.

source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• eps (float) – epsilon to avoid division by zero

• apply_softmax (bool) – whether to apply softmax on the heatmap

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord(heatmap, offset_coords=0, method='argmax')

Returns the retrieved coordinates via specified method from a heatmap. The offset_coords is used to remove the first offset_coords coordinates from the heatmap. This is used to remove the background class (if present).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• offset_coords (int) – number of coordinates to remove

• method (str) – method to retrieve the coordinates

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord_cov(heatmap, method='soft_argmax', eps=1e-06, logit_scale=True, require_grad=True)

Returns the modal coordinates and covariance matrix from a heatmap.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• method (str) – method to retrieve the coordinates

• eps (float) – epsilon to avoid division by zero

• logit_scale (bool) – whether the input is logit scaled

• require_grad (bool) – whether to require gradient for the coordinates

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord_enlarge(heatmap, offset_coords=0, method='argmax', enlarge_factor=1, enlarge_dim=None, enlarge_mode='bilinear')

Returns the retrieved coordinates via specified method from an enlarged heatmap. The offset_coords is used to remove the first offset_coords coordinates from the heatmap. This is used to remove the background class (if present).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• offset_coords (int) – number of coordinates to remove

• method (str) – method to retrieve the coordinates

• enlarge_factor (int) – factor to enlarge the heatmap

• enlarge_dim (tuple[int, int] or None) – dimensions to enlarge the heatmap to

• enlarge_mode (str) – interpolation mode to enlarge the heatmap

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_multiple_coord(heatmaps, window=5, threshold=None, method='argmax')

Returns the multiple coordinates of the maximum value of the heatmap for each batch and channel. Additionally, the scores of the maxima are returned. If threshold is not None, only the maxima with a score higher than the threshold are returned.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• window (tuple) – window size

• threshold (float) – threshold to remove points. If None, no threshold is applied. Default None.

• method (str) – method to retrieve the coordinates

• heatmaps (Tensor) –

Returns:

list of coordinates of the maxima for each batch and

channel, with the first list the batch, the second list the channel and the third list the coordinates

(list(list(list(float)))): list of scores of the maxima for each batch and channel, with

the first list the batch, the second list the channel and the third list the scores.

Return type:

(list(list(list(torch.Tensor))))

landmarker.heatmap.non_maximum_surpression(heatmap, window=3)

Non-Maximum Surpression (NMS)

source: Efficient Non-Maximum Suppression - Neubeck and Van Gool (2006)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (H, W)

• window (int) – window size

Returns:

list of coordinates of the maxima

Return type:

(list(tuple(int, int)))

landmarker.heatmap.non_maximum_surpression_local_soft_argmax(heatmap, window=3)

Returns the coordinates of the maximum value of the heatmap for each batch and channel (landmark), with the soft-argmax function applied on the heatmaps.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (H, W)

• window (int) – window size

Returns:

list of coordinates of the maxima

Return type:

(list(tuple(int, int)))

landmarker.heatmap.weighted_sample_cov(heatmap, coords, eps=1e-06)

Calculate the covariance matrix from a heatmap by calculating the mean of the heatmap values weighted by the heatmap values. source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• eps (float) – epsilon to avoid division by zero

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)