Heatmap Module

Heatmap module.

class landmarker.heatmap.GaussianHeatmapGenerator

Gaussian heatmap generator for generating heatmaps from landmarks.

Parameters:

• nb_landmarks (int) – number of landmarks

• sigmas (float or list[float] or torch.Tensor or np.ndarray) – sigmas of the gaussian heatmap function

• gamma (float or None) – scaling factor of the gaussian heatmap function

• rotation (float or list[float] or torch.Tensor or np.ndarray) – rotation of the gaussian heatmap function

• heatmap_size (tuple[int, int]) – size of the heatmap

• learnable (bool) – whether the sigmas and rotation are learnable

• background (bool) – whether to add a background channel to the heatmap

• all_points (bool) – whether to add a channel with the sum of all the landmarks

• continuous (bool) – whether to use continuous or discrete landmarks

• na_zero (bool) – whether to set the value of the landmarks to zero if they are not available

__init__(nb_landmarks, sigmas=1.0, gamma=None, rotation=0, heatmap_size=(512, 512), learnable=False, background=False, all_points=False, continuous=True, na_zero=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

• nb_landmarks (int) –

• sigmas (float | list[float] | Tensor | ndarray) –

• gamma (float | None) –

• rotation (float | list[float] | Tensor | ndarray) –

• heatmap_size (tuple[int, ...]) –

• learnable (bool) –

• background (bool) –

• all_points (bool) –

• continuous (bool) –

• na_zero (bool) –

Return type:

None

heatmap_fun(landmark_t, coords, covariance, gamma)

Gaussian heatmap function

Parameters:

• landmark_t (torch.Tensor) – coordinates of the landmark (y, x) or (z, y, x)

• coords (torch.Tensor) – coordinates of the pixel (y, x) or (z, y, x)

• covariance (torch.Tensor) – covariance matrix (y, x) or (z, y, x)

• gamma (float or None) – scaling factor of the heatmap function

Returns:

value of the gaussian heatmap function for the given pixel

Return type:

torch.Tensor

class landmarker.heatmap.LaplacianHeatmapGenerator

Laplacian heatmap generator for generating heatmaps from landmarks.

Parameters:

• nb_landmarks (int) – number of landmarks

• sigmas (float or list[float] or torch.Tensor or np.ndarray) – sigmas of the Laplacian heatmap function

• gamma (float or None) – scaling factor of the Laplacian heatmap function

• rotation (float or list[float] or torch.Tensor or np.ndarray) – rotation of the laplacian heatmap function

• heatmap_size (tuple[int, int]) – size of the heatmap

• learnable (bool) – whether the sigmas and rotation are learnable

• background (bool) – whether to add a background channel to the heatmap

• all_points (bool) – whether to add a channel with the sum of all the landmarks

• continuous (bool) – whether to use continuous or discrete landmarks

• na_zero (bool) – whether to set the value of the landmarks to zero if they are not available

__init__(nb_landmarks, sigmas=1.0, gamma=None, rotation=0, heatmap_size=(512, 512), learnable=False, background=False, all_points=False, continuous=True, na_zero=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

• nb_landmarks (int) –

• sigmas (float | list[float] | Tensor | ndarray) –

• gamma (float | None) –

• rotation (float | list[float] | Tensor | ndarray) –

• heatmap_size (tuple[int, ...]) –

• learnable (bool) –

• background (bool) –

• all_points (bool) –

• continuous (bool) –

• na_zero (bool) –

Return type:

None

heatmap_fun(landmark_t, coords, covariance, gamma=None)

Laplacian heatmap function

Parameters:

• landmark_t (torch.Tensor) – coordinates of the landmark (y, x)

• coords (torch.Tensor) – coordinates of the pixel (y, x)

• covariance (torch.Tensor) – covariance matrix (y, x)

• gamma (float or None) – scaling factor of the heatmap function

Returns:

value of the gaussian heatmap function for the given pixel

Return type:

torch.Tensor

landmarker.heatmap.coord_argmax(heatmap, spatial_dims=2)

Returns the coordinates of the maximum value of the heatmap for each batch and channel (landmark).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2) or (B, C, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_cov_from_gaussian_ls(heatmap, gamma=None, ls_library='scipy', spatial_dims=2)

Returns the modal coordinates and covariance matrix from a heatmap through fitting the heatmap on Gaussian distribution with a specicic scaling factor gamma with help of least squares optimization.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• gamma (float) – gamma parameter of the gaussian heatmap generator

• ls_library (str) – library to use for least squares optimization. (scipy or pytorch)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_local_soft_argmax(heatmap, window=5, t=10.0, spatial_dims=2)

Returns coordiantes through applying the local soft-argmax function on the heatmaps.

Source: “Subpixel Heatmap Regression for Facial Landmark Localization” - Bulat et al. (2021)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• window (int) – local window size

• t (float) – temperature that controls the resulting probability map

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_weighted_spatial_mean(heatmap, spatial_dims=2, activation=None, require_grad=False, t=1.0)

Returns the spatial weighted mean of the heatmap. Source: “UGLLI Face Alignment: Estimating Uncertainty with

Gaussian Log-Likelihood Loss” - Kumar et al. (2019)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• activation (str) – activation function to apply to the heatmap

• require_grad (bool) – whether to require gradient for the coordinates

• t (float) –

Returns:

coordinates of shape (B, C, 2) or (B, C, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_weighted_spatial_mean_cov(heatmap, spatial_dims=2, require_grad=False, activation=None)

Returns the spatial weighted mean and the weighted sample covariance of the possitive elements of the heatmap by the heatmap values.

source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• require_grad (bool) – whether to require gradient for the coordinates

• activation (str) – activation function to apply to the heatmap

Returns:

coordinates of shape (B, C, 2) or (B, C, 3) (torch.Tensor): covariance matrix of shape (B, C, 2, 2) or (B, C, 3, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.cov_from_gaussian_ls(heatmap, coords, gamma=None, ls_library='scipy', spatial_dims=2)

Returns covariance matrix from a heatmap through fitting the heatmap on Gaussian distribution with a specicic scaling factor gamma and specified coordinates with help of least squares optimization.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• gamma (float) – gamma parameter of the gaussian heatmap generator

• ls_library (str) – library to use for least squares optimization. (scipy or pytorch)

• spatial_dims (int) –

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord(heatmap, offset_coords=0, method='argmax', spatial_dims=2, require_grad=False)

Returns the retrieved coordinates via specified method from a heatmap. The offset_coords is used to remove the first offset_coords coordinates from the heatmap. This is used to remove the background class (if present).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• offset_coords (int) – number of coordinates to remove

• method (str) – method to retrieve the coordinates

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• require_grad (bool) – whether to require gradient for the coordinates

Returns:

coordinates of shape (B, C, 2) or (B, C, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord_cov(heatmap, method='soft_argmax', require_grad=True, spatial_dims=2)

Returns the modal coordinates and covariance matrix from a heatmap.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• method (str) – method to retrieve the coordinates

• require_grad (bool) – whether to require gradient for the coordinates

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2) or (B, C, 3) (torch.Tensor): covariance matrix of shape (B, C, 2, 2) or (B, C, 3, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord_enlarge(heatmap, offset_coords=0, method='argmax', enlarge_factor=1, enlarge_dim=None, enlarge_mode='bilinear', spatial_dims=2)

Returns the retrieved coordinates via specified method from an enlarged heatmap. The offset_coords is used to remove the first offset_coords coordinates from the heatmap. This is used to remove the background class (if present).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• offset_coords (int) – number of coordinates to remove

• method (str) – method to retrieve the coordinates

• enlarge_factor (int) – factor to enlarge the heatmap

• enlarge_dim (tuple[int, ...] or None) – dimensions to enlarge the heatmap to

• enlarge_mode (str) – interpolation mode to enlarge the heatmap

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_multiple_coord(heatmaps, window=5, threshold=None, method='argmax', spatial_dims=2)

Returns the multiple coordinates of the maximum value of the heatmap for each batch and channel. Additionally, the scores of the maxima are returned. If threshold is not None, only the maxima with a score higher than the threshold are returned.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• window (tuple) – window size

• threshold (float) – threshold to remove points. If None, no threshold is applied. Default None.

• method (str) – method to retrieve the coordinates

• heatmaps (Tensor) –

• spatial_dims (int) –

Returns:

list of coordinates of the maxima for each batch and

channel, with the first list the batch, the second list the channel and the third list the coordinates

(list(list(list(float)))): list of scores of the maxima for each batch and channel, with

the first list the batch, the second list the channel and the third list the scores.

Return type:

(list(list(list(torch.Tensor))))

landmarker.heatmap.non_maximum_surpression(heatmap, window=3)

Non-Maximum Surpression (NMS)

source: Efficient Non-Maximum Suppression - Neubeck and Van Gool (2006)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (H, W)

• window (int) – window size

Returns:

list of coordinates of the maxima

Return type:

(list(tuple(int, int)))

landmarker.heatmap.non_maximum_surpression_local_soft_argmax(heatmap, window=3)

Returns the coordinates of the maximum value of the heatmap for each batch and channel (landmark), with the soft-argmax function applied on the heatmaps.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (H, W)

• window (int) – window size

Returns:

list of coordinates of the maxima

Return type:

(list(tuple(int, int)))

landmarker.heatmap.weighted_sample_cov(heatmap, coords, spatial_dims=2, activation=None)

Calculate the covariance matrix from a heatmap by calculating the mean of the heatmap values weighted by the heatmap values. source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2) or (B, C, 3)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• activation (str) – activation function to apply to the heatmap

Returns:

covariance matrix of shape (B, C, 2, 2) or (B, C, 3, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.windowed_weigthed_sample_cov(heatmap, coords, spatial_dims=2, activation=None)

Calculate the covariance matrix from a heatmap by calculating the mean of the heatmap values weighted by the heatmap values. The window is determined by the distance to the closest edge.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• activation (str) – activation function to apply to the heatmap

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)