Heatmap Module

Heatmap module.

class landmarker.heatmap.GaussianHeatmapGenerator

Gaussian heatmap generator for generating heatmaps from landmarks.

Parameters:

• nb_landmarks (int) – number of landmarks

• sigmas (float or list[float] or torch.Tensor or np.ndarray) – sigmas of the gaussian heatmap function

• gamma (float or None) – scaling factor of the gaussian heatmap function

• rotation (float or list[float] or torch.Tensor or np.ndarray) – rotation of the gaussian heatmap function

• heatmap_size (tuple[int, int]) – size of the heatmap

• learnable (bool) – whether the sigmas and rotation are learnable

• background (bool) – whether to add a background channel to the heatmap

• all_points (bool) – whether to add a channel with the sum of all the landmarks

• continuous (bool) – whether to use continuous or discrete landmarks

• na_zero (bool) – whether to set the value of the landmarks to zero if they are not available

__init__(nb_landmarks, sigmas=1.0, gamma=None, rotation=0, heatmap_size=(512, 512), learnable=False, background=False, all_points=False, continuous=True, na_zero=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

• nb_landmarks (int) –

• sigmas (float | list[float] | Tensor | ndarray) –

• gamma (float | None) –

• rotation (float | list[float] | Tensor | ndarray) –

• heatmap_size (tuple[int, ...]) –

• learnable (bool) –

• background (bool) –

• all_points (bool) –

• continuous (bool) –

• na_zero (bool) –

Return type:

None

heatmap_fun(landmark_t, coords, covariance, gamma)

Gaussian heatmap function

Parameters:

• landmark_t (torch.Tensor) – coordinates of the landmark (y, x) or (z, y, x)

• coords (torch.Tensor) – coordinates of the pixel (y, x) or (z, y, x)

• covariance (torch.Tensor) – covariance matrix (y, x) or (z, y, x)

• gamma (float or None) – scaling factor of the heatmap function

Returns:

value of the gaussian heatmap function for the given pixel

Return type:

torch.Tensor

class landmarker.heatmap.LaplacianHeatmapGenerator

Laplacian heatmap generator for generating heatmaps from landmarks.

Parameters:

• nb_landmarks (int) – number of landmarks

• sigmas (float or list[float] or torch.Tensor or np.ndarray) – sigmas of the Laplacian heatmap function

• gamma (float or None) – scaling factor of the Laplacian heatmap function

• rotation (float or list[float] or torch.Tensor or np.ndarray) – rotation of the laplacian heatmap function

• heatmap_size (tuple[int, int]) – size of the heatmap

• learnable (bool) – whether the sigmas and rotation are learnable

• background (bool) – whether to add a background channel to the heatmap

• all_points (bool) – whether to add a channel with the sum of all the landmarks

• continuous (bool) – whether to use continuous or discrete landmarks

• na_zero (bool) – whether to set the value of the landmarks to zero if they are not available

__init__(nb_landmarks, sigmas=1.0, gamma=None, rotation=0, heatmap_size=(512, 512), learnable=False, background=False, all_points=False, continuous=True, na_zero=False)

Initialize internal Module state, shared by both nn.Module and ScriptModule.

Parameters:

• nb_landmarks (int) –

• sigmas (float | list[float] | Tensor | ndarray) –

• gamma (float | None) –

• rotation (float | list[float] | Tensor | ndarray) –

• heatmap_size (tuple[int, ...]) –

• learnable (bool) –

• background (bool) –

• all_points (bool) –

• continuous (bool) –

• na_zero (bool) –

Return type:

None

heatmap_fun(landmark_t, coords, covariance, gamma=None)

Laplacian heatmap function

Parameters:

• landmark_t (torch.Tensor) – coordinates of the landmark (y, x)

• coords (torch.Tensor) – coordinates of the pixel (y, x)

• covariance (torch.Tensor) – covariance matrix (y, x)

• gamma (float or None) – scaling factor of the heatmap function

Returns:

value of the gaussian heatmap function for the given pixel

Return type:

torch.Tensor

landmarker.heatmap.coord_argmax(heatmap, spatial_dims=2)

Returns the coordinates of the maximum value of the heatmap for each batch and channel (landmark).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2) or (B, C, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_cov_from_gaussian_ls(heatmap, gamma=None, ls_library='scipy', spatial_dims=2)

Returns the modal coordinates and covariance matrix from a heatmap through fitting the heatmap on Gaussian distribution with a specicic scaling factor gamma with help of least squares optimization.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• gamma (float) – gamma parameter of the gaussian heatmap generator

• ls_library (str) – library to use for least squares optimization. (scipy or pytorch)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2) (torch.Tensor): covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_local_soft_argmax(heatmap, window=5, t=10.0, spatial_dims=2)

Returns coordiantes through applying the local soft-argmax function on the heatmaps.

Source: “Subpixel Heatmap Regression for Facial Landmark Localization” - Bulat et al. (2021)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• window (int) – local window size

• t (float) – temperature that controls the resulting probability map

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_weighted_spatial_mean(heatmap, spatial_dims=2, activation=None, require_grad=False, t=1.0)

Returns the spatial weighted mean of the heatmap. Source: “UGLLI Face Alignment: Estimating Uncertainty with

Gaussian Log-Likelihood Loss” - Kumar et al. (2019)

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• activation (str) – activation function to apply to the heatmap

• require_grad (bool) – whether to require gradient for the coordinates

• t (float) –

Returns:

coordinates of shape (B, C, 2) or (B, C, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.coord_weighted_spatial_mean_cov(heatmap, spatial_dims=2, require_grad=False, activation=None)

Returns the spatial weighted mean and the weighted sample covariance of the possitive elements of the heatmap by the heatmap values.

source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• require_grad (bool) – whether to require gradient for the coordinates

• activation (str) – activation function to apply to the heatmap

Returns:

coordinates of shape (B, C, 2) or (B, C, 3) (torch.Tensor): covariance matrix of shape (B, C, 2, 2) or (B, C, 3, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.cov_from_gaussian_ls(heatmap, coords, gamma=None, ls_library='scipy', spatial_dims=2)

Returns covariance matrix from a heatmap through fitting the heatmap on Gaussian distribution with a specicic scaling factor gamma and specified coordinates with help of least squares optimization.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• gamma (float) – gamma parameter of the gaussian heatmap generator

• ls_library (str) – library to use for least squares optimization. (scipy or pytorch)

• spatial_dims (int) –

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord(heatmap, offset_coords=0, method='argmax', spatial_dims=2, require_grad=False)

Returns the retrieved coordinates via specified method from a heatmap. The offset_coords is used to remove the first offset_coords coordinates from the heatmap. This is used to remove the background class (if present).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• offset_coords (int) – number of coordinates to remove

• method (str) – method to retrieve the coordinates

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• require_grad (bool) – whether to require gradient for the coordinates

Returns:

coordinates of shape (B, C, 2) or (B, C, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord_cov(heatmap, method='soft_argmax', require_grad=True, spatial_dims=2)

Returns the modal coordinates and covariance matrix from a heatmap.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• method (str) – method to retrieve the coordinates

• require_grad (bool) – whether to require gradient for the coordinates

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2) or (B, C, 3) (torch.Tensor): covariance matrix of shape (B, C, 2, 2) or (B, C, 3, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.heatmap_to_coord_enlarge(heatmap, offset_coords=0, method='argmax', enlarge_factor=1, enlarge_dim=None, enlarge_mode='bilinear', spatial_dims=2)

Returns the retrieved coordinates via specified method from an enlarged heatmap. The offset_coords is used to remove the first offset_coords coordinates from the heatmap. This is used to remove the background class (if present).

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• offset_coords (int) – number of coordinates to remove

• method (str) – method to retrieve the coordinates

• enlarge_factor (int) – factor to enlarge the heatmap

• enlarge_dim (tuple[int, ...] or None) – dimensions to enlarge the heatmap to

• enlarge_mode (str) – interpolation mode to enlarge the heatmap

• spatial_dims (int) – number of spatial dimensions (2 or 3)

Returns:

coordinates of shape (B, C, 2)

Return type:

(torch.Tensor)

landmarker.heatmap.weighted_sample_cov(heatmap, coords, spatial_dims=2, activation=None)

Calculate the covariance matrix from a heatmap by calculating the mean of the heatmap values weighted by the heatmap values. source: https://en.wikipedia.org/wiki/Weighted_arithmetic_mean#Weighted_sample_covariance

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W) or (B, C, D, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2) or (B, C, 3)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• activation (str) – activation function to apply to the heatmap

Returns:

covariance matrix of shape (B, C, 2, 2) or (B, C, 3, 3)

Return type:

(torch.Tensor)

landmarker.heatmap.windowed_weigthed_sample_cov(heatmap, coords, spatial_dims=2, activation=None)

Calculate the covariance matrix from a heatmap by calculating the mean of the heatmap values weighted by the heatmap values. The window is determined by the distance to the closest edge.

Parameters:

• heatmap (torch.Tensor) – heatmap of shape (B, C, H, W)

• coords (torch.Tensor) – coordinates of shape (B, C, 2)

• spatial_dims (int) – number of spatial dimensions (2 or 3)

• activation (str) – activation function to apply to the heatmap

Returns:

covariance matrix of shape (B, C, 2, 2)

Return type:

(torch.Tensor)