mlguess.torch package

mlguess.torch package#

Submodules#

mlguess.torch.class_losses module#

class mlguess.torch.class_losses.EDLDigammaLoss(num_classes, annealing_step, weights=None)#

Bases: Module

forward(output, target, epoch_num, device=None)#

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

class mlguess.torch.class_losses.EDLLoss(func, num_classes, annealing_step, weights=None)#

Bases: Module

forward(y, alpha, epoch_num, device=None)#

Define the computation performed at every call.

Should be overridden by all subclasses.

Note

Although the recipe for forward pass needs to be defined within this function, one should call the Module instance afterwards instead of this since the former takes care of running the registered hooks while the latter silently ignores them.

mlguess.torch.class_losses.edl_digamma_loss(output, target, epoch_num, num_classes, annealing_step, weights=None, device=None)#

Compute the Evidence Deep Learning (EDL) loss with the digamma function of evidence.

Parameters:

  • output (torch.Tensor) – Model output tensor.

  • target (torch.Tensor) – Target values.

  • epoch_num (int) – The current epoch number.

  • num_classes (int) – The number of classes.

  • annealing_step (int) – The step at which annealing occurs.

  • weights (optional, torch.Tensor) – Weights to apply to the loss. Defaults to None.

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The computed EDL loss with digamma evidence.

Return type:

torch.Tensor

mlguess.torch.class_losses.edl_log_loss(output, target, epoch_num, num_classes, annealing_step, weights=None, device=None)#

Compute the Evidence Deep Learning (EDL) loss with the logarithm of evidence.

Parameters:

  • output (torch.Tensor) – Model output tensor.

  • target (torch.Tensor) – Target values.

  • epoch_num (int) – The current epoch number.

  • num_classes (int) – The number of classes.

  • annealing_step (int) – The step at which annealing occurs.

  • weights (optional, torch.Tensor) – Weights to apply to the loss. Defaults to None.

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The computed EDL loss with logarithmic evidence.

Return type:

torch.Tensor

mlguess.torch.class_losses.edl_loss(func, y, alpha, epoch_num, num_classes, annealing_step, weights=None, device=None)#

Compute the Evidence Deep Learning (EDL) loss.

Parameters:

  • func (callable) – Function to apply to alpha (e.g., log, softplus).

  • y (torch.Tensor) – Target values.

  • alpha (torch.Tensor) – The Dirichlet parameters (alpha).

  • epoch_num (int) – The current epoch number.

  • num_classes (int) – The number of classes.

  • annealing_step (int) – The step at which annealing occurs.

  • weights (optional, torch.Tensor) – Weights to apply to the loss. Defaults to None.

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The computed EDL loss.

Return type:

torch.Tensor

mlguess.torch.class_losses.edl_mse_loss(output, target, epoch_num, num_classes, annealing_step, weights=None, device=None)#

Compute the Evidence Deep Learning (EDL) loss with mean squared error.

Parameters:

  • output (torch.Tensor) – Model output tensor.

  • target (torch.Tensor) – Target values.

  • epoch_num (int) – The current epoch number.

  • num_classes (int) – The number of classes.

  • annealing_step (int) – The step at which annealing occurs.

  • weights (optional, torch.Tensor) – Weights to apply to the loss. Defaults to None.

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The computed EDL loss with MSE.

Return type:

torch.Tensor

mlguess.torch.class_losses.exp_evidence(y)#

Apply the exponential function to the input tensor with clamping.

Parameters:

y (torch.Tensor) – Input tensor.

Returns:

Tensor after applying the exponential function with clamping.

Return type:

torch.Tensor

mlguess.torch.class_losses.get_device()#

Get the device for PyTorch operations.

Returns:

The device to use, either “cuda” if CUDA is available, otherwise “cpu”.

Return type:

torch.device

mlguess.torch.class_losses.kl_divergence(alpha, num_classes, device=None)#

Compute the Kullback-Leibler divergence for a Dirichlet distribution.

Parameters:

  • alpha (torch.Tensor) – The Dirichlet parameters (alpha).

  • num_classes (int) – The number of classes.

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The KL divergence between the given Dirichlet distribution and a uniform distribution.

Return type:

torch.Tensor

mlguess.torch.class_losses.loglikelihood_loss(y, alpha, device=None)#

Compute the log-likelihood loss for a Dirichlet distribution.

Parameters:

  • y (torch.Tensor) – Target values.

  • alpha (torch.Tensor) – The Dirichlet parameters (alpha).

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The computed log-likelihood loss.

Return type:

torch.Tensor

mlguess.torch.class_losses.mse_loss(y, alpha, epoch_num, num_classes, annealing_step, device=None)#

Compute the mean squared error loss with KL divergence for Dirichlet distributions.

Parameters:

  • y (torch.Tensor) – Target values.

  • alpha (torch.Tensor) – The Dirichlet parameters (alpha).

  • epoch_num (int) – The current epoch number.

  • num_classes (int) – The number of classes.

  • annealing_step (int) – The step at which annealing occurs.

  • device (optional, torch.device) – Device to perform computation on. Defaults to None, which uses the default device.

Returns:

The computed MSE loss with KL divergence.

Return type:

torch.Tensor

mlguess.torch.class_losses.relu_evidence(y)#

Apply the ReLU activation function to the input tensor.

Parameters:

y (torch.Tensor) – Input tensor.

Returns:

Tensor after applying the ReLU activation function.

Return type:

torch.Tensor

mlguess.torch.class_losses.softplus_evidence(y)#

Apply the Softplus activation function to the input tensor.

Parameters:

y (torch.Tensor) – Input tensor.

Returns:

Tensor after applying the Softplus activation function.

Return type:

torch.Tensor

mlguess.torch.layers module#

class mlguess.torch.layers.LinearNormalGamma(in_channels, out_channels, spectral_norm=True)#

Bases: Module

A linear layer with a Normal-Gamma distribution parameterization.

This module applies a linear transformation to the input, followed by reshaping and parameter extraction for a Normal-Gamma distribution. The parameters are used to estimate the mean (mu), variance (logv), and shape parameters of the Gamma distribution (logalpha and logbeta).

linear#

A linear layer with spectral normalization.

Type:

nn.Module

Parameters:

  • in_channels (int) – The number of input features.

  • out_channels (int) – The number of output features.

evidence(x)#

Applies a log transformation to the input with a shift.

Parameters:

x (torch.Tensor) – The input tensor.

Returns:

The transformed tensor.

Return type:

torch.Tensor

forward(x)#

Forward pass of the module.

Parameters:

x (torch.Tensor) – The input tensor of shape (batch_size, in_channels).

Returns:

A tuple containing:

  • mu (torch.Tensor): The mean of the distribution.

  • v (torch.Tensor): The variance of the distribution.

  • alpha (torch.Tensor): The shape parameter of the Gamma distribution.

  • beta (torch.Tensor): The rate parameter of the Gamma distribution.

Return type:

tuple

mlguess.torch.mc_dropout module#

mlguess.torch.mc_dropout.enable_dropout(model)#

Function to enable the dropout layers during test-time

mlguess.torch.mc_dropout.monte_carlo_dropout(data_loader, forward_passes, model, n_classes, n_samples, batch_size=1024, uncertainty=False)#

Function to get the monte-carlo samples and uncertainty estimates through multiple forward passes

Parameters:

  • data_loader (Object. Data loader object from the data loader module)

  • forward_passes (Integer. Number of monte-carlo samples/forward passes)

  • model (Object. Keras model)

  • n_classes (Integer. Number of classes in the dataset)

  • n_samples (Integer. Number of samples in the test set)

  • batch_size (Integer. Number of samples per gradient update)

  • uncertainty (Boolean. Whether to calculate uncertainty)

mlguess.torch.models module#

class mlguess.torch.models.CategoricalDNN(input_size, output_size, layer_size=[1000], dr=[0.5], batch_norm=True, lng=False, softmax=False, weight_init=False, num_layers=None)#

Bases: DNN

calc_uncertainty(y_pred)#
predict(x, return_uncertainties=True, **kwargs)#

Make predictions with the model.

Parameters:

  • input (torch.Tensor) – Input tensor.

  • y_scaler (optional, callable) – A scaler to inverse transform predictions. Default is None.

  • return_uncertainties (bool) – Whether to return uncertainties along with predictions. Default is True.

  • return_tuple (bool) – Whether to return the output as a tuple. Default is False.

Returns:

  • If return_tuple is True, returns a tuple (output).

  • Otherwise, returns concatenated output tensor.

Return type:

tuple or torch.Tensor

predict_dropout(input)#

Perform Monte Carlo Dropout predictions.

Parameters:

  • x (torch.Tensor) – Input tensor.

  • mc_forward_passes (int) – Number of Monte Carlo forward passes. Default is 10.

  • batch_size (optional, int) – Batch size for processing. Default is None.

Returns:

A tuple containing:

  • pred_probs (numpy.ndarray): Mean predicted probabilities.

  • aleatoric (numpy.ndarray): Aleatoric uncertainty.

  • epistemic (numpy.ndarray): Epistemic uncertainty.

  • entropy (numpy.ndarray): Entropy across multiple forward passes.

  • mutual_info (numpy.ndarray): Mutual information across multiple forward passes.

Return type:

tuple

predict_uncertainty(input)#

Estimate uncertainties of predictions.

Parameters:

  • input (tuple of torch.Tensor) – Tuple containing (mu, v, alpha, beta) tensors.

  • y_scaler (optional, callable) – A scaler to inverse transform predictions. Default is None.

Returns:

A tuple containing:

  • mu (torch.Tensor): Mean predictions.

  • aleatoric (torch.Tensor): Aleatoric uncertainty.

  • epistemic (torch.Tensor): Epistemic uncertainty.

  • total (torch.Tensor): Total uncertainty (aleatoric + epistemic).

Return type:

tuple

class mlguess.torch.models.DNN(input_size, output_size, layer_size=[1000], dr=[0.5], batch_norm=True, lng=False, softmax=False, weight_init=False, num_layers=None)#

Bases: Module

Initialize the Deep Neural Network (DNN) model.

Parameters:

  • input_size (int or list of int) – Number of input features or a list of sizes for each input.

  • output_size (int or list of int) – Number of output features or a list of sizes for each output.

  • layer_size (list of int) – List of sizes for hidden layers. Default is [1000].

  • dr (list of float) – Dropout rates for each layer. Default is [0.5].

  • batch_norm (bool) – Whether to use batch normalization. Default is True.

  • lng (bool) – Whether to use LinearNormalGamma layer at the end. Default is False.

  • weight_init (bool) – Whether to initialize weights. Default is False.

  • num_layers (int) – Number of layers to create if layer_size is a single number. Default is None.

block(input_size, output_size, dr, batch_norm)#

Create a block of layers for the network.

Parameters:

  • input_size (int) – Number of input features.

  • output_size (int) – Number of output features.

  • dr (float) – Dropout rate.

  • batch_norm (bool) – Whether to apply batch normalization.

Returns:

A list of layers constituting the block.

Return type:

list

forward(x)#

Perform a forward pass through the network.

Parameters:

x (torch.Tensor) – Input tensor.

Returns:

Output tensor after passing through the network.

Return type:

torch.Tensor

classmethod from_config(conf, device='cpu')#

Create a model instance from configuration.

Parameters:

  • conf (dict) – Configuration dictionary with model parameters.

  • device (str) – Device to load the model onto. Default is “cpu”.

Returns:

The initialized model.

Return type:

DNN

load_weights(weights_path: str) → None#

Load model weights from a file.

Parameters:

weights_path (str) – Path to the weights file (.pt).

Returns:

None

predict(input, y_scaler=None, return_uncertainties=True, return_tuple=False)#

Make predictions with the model.

Parameters:

  • input (torch.Tensor) – Input tensor.

  • y_scaler (optional, callable) – A scaler to inverse transform predictions. Default is None.

  • return_uncertainties (bool) – Whether to return uncertainties along with predictions. Default is True.

  • return_tuple (bool) – Whether to return the output as a tuple. Default is False.

Returns:

  • If return_tuple is True, returns a tuple (output).

  • Otherwise, returns concatenated output tensor.

Return type:

tuple or torch.Tensor

predict_dropout(x, mc_forward_passes=10, batch_size=16)#

Perform Monte Carlo Dropout predictions.

Parameters:

  • x (torch.Tensor) – Input tensor.

  • mc_forward_passes (int) – Number of Monte Carlo forward passes. Default is 10.

  • batch_size (optional, int) – Batch size for processing. Default is None.

Returns:

A tuple containing:

  • pred_probs (numpy.ndarray): Mean predicted probabilities.

  • aleatoric (numpy.ndarray): Aleatoric uncertainty.

  • epistemic (numpy.ndarray): Epistemic uncertainty.

  • entropy (numpy.ndarray): Entropy across multiple forward passes.

  • mutual_info (numpy.ndarray): Mutual information across multiple forward passes.

Return type:

tuple

predict_uncertainty(input, y_scaler=None)#

Estimate uncertainties of predictions.

Parameters:

  • input (tuple of torch.Tensor) – Tuple containing (mu, v, alpha, beta) tensors.

  • y_scaler (optional, callable) – A scaler to inverse transform predictions. Default is None.

Returns:

A tuple containing:

  • mu (torch.Tensor): Mean predictions.

  • aleatoric (torch.Tensor): Aleatoric uncertainty.

  • epistemic (torch.Tensor): Epistemic uncertainty.

  • total (torch.Tensor): Total uncertainty (aleatoric + epistemic).

Return type:

tuple

mlguess.torch.models.get_device()#

Determine the computing device to use.

Checks if CUDA is available and returns the appropriate device (either “cuda” or “cpu”).

Returns:

The device to be used for computation.

Return type:

torch.device

mlguess.torch.models.init_weights(net, init_type='normal', init_gain=0.0, verbose=True)#

Initialize network weights using the specified method.

Parameters:

  • net (nn.Module) – The network whose weights are to be initialized.

  • init_type (str) – The type of initialization method to use. Options are: ‘normal’, ‘xavier’, ‘kaiming’, ‘orthogonal’. Default is ‘normal’.

  • init_gain (float) – Scaling factor for ‘normal’, ‘xavier’, and ‘orthogonal’ methods. Default is 0.0.

  • verbose (bool) – If True, logs the initialization method. Default is True.

Raises:

NotImplementedError – If an unsupported initialization method is specified.

Notes

  • ‘normal’: Gaussian distribution with mean 0 and specified standard deviation.

  • ‘xavier’: Xavier initialization.

  • ‘kaiming’: Kaiming initialization.

  • ‘orthogonal’: Orthogonal initialization.

mlguess.torch.models.seed_everything(seed=1234)#

Seed all random number generators for reproducibility.

Parameters:

seed (int) – The seed value to use for all random number generators. Default is 1234.

Notes

This function seeds: - Python’s random module - NumPy’s random module - PyTorch’s CPU and GPU random number generators - Sets environment variable to control hash seed - Configures PyTorch’s cuDNN to be deterministic and benchmark mode.

mlguess.torch.regression_losses module#

Torch losses for regression models

class mlguess.torch.regression_losses.EvidenceRegularizer(size_average=None, reduce=None, reduction: str = 'mean', coef=0.1)#

Bases: _Loss

Regularization for the regression prior network. If self.factor increases, the model output the wider(high confidence interval) predictions.

Reference: https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf Source: deargen/MT-ENet

forward(gamma: Tensor, nu: Tensor, alpha: Tensor, target: Tensor) → Tensor#

Forward pass through the loss.

Parameters:

  • gamma (torch.Tensor) – gamma output value of the evidential network

  • nu (torch.Tensor) – nu output value of the evidential network

  • alpha (torch.Tensor) – alpha output value of the evidential network

  • target (torch.Tensor) – target value

Returns:

(Tensor) prior network regularization Loss = |y - gamma|*(2*nu + alpha) * factor

class mlguess.torch.regression_losses.EvidentialMarginalLikelihood(size_average=None, reduce=None, reduction: str = 'mean')#

Bases: _Loss

Marginal likelihood error of prior network. The target value is not a distribution (mu, std), but a just value.

Reference: Amini et al. 2020 (https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf) Source: deargen/MT-ENet

forward(gamma: Tensor, nu: Tensor, alpha: Tensor, beta: Tensor, target: Tensor) → Tensor#

Conduct the forward pass through the loss.

Parameters:

  • gamma (torch.Tensor) – gamma output value of the evidential network

  • nu (torch.Tensor) – nu output value of the evidential network

  • alpha (torch.Tensor) – alpha output value of the evidential network

  • beta (torch.Tensor) – beta output value of the evidential network

  • target (torch.Tensor) – target value

Returns:

(Tensor) Negative log marginal likelihood of EvidentialNet

class mlguess.torch.regression_losses.EvidentialRegressionLoss(coef=1.0)#

Bases: object

Class for computing Evidential Regression Loss, which includes the Normal Inverse Gamma negative log likelihood and a regularization term.

Parameters:

coef (float, optional) – Coefficient for the regularization term. Defaults to 1.0.

normal_inverse_gamma_nll(y, gamma, v, alpha, beta)#

Compute the Normal Inverse Gamma Negative Log Likelihood (NLL) for Deep Evidential Regression.

Parameters:

  • y (torch.Tensor) – Target values.

  • gamma (torch.Tensor) – Mean of the Normal-Inverse Gamma distribution.

  • v (torch.Tensor) – Degrees of freedom of the distribution.

  • alpha (torch.Tensor) – Shape parameter of the Normal-Inverse Gamma distribution.

  • beta (torch.Tensor) – Scale parameter of the Normal-Inverse Gamma distribution.

Returns:

The computed negative log likelihood.

Return type:

torch.Tensor

Reference: https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf Source: hxu296/torch-evidental-deep-learning

normal_inverse_gamma_reg(y, gamma, v, alpha, beta)#

Compute the Normal Inverse Gamma Regularizer for Deep Evidential Regression.

Parameters:

  • y (torch.Tensor) – Target values.

  • gamma (torch.Tensor) – Mean of the Normal-Inverse Gamma distribution.

  • v (torch.Tensor) – Degrees of freedom of the distribution.

  • alpha (torch.Tensor) – Shape parameter of the Normal-Inverse Gamma distribution.

  • beta (torch.Tensor) – Scale parameter of the Normal-Inverse Gamma distribution.

Returns:

The computed regularization term.

Return type:

torch.Tensor

class mlguess.torch.regression_losses.LipschitzMSELoss(tol=1e-08, coef=0.1, reduction='mean')#

Bases: Module

Compute the Lipschitz MSE loss, which includes the Evidential Marginal Likelihood, Evidence Regularizer, and a modified MSE term.

Parameters:

  • tol (float, optional) – Tolerance value to avoid division by zero. Defaults to 1e-8.

  • coef (float, optional) – Coefficient for the regularization term. Defaults to 0.1.

  • reduction (str, optional) – Specifies the method to reduce the loss over the batch. Can be ‘mean’, ‘sum’, or ‘none’. Defaults to ‘mean’.

forward(gamma, nu, alpha, beta, target)#

Compute the total Lipschitz MSE Loss.

Parameters:

  • gamma (torch.Tensor) – Output value of the evidential network for gamma.

  • nu (torch.Tensor) – Output value of the evidential network for nu.

  • alpha (torch.Tensor) – Output value of the evidential network for alpha.

  • beta (torch.Tensor) – Output value of the evidential network for beta.

  • target (torch.Tensor) – True labels.

Returns:

The total computed loss.

Return type:

torch.Tensor

mlguess.torch.regression_losses.check_mse_efficiency_alpha(nu, alpha, beta)#

Check the MSE loss (gamma - y)^2 can make negative gradients for alpha, which is a pseudo observation of the normal-inverse-gamma. We can use this to check the MSE loss can success(increase the pseudo observation, alpha).

Reference: https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf Source: deargen/MT-ENet

Parameters:

  • nu (torch.Tensor) – nu output value of the evidential network

  • alpha (torch.Tensor) – alpha output value of the evidential network

  • beta (torch.Tensor) – beta output value of the evidential network

Returns:

partial f / partial alpha(numpy.array) where f => the NLL loss (BayesianDTI.loss.MarginalLikelihood)

mlguess.torch.regression_losses.check_mse_efficiency_nu(gamma, nu, alpha, beta)#

Check the MSE loss (gamma - y)^2 can make negative gradients for nu, which is a pseudo observation of the normal-inverse-gamma. We can use this to check the MSE loss can success(increase the pseudo observation, nu).

Reference: https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf Source: deargen/MT-ENet

Parameters:

  • gamma (torch.Tensor) – gamma output value of the evidential network

  • nu (torch.Tensor) – nu output value of the evidential network

  • alpha (torch.Tensor) – alpha output value of the evidential network

  • beta (torch.Tensor) – beta output value of the evidential network

Returns:

partial f / partial nu(torch.Tensor) where f => the NLL loss (BayesianDTI.loss.MarginalLikelihood)

mlguess.torch.regression_losses.get_mse_coef(gamma, nu, alpha, beta, y)#

Return the coefficient of the MSE loss for each prediction. By assigning the coefficient to each MSE value, it clips the gradient of the MSE based on the threshold values U_nu, U_alpha, which are calculated by check_mse_efficiency_* functions.

Reference: https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf Source: deargen/MT-ENet

Parameters:

  • gamma ([FloatTensor]) – the output of the ENet.

  • nu ([FloatTensor]) – the output of the ENet.

  • alpha ([FloatTensor]) – the output of the ENet.

  • beta ([FloatTensor]) – the output of the ENet.

  • y ([FloatTensor]) – true labels.

Returns:

[0.0-1.0], the coefficient of the MSE for each prediction.

Return type:

[FloatTensor]

mlguess.torch.regression_losses.modified_mse(gamma, nu, alpha, beta, target, reduction='mean')#

Compute the Lipschitz Mean Squared Error (MSE) loss as described in “Improving Evidential Deep Learning via Multitask Learning.”

Parameters:

  • gamma (torch.Tensor) – Output of the evidential network.

  • nu (torch.Tensor) – Output of the evidential network.

  • alpha (torch.Tensor) – Output of the evidential network.

  • beta (torch.Tensor) – Output of the evidential network.

  • target (torch.Tensor) – True labels.

  • reduction (str, optional) – Specifies the reduction to apply to the output. Can be ‘mean’, ‘sum’, or ‘none’. Defaults to ‘mean’.

Returns:

The computed modified MSE loss.

Return type:

torch.Tensor

Reference: https://www.mit.edu/~amini/pubs/pdf/deep-evidential-regression.pdf Source: deargen/MT-ENet

Module contents#

Contents