`utilities` – Utility Functions and Classes ¶

Contents

utilities – Utility Functions and Classes

Utilities for tensor manipulation ¶

neuralnet_pytorch.utils.tensor_utils.dimshuffle(x: torch.Tensor, pattern)[source]¶

Reorders the dimensions of this variable, optionally inserting broadcasted dimensions. Inspired by Theano’s dimshuffle.

Parameters:	x – Input tensor. pattern – List/tuple of int mixed with x for broadcastable dimensions.
Returns:	a tensor whose shape matches pattern.

Examples

To create a 3D view of a [2D] matrix, call dimshuffle(x, [0,'x',1]). This will create a 3D view such that the middle dimension is an implicit broadcasted dimension. To do the same thing on the transpose of that matrix, call dimshuffle(x, [1, 'x', 0]).

See also

shape_padleft() shape_padright() swapaxes()

neuralnet_pytorch.utils.tensor_utils.shape_padleft(x: torch.Tensor, n_ones=1)[source]¶

Reshape x by left-padding the shape with n_ones 1s. Inspired by Theano’s shape_padleft.

Parameters:	x – variable to be reshaped. n_ones – number of 1s to pad.

See also

dimshuffle() shape_padright() swapaxes()

neuralnet_pytorch.utils.tensor_utils.shape_padright(x: torch.Tensor, n_ones=1)[source]¶

Reshape x by right-padding the shape with n_ones 1s. Inspired by Theano’s shape_padright.

Parameters:	x – variable to be reshaped. n_ones – number of 1s to pad.

See also

dimshuffle() shape_padleft() swapaxes()

neuralnet_pytorch.utils.tensor_utils.swapaxes(y: torch.Tensor, axis1, axis2)[source]¶

Swaps two given axes in the input tensor. If the input is of shape \((n_1, n_2, ..., n_{axis1}, ..., n_{axis2}, ...)\), the output will be \((n_1, n_2, ..., n_{axis2}, ..., n_{axis1}, ...)\). Can be seen as a generalization of transpose. Taken from Theano’s swapaxes.

Parameters:	y – a tensor. axis1 – an axis to be swapped. axis2 – another axis to be swapped.
Returns:	the axis-swapped tensor.

See also

dimshuffle() shape_padleft() shape_padright()

neuralnet_pytorch.utils.tensor_utils.ravel_index(indices: torch.Tensor, shape)[source]¶

Finds the linear index of index of a tensor of shape when it is flattened.

Parameters:	indices – a tensor containing indices to be linearized. shape – shape of the tensor w.r.t the index tensor.
Returns:	the linear index of the element having index.

Examples

>>> import torch as T
>>> import numpy as np
>>> import neuralnet_pytorch as nnt
>>> shape = (2, 3, 5)
>>> a = T.arange(np.prod(shape)).view(*shape)
>>> indices = T.tensor([[1, 0, 1, 1], [0, 1, 2, 1], [1, 0, 4, 3]]).long()
>>> print(a[list(indices)])
tensor([16.,  5., 29., 23.])
>>> linear_indices = nnt.utils.ravel_index(indices, shape)
>>> print(linear_indices)
tensor([16,  5, 29, 23])
>>> print(a.flatten()[linear_indices])
tensor([16.,  5., 29., 23.])

neuralnet_pytorch.utils.tensor_utils.tile(x: torch.Tensor, dims)[source]¶

Repeats x along dims. Behaves like numpy.tile().

Parameters:	x – a `torch.Tensor`. dims – the number of times to tile this tensor along each dimension.
Returns:	the tiled tensor.

neuralnet_pytorch.utils.tensor_utils.repeat(input: torch.Tensor, repeats, dim=None)[source]¶

Repeats elements of a tensor like numpy.repeat().

Parameters:	input – a `torch.Tensor`. repeats – the number of times to repeat this tensor along dim. dim – the dimension to repeat. If not specified, the method is applied to the flattened tensor. Default: `None`.
Returns:	the repeated tensor.

neuralnet_pytorch.utils.tensor_utils.block_diag(*blocks)[source]¶

Modified from scipy.linalg.block_diag. Creates a block diagonal matrix from provided arrays. Given the inputs A, B and C, the output will have these arrays arranged on the diagonal:

[[A, 0, 0],
 [0, B, 0],
 [0, 0, C]]

Parameters:	blocks – an iterator of tensors, up to 2-D. A 1-D tensor of length n is treated as a 2-D array with shape (1,n).
Returns:	a tensor with A, B, C, … on the diagonal. Has the same dtype as A.

Notes

If all the input arrays are square, the output is known as a block diagonal matrix.

See also

block_diag_sparse()

Examples

>>> from neuralnet_pytorch.utils import block_diag
>>> A = T.tensor([[1, 0],
...               [0, 1]])
>>> B = T.tensor([[3, 4, 5],
...               [6, 7, 8]])
>>> C = T.tensor([[7]])
>>> block_diag(A, B, C)
tensor([[1 0 0 0 0 0]
        [0 1 0 0 0 0]
        [0 0 3 4 5 0]
        [0 0 6 7 8 0]
        [0 0 0 0 0 7]])
>>> block_diag(T.tensor([1.0]), T.tensor([2, 3]), T.tensor([[4, 5], [6, 7]]))
tensor([[ 1.,  0.,  0.,  0.,  0.],
        [ 0.,  2.,  3.,  0.,  0.],
        [ 0.,  0.,  0.,  4.,  5.],
        [ 0.,  0.,  0.,  6.,  7.]])

neuralnet_pytorch.utils.tensor_utils.block_diag_sparse(a: torch.Tensor, dense=False)[source]¶

Creates a sparse block diagonal matrix from the provided array. Given the input tensor of size (n, r, c), the output will have the matrices of the last two indices arranged on the diagonal:

[[a[0], 0, 0],
 [0, a[1], 0],
 [0, 0, a[2]]]

Parameters:	a – a tensor of size `(n, r, c)`. dense – whether to return a dense matrix. Default: `False`.
Returns:	a tensor with a[0], a[1], a[2], … on the diagonal. Has the same dtype as a.

Notes

This function is for square matrices only. For general cases, use block_diag().

See also

block_diag()

Examples

>>> from neuralnet_pytorch.utils import block_diag_sparse
>>> import numpy as np
>>> a = T.arange(3 * 2 * 4).view(3, 2, 4)
>>> block_diag_sparse(a)
tensor(indices=tensor([[ 0,  0,  0,  0,  1,  1,  1,  1,  2,  2,  2,  2,  3,  3,
                         3,  3,  4,  4,  4,  4,  5,  5,  5,  5],
                       [ 0,  1,  2,  3,  0,  1,  2,  3,  4,  5,  6,  7,  4,  5,
                         6,  7,  8,  9, 10, 11,  8,  9, 10, 11]]),
       values=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13,
                      14, 15, 16, 17, 18, 19, 20, 21, 22, 23]),
       size=(6, 12), nnz=24, layout=torch.sparse_coo)
>>> block_diag_sparse(a, dense=True)
tensor([[ 0,  1,  2,  3,  0,  0,  0,  0,  0,  0,  0,  0],
        [ 4,  5,  6,  7,  0,  0,  0,  0,  0,  0,  0,  0],
        [ 0,  0,  0,  0,  8,  9, 10, 11,  0,  0,  0,  0],
        [ 0,  0,  0,  0, 12, 13, 14, 15,  0,  0,  0,  0],
        [ 0,  0,  0,  0,  0,  0,  0,  0, 16, 17, 18, 19],
        [ 0,  0,  0,  0,  0,  0,  0,  0, 20, 21, 22, 23]])

neuralnet_pytorch.utils.tensor_utils.get_bilinear_weights(x: torch.Tensor, y: torch.Tensor, h: int, w: int, border_mode='nearest')[source]¶

Returns bilinear weights used in bilinear interpolation.

Parameters:	x – floating point coordinates along the x-axis. y – floating point coordinates along the y-axis. h – height of the 2D array. w – width of the 2D array border_mode – strategy to deal with borders. Choices are `'nearest'` (default), `'mirror'`, and `'wrap'`.
Returns:	the weights for bilinear interpolation and the integer coordinates.

neuralnet_pytorch.utils.tensor_utils.interpolate_bilinear(im: torch.Tensor, x: torch.Tensor, y: torch.Tensor, output_shape=None, border_mode='nearest')[source]¶

Returns a batch of interpolated images. Used for Spatial Transformer Network. Works like torch.grid_sample.

Parameters:

im – a batch of input images
x – floating point coordinates along the x-axis. Should be in the range [-1, 1].
y – floating point coordinates along the y-axis. Should be in the range [-1, 1].
output_shape – output shape. A tuple of height and width. If not specified, output will have the same shape as input.
border_mode – strategy to deal with borders. Choices are 'nearest' (default), 'mirror', and 'wrap'.

Returns:

the bilinear interpolated batch of images.

neuralnet_pytorch.utils.tensor_utils.batch_pairwise_sqdist(x: torch.Tensor, y: torch.Tensor, c_code=False)[source]¶

Calculates the pair-wise square distance between two sets of points. To get the Euclidean distance, explicit square root needs to be applied to the output.

Parameters:	x – a tensor of shape `(..., nx, d)`. If the tensor dimension is 2, the tensor batch dim is broadcasted. y – a tensor of shape `(..., ny, d)`. If the tensor dimension is 2, the tensor batch dim is broadcasted. c_code – whether to use a C++ implementation. Default: `True` when the CUDA extension is installed. `False` otherwise.
Returns:	a tensor containing the exhaustive square distance between every pair of points in x and y from the same batch.

neuralnet_pytorch.utils.tensor_utils.gram_matrix(x: torch.Tensor) → torch.Tensor[source]¶

Computes the Gram matrix given a 4D tensor.

Parameters:	x – a 4D tensor.
Returns:	the Gram matrix of x.

neuralnet_pytorch.utils.tensor_utils.var(x: torch.Tensor, dim=None, unbiased=True, keepdim=False)[source]¶

Calculates the variance of x along dim. Exists because torch.var sometimes causes some error in backward pass.

Parameters:	x – a tensor. dim – the dimension along which to calculate the variance. Can be `int`/`list`/`tuple`. unbiased – whether to use an unbiased estimate. Default: `True`. keepdim – whether to keep the reduced dims as `1`. Default: `False`.
Returns:	the variance of x

neuralnet_pytorch.utils.tensor_utils.std(x: torch.Tensor, dim=None, unbiased=True, keepdim=False)[source]¶

Calculates the standard deviation of x along dim. Exists because torch.std sometimes causes some error in backward pass.

Parameters:	x – a tensor. dim – the dimension along which to calculate the variance. Can be `int`/`list`/`tuple`. unbiased – whether to use an unbiased estimate. Default: `True`. keepdim – whether to keep the reduced dims as `1`. Default: `False`.
Returns:	the standard deviation of x

neuralnet_pytorch.utils.tensor_utils.break_dim(x: torch.Tensor, dim: int, sizes=(-1, ))[source]¶

Break input tensor at dim into sizes.

Parameters:	x – an input tensor. dim – position at which the tensor is broken. sizes – sizes that the broken tensor is reshaped into.
Returns:	a tensor with shape at dim is sizes.

Utilities for Pytorch layers ¶

neuralnet_pytorch.utils.layer_utils.validate(func)[source]¶: A decorator to make sure output shape is a tuple of int s.

neuralnet_pytorch.utils.layer_utils.no_dim_change_op(cls)[source]¶

A decorator to overwrite output_shape to an op that does not change the tensor shape.

Parameters:	cls – a subclass of `Module`.

neuralnet_pytorch.utils.layer_utils.add_simple_repr(cls)[source]¶

A decorator to add a simple repr to the designated class.

Parameters:	cls – a subclass of `Module`.

neuralnet_pytorch.utils.layer_utils.add_custom_repr(cls)[source]¶

A decorator to add a custom repr to the designated class. User should define extra_repr for the decorated class.

Parameters:	cls – a subclass of `Module`.

neuralnet_pytorch.utils.layer_utils.get_non_none(array)[source]¶

Gets the first item that is not None from the given array.

Parameters:	array – an arbitrary array that is iterable.
Returns:	the first item that is not `None`.

Utilities for data handling ¶

class neuralnet_pytorch.utils.data_utils.DataLoader(dataset, batch_size=1, shuffle=False, sampler=None, batch_sampler=None, num_workers=0, collate_fn=None, pin_memory=False, drop_last=False, num_cached=10, **kwargs)[source]¶

A lightweight data loader. Works comparably to Pytorch’s torch.utils.data.Dataloader when the workload is light, but it initializes much faster. It is totally compatible with torch.utils.data.Dataset and can be an alternative for torch.utils.data.Dataloader.

Parameters:

dataset – an instance of torch.utils.data.Dataset.
batch_size – how many samples per batch to load. Default: 1.
shuffle – whether to shuffle in each iteration. Default: False.
sampler – defines the strategy to draw samples from the dataset. If specified, shuffle must be False.
batch_sampler – like sampler, but returns a batch of indices at a time. Mutually exclusive with batch_size, shuffle, sampler, and drop_last.
num_workers – number of threads to be used.
collate_fn – function to specify how a batch is loaded.
pin_memory – if True, the data loader will copy Tensors into CUDA pinned memory before returning them. If your data elements are a custom type, or your collate_fn returns a batch that is a custom type. Default: False.
num_cached – number of batches to be cached.
drop_last – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False and the size of dataset is not divisible by the batch size, then the last batch will be smaller. Default: False.
kwargs – arguments what will not be used. For compatibility with torch.utils.data.Dataloader only.

batches¶: contains batches of data when iterating over the dataset.

class neuralnet_pytorch.utils.data_utils.DataPrefetcher(loader, transform=None, device=device(type='cuda'))[source]¶

A prefetcher for torch.utils.data.Dataloader. Adapted from https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py.

Parameters:	loader – an instance of `torch.utils.data.Dataloader`. transform – a function to preprocess each batch in GPU. Default: `None`.

Examples

This prefetcher can be added seamlessly to the current code pipeline.

from torch.utils.data import DataLoader
import neuralnet_pytorch as nnt

...
loader = DataLoader(dataset, ...)
loader = nnt.DataPrefetcher(loader)
...

next_data¶: the prefetched data.

stream¶: an instance of torch.cuda.Stream.

neuralnet_pytorch.utils.data_utils.truncated_normal(x: torch.Tensor, a=-1, b=1, mean=0.0, std=1.0)[source]¶

Initializes a tensor from a truncated normal distribution.

Parameters:	x – a `torch.Tensor`. a – lower bound of the truncated normal. b – higher bound of the truncated normal. mean – mean of the truncated normal. std – standard deviation of the truncated normal.
Returns:	`None`.

neuralnet_pytorch.utils.data_utils.batch_set_value(params, values)[source]¶

Sets values of a tensor to another.

Parameters:	params – a `torch.Tensor`. values – a `numpy.ndarray` of the same shape as params.
Returns:	`None`.

neuralnet_pytorch.utils.data_utils.bulk_to_cuda(xs, non_blocking=False)[source]¶

Moves a list of numpy.ndarray to tensors.

Parameters:	xs – a tuple of `numpy.ndarray` s.
Returns:	a tuple of `torch.Tensor` s.

neuralnet_pytorch.utils.data_utils.bulk_to_cuda_sparse(xs, non_blocking=False)[source]¶

Moves a list sparse matrices to cuda tensor.

Parameters:	x – a list/tuple of `scipy.sparse.coo.coo_matrix`.
Returns:	a `torch.Tensor`.

neuralnet_pytorch.utils.data_utils.bulk_to_numpy(xs)[source]¶

Moves a list of tensors to numpy.ndarray.

Parameters:	xs – a list/tuple of `torch.Tensor` s.
Returns:	a tuple of `numpy.ndarray` s.

neuralnet_pytorch.utils.data_utils.to_cuda(x: numpy.ndarray, non_blocking=False)[source]¶

Moves a numpy to tensor.

Parameters:	x – a `numpy.ndarray`.
Returns:	a `torch.Tensor`.

neuralnet_pytorch.utils.data_utils.to_cuda_sparse(coo, non_blocking=False)[source]¶

Moves a sparse matrix to cuda tensor.

Parameters:	x – a `scipy.sparse.coo.coo_matrix`.
Returns:	a `torch.Tensor`.

neuralnet_pytorch.utils.data_utils.to_numpy(x: torch.Tensor)[source]¶

Moves a tensor to numpy.

Parameters:	x – a `torch.Tensor`.
Returns:	a `numpy.ndarray`.

neuralnet_pytorch.utils.data_utils.batch_to_device(batch, *args, **kwargs)[source]¶

Moves a batch to the specified device.

Parameters:	batch – a `torch.Tensor` or an iterable of `torch.Tensor`.
Returns:	a copy of the original batch that on the specified device.

neuralnet_pytorch.utils.data_utils.batch_to_cuda(batch, *args, **kwargs)[source]¶

Moves a batch to the default CUDA device.

Parameters:	batch – a `torch.Tensor` or an iterable of `torch.Tensor`.
Returns:	a copy of the original batch that on the default CUDA device.

neuralnet_pytorch.utils.data_utils.batch_set_tensor(params, values)[source]¶

Sets values of a tensor to another.

Parameters:	params – a `torch.Tensor`. values – a `torch.Tensor` of the same shape as params.
Returns:	`None`.

class neuralnet_pytorch.utils.data_utils.ReadWriteLock[source]¶

A lock object that allows many simultaneous read locks, but only one write lock. From https://www.oreilly.com/library/view/python-cookbook/0596001673/ch06s04.html.

acquire_read()[source]¶: Acquire a read lock. Blocks only if a thread has acquired the write lock.

acquire_write()[source]¶: Acquire a write lock. Blocks until there are no acquired read or write locks.

release_read()[source]¶: Release a read lock.

release_write()[source]¶: Release a write lock.

Utilities for activation functions ¶

neuralnet_pytorch.utils.activation_utils.relu(x: torch.Tensor, **kwargs)[source]¶: ReLU activation.

neuralnet_pytorch.utils.activation_utils.linear(x: torch.Tensor, **kwargs)[source]¶: Linear activation.

neuralnet_pytorch.utils.activation_utils.lrelu(x: torch.Tensor, **kwargs)[source]¶: Leaky ReLU activation.

neuralnet_pytorch.utils.activation_utils.tanh(x: torch.Tensor, **kwargs)[source]¶: Hyperbolic tangent activation.

neuralnet_pytorch.utils.activation_utils.sigmoid(x: torch.Tensor, **kwargs)[source]¶: Sigmoid activation.

neuralnet_pytorch.utils.activation_utils.elu(x: torch.Tensor, **kwargs)[source]¶: ELU activation.

neuralnet_pytorch.utils.activation_utils.selu(x: torch.Tensor, **kwargs)[source]¶: SELU activation.

neuralnet_pytorch.utils.activation_utils.softmax(x: torch.Tensor, **kwargs)[source]¶: Softmax activation.

neuralnet_pytorch.utils.activation_utils.function(activation, **kwargs)[source]¶

returns the activation. Can be str or callable. For str, possible choices are None, 'linear', 'relu', 'lrelu', 'tanh', 'sigmoid', 'elu', 'softmax', and 'selu'.

Parameters:	activation – name of the activation function.
Returns:	activation function

Utilities for computer vision applications ¶

neuralnet_pytorch.utils.cv_utils.rgb2gray(img: torch.Tensor)[source]¶

Converts a batch of RGB images to gray.

Parameters:	img – a batch of RGB image tensors.
Returns:	a batch of gray images.

neuralnet_pytorch.utils.cv_utils.rgb2ycbcr(img: torch.Tensor)[source]¶

Converts a batch of RGB images to YCbCr.

Parameters:	img – a batch of RGB image tensors.
Returns:	a batch of YCbCr images.

neuralnet_pytorch.utils.cv_utils.rgba2rgb(img: torch.Tensor)[source]¶

Converts a batch of RGBA images to RGB.

Parameters:	img – an batch of RGBA image tensors.
Returns:	a batch of RGB images.

neuralnet_pytorch.utils.cv_utils.ycbcr2rgb(img: torch.Tensor)[source]¶

Converts a batch of YCbCr images to RGB.

Parameters:	img – a batch of YCbCr image tensors.
Returns:	a batch of RGB images.

neuralnet_pytorch.utils.cv_utils.pc2vox(pc: torch.Tensor, vox_size=32, sigma=0.005, analytical_gauss_norm=True)[source]¶

Converts a centered point cloud to voxel representation.

Parameters:	pc – a batch of centered point clouds. vox_size – resolution of the voxel field. Default: 32. sigma – std of the Gaussian blur that determines the area of effect of each point. analytical_gauss_norm – whether to use a analytically precomputed normalization term.
Returns:	the voxel representation of input.

neuralnet_pytorch.utils.cv_utils.pc2vox_fast(pc: torch.Tensor, voxel_size=32, grid_size=1.0, filter_outlier=True, c_code=False)[source]¶

A fast conversion from a centered point cloud to voxel representation.

Parameters:	pc – a batch of centered point clouds. voxel_size – resolution of the voxel field. Default: 32. grid_size – range of the point clouds. Default: 1. filter_outlier – whether to filter point outside of grid_size. Default: `True`. c_code – whether to use a C++ implementation. Default: `True`.
Returns:	the voxel representation of input.

Utilities for Numpy arrays ¶

neuralnet_pytorch.utils.numpy_utils.is_outlier(x: numpy.ndarray, thresh=3.5)[source]¶

Returns a boolean array with True if points are outliers and False otherwise. Adapted from https://stackoverflow.com/a/11886564/4591601.

Parameters:	x – an `nxd` array of observations thresh – the modified z-score to use as a threshold. Observations with a modified z-score (based on the median absolute deviation) greater than this value will be classified as outliers.
Returns:	a `nx1` boolean array.

References

Boris Iglewicz and David Hoaglin (1993), “Volume 16: How to Detect and Handle Outliers”, The ASQC Basic References in Quality Control: Statistical Techniques, Edward F. Mykytka, Ph.D., Editor.

neuralnet_pytorch.utils.numpy_utils.smooth(x, beta=0.9, window='hanning')[source]¶

Smoothens the data using a window with requested size. This method is based on the convolution of a scaled window with the signal. The signal is prepared by introducing reflected copies of the signal (with the window size) in both ends so that transient parts are minimized in the begining and end part of the output signal.

Parameters:	x – the input signal. beta – the weighted moving average coeff. Window length is \(1 / (1 - \beta)\). window – the type of window from `'flat'`, `'hanning'`, `'hamming'`, `'bartlett'`, and `'blackman'`. Flat window will produce a moving average smoothing.
Returns:	the smoothed signal.

Examples

t = linspace(-2, 2, .1)
x = sin(t) + randn(len(t)) * .1
y = smooth(x)

Miscellaneous utilities ¶

neuralnet_pytorch.utils.misc_utils.time_cuda_module(f, *args, **kwargs)[source]¶

Measures the time taken by a Pytorch module.

Parameters:	f – a Pytorch module. args – arguments to be passed to f. kwargs – keyword arguments to be passed to f.
Returns:	the time (in second) that f takes.

neuralnet_pytorch.utils.misc_utils.slack_message(username: str, message: str, channel: str, token: str, **kwargs)[source]¶

Sends a slack message to the specified chatroom.

Parameters:	username – Slack username. message – message to be sent. channel – Slack channel. token – Slack chatroom token. kwargs – additional keyword arguments to slack’s `api_call()`.
Returns:	`None`.