Welcome to Neuralnet-Pytorch’s documentation!

Personally, going from Theano to Pytorch is pretty much like time traveling from 90s to the modern day. However, we feel like despite having a lot of bells and whistles, Pytorch is still missing many elements that are confirmed to never be added to the library. Therefore, this library is written to supplement more features to the current magical Pytorch. All the modules in the package directly subclass the corresponding modules from Pytorch, so everything should still be familiar. For example, the following snippet in Pytorch

from torch import nn
model = nn.Sequential(
    nn.Conv2d(1, 20, 5, padding=2),
    nn.ReLU(),
    nn.Conv2d(20, 64, 5, padding=2),
    nn.ReLU()
)

can be rewritten in Neuralnet-pytorch as

import neuralnet_pytorch as nnt
model = nnt.Sequential(
    nnt.Conv2d(1, 20, 5, padding='half', activation='relu'),
    nnt.Conv2d(20, 64, 5, padding='half', activation='relu')
)

which is the same as the native Pytorch, or

import neuralnet_pytorch as nnt
model = nnt.Sequential(input_shape=1)
model.add_module('conv1', nnt.Conv2d(model.output_shape, 20, 5, padding='half', activation='relu'))
model.add_module('conv2', nnt.Conv2d(model.output_shape, 64, 5, padding='half', activation='relu'))

which frees you from doing a lot of manual calculations when adding one layer on top of another. Theano folks will also find some reminiscence as many functions are highly inspired by Theano.

Installation

Requirements

Pytorch

Neuralnet-pytorch is built on top of Pytorch, so obviously Pytorch is needed. Please refer to the official Pytorch website for installation details.

Other dependencies

In Neuralnet-pytorch, we use several backends to visualize training, so it is necessary to install some additional packages. For convenience, installing the package will install all the required dependencies. Optional dependencies can be installed as instructed below.

Install Neuralnet-pytorch

There are two ways to install Neuralnet-pytorch: via PyPi and Github. At the moment, the package is not available on Conda yet.

From PyPi

The easiest and quickest way to get Neuralnet-pytorch is to install the package from Pypi. In a Terminal session, simply type

pip install neuralnet-pytorch

From Github

To install the bleeding-edge version, which is highly recommended, run

pip install git+git://github.com/justanhduc/neuralnet-pytorch.git@master

To install the package with optional dependencies, try

pip install "neuralnet-pytorch[option] @ git+git://github.com/justanhduc/neuralnet-pytorch.git@master"

in which option can be gin/geom/visdom/slack.

We also provide a version with some fancy Cuda/C++ implementations that are implemented or collected from various sources. To install this version, run

pip install neuralnet-pytorch --cuda-ext

Uninstall Neuralnet-pytorch

Simply use pip to uninstall the package

pip uninstall neuralnet-pytorch

Why would you want to do that anyway?

Upgrade Neuralnet-pytorch

Use pip with -U or --upgrade option

pip install -U neuralnet-pytorch

However, for maximal experience, please considering using the bleeding-edge version on Github.

Reinstall Neuralnet-pytorch

If you want to reinstall Neuralnet-pytorch, please uninstall and then install it again. When reinstalling the package, we recommend to use --no-cache-dir option as pip caches the previously built binaries

pip uninstall neuralnet-pytorch
pip install neuralnet-pytorch --no-cache-dir

Reference Manual

To check if you have installed Neuralnet-pytorch correctly, try

>>> import neuralnet_pytorch as nnt

If there is any error, please check your installation again (see Installation).

The basic contents of our package are as follows.

Neuralnet Layers

layers – Basic Layers

This section describes all the backbone modules of Neuralnet-pytorch.

Abstract Layers
Attributes

The following classes equip the plain torch modules with more bells and whistles. Also, some features are deeply integrated into Neuralnet-pytorch, which enables faster and more convenient training and testing of your neural networks.

class neuralnet_pytorch.layers.abstract._LayerMethod[source]

This mixin class contains various attributes to extend torch modules.

load(param_file, eval=True)[source]

Load the numpy.ndarray weights from file.

Parameters:
  • param_file – path to the weight file.
  • eval – whether to use evaluation mode or not.
params

Return a tuple of all the parameters in the module.

regularizable

Returns a tuple of parameters to be regularized.

reset_parameters()[source]

This overloads the torch.Module.reset_parameters() of the module. Used for custom weight initialization.

save(param_file)[source]

Save the weights of the model in numpy.nrdarray format.

Parameters:param_file – path to the weight file.
trainable

Return a tuple of all parameters with requires_grad set to True.

class neuralnet_pytorch.layers.MultiSingleInputModule(*modules_or_tensors)[source]

This is an abstract class. This class computes the results of multiple modules given an input tensor, then fuses the results.

Parameters:modules_or_tensors – a list of modules or tensors whose results are fused together.
input_shape

a list of input shapes of the incoming modules and tensors.

class neuralnet_pytorch.layers.MultiMultiInputModule(*modules_or_tensors)[source]

Similar to MultiSingleInputModule, but each module has its own input tensor.

Extended Pytorch Abstract Layers
class neuralnet_pytorch.layers.Module(input_shape=None)[source]

Similar to torch.nn.Module, but extended by _LayerMethod. All the usages in native Pytorch are preserved.

Parameters:input_shape – shape of the tensor to be input to the modules. Can be a list, tuple, nested list/tuple or an integer.
class neuralnet_pytorch.layers.Sequential(*args, input_shape=None)[source]

Similar to torch.nn.Sequential, but extended by _LayerMethod. All the usages in native Pytorch are preserved.

Parameters:
  • args – a list of modules as in torch.nn.Sequential.
  • input_shape – shape of the input tensor. If None, the functionality is the same as torch.nn.Sequential.
Quick-and-dirty Layers
class neuralnet_pytorch.layers.Lambda(func, input_shape=None, output_shape=None, **kwargs)[source]

Wraps a function as a Module.

Parameters:
  • func – a callable function.
  • input_shape – shape of the input tensor.
  • output_shape – shape of the output tensor. If None, the output shape is calculated by performing a forward pass.
  • kwargs – keyword arguments required by func.

Examples

You can easily wrap a torch function

import torch as T
import neuralnet_pytorch as nnt

a, b = T.rand(3, 1), T.rand(3, 2)
cat = nnt.Lambda(T.cat, dim=1)
c = cat((a, b))
print(c.shape)

Also, it works for any self-defined function as well

import neuralnet_pytorch as nnt

def foo(x, y):
    return x + y

a = T.rand(3, 3)
print(a)
foo_sum = nnt.Lambda(foo, y=1.)
res = foo_sum(a)
print(res)
Common Layers
Extended Pytorch Common Layers
class neuralnet_pytorch.layers.Conv2d(input_shape, out_channels, kernel_size, stride=1, padding='half', dilation=1, groups=1, bias=True, activation=None, weights_init=None, bias_init=None, **kwargs)[source]

Extends torch.nn.Conv2d with _LayerMethod.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.ConvTranspose2d(input_shape, out_channels, kernel_size, stride=1, padding='half', output_padding=0, groups=1, bias=True, dilation=1, activation='linear', weights_init=None, bias_init=None, output_size=None, **kwargs)[source]

Extends torch.nn.ConvTranspose2d with _LayerMethod.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – dilation * (kernel_size - 1) - padding zero-padding will be added to both sides of each dimension in the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep' which are common padding schemes. Default: 'half'.
  • output_padding – additional size added to one side of each dimension in the output shape. Default: 0
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • dilation – spacing between kernel elements. Default: 1.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • output_size – size of the output tensor. If None, the shape is automatically determined.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.FC(input_shape, out_features, bias=True, activation=None, weights_init=None, bias_init=None, flatten=False, keepdim=True, **kwargs)[source]

AKA fully connected layer in deep learning literature. This class extends torch.nn.Linear by _LayerMethod.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • bias (bool) – if set to False, the layer will not learn an additive bias. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • flatten (bool) – whether to flatten input tensor into 2D matrix. Default: False.
  • keepdim (bool) – whether to keep the output dimension when out_features is 1.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.Softmax(input_shape, out_features, dim=1, weights_init=None, bias_init=None, **kwargs)[source]

A special case of FC with softmax activation function.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • dim (int) – dimension to apply softmax. Default: 1.
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • kwargs – extra keyword arguments to pass to activation.
Extra Layers
class neuralnet_pytorch.layers.Activation(activation='relu', input_shape=None, **kwargs)[source]

Applies a non-linear function to the incoming input.

Parameters:
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • input_shape – shape of the input tensor. Can be None.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.ConvNormAct(input_shape, out_channels, kernel_size, stride=1, padding='half', dilation=1, groups=1, bias=True, activation='relu', weights_init=None, bias_init=None, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, no_scale=False, norm_layer='bn', **kwargs)[source]

Fuses convolution, normalization and activation together.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum (float) – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', 'ln' or a callable. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.DepthwiseSepConv2D(input_shape, out_channels, kernel_size, depth_mul=1, stride=1, padding='half', dilation=1, bias=True, activation=None, **kwargs)[source]

Performs depthwise separable convolution in image processing.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size (int) – size of the convolving kernel.
  • depth_mul – depth multiplier for intermediate result of depthwise convolution
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.FCNormAct(input_shape, out_features, bias=True, activation=None, weights_init=None, bias_init=None, flatten=False, keepdim=True, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, no_scale=False, norm_layer='bn', **kwargs)[source]

Fuses fully connected, normalization and activation together.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • bias (bool) – if set to False, the layer will not learn an additive bias. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • flatten (bool) – whether to flatten input tensor into 2D matrix. Default: False.
  • keepdim (bool) – whether to keep the output dimension when out_features is 1.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum (float) – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', and 'ln', or a callable function. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation and norm_layer.
class neuralnet_pytorch.layers.ResNetBasicBlock(input_shape, out_channels, kernel_size=3, stride=1, padding='half', dilation=1, activation='relu', base_width=64, downsample=None, groups=1, block=None, weights_init=None, norm_layer='bn', **kwargs)[source]

A basic block to build ResNet (https://arxiv.org/abs/1512.03385).

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • downsample – a module to process the residual branch when output shape is different from input shape. If None, a simple ConvNormAct is used.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • block – a function to construct the main branch of the block. If None, a simple block as described in the paper is used.
  • weights_init – a kernel initialization method from torch.nn.init.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', and 'ln'. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation.
expansion

expansion coefficients of the number of output channels. Default: 1.

Type:int
class neuralnet_pytorch.layers.ResNetBottleneckBlock(input_shape, out_channels, kernel_size=3, stride=1, padding='half', dilation=1, activation='relu', base_width=64, downsample=None, groups=1, block=None, weights_init=None, norm_layer='bn', **kwargs)[source]
A bottleneck block to build ResNet (https://arxiv.org/abs/1512.03385).
Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • downsample – a module to process the residual branch when output shape is different from input shape. If None, a simple ConvNormAct is used.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • block – a function to construct the main branch of the block. If None, a simple block as described in the paper is used.
  • weights_init – a kernel initialization method from torch.nn.init.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', and 'ln'. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation.
expansion

expansion coefficients of the number of output channels. Default: 4.

Type:int
class neuralnet_pytorch.layers.StackingConv(input_shape, out_channels, kernel_size, num_layers, stride=1, padding='half', dilation=1, bias=True, activation='relu', weights_init=None, bias_init=None, norm_method=None, groups=1, **kwargs)[source]

Stacks multiple convolution layers together.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • num_layer (int) – number of convolutional layers.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • norm_method – normalization method to be used. Choices are 'bn', 'in', and 'ln'. Default: 'bn'.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • kwargs – extra keyword arguments to pass to activation.
Graph Learning Layers
class neuralnet_pytorch.layers.GraphConv(input_shape, out_features, bias=True, activation=None, **kwargs)[source]

Performs graph convolution as described in https://arxiv.org/abs/1609.02907. Adapted from https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.BatchGraphConv(input_shape, out_features, bias=True, activation=None, **kwargs)[source]

Performs graph convolution as described in https://arxiv.org/abs/1609.02907 on a batch of graphs. Adapted from https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py.

Parameters:
  • input_shape – shape of the input tensor. The first dim should be batch size. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.GraphXConv(input_shape, out_features, out_instances=None, rank=None, bias=True, activation=None, weights_init=None, bias_init=None, **kwargs)[source]

Performs GraphX Convolution as described here. Disclaimer: I am the first author of the paper.

Parameters:
  • input_shape – shape of the input tensor. The first dim should be batch size. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • out_instances (int) – resolution of the output point clouds. If not specified, output will have the same resolution as input. Default: None.
  • rank – if specified and smaller than num_out_points, the mixing matrix will be broken into a multiplication of two matrices of sizes (num_out_points, rank) and (rank, input_shape[1]).
  • bias – whether to use bias. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • kwargs – extra keyword arguments to pass to activation.
Multi-module Layers
class neuralnet_pytorch.layers.Sum(*modules_or_tensors)[source]

Sums the outputs of multiple modules given an input tensor. A subclass of MultiSingleInputModule.

See also

MultiSingleInputModule MultiMultiInputModule SequentialSum ConcurrentSum Cat SequentialCat ConcurrentCat

class neuralnet_pytorch.layers.ConcurrentSum(*modules_or_tensors)[source]

Sums the outputs of multiple modules given input tensors. A subclass of MultiMultiInputModule.

See also

MultiSingleInputModule MultiMultiInputModule Sum SequentialSum Cat SequentialCat ConcurrentCat

class neuralnet_pytorch.layers.SequentialSum(*modules)[source]

Sums the intermediate outputs of multiple sequential modules given an input tensor. A subclass of Sum.

See also

MultiSingleInputModule MultiMultiInputModule Sum ConcurrentSum Cat SequentialCat ConcurrentCat

normalization – Normalization Layers

Extended Normalization Layers
class neuralnet_pytorch.layers.BatchNorm1d(input_shape, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, activation=None, no_scale=False, **kwargs)[source]

Performs batch normalization on 1D signals.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.BatchNorm2d(input_shape, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, activation=None, no_scale=False, **kwargs)[source]

Performs batch normalization on 2D signals.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.LayerNorm(input_shape, eps=1e-05, elementwise_affine=True, activation=None, **kwargs)[source]

Performs layer normalization on input tensor.

Parameters:
  • input_shape –

    input shape from an expected input of size

    \[[\text{input_shape}[0] \times \text{input_shape}[1] \times \ldots \times \text{input_shape}[-1]]\]

    If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size.

  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • elementwise_affine – a boolean value that when set to True, this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.InstanceNorm1d(input_shape, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False, activation=None, **kwargs)[source]

Performs instance normalization on 1D signals.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.InstanceNorm2d(input_shape, eps=1e-05, momentum=0.1, affine=True, track_running_stats=False, activation=None, **kwargs)[source]

Performs instance normalization on 2D signals.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.GroupNorm(input_shape, num_groups, eps=1e-05, affine=True, activation=None, **kwargs)[source]
Performs instance normalization on 2D signals.
Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • num_groups (int) – number of channels expected in input
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • kwargs – extra keyword arguments to pass to activation.
Custom Lormalization Layers
class neuralnet_pytorch.layers.FeatureNorm1d(input_shape, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, activation=None, no_scale=False, **kwargs)[source]

Performs batch normalization over the last dimension of the input.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.AdaIN(module, dim=(2, 3))[source]

The original Adaptive Instance Normalization from https://arxiv.org/abs/1703.06868.

\(Y_1 = \text{module}(X)\)

\(Y_2 = \text{module}(X)\)

\(Y = \sigma_{Y_2} * (Y_1 - \mu_{Y_1}) / \sigma_{Y_1} + \mu_{Y_2}\)

Parameters:
  • module – a torch module which generates target feature maps.
  • dim – dimension to reduce in the target feature maps. Default: (2, 3).
class neuralnet_pytorch.layers.MultiModuleAdaIN(module1, module2, dim1=(2, 3), dim2=(2, 3))[source]

A modified Adaptive Instance Normalization from https://arxiv.org/abs/1703.06868.

\(Y_1 = \text{module1}(X)\)

\(Y_2 = \text{module2}(X)\)

\(Y = \sigma_{Y_2} * (Y_1 - \mu_{Y_1}) / \sigma_{Y_1} + \mu_{Y_2}\)

Parameters:
  • module1 – a torch module which generates target feature maps.
  • module2 – a torch module which generates style feature maps.
  • dim1 – dimension to reduce in the target feature maps. Default: (2, 3).
  • dim2 – dimension to reduce in the style feature maps. Default: (2, 3).
class neuralnet_pytorch.layers.MultiInputAdaIN(module1, module2, dim1=(2, 3), dim2=(2, 3))[source]

A modified Adaptive Instance Normalization from https://arxiv.org/abs/1703.06868.

\(Y_1 = \text{module1}(X_1)\)

\(Y_2 = \text{module2}(X_2)\)

\(Y = \sigma_{Y_2} * (Y_1 - \mu_{Y_1}) / \sigma_{Y_1} + \mu_{Y_2}\)

Parameters:
  • module1 – a torch module which generates target feature maps.
  • module2 – a torch module which generates style feature maps.
  • dim1 – dimension to reduce in the target feature maps. Default: (2, 3).
  • dim2 – dimension to reduce in the style feature maps. Default: (2, 3).

resizing – Resizing Layers

Extended Resizing Layers
class neuralnet_pytorch.resizing.Interpolate(size=None, scale_factor=None, mode='bilinear', align_corners=None, input_shape=None)

Down/Upsamples a given multi-channel 1D (temporal), 2D (spatial) or 3D (volumetric) data.

Parameters:
  • size – output spatial sizes. Mutually exclusive with scale_factor.
  • scale_factor – float or tuple of floats. Multiplier for spatial size. Has to match input size if it is a tuple. Mutually exclusive with size.
  • mode – talgorithm used for upsampling: 'nearest', 'linear', 'bilinear', 'bicubic', 'trilinear', and 'area'. Default: 'nearest'.
  • align_corners – if True, the corner pixels of the input and output tensors are aligned, and thus preserving the values at those pixels. If False, the input and output tensors are aligned by the corner points of their corner pixels, and the interpolation uses edge value padding for out-of-boundary values, making this operation independent of input size when scale_factor is kept the same. This only has effect when mode is 'linear', 'bilinear', or 'trilinear'. Default: False.
  • input_shape – shape of the input tensor. Optional.
class neuralnet_pytorch.resizing.AvgPool2d(kernel_size, stride=None, padding=0, ceil_mode=False, count_include_pad=False, input_shape=None)

Applies a 2D average pooling over an input signal composed of several input planes.

Parameters:
  • kernel_size – the size of the window.
  • stride – the stride of the window. Default value is kernel_size.
  • padding – implicit zero padding to be added on both sides.
  • ceil_mode – when True, will use ceil instead of floor to compute the output shape.
  • count_include_pad – when True, will include the zero-padding in the averaging calculation.
  • input_shape – shape of the input image. Optional.
class neuralnet_pytorch.resizing.MaxPool2d(kernel_size, stride=None, padding=0, dilation=1, return_indices=False, ceil_mode=False, input_shape=None)

Applies a 2D max pooling over an input signal composed of several input planes.

Parameters:
  • kernel_size – the size of the window.
  • stride – the stride of the window. Default value is kernel_size.
  • padding – implicit zero padding to be added on both sides.
  • dilation – a parameter that controls the stride of elements in the window.
  • return_indices – if True, will return the max indices along with the outputs. Useful for torch.nn.MaxUnpool2d later.
  • ceil_mode – when True, will use ceil instead of floor to compute the output shape.
  • input_shape – shape of the input image. Optional.
Custom Resizing Layers
class neuralnet_pytorch.resizing.GlobalAvgPool2D(keepdim=False, input_shape=None)

Applies a 2D global average pooling over an input signal composed of several input planes.

Parameters:
  • keepdim (bool) – whether to keep the collapsed dim as (1, 1). Default: False.
  • input_shape – shape of the input image. Optional.
class neuralnet_pytorch.resizing.Cat(dim=1, *modules_or_tensors)

Concatenates the outputs of multiple modules given an input tensor. A subclass of MultiSingleInputModule.

See also

MultiSingleInputModule MultiMultiInputModule Sum SequentialSum ConcurrentSum SequentialCat ConcurrentCat

class neuralnet_pytorch.resizing.ConcurrentCat(dim=1, *modules_or_tensors)

Concatenates the outputs of multiple modules given input tensors. A subclass of MultiMultiInputModule.

See also

MultiSingleInputModule MultiMultiInputModule Sum SequentialSum ConcurrentSum Cat SequentialCat

class neuralnet_pytorch.resizing.SequentialCat(dim=1, *modules_or_tensors)

Concatenates the intermediate outputs of multiple sequential modules given an input tensor. A subclass of Cat.

See also

MultiSingleInputModule MultiMultiInputModule Sum SequentialSum ConcurrentSum Cat ConcurrentCat

class neuralnet_pytorch.resizing.Reshape(shape, input_shape=None)

Reshapes the input tensor to the specified shape.

Parameters:
  • shape – new shape of the tensor. One dim can be set to -1 to let torch automatically calculate the suitable value.
  • input_shape – shape of the input tensor. Optional.
class neuralnet_pytorch.resizing.Flatten(start_dim=0, end_dim=-1, input_shape=None)

Collapses some adjacent dims.

Parameters:
  • start_dim – dim where flattening starts.
  • end_dim – dim where flattening ends.
  • input_shape – shape of the input tensor. Optional.
class neuralnet_pytorch.resizing.DimShuffle(pattern, input_shape=None)

Reorder the dimensions of this variable, optionally inserting broadcasted dimensions. Inspired by Theano’s dimshuffle.

Parameters:
  • pattern – List/tuple of int mixed with ‘x’ for broadcastable dimensions.
  • input_shape – shape of the input tensor. Optional.

Monitor Network Training

monitor – Monitor Training of Neural Networks

class neuralnet_pytorch.monitor.Monitor(model_name=None, root=None, current_folder=None, print_freq=100, num_iters=None, prefix='run', use_visdom=False, use_tensorboard=False, send_slack=False, with_git=False, **kwargs)[source]

Collects statistics and displays the results using various backends. The collected stats are stored in ‘<root>/<model_name>/<prefix><#id>’ where #id is automatically assigned each time a new run starts.

Examples

The following snippet shows how to plot smoothed training losses and save images from the current iteration, and then display them every 100 iterations.

from neuralnet_pytorch import monitor as mon

mon.model_name = 'foo-model'
mon.set_path()
mon.print_freq = 100

...
for epoch in mon.iter_epoch(range(n_epochs)):
    for data in mon.iter_batch(data_loader):
        loss = net(data)
        mon.plot('training loss', loss, smooth=.99, filter_outliers=True)
        mon.imwrite('input images', data['images'], latest_only=True)
...
Parameters:
  • model_name (str) – name of the model folder. Default: None.
  • root (str) – path to store the collected statistics. Default: None.
  • current_folder (str) – if given, all the stats will be loaded from the given folder. Default: None.
  • print_freq (int) – statistics display frequency. Unit is iteration. Default: None.
  • num_iters (int) – number of iterations per epoch. If specified, training iteration percentage will be displayed along with epoch. Otherwise, it will be automatically calculated in the first epoch. Default: 100.
  • prefix (str) – predix for folder name of of each run. Default: 'run'.
  • use_visdom (bool) – whether to use Visdom for real-time monitoring. Default: False.
  • use_tensorboard (bool) – whether to use Tensorboard for real-time monitoring. Default: False.
  • send_slack (bool) – whether to send the statistics to Slack chatroom. Default: False.
  • with_git (bool) – whether to retrieve git information. Default: False.
  • kwargs – some miscellaneous options for Visdom and other functions.
path

contains all the runs of model_name.

current_folder

path to the current run.

vis

an instance of Visdom when use_visdom is set to True.

writer

an instance of Tensorboard’s SummaryWriter when use_tensorboard is set to True.

plot_folder

path to the folder containing the collected plots.

file_folder

path to the folder containing the collected files.

image_folder

path to the folder containing the collected images.

hist_folder

path to the folder containing the collected histograms.

clear_hist_stats(key)[source]

removes the collected statistics for histogram plot of the specified key.

Parameters:key – the name of the histogram collection.
Returns:None.
clear_mat_stats(key)[source]

removes the collected statistics for matrix plot of the specified key.

Parameters:key – the name of the matrix collection.
Returns:None.
clear_num_stats(key)[source]

removes the collected statistics for scalar plot of the specified key.

Parameters:key – the name of the scalar collection.
Returns:None.
epoch

returns the current epoch.

Returns:_last_epoch.
hist_stats

returns the collected tensors from beginning.

Returns:_hist_since_beginning.
iter

returns the current iteration.

Returns:_iter.
iter_batch(iterator)[source]

tracks training iteration and returns the item in iterator.

Parameters:iterator – the batch iterator. For e.g., enumerator(loader).
Returns:a generator over iterator.

Examples

>>> from neuralnet_pytorch import monitor as mon
>>> mon.print_freq = 1000
>>> data_loader = ...
>>> num_epochs = 10
>>> for epoch in mon.iter_epoch(range(num_epochs)):
...     for idx, data in mon.iter_batch(enumerate(data_loader)):
...         # do something here

See also

iter_epoch()

iter_epoch(iterator)[source]

tracks training epoch and returns the item in iterator.

Parameters:iterator – the epoch iterator. For e.g., range(num_epochs).
Returns:a generator over iterator.

Examples

>>> from neuralnet_pytorch import monitor as mon
>>> mon.print_freq = 1000
>>> num_epochs = 10
>>> for epoch in mon.iter_epoch(range(mon.epoch, num_epochs))
...     # do something here

See also

iter_batch()

load(file, method='pickle', version=-1, **kwargs)[source]

loads from the given file.

Parameters:
  • file – name of the saved file without version.
  • method –

    str or callable. If callable, it should be a custom method to load object. There are 3 types of str.

    'pickle': use pickle.dump() to store object.

    'torch': use torch.save() to store object.

    'txt': use numpy.savetxt() to store object.

    Default: 'pickle'.

  • version – the version of the saved file to load. Default: -1 (loads the latest version of the saved file).
  • kwargs – additional keyword arguments to the underlying load function.
Returns:

None.
mat_stats

returns the collected scalar statistics from beginning.

Returns:_num_since_beginning.
model_name

returns the name of the model.

Returns:_model_name.
num_stats

returns the collected scalar statistics from beginning.

Returns:_num_since_beginning.
prefix

returns the prefix of saved folders.

Returns:_prefix.
read_log(log)[source]

reads a saved log file.

Parameters:log – name of the log file.
Returns:contents of the log file.
reset()[source]

factory-resets the monitor object. This includes clearing all the collected data, set the iteration and epoch counters to 0, and reset the timer.

Returns:None.
run_training(net, solver: torch.optim.optimizer.Optimizer, train_loader, n_epochs: int, closure=None, eval_loader=None, valid_freq=None, start_epoch=None, scheduler=None, scheduler_iter=False, device=None, *args, **kwargs)[source]

Runs the training loop for the given neural network.

Parameters:
  • net – must be an instance of Net and Module.
  • solver – a solver for optimization.
  • train_loader – provides training data for neural net.
  • n_epochs – number of training epochs.
  • closure – a method to calculate loss in each optimization step. Optional.
  • eval_loader – provides validation data for neural net. Optional.
  • valid_freq – indicates how often validation is run. In effect if only eval_loader is given.
  • start_epoch – the epoch from which training will continue. If None, training counter will be set to 0.
  • scheduler – a learning rate scheduler. Default: None.
  • scheduler_iter – if True, scheduler will run every iteration. Otherwise, it will step every epoch. Default: False.
  • device – device to perform calculation. Default: None.
  • args – additional arguments that will be passed to neural net.
  • kwargs – additional keyword arguments that will be passed to neural net.
Returns:

None.

Examples

import neuralnet_pytorch as nnt
from neuralnet_pytorch import monitor as mon

class MyNet(nnt.Net, nnt.Module):
    ...

    def train_procedure(batch, *args, **kwargs):
        loss = ...
        mon.plot('train loss', loss)
        return loss

    def eval_procedure(batch, *args, **kwargs):
        pred = ...
        loss = ...
        acc = ...
        mon.plot('eval loss', loss)
        mon.plot('eval accuracy', acc)

# define the network, and training and testing loaders
net = MyNet(...)
train_loader = ...
eval_loader = ...
solver = ...
scheduler = ...

# instantiate a Monitor object
mon.model_name = 'my_net'
mon.print_freq = 100
mon.set_path()

# collect the parameters of the network
def save_checkpoint():
    states = {
        'states': mon.epoch,
        'model_state_dict': net.state_dict(),
        'opt_state_dict': solver.state_dict()
    }
    if scheduler is not None:
        states['scheduler_state_dict'] = scheduler.state_dict()

    mon.dump(name='training.pt', obj=states, type='torch', keep=5)

# save a checkpoint after each epoch and keep only the 5 latest checkpoints
mon.schedule(save_checkpoint)
print('Training...')

# run the training loop
mon.run_training(net, solver, train_loader, n_epochs, eval_loader=eval_loader, scheduler=scheduler,
                 valid_freq=val_freq)
print('Training finished!')
Parameters:
  • solver –
  • scheduler –
  • scheduler –
schedule(func, when=None, *args, **kwargs)[source]

uses to schedule a routine during every epoch in run_training().

Parameters:
  • func – a routine to be executed in run_training().
  • when – the moment when the func is executed. For the moment, choices are: 'begin_epoch', 'end_epoch', 'begin_iter', and 'end_iter'. Default: 'begin_epoch'.
  • args – additional arguments to func.
  • kwargs – additional keyword arguments to func.
Returns:

None

monitor – Track Intermediate Results

neuralnet_pytorch.monitor.track(name, x, direction=None)[source]

An identity function that registers hooks to track the value and gradient of the specified tensor.

Here is an example of how to track an intermediate output

input = ...
conv1 = nnt.track('op', nnt.Conv2d(shape, 4, 3), 'all')
conv2 = nnt.Conv2d(conv1.output_shape, 5, 3)
intermediate = conv1(input)
output = nnt.track('conv2_output', conv2(intermediate), 'all')
loss = T.sum(output ** 2)
loss.backward(retain_graph=True)
d_inter = T.autograd.grad(loss, intermediate, retain_graph=True)
d_out = T.autograd.grad(loss, output)
tracked = nnt.eval_tracked_variables()

testing.assert_allclose(tracked['conv2_output'], nnt.utils.to_numpy(output))
testing.assert_allclose(np.stack(tracked['grad_conv2_output']), nnt.utils.to_numpy(d_out[0]))
testing.assert_allclose(tracked['op'], nnt.utils.to_numpy(intermediate))
for d_inter_, tracked_d_inter_ in zip(d_inter, tracked['grad_op_output']):
    testing.assert_allclose(tracked_d_inter_, nnt.utils.to_numpy(d_inter_))
Parameters:
  • name – name of the tracked tensor.
  • x – tensor or module to be tracked. If module, the output of the module will be tracked.
  • direction –

    there are 4 options

    None: tracks only value.

    'forward': tracks only value.

    'backward': tracks only gradient.

    'all': tracks both value and gradient.

    Default: None.
Returns:

x.
neuralnet_pytorch.monitor.get_tracked_variables(name=None, return_name=False)[source]

Gets tracked variable given name.

Parameters:
  • name – name of the tracked variable. can be str or``list``/tuple of str``s. If ``None, all the tracked variables will be returned.
  • return_name – whether to return the names of the tracked variables.
Returns:

the tracked variables.
neuralnet_pytorch.monitor.eval_tracked_variables()[source]

Retrieves the values of tracked variables.

Returns:a dictionary containing the values of tracked variables associated with the given names.

Optimization

optimization – Extra Optimization Schemes

Extra Optimizers
class neuralnet_pytorch.optim.AdaBound(params, lr=0.001, betas=(0.9, 0.999), final_lr=0.1, gamma=0.001, eps=1e-08, weight_decay=0, amsbound=False)[source]

Implements AdaBound algorithm proposed in Adaptive Gradient Methods with Dynamic Bound of Learning Rate.

Parameters:
  • params – iterable of parameters to optimize or dicts defining. parameter groups
  • lr – Adam learning rate. Default: 1e-3.
  • betas – coefficients used for computing running averages of gradient and its square. Default: (0.9, 0.999).
  • final_lr – final (SGD) learning rate. Default: 0.1.
  • gamma – convergence speed of the bound functions. Default: 1e-3.
  • eps – term added to the denominator to improve numerical stability. Default: 1e-8.
  • weight_decay – weight decay (L2 penalty). Default: 0.
  • amsbound (bool) – whether to use the AMSBound variant of this algorithm.
class neuralnet_pytorch.optim.Lookahead(optimizer, la_steps=5, alpha=0.8, pullback_momentum=None)[source]

PyTorch implementation of the lookahead wrapper. Lookahead Optimizer: https://arxiv.org/abs/1907.08610.

Parameters:
  • optimizer – an usual optimizer such as Adam.
  • la_steps – number of lookahead steps. Default: 5.
  • alpha – linear interpolation coefficient. Default: 0.8.
  • pullback_momentum – either 'reset', 'pullback', or None. Default: None.
class neuralnet_pytorch.optim.NAdam(params, lr=0.001, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, decay=<function NAdam.<lambda>>)[source]

Adaptive moment with Nesterov gradients.

http://cs229.stanford.edu/proj2015/054_report.pdf

Parameters:
  • params – iterable of parameters to optimize or dicts defining parameter groups
  • lr – learning rate (default: 1e-3)
  • betas – coefficients used for computing running averages of gradient and its square (default: (0.9, 0.999))
  • eps – term added to the denominator to improve numerical stability (default: 1e-8)
  • weight_decay – weight decay (L2 penalty) (default: 0)
  • decay – a decay scheme for betas[0]. Default: \(\beta * (1 - 0.5 * 0.96^{\frac{t}{250}})\) where t is the training step.
Extra LR Schedulers
class neuralnet_pytorch.optim.lr_scheduler.InverseLR(optimizer, gamma, last_epoch=-1)[source]

Decreases lr every iteration by the inverse of gamma times iteration plus 1. \(\text{lr} = \text{lr} / (1 + \gamma * t)\).

Parameters:
  • optimizer – wrapped optimizer.
  • gamma – decrease coefficient.
  • last_epoch (int) – the index of last epoch. Default: -1.
class neuralnet_pytorch.optim.lr_scheduler.WarmRestart(optimizer, T_max, T_mul=1, eta_min=0, last_epoch=-1)[source]

Step should be used in batch iteration loop. Putting step in the epoch loop results in wrong behavior of the restart. One must not pass the iteration number to step.

Parameters:
  • optimizer – wrapped optimizer.
  • T_max – maximum number of iterations.
  • T_mul – multiplier for T_max.
  • eta_min – minimum learning rate. Default: 0.
  • last_epoch – the index of last epoch. Default: -1.

Additional Metrics

metrics – Common Metrics

neuralnet_pytorch.metrics.huber_loss(x, y, reduce='mean')[source]

An alias for torch.nn.functional.smooth_l1_loss().

neuralnet_pytorch.metrics.first_derivative_loss(x, y, p=2)[source]

Calculates lp loss between the first derivatives of the inputs.

Parameters:
  • x – a torch.Tensor.
  • y – a torch.Tensor of the same shape as x.
  • p – order of the norm.
Returns:

the scalar loss between the first derivatives of the inputs.
neuralnet_pytorch.metrics.lp_loss(x, y, p=2, reduction='mean')[source]

Calculates p-norm of (x - y).

Parameters:
  • x – a torch.Tensor.
  • y – a torch.Tensor of the same shape as x.
  • p – order of the norm.
  • reduction – 'mean' or 'sum'.
Returns:

the p-norm of (x - y).
neuralnet_pytorch.metrics.ssim(img1, img2, max_val=1.0, filter_size=11, filter_sigma=1.5, k1=0.01, k2=0.03, cs_map=False)[source]

Returns the Structural Similarity Map between img1 and img2. This function attempts to match the functionality of ssim_index_new.m by Zhou Wang: http://www.cns.nyu.edu/~lcv/ssim/msssim.zip

Parameters:
  • img1 – a 4D torch.Tensor.
  • img2 – a 4D torch.Tensor of the same shape as img1.
  • max_val – the dynamic range of the images (i.e., the difference between the maximum the and minimum allowed values).
  • filter_size – size of blur kernel to use (will be reduced for small images).
  • filter_sigma – standard deviation for Gaussian blur kernel (will be reduced for small images).
  • k1 – constant used to maintain stability in the SSIM calculation (0.01 in the original paper).
  • k2 – constant used to maintain stability in the SSIM calculation (0.03 in the original paper).
Returns:

pair containing the mean SSIM and contrast sensitivity between img1 and img2.
Raise:

RuntimeError: If input images don’t have the same shape or don’t have four dimensions: [batch_size, height, width, depth].
neuralnet_pytorch.metrics.psnr(x, y)[source]

Peak-signal-to-noise ratio for [0,1] images.

Parameters:
  • x – a torch.Tensor.
  • y – a torch.Tensor of the same shape as x.
neuralnet_pytorch.metrics.chamfer_loss(xyz1, xyz2, reduce='mean', c_code=False)[source]

Calculates the Chamfer distance between two batches of point clouds. The Pytorch code is adapted from DenseLidarNet. The CUDA code is adapted from AtlasNet.

Parameters:
  • xyz1 – a point cloud of shape (b, n1, k) or (n1, k).
  • xyz2 – a point cloud of shape (b, n2, k) or (n2, k).
  • reduce – 'mean' or 'sum'. Default: 'mean'.
  • c_code – whether to use CUDA implementation. This version is much more memory-friendly and slightly faster.
Returns:

the Chamfer distance between the inputs.
neuralnet_pytorch.metrics.emd_loss(xyz1, xyz2, reduce='mean', sinkhorn=False)[source]

Calculates the Earth Mover Distance (or Wasserstein metric) between two sets of points.

Parameters:
  • xyz1 – a point cloud of shape (b, n1, k) or (n1, k).
  • xyz2 – a point cloud of shape (b, n2, k) or (n2, k).
  • reduce – 'mean' or 'sum'. Default: 'mean'.
  • sinkhorn – whether to use the Sinkhorn approximation of the Wasserstein distance. False will fall back to a CUDA implementation, which is only available if the CUDA-extended neuralnet-pytorch is installed. Default: True.
Returns:

the EMD between the inputs.
neuralnet_pytorch.metrics.tv_reg(y)[source]

Total variation regularization.

Parameters:y – a tensor of at least 2D. The last 2 dimensions will be regularized.
Returns:the total variation loss.
neuralnet_pytorch.metrics.spectral_norm(module, name='weight', n_power_iterations=1, eps=1e-12, dim=None)[source]

Applies torch.nn.utils.spectral_norm() recursively to module and all of its submodules.

Parameters:
  • module – containing module.
  • name – name of weight parameter. Default: 'weight'.
  • n_power_iterations – number of power iterations to calculate spectral norm.
  • eps – epsilon for numerical stability in calculating norms.
  • dim – dimension corresponding to number of outputs, the default is 0, except for modules that are instances of ConvTranspose{1,2,3}d, when it is 1.
Returns:

the original module with the spectral norm hook.

Extras

utilities – Utility Functions and Classes

Utilities for tensor manipulation
neuralnet_pytorch.utils.tensor_utils.dimshuffle(x: torch.Tensor, pattern)[source]

Reorders the dimensions of this variable, optionally inserting broadcasted dimensions. Inspired by Theano’s dimshuffle.

Parameters:
  • x – Input tensor.
  • pattern – List/tuple of int mixed with x for broadcastable dimensions.
Returns:

a tensor whose shape matches pattern.

Examples

To create a 3D view of a [2D] matrix, call dimshuffle(x, [0,'x',1]). This will create a 3D view such that the middle dimension is an implicit broadcasted dimension. To do the same thing on the transpose of that matrix, call dimshuffle(x, [1, 'x', 0]).

See also

shape_padleft() shape_padright() swapaxes()

neuralnet_pytorch.utils.tensor_utils.shape_padleft(x: torch.Tensor, n_ones=1)[source]

Reshape x by left-padding the shape with n_ones 1s. Inspired by Theano’s shape_padleft.

Parameters:
  • x – variable to be reshaped.
  • n_ones – number of 1s to pad.

See also

dimshuffle() shape_padright() swapaxes()

neuralnet_pytorch.utils.tensor_utils.shape_padright(x: torch.Tensor, n_ones=1)[source]

Reshape x by right-padding the shape with n_ones 1s. Inspired by Theano’s shape_padright.

Parameters:
  • x – variable to be reshaped.
  • n_ones – number of 1s to pad.

See also

dimshuffle() shape_padleft() swapaxes()

neuralnet_pytorch.utils.tensor_utils.swapaxes(y: torch.Tensor, axis1, axis2)[source]

Swaps two given axes in the input tensor. If the input is of shape \((n_1, n_2, ..., n_{axis1}, ..., n_{axis2}, ...)\), the output will be \((n_1, n_2, ..., n_{axis2}, ..., n_{axis1}, ...)\). Can be seen as a generalization of transpose. Taken from Theano’s swapaxes.

Parameters:
  • y – a tensor.
  • axis1 – an axis to be swapped.
  • axis2 – another axis to be swapped.
Returns:

the axis-swapped tensor.

See also

dimshuffle() shape_padleft() shape_padright()

neuralnet_pytorch.utils.tensor_utils.ravel_index(indices: torch.Tensor, shape)[source]

Finds the linear index of index of a tensor of shape when it is flattened.

Parameters:
  • indices – a tensor containing indices to be linearized.
  • shape – shape of the tensor w.r.t the index tensor.
Returns:

the linear index of the element having index.

Examples

>>> import torch as T
>>> import numpy as np
>>> import neuralnet_pytorch as nnt
>>> shape = (2, 3, 5)
>>> a = T.arange(np.prod(shape)).view(*shape)
>>> indices = T.tensor([[1, 0, 1, 1], [0, 1, 2, 1], [1, 0, 4, 3]]).long()
>>> print(a[list(indices)])
tensor([16.,  5., 29., 23.])
>>> linear_indices = nnt.utils.ravel_index(indices, shape)
>>> print(linear_indices)
tensor([16,  5, 29, 23])
>>> print(a.flatten()[linear_indices])
tensor([16.,  5., 29., 23.])
neuralnet_pytorch.utils.tensor_utils.tile(x: torch.Tensor, dims)[source]

Repeats x along dims. Behaves like numpy.tile().

Parameters:
  • x – a torch.Tensor.
  • dims – the number of times to tile this tensor along each dimension.
Returns:

the tiled tensor.
neuralnet_pytorch.utils.tensor_utils.repeat(input: torch.Tensor, repeats, dim=None)[source]

Repeats elements of a tensor like numpy.repeat().

Parameters:
  • input – a torch.Tensor.
  • repeats – the number of times to repeat this tensor along dim.
  • dim – the dimension to repeat. If not specified, the method is applied to the flattened tensor. Default: None.
Returns:

the repeated tensor.
neuralnet_pytorch.utils.tensor_utils.block_diag(*blocks)[source]

Modified from scipy.linalg.block_diag. Creates a block diagonal matrix from provided arrays. Given the inputs A, B and C, the output will have these arrays arranged on the diagonal:

[[A, 0, 0],
 [0, B, 0],
 [0, 0, C]]
Parameters:blocks – an iterator of tensors, up to 2-D. A 1-D tensor of length n is treated as a 2-D array with shape (1,n).
Returns:a tensor with A, B, C, … on the diagonal. Has the same dtype as A.

Notes

If all the input arrays are square, the output is known as a block diagonal matrix.

See also

block_diag_sparse()

Examples

>>> from neuralnet_pytorch.utils import block_diag
>>> A = T.tensor([[1, 0],
...               [0, 1]])
>>> B = T.tensor([[3, 4, 5],
...               [6, 7, 8]])
>>> C = T.tensor([[7]])
>>> block_diag(A, B, C)
tensor([[1 0 0 0 0 0]
        [0 1 0 0 0 0]
        [0 0 3 4 5 0]
        [0 0 6 7 8 0]
        [0 0 0 0 0 7]])
>>> block_diag(T.tensor([1.0]), T.tensor([2, 3]), T.tensor([[4, 5], [6, 7]]))
tensor([[ 1.,  0.,  0.,  0.,  0.],
        [ 0.,  2.,  3.,  0.,  0.],
        [ 0.,  0.,  0.,  4.,  5.],
        [ 0.,  0.,  0.,  6.,  7.]])
neuralnet_pytorch.utils.tensor_utils.block_diag_sparse(a: torch.Tensor, dense=False)[source]

Creates a sparse block diagonal matrix from the provided array. Given the input tensor of size (n, r, c), the output will have the matrices of the last two indices arranged on the diagonal:

[[a[0], 0, 0],
 [0, a[1], 0],
 [0, 0, a[2]]]
Parameters:
  • a – a tensor of size (n, r, c).
  • dense – whether to return a dense matrix. Default: False.
Returns:

a tensor with a[0], a[1], a[2], … on the diagonal. Has the same dtype as a.

Notes

This function is for square matrices only. For general cases, use block_diag().

See also

block_diag()

Examples

>>> from neuralnet_pytorch.utils import block_diag_sparse
>>> import numpy as np
>>> a = T.arange(3 * 2 * 4).view(3, 2, 4)
>>> block_diag_sparse(a)
tensor(indices=tensor([[ 0,  0,  0,  0,  1,  1,  1,  1,  2,  2,  2,  2,  3,  3,
                         3,  3,  4,  4,  4,  4,  5,  5,  5,  5],
                       [ 0,  1,  2,  3,  0,  1,  2,  3,  4,  5,  6,  7,  4,  5,
                         6,  7,  8,  9, 10, 11,  8,  9, 10, 11]]),
       values=tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13,
                      14, 15, 16, 17, 18, 19, 20, 21, 22, 23]),
       size=(6, 12), nnz=24, layout=torch.sparse_coo)
>>> block_diag_sparse(a, dense=True)
tensor([[ 0,  1,  2,  3,  0,  0,  0,  0,  0,  0,  0,  0],
        [ 4,  5,  6,  7,  0,  0,  0,  0,  0,  0,  0,  0],
        [ 0,  0,  0,  0,  8,  9, 10, 11,  0,  0,  0,  0],
        [ 0,  0,  0,  0, 12, 13, 14, 15,  0,  0,  0,  0],
        [ 0,  0,  0,  0,  0,  0,  0,  0, 16, 17, 18, 19],
        [ 0,  0,  0,  0,  0,  0,  0,  0, 20, 21, 22, 23]])
neuralnet_pytorch.utils.tensor_utils.get_bilinear_weights(x: torch.Tensor, y: torch.Tensor, h: int, w: int, border_mode='nearest')[source]

Returns bilinear weights used in bilinear interpolation.

Parameters:
  • x – floating point coordinates along the x-axis.
  • y – floating point coordinates along the y-axis.
  • h – height of the 2D array.
  • w – width of the 2D array
  • border_mode – strategy to deal with borders. Choices are 'nearest' (default), 'mirror', and 'wrap'.
Returns:

the weights for bilinear interpolation and the integer coordinates.
neuralnet_pytorch.utils.tensor_utils.interpolate_bilinear(im: torch.Tensor, x: torch.Tensor, y: torch.Tensor, output_shape=None, border_mode='nearest')[source]

Returns a batch of interpolated images. Used for Spatial Transformer Network. Works like torch.grid_sample.

Parameters:
  • im – a batch of input images
  • x – floating point coordinates along the x-axis. Should be in the range [-1, 1].
  • y – floating point coordinates along the y-axis. Should be in the range [-1, 1].
  • output_shape – output shape. A tuple of height and width. If not specified, output will have the same shape as input.
  • border_mode – strategy to deal with borders. Choices are 'nearest' (default), 'mirror', and 'wrap'.
Returns:

the bilinear interpolated batch of images.
neuralnet_pytorch.utils.tensor_utils.batch_pairwise_sqdist(x: torch.Tensor, y: torch.Tensor, c_code=False)[source]

Calculates the pair-wise square distance between two sets of points. To get the Euclidean distance, explicit square root needs to be applied to the output.

Parameters:
  • x – a tensor of shape (..., nx, d). If the tensor dimension is 2, the tensor batch dim is broadcasted.
  • y – a tensor of shape (..., ny, d). If the tensor dimension is 2, the tensor batch dim is broadcasted.
  • c_code – whether to use a C++ implementation. Default: True when the CUDA extension is installed. False otherwise.
Returns:

a tensor containing the exhaustive square distance between every pair of points in x and y from the same batch.
neuralnet_pytorch.utils.tensor_utils.gram_matrix(x: torch.Tensor) → torch.Tensor[source]

Computes the Gram matrix given a 4D tensor.

Parameters:x – a 4D tensor.
Returns:the Gram matrix of x.
neuralnet_pytorch.utils.tensor_utils.var(x: torch.Tensor, dim=None, unbiased=True, keepdim=False)[source]

Calculates the variance of x along dim. Exists because torch.var sometimes causes some error in backward pass.

Parameters:
  • x – a tensor.
  • dim – the dimension along which to calculate the variance. Can be int/list/tuple.
  • unbiased – whether to use an unbiased estimate. Default: True.
  • keepdim – whether to keep the reduced dims as 1. Default: False.
Returns:

the variance of x
neuralnet_pytorch.utils.tensor_utils.std(x: torch.Tensor, dim=None, unbiased=True, keepdim=False)[source]

Calculates the standard deviation of x along dim. Exists because torch.std sometimes causes some error in backward pass.

Parameters:
  • x – a tensor.
  • dim – the dimension along which to calculate the variance. Can be int/list/tuple.
  • unbiased – whether to use an unbiased estimate. Default: True.
  • keepdim – whether to keep the reduced dims as 1. Default: False.
Returns:

the standard deviation of x
neuralnet_pytorch.utils.tensor_utils.break_dim(x: torch.Tensor, dim: int, sizes=(-1, ))[source]

Break input tensor at dim into sizes.

Parameters:
  • x – an input tensor.
  • dim – position at which the tensor is broken.
  • sizes – sizes that the broken tensor is reshaped into.
Returns:

a tensor with shape at dim is sizes.
Utilities for Pytorch layers
neuralnet_pytorch.utils.layer_utils.validate(func)[source]

A decorator to make sure output shape is a tuple of int s.

neuralnet_pytorch.utils.layer_utils.no_dim_change_op(cls)[source]

A decorator to overwrite output_shape to an op that does not change the tensor shape.

Parameters:cls – a subclass of Module.
neuralnet_pytorch.utils.layer_utils.add_simple_repr(cls)[source]
A decorator to add a simple repr to the designated class.
Parameters:cls – a subclass of Module.
neuralnet_pytorch.utils.layer_utils.add_custom_repr(cls)[source]

A decorator to add a custom repr to the designated class. User should define extra_repr for the decorated class.

Parameters:cls – a subclass of Module.
neuralnet_pytorch.utils.layer_utils.get_non_none(array)[source]

Gets the first item that is not None from the given array.

Parameters:array – an arbitrary array that is iterable.
Returns:the first item that is not None.
Utilities for data handling
class neuralnet_pytorch.utils.data_utils.DataLoader(dataset, batch_size=1, shuffle=False, sampler=None, batch_sampler=None, num_workers=0, collate_fn=None, pin_memory=False, drop_last=False, num_cached=10, **kwargs)[source]

A lightweight data loader. Works comparably to Pytorch’s torch.utils.data.Dataloader when the workload is light, but it initializes much faster. It is totally compatible with torch.utils.data.Dataset and can be an alternative for torch.utils.data.Dataloader.

Parameters:
  • dataset – an instance of torch.utils.data.Dataset.
  • batch_size – how many samples per batch to load. Default: 1.
  • shuffle – whether to shuffle in each iteration. Default: False.
  • sampler – defines the strategy to draw samples from the dataset. If specified, shuffle must be False.
  • batch_sampler – like sampler, but returns a batch of indices at a time. Mutually exclusive with batch_size, shuffle, sampler, and drop_last.
  • num_workers – number of threads to be used.
  • collate_fn – function to specify how a batch is loaded.
  • pin_memory – if True, the data loader will copy Tensors into CUDA pinned memory before returning them. If your data elements are a custom type, or your collate_fn returns a batch that is a custom type. Default: False.
  • num_cached – number of batches to be cached.
  • drop_last – set to True to drop the last incomplete batch, if the dataset size is not divisible by the batch size. If False and the size of dataset is not divisible by the batch size, then the last batch will be smaller. Default: False.
  • kwargs – arguments what will not be used. For compatibility with torch.utils.data.Dataloader only.
batches

contains batches of data when iterating over the dataset.

class neuralnet_pytorch.utils.data_utils.DataPrefetcher(loader, transform=None, device=device(type='cuda'))[source]

A prefetcher for torch.utils.data.Dataloader. Adapted from https://github.com/NVIDIA/apex/blob/master/examples/imagenet/main_amp.py.

Parameters:
  • loader – an instance of torch.utils.data.Dataloader.
  • transform – a function to preprocess each batch in GPU. Default: None.

Examples

This prefetcher can be added seamlessly to the current code pipeline.

from torch.utils.data import DataLoader
import neuralnet_pytorch as nnt

...
loader = DataLoader(dataset, ...)
loader = nnt.DataPrefetcher(loader)
...
next_data

the prefetched data.

stream

an instance of torch.cuda.Stream.

neuralnet_pytorch.utils.data_utils.truncated_normal(x: torch.Tensor, a=-1, b=1, mean=0.0, std=1.0)[source]

Initializes a tensor from a truncated normal distribution.

Parameters:
  • x – a torch.Tensor.
  • a – lower bound of the truncated normal.
  • b – higher bound of the truncated normal.
  • mean – mean of the truncated normal.
  • std – standard deviation of the truncated normal.
Returns:

None.
neuralnet_pytorch.utils.data_utils.batch_set_value(params, values)[source]

Sets values of a tensor to another.

Parameters:
  • params – a torch.Tensor.
  • values – a numpy.ndarray of the same shape as params.
Returns:

None.
neuralnet_pytorch.utils.data_utils.bulk_to_cuda(xs, non_blocking=False)[source]

Moves a list of numpy.ndarray to tensors.

Parameters:xs – a tuple of numpy.ndarray s.
Returns:a tuple of torch.Tensor s.
neuralnet_pytorch.utils.data_utils.bulk_to_cuda_sparse(xs, non_blocking=False)[source]

Moves a list sparse matrices to cuda tensor.

Parameters:x – a list/tuple of scipy.sparse.coo.coo_matrix.
Returns:a torch.Tensor.
neuralnet_pytorch.utils.data_utils.bulk_to_numpy(xs)[source]

Moves a list of tensors to numpy.ndarray.

Parameters:xs – a list/tuple of torch.Tensor s.
Returns:a tuple of numpy.ndarray s.
neuralnet_pytorch.utils.data_utils.to_cuda(x: numpy.ndarray, non_blocking=False)[source]

Moves a numpy to tensor.

Parameters:x – a numpy.ndarray.
Returns:a torch.Tensor.
neuralnet_pytorch.utils.data_utils.to_cuda_sparse(coo, non_blocking=False)[source]

Moves a sparse matrix to cuda tensor.

Parameters:x – a scipy.sparse.coo.coo_matrix.
Returns:a torch.Tensor.
neuralnet_pytorch.utils.data_utils.to_numpy(x: torch.Tensor)[source]

Moves a tensor to numpy.

Parameters:x – a torch.Tensor.
Returns:a numpy.ndarray.
neuralnet_pytorch.utils.data_utils.batch_to_device(batch, *args, **kwargs)[source]

Moves a batch to the specified device.

Parameters:batch – a torch.Tensor or an iterable of torch.Tensor.
Returns:a copy of the original batch that on the specified device.
neuralnet_pytorch.utils.data_utils.batch_to_cuda(batch, *args, **kwargs)[source]

Moves a batch to the default CUDA device.

Parameters:batch – a torch.Tensor or an iterable of torch.Tensor.
Returns:a copy of the original batch that on the default CUDA device.
neuralnet_pytorch.utils.data_utils.batch_set_tensor(params, values)[source]

Sets values of a tensor to another.

Parameters:
  • params – a torch.Tensor.
  • values – a torch.Tensor of the same shape as params.
Returns:

None.
class neuralnet_pytorch.utils.data_utils.ReadWriteLock[source]

A lock object that allows many simultaneous read locks, but only one write lock. From https://www.oreilly.com/library/view/python-cookbook/0596001673/ch06s04.html.

acquire_read()[source]

Acquire a read lock. Blocks only if a thread has acquired the write lock.

acquire_write()[source]

Acquire a write lock. Blocks until there are no acquired read or write locks.

release_read()[source]

Release a read lock.

release_write()[source]

Release a write lock.

Utilities for activation functions
neuralnet_pytorch.utils.activation_utils.relu(x: torch.Tensor, **kwargs)[source]

ReLU activation.

neuralnet_pytorch.utils.activation_utils.linear(x: torch.Tensor, **kwargs)[source]

Linear activation.

neuralnet_pytorch.utils.activation_utils.lrelu(x: torch.Tensor, **kwargs)[source]

Leaky ReLU activation.

neuralnet_pytorch.utils.activation_utils.tanh(x: torch.Tensor, **kwargs)[source]

Hyperbolic tangent activation.

neuralnet_pytorch.utils.activation_utils.sigmoid(x: torch.Tensor, **kwargs)[source]

Sigmoid activation.

neuralnet_pytorch.utils.activation_utils.elu(x: torch.Tensor, **kwargs)[source]

ELU activation.

neuralnet_pytorch.utils.activation_utils.selu(x: torch.Tensor, **kwargs)[source]

SELU activation.

neuralnet_pytorch.utils.activation_utils.softmax(x: torch.Tensor, **kwargs)[source]

Softmax activation.

neuralnet_pytorch.utils.activation_utils.function(activation, **kwargs)[source]

returns the activation. Can be str or callable. For str, possible choices are None, 'linear', 'relu', 'lrelu', 'tanh', 'sigmoid', 'elu', 'softmax', and 'selu'.

Parameters:activation – name of the activation function.
Returns:activation function
Utilities for computer vision applications
neuralnet_pytorch.utils.cv_utils.rgb2gray(img: torch.Tensor)[source]

Converts a batch of RGB images to gray.

Parameters:img – a batch of RGB image tensors.
Returns:a batch of gray images.
neuralnet_pytorch.utils.cv_utils.rgb2ycbcr(img: torch.Tensor)[source]

Converts a batch of RGB images to YCbCr.

Parameters:img – a batch of RGB image tensors.
Returns:a batch of YCbCr images.
neuralnet_pytorch.utils.cv_utils.rgba2rgb(img: torch.Tensor)[source]

Converts a batch of RGBA images to RGB.

Parameters:img – an batch of RGBA image tensors.
Returns:a batch of RGB images.
neuralnet_pytorch.utils.cv_utils.ycbcr2rgb(img: torch.Tensor)[source]

Converts a batch of YCbCr images to RGB.

Parameters:img – a batch of YCbCr image tensors.
Returns:a batch of RGB images.
neuralnet_pytorch.utils.cv_utils.pc2vox(pc: torch.Tensor, vox_size=32, sigma=0.005, analytical_gauss_norm=True)[source]

Converts a centered point cloud to voxel representation.

Parameters:
  • pc – a batch of centered point clouds.
  • vox_size – resolution of the voxel field. Default: 32.
  • sigma – std of the Gaussian blur that determines the area of effect of each point.
  • analytical_gauss_norm – whether to use a analytically precomputed normalization term.
Returns:

the voxel representation of input.
neuralnet_pytorch.utils.cv_utils.pc2vox_fast(pc: torch.Tensor, voxel_size=32, grid_size=1.0, filter_outlier=True, c_code=False)[source]

A fast conversion from a centered point cloud to voxel representation.

Parameters:
  • pc – a batch of centered point clouds.
  • voxel_size – resolution of the voxel field. Default: 32.
  • grid_size – range of the point clouds. Default: 1.
  • filter_outlier – whether to filter point outside of grid_size. Default: True.
  • c_code – whether to use a C++ implementation. Default: True.
Returns:

the voxel representation of input.
Utilities for Numpy arrays
neuralnet_pytorch.utils.numpy_utils.is_outlier(x: numpy.ndarray, thresh=3.5)[source]

Returns a boolean array with True if points are outliers and False otherwise. Adapted from https://stackoverflow.com/a/11886564/4591601.

Parameters:
  • x – an nxd array of observations
  • thresh – the modified z-score to use as a threshold. Observations with a modified z-score (based on the median absolute deviation) greater than this value will be classified as outliers.
Returns:

a nx1 boolean array.

References

Boris Iglewicz and David Hoaglin (1993), “Volume 16: How to Detect and Handle Outliers”, The ASQC Basic References in Quality Control: Statistical Techniques, Edward F. Mykytka, Ph.D., Editor.

neuralnet_pytorch.utils.numpy_utils.smooth(x, beta=0.9, window='hanning')[source]

Smoothens the data using a window with requested size. This method is based on the convolution of a scaled window with the signal. The signal is prepared by introducing reflected copies of the signal (with the window size) in both ends so that transient parts are minimized in the begining and end part of the output signal.

Parameters:
  • x – the input signal.
  • beta – the weighted moving average coeff. Window length is \(1 / (1 - \beta)\).
  • window – the type of window from 'flat', 'hanning', 'hamming', 'bartlett', and 'blackman'. Flat window will produce a moving average smoothing.
Returns:

the smoothed signal.

Examples

t = linspace(-2, 2, .1)
x = sin(t) + randn(len(t)) * .1
y = smooth(x)
Miscellaneous utilities
neuralnet_pytorch.utils.misc_utils.time_cuda_module(f, *args, **kwargs)[source]

Measures the time taken by a Pytorch module.

Parameters:
  • f – a Pytorch module.
  • args – arguments to be passed to f.
  • kwargs – keyword arguments to be passed to f.
Returns:

the time (in second) that f takes.
neuralnet_pytorch.utils.misc_utils.slack_message(username: str, message: str, channel: str, token: str, **kwargs)[source]

Sends a slack message to the specified chatroom.

Parameters:
  • username – Slack username.
  • message – message to be sent.
  • channel – Slack channel.
  • token – Slack chatroom token.
  • kwargs – additional keyword arguments to slack’s api_call().
Returns:

None.

Gin-config

gin – Gin-config

This page presents a list of preserved keywords for Gin-config.

Optimizer Keywords
'adabound' - neuralnet_pytorch.optim.AdaBound
'adadelta' - torch.optim.Adadelta
'adagrad' - torch.optim.Adagrad
'adam' - torch.optim.Adam
'adamax' - torch.optim.Adamax
'adamw' - torch.optim.AdamW
'asgd' - torch.optim.ASGD
'lbfgs' - torch.optim.LBFGS
'lookahead' - neuralnet_pytorch.optim.Lookahead
'nadam' - neuralnet_pytorch.optim.NAdam
'rmsprop' - torch.optim.RMSprop
'rprop' - torch.optim.Rprop
'sgd' - torch.optim.SGD
'sparse_adam' - torch.optim.SparseAdam

If Automatic Mixed Precision is installed, there are a few extra choices coming from the package

'fusedadam' - apex.optimizers.FusedAdam
'fusedlamb' - apex.optimizers.FusedLAMB
'fusednovograd' - apex.optimizers.FusedNovoGrad
'fusedsgd' - apex.optimizers.FusedSGD
Learning Rate Scheduler Keywords
'lambda_lr' - torch.optim.lr_scheduler.LambdaLR
'step_lr' - torch.optim.lr_scheduler.StepLR
'multistep_lr' - torch.optim.lr_scheduler.MultiStepLR
'exp_lr' - torch.optim.lr_scheduler.ExponentialLR
'cosine_lr' - torch.optim.lr_scheduler.CosineAnnealingLR
'plateau_lr' - torch.optim.lr_scheduler.ReduceLROnPlateau
'cyclic_lr' - torch.optim.lr_scheduler.CyclicLR
'inverse_lr' - neuralnet_pytorch.optim.lr_scheduler.InverseLR
'warm_restart' - neuralnet_pytorch.optim.lr_scheduler.WarmRestart
Loss Keywords
'l1loss' - torch.nn.L1Loss
'mseloss' - torch.nn.MSELoss
'celoss' - torch.nn.CrossEntropyLoss
'bceloss' - torch.nn.BCELoss
'bcelogit_loss' - torch.nn.BCEWithLogitsLoss
'nllloss' - torch.nn.NLLLoss
'kldloss' - torch.nn.KLDivLoss
'huberloss' - torch.nn.SmoothL1Loss
'cosineembed_loss' - torch.nn.CosineEmbeddingLoss
Activation Keywords
'elu' - torch.nn.ELU
'hardshrink' - torch.nn.Hardshrink
'hardtanh' - torch.nn.Hardtanh
'lrelu' - torch.nn.LeakyReLU
'logsig' - torch.nn.LogSigmoid
'multihead_att' - torch.nn.MultiheadAttention
'prelu' - torch.nn.PReLU
'relu' - torch.nn.ReLU
'rrelu' - torch.nn.RReLU
'selu' - torch.nn.SELU
'celu' - torch.nn.CELU
'sigmoid' - torch.nn.Sigmoid
'softplus' - torch.nn.Softplus
'softshrink' - torch.nn.Softshrink
'softsign' - torch.nn.Softsign
'tanh' - torch.nn.Tanh
'tanhshrink' - torch.nn.Tanhshrink
'threshold' - torch.nn.Threshold
Data Type Keywords
'float16' - torch.float16
'float32' - torch.float32
'float64' - torch.float64
'int8' - torch.int8
'int16' - torch.int16
'int32' - torch.int32
'int64' - torch.int64
'complex32' - torch.complex32
'complex64' - torch.complex64
'complex128' - torch.complex128
'float' - torch.float
'short' - torch.short
'long' - torch.long
'half' - torch.half
'uint8' - torch.uint8
'int' - torch.int

Pretrained Networks

zoo – Model Zoo - Pretrained Neural Nets

Residual Neural Networks

Adapted PyTorch’s officially pretrained ResNets on ImageNet.

class neuralnet_pytorch.zoo.resnet.ResNet(block, layers, num_classes=1000, zero_init_residual=False, groups=1, width_per_group=64, replace_stride_with_dilation=None, norm_layer=None, default_init=True)[source]
neuralnet_pytorch.zoo.resnet.resnet18(pretrained=False, progress=True, **kwargs)[source]

ResNet-18 model from “Deep Residual Learning for Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.resnet34(pretrained=False, progress=True, **kwargs)[source]

ResNet-34 model from “Deep Residual Learning for Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.resnet50(pretrained=False, progress=True, **kwargs)[source]

ResNet-50 model from “Deep Residual Learning for Image Recognition” :param pretrained: If True, returns a model pre-trained on ImageNet :type pretrained: bool :param progress: If True, displays a progress bar of the download to stderr :type progress: bool

neuralnet_pytorch.zoo.resnet.resnet101(pretrained=False, progress=True, **kwargs)[source]

ResNet-101 model from “Deep Residual Learning for Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.resnet152(pretrained=False, progress=True, **kwargs)[source]

ResNet-152 model from “Deep Residual Learning for Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.resnext50_32x4d(pretrained=False, progress=True, **kwargs)[source]

ResNeXt-50 32x4d model from “Aggregated Residual Transformation for Deep Neural Networks”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.resnext101_32x8d(pretrained=False, progress=True, **kwargs)[source]

ResNeXt-101 32x8d model from “Aggregated Residual Transformation for Deep Neural Networks”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.wide_resnet50_2(pretrained=False, progress=True, **kwargs)[source]

Wide ResNet-50-2 model from “Wide Residual Networks” The model is the same as ResNet except for the bottleneck number of channels which is twice larger in every block. The number of channels in outer 1x1 convolutions is the same, e.g. last block in ResNet-50 has 2048-512-2048 channels, and in Wide ResNet-50-2 has 2048-1024-2048.

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.resnet.wide_resnet101_2(pretrained=False, progress=True, **kwargs)[source]

Wide ResNet-101-2 model from “Wide Residual Networks” The model is the same as ResNet except for the bottleneck number of channels which is twice larger in every block. The number of channels in outer 1x1 convolutions is the same, e.g. last block in ResNet-50 has 2048-512-2048 channels, and in Wide ResNet-50-2 has 2048-1024-2048.

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
VGG

Adapted PyTorch’s officially pretrained VGG on ImageNet.

class neuralnet_pytorch.zoo.vgg.VGG(features, num_classes=1000, default_init=True)[source]
neuralnet_pytorch.zoo.vgg.vgg11(pretrained=False, progress=True, **kwargs)[source]

VGG 11-layer model (configuration “A”) from “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg11_bn(pretrained=False, progress=True, **kwargs)[source]

VGG 11-layer model (configuration “A”) with batch normalization “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg13(pretrained=False, progress=True, **kwargs)[source]

VGG 13-layer model (configuration “B”) “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg13_bn(pretrained=False, progress=True, **kwargs)[source]

VGG 13-layer model (configuration “B”) with batch normalization “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg16(pretrained=False, progress=True, **kwargs)[source]

VGG 16-layer model (configuration “D”) “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg16_bn(pretrained=False, progress=True, **kwargs)[source]

VGG 16-layer model (configuration “D”) with batch normalization “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg19_bn(pretrained=False, progress=True, **kwargs)[source]

VGG 19-layer model (configuration ‘E’) with batch normalization “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.
neuralnet_pytorch.zoo.vgg.vgg19(pretrained=False, progress=True, **kwargs)[source]

VGG 19-layer model (configuration “E”) “Very Deep Convolutional Networks For Large-Scale Image Recognition”

Parameters:
  • pretrained – If True, returns a model pre-trained on ImageNet.
  • progress – If True, displays a progress bar of the download to stderr.

License

MIT License

Copyright (c) 2019 Duc Nguyen

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

Contact

I am Nguyen Anh Duc, a PhD candidate in Computer Vision at Yonsei University. This package is written during my little free time when doing PhD. Most but not all the written modules are properly checked. Therefore, buggy performance is inevitable. No replacements or refunds are guaranteed for such situations. If you want to collaborate or contribute, feel free to make a PR/issue on Github or email me at adnguyen@yonsei.ac.kr.