layers – Basic Layers

This section describes all the backbone modules of Neuralnet-pytorch.

Abstract Layers

Attributes

The following classes equip the plain torch modules with more bells and whistles. Also, some features are deeply integrated into Neuralnet-pytorch, which enables faster and more convenient training and testing of your neural networks.

class neuralnet_pytorch.layers.abstract._LayerMethod[source]

This mixin class contains various attributes to extend torch modules.

load(param_file, eval=True)[source]

Load the numpy.ndarray weights from file.

Parameters:
  • param_file – path to the weight file.
  • eval – whether to use evaluation mode or not.
params

Return a tuple of all the parameters in the module.

regularizable

Returns a tuple of parameters to be regularized.

reset_parameters()[source]

This overloads the torch.Module.reset_parameters() of the module. Used for custom weight initialization.

save(param_file)[source]

Save the weights of the model in numpy.nrdarray format.

Parameters:param_file – path to the weight file.
trainable

Return a tuple of all parameters with requires_grad set to True.

class neuralnet_pytorch.layers.MultiSingleInputModule(*modules_or_tensors)[source]

This is an abstract class. This class computes the results of multiple modules given an input tensor, then fuses the results.

Parameters:modules_or_tensors – a list of modules or tensors whose results are fused together.
input_shape

a list of input shapes of the incoming modules and tensors.

class neuralnet_pytorch.layers.MultiMultiInputModule(*modules_or_tensors)[source]

Similar to MultiSingleInputModule, but each module has its own input tensor.

Extended Pytorch Abstract Layers

class neuralnet_pytorch.layers.Module(input_shape=None)[source]

Similar to torch.nn.Module, but extended by _LayerMethod. All the usages in native Pytorch are preserved.

Parameters:input_shape – shape of the tensor to be input to the modules. Can be a list, tuple, nested list/tuple or an integer.
class neuralnet_pytorch.layers.Sequential(*args, input_shape=None)[source]

Similar to torch.nn.Sequential, but extended by _LayerMethod. All the usages in native Pytorch are preserved.

Parameters:
  • args – a list of modules as in torch.nn.Sequential.
  • input_shape – shape of the input tensor. If None, the functionality is the same as torch.nn.Sequential.

Quick-and-dirty Layers

class neuralnet_pytorch.layers.Lambda(func, input_shape=None, output_shape=None, **kwargs)[source]

Wraps a function as a Module.

Parameters:
  • func – a callable function.
  • input_shape – shape of the input tensor.
  • output_shape – shape of the output tensor. If None, the output shape is calculated by performing a forward pass.
  • kwargs – keyword arguments required by func.

Examples

You can easily wrap a torch function

import torch as T
import neuralnet_pytorch as nnt

a, b = T.rand(3, 1), T.rand(3, 2)
cat = nnt.Lambda(T.cat, dim=1)
c = cat((a, b))
print(c.shape)

Also, it works for any self-defined function as well

import neuralnet_pytorch as nnt

def foo(x, y):
    return x + y

a = T.rand(3, 3)
print(a)
foo_sum = nnt.Lambda(foo, y=1.)
res = foo_sum(a)
print(res)

Common Layers

Extended Pytorch Common Layers

class neuralnet_pytorch.layers.Conv2d(input_shape, out_channels, kernel_size, stride=1, padding='half', dilation=1, groups=1, bias=True, activation=None, weights_init=None, bias_init=None, **kwargs)[source]

Extends torch.nn.Conv2d with _LayerMethod.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.ConvTranspose2d(input_shape, out_channels, kernel_size, stride=1, padding='half', output_padding=0, groups=1, bias=True, dilation=1, activation='linear', weights_init=None, bias_init=None, output_size=None, **kwargs)[source]

Extends torch.nn.ConvTranspose2d with _LayerMethod.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – dilation * (kernel_size - 1) - padding zero-padding will be added to both sides of each dimension in the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep' which are common padding schemes. Default: 'half'.
  • output_padding – additional size added to one side of each dimension in the output shape. Default: 0
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • dilation – spacing between kernel elements. Default: 1.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • output_size – size of the output tensor. If None, the shape is automatically determined.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.FC(input_shape, out_features, bias=True, activation=None, weights_init=None, bias_init=None, flatten=False, keepdim=True, **kwargs)[source]

AKA fully connected layer in deep learning literature. This class extends torch.nn.Linear by _LayerMethod.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • bias (bool) – if set to False, the layer will not learn an additive bias. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • flatten (bool) – whether to flatten input tensor into 2D matrix. Default: False.
  • keepdim (bool) – whether to keep the output dimension when out_features is 1.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.Softmax(input_shape, out_features, dim=1, weights_init=None, bias_init=None, **kwargs)[source]

A special case of FC with softmax activation function.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • dim (int) – dimension to apply softmax. Default: 1.
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • kwargs – extra keyword arguments to pass to activation.

Extra Layers

class neuralnet_pytorch.layers.Activation(activation='relu', input_shape=None, **kwargs)[source]

Applies a non-linear function to the incoming input.

Parameters:
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • input_shape – shape of the input tensor. Can be None.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.ConvNormAct(input_shape, out_channels, kernel_size, stride=1, padding='half', dilation=1, groups=1, bias=True, activation='relu', weights_init=None, bias_init=None, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, no_scale=False, norm_layer='bn', **kwargs)[source]

Fuses convolution, normalization and activation together.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum (float) – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', 'ln' or a callable. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.DepthwiseSepConv2D(input_shape, out_channels, kernel_size, depth_mul=1, stride=1, padding='half', dilation=1, bias=True, activation=None, **kwargs)[source]

Performs depthwise separable convolution in image processing.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size (int) – size of the convolving kernel.
  • depth_mul – depth multiplier for intermediate result of depthwise convolution
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.FCNormAct(input_shape, out_features, bias=True, activation=None, weights_init=None, bias_init=None, flatten=False, keepdim=True, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True, no_scale=False, norm_layer='bn', **kwargs)[source]

Fuses fully connected, normalization and activation together.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • bias (bool) – if set to False, the layer will not learn an additive bias. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • flatten (bool) – whether to flatten input tensor into 2D matrix. Default: False.
  • keepdim (bool) – whether to keep the output dimension when out_features is 1.
  • eps – a value added to the denominator for numerical stability. Default: 1e-5.
  • momentum (float) – the value used for the running_mean and running_var computation. Can be set to None for cumulative moving average (i.e. simple average). Default: 0.1.
  • affine – a boolean value that when set to True, this module has learnable affine parameters. Default: True.
  • track_running_stats – a boolean value that when set to True, this module tracks the running mean and variance, and when set to False, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: True.
  • no_scale (bool) – whether to use a trainable scale parameter. Default: True.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', and 'ln', or a callable function. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation and norm_layer.
class neuralnet_pytorch.layers.ResNetBasicBlock(input_shape, out_channels, kernel_size=3, stride=1, padding='half', dilation=1, activation='relu', base_width=64, downsample=None, groups=1, block=None, weights_init=None, norm_layer='bn', **kwargs)[source]

A basic block to build ResNet (https://arxiv.org/abs/1512.03385).

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • downsample – a module to process the residual branch when output shape is different from input shape. If None, a simple ConvNormAct is used.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • block – a function to construct the main branch of the block. If None, a simple block as described in the paper is used.
  • weights_init – a kernel initialization method from torch.nn.init.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', and 'ln'. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation.
expansion

expansion coefficients of the number of output channels. Default: 1.

Type:int
class neuralnet_pytorch.layers.ResNetBottleneckBlock(input_shape, out_channels, kernel_size=3, stride=1, padding='half', dilation=1, activation='relu', base_width=64, downsample=None, groups=1, block=None, weights_init=None, norm_layer='bn', **kwargs)[source]
A bottleneck block to build ResNet (https://arxiv.org/abs/1512.03385).
Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • downsample – a module to process the residual branch when output shape is different from input shape. If None, a simple ConvNormAct is used.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • block – a function to construct the main branch of the block. If None, a simple block as described in the paper is used.
  • weights_init – a kernel initialization method from torch.nn.init.
  • norm_layer – normalization method to be used. Choices are 'bn', 'in', and 'ln'. Default: 'bn'.
  • kwargs – extra keyword arguments to pass to activation.
expansion

expansion coefficients of the number of output channels. Default: 4.

Type:int
class neuralnet_pytorch.layers.StackingConv(input_shape, out_channels, kernel_size, num_layers, stride=1, padding='half', dilation=1, bias=True, activation='relu', weights_init=None, bias_init=None, norm_method=None, groups=1, **kwargs)[source]

Stacks multiple convolution layers together.

Parameters:
  • input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
  • out_channels (int) – number of channels produced by the convolution.
  • kernel_size – size of the convolving kernel.
  • num_layer (int) – number of convolutional layers.
  • stride – stride of the convolution. Default: 1.
  • padding – zero-padding added to both sides of the input. Besides tuple/list/integer, it accepts str such as 'half' and 'valid', which is similar the Theano, and 'ref' and 'rep', which are common padding schemes. Default: 'half'.
  • dilation – spacing between kernel elements. Default: 1.
  • bias (bool) – if True, adds a learnable bias to the output. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • norm_method – normalization method to be used. Choices are 'bn', 'in', and 'ln'. Default: 'bn'.
  • groups (int) – number of blocked connections from input channels to output channels. Default: 1.
  • kwargs – extra keyword arguments to pass to activation.

Graph Learning Layers

class neuralnet_pytorch.layers.GraphConv(input_shape, out_features, bias=True, activation=None, **kwargs)[source]

Performs graph convolution as described in https://arxiv.org/abs/1609.02907. Adapted from https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py.

Parameters:
  • input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.BatchGraphConv(input_shape, out_features, bias=True, activation=None, **kwargs)[source]

Performs graph convolution as described in https://arxiv.org/abs/1609.02907 on a batch of graphs. Adapted from https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py.

Parameters:
  • input_shape – shape of the input tensor. The first dim should be batch size. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • kwargs – extra keyword arguments to pass to activation.
class neuralnet_pytorch.layers.GraphXConv(input_shape, out_features, out_instances=None, rank=None, bias=True, activation=None, weights_init=None, bias_init=None, **kwargs)[source]

Performs GraphX Convolution as described here. Disclaimer: I am the first author of the paper.

Parameters:
  • input_shape – shape of the input tensor. The first dim should be batch size. If an integer is passed, it is treated as the size of each input sample.
  • out_features (int) – size of each output sample.
  • out_instances (int) – resolution of the output point clouds. If not specified, output will have the same resolution as input. Default: None.
  • rank – if specified and smaller than num_out_points, the mixing matrix will be broken into a multiplication of two matrices of sizes (num_out_points, rank) and (rank, input_shape[1]).
  • bias – whether to use bias. Default: True.
  • activation – non-linear function to activate the linear result. It accepts any callable function as well as a recognizable str. A list of possible str is in function().
  • weights_init – a kernel initialization method from torch.nn.init.
  • bias_init – a bias initialization method from torch.nn.init.
  • kwargs – extra keyword arguments to pass to activation.

Multi-module Layers

class neuralnet_pytorch.layers.Sum(*modules_or_tensors)[source]

Sums the outputs of multiple modules given an input tensor. A subclass of MultiSingleInputModule.

See also

MultiSingleInputModule MultiMultiInputModule SequentialSum ConcurrentSum Cat SequentialCat ConcurrentCat

class neuralnet_pytorch.layers.ConcurrentSum(*modules_or_tensors)[source]

Sums the outputs of multiple modules given input tensors. A subclass of MultiMultiInputModule.

See also

MultiSingleInputModule MultiMultiInputModule Sum SequentialSum Cat SequentialCat ConcurrentCat

class neuralnet_pytorch.layers.SequentialSum(*modules)[source]

Sums the intermediate outputs of multiple sequential modules given an input tensor. A subclass of Sum.

See also

MultiSingleInputModule MultiMultiInputModule Sum ConcurrentSum Cat SequentialCat ConcurrentCat