layers
– Basic Layers¶
This section describes all the backbone modules of Neuralnetpytorch.
Contents
Abstract Layers¶
Attributes¶
The following classes equip the plain torch
modules with more bells and whistles.
Also, some features are deeply integrated into Neuralnetpytorch,
which enables faster and more convenient training and testing of your neural networks.

class
neuralnet_pytorch.layers.abstract.
_LayerMethod
[source]¶ This mixin class contains various attributes to extend
torch
modules.
load
(param_file, eval=True)[source]¶ Load the numpy.ndarray weights from file.
Parameters:  param_file – path to the weight file.
 eval – whether to use evaluation mode or not.

params
¶ Return a tuple of all the parameters in the module.

regularizable
¶ Returns a tuple of parameters to be regularized.

reset_parameters
()[source]¶ This overloads the
torch.Module.reset_parameters()
of the module. Used for custom weight initialization.

save
(param_file)[source]¶ Save the weights of the model in
numpy.nrdarray
format.Parameters: param_file – path to the weight file.

trainable
¶ Return a tuple of all parameters with
requires_grad
set to True.


class
neuralnet_pytorch.layers.
MultiSingleInputModule
(*modules_or_tensors)[source]¶ This is an abstract class. This class computes the results of multiple modules given an input tensor, then fuses the results.
Parameters: modules_or_tensors – a list of modules or tensors whose results are fused together. 
input_shape
¶ a list of input shapes of the incoming modules and tensors.


class
neuralnet_pytorch.layers.
MultiMultiInputModule
(*modules_or_tensors)[source]¶ Similar to
MultiSingleInputModule
, but each module has its own input tensor.
Extended Pytorch Abstract Layers¶

class
neuralnet_pytorch.layers.
Module
(input_shape=None)[source]¶ Similar to
torch.nn.Module
, but extended by_LayerMethod
. All the usages in native Pytorch are preserved.Parameters: input_shape – shape of the tensor to be input to the modules. Can be a list, tuple, nested list/tuple or an integer.

class
neuralnet_pytorch.layers.
Sequential
(*args, input_shape=None)[source]¶ Similar to
torch.nn.Sequential
, but extended by_LayerMethod
. All the usages in native Pytorch are preserved.Parameters:  args – a list of modules as in
torch.nn.Sequential
.  input_shape – shape of the input tensor. If
None
, the functionality is the same astorch.nn.Sequential
.
 args – a list of modules as in
Quickanddirty Layers¶

class
neuralnet_pytorch.layers.
Lambda
(func, input_shape=None, output_shape=None, **kwargs)[source]¶ Wraps a function as a
Module
.Parameters:  func – a callable function.
 input_shape – shape of the input tensor.
 output_shape – shape of the output tensor.
If
None
, the output shape is calculated by performing a forward pass.  kwargs – keyword arguments required by func.
Examples
You can easily wrap a
torch
functionimport torch as T import neuralnet_pytorch as nnt a, b = T.rand(3, 1), T.rand(3, 2) cat = nnt.Lambda(T.cat, dim=1) c = cat((a, b)) print(c.shape)
Also, it works for any selfdefined function as well
import neuralnet_pytorch as nnt def foo(x, y): return x + y a = T.rand(3, 3) print(a) foo_sum = nnt.Lambda(foo, y=1.) res = foo_sum(a) print(res)
Common Layers¶
Extended Pytorch Common Layers¶

class
neuralnet_pytorch.layers.
Conv2d
(input_shape, out_channels, kernel_size, stride=1, padding='half', dilation=1, groups=1, bias=True, activation=None, weights_init=None, bias_init=None, **kwargs)[source]¶ Extends
torch.nn.Conv2d
with_LayerMethod
.Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size – size of the convolving kernel.
 stride – stride of the convolution. Default: 1.
 padding – zeropadding added to both sides of the input.
Besides tuple/list/integer, it accepts
str
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
, which are common padding schemes. Default:'half'
.  dilation – spacing between kernel elements. Default: 1.
 groups (int) – number of blocked connections from input channels to output channels. Default: 1.
 bias (bool) – if
True
, adds a learnable bias to the output. Default:True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
ConvTranspose2d
(input_shape, out_channels, kernel_size, stride=1, padding='half', output_padding=0, groups=1, bias=True, dilation=1, activation='linear', weights_init=None, bias_init=None, output_size=None, **kwargs)[source]¶ Extends
torch.nn.ConvTranspose2d
with_LayerMethod
.Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size – size of the convolving kernel.
 stride – stride of the convolution. Default: 1.
 padding –
dilation * (kernel_size  1)  padding
zeropadding will be added to both sides of each dimension in the input. Besides tuple/list/integer, it acceptsstr
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
which are common padding schemes. Default:'half'
.  output_padding – additional size added to one side of each dimension in the output shape. Default: 0
 groups (int) – number of blocked connections from input channels to output channels. Default: 1.
 bias (bool) – if
True
, adds a learnable bias to the output. Default:True
.  dilation – spacing between kernel elements. Default: 1.
 activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  output_size – size of the output tensor. If
None
, the shape is automatically determined.  kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
FC
(input_shape, out_features, bias=True, activation=None, weights_init=None, bias_init=None, flatten=False, keepdim=True, **kwargs)[source]¶ AKA fully connected layer in deep learning literature. This class extends
torch.nn.Linear
by_LayerMethod
.Parameters:  input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
 out_features (int) – size of each output sample.
 bias (bool) – if set to
False
, the layer will not learn an additive bias. Default:True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  flatten (bool) – whether to flatten input tensor into 2D matrix. Default:
False
.  keepdim (bool) – whether to keep the output dimension when out_features is 1.
 kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
Softmax
(input_shape, out_features, dim=1, weights_init=None, bias_init=None, **kwargs)[source]¶ A special case of
FC
with softmax activation function.Parameters:  input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
 out_features (int) – size of each output sample.
 dim (int) – dimension to apply softmax. Default: 1.
 weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  kwargs – extra keyword arguments to pass to activation.
Extra Layers¶

class
neuralnet_pytorch.layers.
Activation
(activation='relu', input_shape=None, **kwargs)[source]¶ Applies a nonlinear function to the incoming input.
Parameters:  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  input_shape – shape of the input tensor. Can be
None
.  kwargs – extra keyword arguments to pass to activation.
 activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable

class
neuralnet_pytorch.layers.
ConvNormAct
(input_shape, out_channels, kernel_size, stride=1, padding='half', dilation=1, groups=1, bias=True, activation='relu', weights_init=None, bias_init=None, eps=1e05, momentum=0.1, affine=True, track_running_stats=True, no_scale=False, norm_layer='bn', **kwargs)[source]¶ Fuses convolution, normalization and activation together.
Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size – size of the convolving kernel.
 stride – stride of the convolution. Default: 1.
 padding – zeropadding added to both sides of the input.
Besides tuple/list/integer, it accepts
str
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
, which are common padding schemes. Default:'half'
.  dilation – spacing between kernel elements. Default: 1.
 groups (int) – number of blocked connections from input channels to output channels. Default: 1.
 bias (bool) – if
True
, adds a learnable bias to the output. Default:True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  eps – a value added to the denominator for numerical stability. Default: 1e5.
 momentum (float) – the value used for the running_mean and running_var
computation. Can be set to
None
for cumulative moving average (i.e. simple average). Default: 0.1.  affine – a boolean value that when set to
True
, this module has learnable affine parameters. Default:True
.  track_running_stats – a boolean value that when set to
True
, this module tracks the running mean and variance, and when set toFalse
, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default:True
.  no_scale (bool) – whether to use a trainable scale parameter. Default:
True
.  norm_layer – normalization method to be used.
Choices are
'bn'
,'in'
,'ln'
or a callable. Default:'bn'
.  kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
DepthwiseSepConv2D
(input_shape, out_channels, kernel_size, depth_mul=1, stride=1, padding='half', dilation=1, bias=True, activation=None, **kwargs)[source]¶ Performs depthwise separable convolution in image processing.
Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size (int) – size of the convolving kernel.
 depth_mul – depth multiplier for intermediate result of depthwise convolution
 padding – zeropadding added to both sides of the input.
Besides tuple/list/integer, it accepts
str
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
, which are common padding schemes. Default:'half'
.  dilation – spacing between kernel elements. Default: 1.
 bias (bool) – if
True
, adds a learnable bias to the output. Default:True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
FCNormAct
(input_shape, out_features, bias=True, activation=None, weights_init=None, bias_init=None, flatten=False, keepdim=True, eps=1e05, momentum=0.1, affine=True, track_running_stats=True, no_scale=False, norm_layer='bn', **kwargs)[source]¶ Fuses fully connected, normalization and activation together.
Parameters:  input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
 out_features (int) – size of each output sample.
 bias (bool) – if set to
False
, the layer will not learn an additive bias. Default:True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  flatten (bool) – whether to flatten input tensor into 2D matrix. Default:
False
.  keepdim (bool) – whether to keep the output dimension when out_features is 1.
 eps – a value added to the denominator for numerical stability. Default: 1e5.
 momentum (float) – the value used for the running_mean and running_var
computation. Can be set to
None
for cumulative moving average (i.e. simple average). Default: 0.1.  affine – a boolean value that when set to
True
, this module has learnable affine parameters. Default:True
.  track_running_stats – a boolean value that when set to
True
, this module tracks the running mean and variance, and when set toFalse
, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default:True
.  no_scale (bool) – whether to use a trainable scale parameter. Default:
True
.  norm_layer – normalization method to be used.
Choices are
'bn'
,'in'
, and'ln'
, or a callable function. Default:'bn'
.  kwargs – extra keyword arguments to pass to activation and norm_layer.

class
neuralnet_pytorch.layers.
ResNetBasicBlock
(input_shape, out_channels, kernel_size=3, stride=1, padding='half', dilation=1, activation='relu', base_width=64, downsample=None, groups=1, block=None, weights_init=None, norm_layer='bn', **kwargs)[source]¶ A basic block to build ResNet (https://arxiv.org/abs/1512.03385).
Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size – size of the convolving kernel.
 stride – stride of the convolution. Default: 1.
 padding – zeropadding added to both sides of the input.
Besides tuple/list/integer, it accepts
str
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
, which are common padding schemes. Default:'half'
.  dilation – spacing between kernel elements. Default: 1.
 activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  downsample – a module to process the residual branch when output shape is different from input shape.
If
None
, a simpleConvNormAct
is used.  groups (int) – number of blocked connections from input channels to output channels. Default: 1.
 block – a function to construct the main branch of the block.
If
None
, a simple block as described in the paper is used.  weights_init – a kernel initialization method from
torch.nn.init
.  norm_layer – normalization method to be used. Choices are
'bn'
,'in'
, and'ln'
. Default:'bn'
.  kwargs – extra keyword arguments to pass to activation.

expansion
¶ expansion coefficients of the number of output channels. Default: 1.
Type: int

class
neuralnet_pytorch.layers.
ResNetBottleneckBlock
(input_shape, out_channels, kernel_size=3, stride=1, padding='half', dilation=1, activation='relu', base_width=64, downsample=None, groups=1, block=None, weights_init=None, norm_layer='bn', **kwargs)[source]¶  A bottleneck block to build ResNet (https://arxiv.org/abs/1512.03385).
Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size – size of the convolving kernel.
 stride – stride of the convolution. Default: 1.
 padding – zeropadding added to both sides of the input.
Besides tuple/list/integer, it accepts
str
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
, which are common padding schemes. Default:'half'
.  dilation – spacing between kernel elements. Default: 1.
 activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  downsample – a module to process the residual branch when output shape is different from input shape.
If
None
, a simpleConvNormAct
is used.  groups (int) – number of blocked connections from input channels to output channels. Default: 1.
 block – a function to construct the main branch of the block.
If
None
, a simple block as described in the paper is used.  weights_init – a kernel initialization method from
torch.nn.init
.  norm_layer – normalization method to be used. Choices are
'bn'
,'in'
, and'ln'
. Default:'bn'
.  kwargs – extra keyword arguments to pass to activation.

expansion
¶ expansion coefficients of the number of output channels. Default: 4.
Type: int

class
neuralnet_pytorch.layers.
StackingConv
(input_shape, out_channels, kernel_size, num_layers, stride=1, padding='half', dilation=1, bias=True, activation='relu', weights_init=None, bias_init=None, norm_method=None, groups=1, **kwargs)[source]¶ Stacks multiple convolution layers together.
Parameters:  input_shape – shape of the 4D input image. If a single integer is passed, it is treated as the number of input channels and other sizes are unknown.
 out_channels (int) – number of channels produced by the convolution.
 kernel_size – size of the convolving kernel.
 num_layer (int) – number of convolutional layers.
 stride – stride of the convolution. Default: 1.
 padding – zeropadding added to both sides of the input.
Besides tuple/list/integer, it accepts
str
such as'half'
and'valid'
, which is similar the Theano, and'ref'
and'rep'
, which are common padding schemes. Default:'half'
.  dilation – spacing between kernel elements. Default: 1.
 bias (bool) – if
True
, adds a learnable bias to the output. Default:True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  norm_method – normalization method to be used. Choices are
'bn'
,'in'
, and'ln'
. Default:'bn'
.  groups (int) – number of blocked connections from input channels to output channels. Default: 1.
 kwargs – extra keyword arguments to pass to activation.
Graph Learning Layers¶

class
neuralnet_pytorch.layers.
GraphConv
(input_shape, out_features, bias=True, activation=None, **kwargs)[source]¶ Performs graph convolution as described in https://arxiv.org/abs/1609.02907. Adapted from https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py.
Parameters:  input_shape – shape of the input tensor. If an integer is passed, it is treated as the size of each input sample.
 out_features (int) – size of each output sample.
 activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
BatchGraphConv
(input_shape, out_features, bias=True, activation=None, **kwargs)[source]¶ Performs graph convolution as described in https://arxiv.org/abs/1609.02907 on a batch of graphs. Adapted from https://github.com/tkipf/pygcn/blob/master/pygcn/layers.py.
Parameters:  input_shape – shape of the input tensor. The first dim should be batch size. If an integer is passed, it is treated as the size of each input sample.
 out_features (int) – size of each output sample.
 activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  kwargs – extra keyword arguments to pass to activation.

class
neuralnet_pytorch.layers.
GraphXConv
(input_shape, out_features, out_instances=None, rank=None, bias=True, activation=None, weights_init=None, bias_init=None, **kwargs)[source]¶ Performs GraphX Convolution as described here. Disclaimer: I am the first author of the paper.
Parameters:  input_shape – shape of the input tensor. The first dim should be batch size. If an integer is passed, it is treated as the size of each input sample.
 out_features (int) – size of each output sample.
 out_instances (int) – resolution of the output point clouds.
If not specified, output will have the same resolution as input.
Default:
None
.  rank – if specified and smaller than num_out_points, the mixing matrix will
be broken into a multiplication of two matrices of sizes
(num_out_points, rank)
and(rank, input_shape[1])
.  bias – whether to use bias.
Default:
True
.  activation – nonlinear function to activate the linear result.
It accepts any callable function
as well as a recognizable
str
. A list of possiblestr
is infunction()
.  weights_init – a kernel initialization method from
torch.nn.init
.  bias_init – a bias initialization method from
torch.nn.init
.  kwargs – extra keyword arguments to pass to activation.
Multimodule Layers¶

class
neuralnet_pytorch.layers.
Sum
(*modules_or_tensors)[source]¶ Sums the outputs of multiple modules given an input tensor. A subclass of
MultiSingleInputModule
.

class
neuralnet_pytorch.layers.
ConcurrentSum
(*modules_or_tensors)[source]¶ Sums the outputs of multiple modules given input tensors. A subclass of
MultiMultiInputModule
.