deel.lip.layers module

This module extends original keras layers, in order to add k lipschitz constraint via reparametrization. Currently, are implemented:

• Dense layer:

  as SpectralDense (and as FrobeniusDense when the layer has a single output)

• Conv2D layer:

  as SpectralConv2D (and as FrobeniusConv2D when the layer has a single output)

• AveragePooling:

  as ScaledAveragePooling

• GlobalAveragePooling2D:

  as ScaledGlobalAveragePooling2D

By default the layers are 1 Lipschitz almost everywhere, which is efficient for wasserstein distance estimation. However for other problems (such as adversarial robustness) the user may want to use layers that are at most 1 lipschitz, this can be done by setting the param niter_bjorck=0.

class deel.lip.layers.Condensable

Bases: abc.ABC

Some Layers don’t optimize directly the kernel, this means that the kernel stored in the layer is not the kernel used to make predictions (called W_bar), to address this, these layers can implement the condense() function that make self.kernel equal to W_bar.

This operation also allow the turn the lipschitz layer to it keras equivalent ie. The Dense layer that have the same predictions as the trained SpectralDense.

abstract condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns

None

abstract vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns

self as super type

class deel.lip.layers.FrobeniusConv2D(*args, **kwargs)

Bases: tensorflow.python.keras.layers.convolutional.Conv2D, deel.lip.layers.LipschitzLayer, deel.lip.layers.Condensable

Same as SpectralConv2D but in the case of a single output.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x)

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns

self as super type

class deel.lip.layers.FrobeniusDense(*args, **kwargs)

Bases: tensorflow.python.keras.layers.core.Dense, deel.lip.layers.LipschitzLayer, deel.lip.layers.Condensable

Same a SpectralDense, but in the case of a single output.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x)

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns

self as super type

class deel.lip.layers.LipschitzLayer

Bases: abc.ABC

This class allow to set lipschitz factor of a layer. Lipschitz layer must inherit this class to allow user to set the lipschitz factor.

Warning

This class only regroup useful functions when developing new Lipschitz layers. But it does not ensure any property about the layer. This means that inheriting from this class won’t ensure anything about the lipschitz constant.

coef_lip = None

define correction coefficient (ie. Lipschitz bound ) of the layer ( multiply the output of the layer by this constant )

k_coef_lip = 1.0

variable used to store the lipschitz factor

set_klip_factor(klip_factor)

Allow to set the lipschitz factor of a layer.

Parameters

klip_factor – the Lipschitz factor the user want to ensure.

Returns

None

class deel.lip.layers.ScaledAveragePooling2D(*args, **kwargs)

Bases: tensorflow.python.keras.layers.pooling.AveragePooling2D, deel.lip.layers.LipschitzLayer

Average pooling operation for spatial data, but with a lipschitz bound.

Parameters

• pool_size – integer or tuple of 2 integers, factors by which to downscale (vertical, horizontal). (2, 2) will halve the input in both spatial dimension. If only one integer is specified, the same window length will be used for both dimensions.

• strides – Integer, tuple of 2 integers, or None. Strides values. If None, it will default to pool_size.

• padding – One of “valid” or “same” (case-insensitive).

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• k_coef_lip – the lipschitz factor to ensure

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, pooled_rows, pooled_cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, pooled_rows, pooled_cols).

This documentation reuse the body of the original keras.layers.AveragePooling2D doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=None)

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

class deel.lip.layers.ScaledGlobalAveragePooling2D(*args, **kwargs)

Bases: tensorflow.python.keras.layers.pooling.GlobalAveragePooling2D, deel.lip.layers.LipschitzLayer

Global average pooling operation for spatial data with Lipschitz bound.

Parameters

data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape: 2D tensor with shape (batch_size, channels).

This documentation reuse the body of the original keras.layers.GlobalAveragePooling doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=None)

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

class deel.lip.layers.ScaledL2NormPooling2D(*args, **kwargs)

Bases: tensorflow.python.keras.layers.pooling.AveragePooling2D, deel.lip.layers.LipschitzLayer

Average pooling operation for spatial data, with a lipschitz bound. This pooling operation is norm preserving (aka gradient=1 almost everywhere).

[1]Y.-L.Boureau, J.Ponce, et Y.LeCun, « A Theoretical Analysis of Feature Pooling in Visual Recognition »,p.8.

Parameters

• pool_size – integer or tuple of 2 integers, factors by which to downscale (vertical, horizontal). (2, 2) will halve the input in both spatial dimension. If only one integer is specified, the same window length will be used for both dimensions.

• strides – Integer, tuple of 2 integers, or None. Strides values. If None, it will default to pool_size.

• padding – One of “valid” or “same” (case-insensitive).

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• k_coef_lip – the lipschitz factor to ensure

• eps_grad_sqrt – Epsilon value to avoid numerical instability due to non-defined gradient at 0 in the sqrt function

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, pooled_rows, pooled_cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, pooled_rows, pooled_cols).

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=None)

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

class deel.lip.layers.SpectralConv2D(*args, **kwargs)

Bases: tensorflow.python.keras.layers.convolutional.Conv2D, deel.lip.layers.LipschitzLayer, deel.lip.layers.Condensable

This class is a Conv2D Layer constrained such that all singular of it’s kernel are 1. The computation based on BjorckNormalizer algorithm. As this is not enough to ensure 1 Lipschitzity a coertive coefficient is applied on the output. The computation is done in three steps:

1. reduce the largest singular value to 1, using iterated power method.

2. increase other singular values to 1, using BjorckNormalizer algorithm.

3. divide the output by the Lipschitz bound to ensure k Lipschitzity.

Parameters

• filters – Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

• kernel_size – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.

• strides – An integer or tuple/list of 2 integers, specifying the strides of the convolution along the height and width. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.

• padding – one of “valid” or “same” (case-insensitive).

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• dilation_rate – an integer or tuple/list of 2 integers, specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.

• activation – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

• use_bias – Boolean, whether the layer uses a bias vector.

• kernel_initializer – Initializer for the kernel weights matrix.

• bias_initializer – Initializer for the bias vector.

• kernel_regularizer – Regularizer function applied to the kernel weights matrix.

• bias_regularizer – Regularizer function applied to the bias vector.

• activity_regularizer – Regularizer function applied to the output of the layer (its “activation”)..

• kernel_constraint – Constraint function applied to the kernel matrix.

• bias_constraint – Constraint function applied to the bias vector.

• k_coef_lip – lipschitz constant to ensure

• niter_spectral – number of iteration to find the maximum singular value.

• niter_bjorck – number of iteration with BjorckNormalizer algorithm.

This documentation reuse the body of the original keras.layers.Conv2D doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=None)

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns

self as super type

class deel.lip.layers.SpectralDense(*args, **kwargs)

Bases: tensorflow.python.keras.layers.core.Dense, deel.lip.layers.LipschitzLayer, deel.lip.layers.Condensable

This class is a Dense Layer constrained such that all singular of it’s kernel are 1. The computation based on BjorckNormalizer algorithm. The computation is done in two steps:

1. reduce the larget singular value to 1, using iterated power method.

2. increase other singular values to 1, using BjorckNormalizer algorithm.

Parameters

• units – Positive integer, dimensionality of the output space.

• activation – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

• use_bias – Boolean, whether the layer uses a bias vector.

• kernel_initializer – Initializer for the kernel weights matrix.

• bias_initializer – Initializer for the bias vector.

• kernel_regularizer – Regularizer function applied to the kernel weights matrix.

• bias_regularizer – Regularizer function applied to the bias vector.

• activity_regularizer – Regularizer function applied to the output of the layer (its “activation”)..

• kernel_constraint – Constraint function applied to the kernel weights matrix.

• bias_constraint – Constraint function applied to the bias vector.

• k_coef_lip – lipschitz constant to ensure

• niter_spectral – number of iteration to find the maximum singular value.

• niter_bjorck – number of iteration with BjorckNormalizer algorithm.

Input shape:

N-D tensor with shape: (batch_size, …, input_dim). The most common situation would be a 2D input with shape (batch_size, input_dim).

Output shape:

N-D tensor with shape: (batch_size, …, units). For instance, for a 2D input with shape (batch_size, input_dim), the output would have shape (batch_size, units).

This documentation reuse the body of the original keras.layers.Dense doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call.

This is typically used to create the weights of Layer subclasses.

Parameters

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=None)

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Returns

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns

self as super type