deel.lip.layers module

This module extends original keras layers, in order to add k lipschitz constraint via reparametrization. Currently, are implemented:

• Dense layer:

  as SpectralDense (and as FrobeniusDense when the layer has a single output)

• Conv2D layer:

  as SpectralConv2D (and as FrobeniusConv2D when the layer has a single output)

• AveragePooling:

  as ScaledAveragePooling

• GlobalAveragePooling2D:

  as ScaledGlobalAveragePooling2D

By default the layers are 1 Lipschitz almost everywhere, which is efficient for wasserstein distance estimation. However for other problems (such as adversarial robustness) the user may want to use layers that are at most 1 lipschitz, this can be done by setting the param eps_bjorck=None.

class deel.lip.layers.Condensable

Bases: ABC

Some Layers don’t optimize directly the kernel, this means that the kernel stored in the layer is not the kernel used to make predictions (called W_bar), to address this, these layers can implement the condense() function that make self.kernel equal to W_bar.

This operation also allow the turn the lipschitz layer to it keras equivalent ie. The Dense layer that have the same predictions as the trained SpectralDense.

abstract condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns:

None

abstract vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns:

self as super type

class deel.lip.layers.FrobeniusConv2D(*args, **kwargs)

Bases: Conv2D, LipschitzLayer, Condensable

Same as SpectralConv2D but in the case of a single output.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns:

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns:

self as super type

class deel.lip.layers.FrobeniusDense(*args, **kwargs)

Bases: Dense, LipschitzLayer, Condensable

Identical and faster than a SpectralDense in the case of a single output. In the multi-neurons setting, this layer can be used: - as a classical Frobenius Dense normalization (disjoint_neurons=False) - as a stacking of 1 lipschitz independent neurons (each output is 1-lipschitz, but the no orthogonality is enforced between outputs ) (disjoint_neurons=True).

Warning :

default is disjoint_neurons = True

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns:

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns:

self as super type

class deel.lip.layers.InvertibleDownSampling(*args, **kwargs)

Bases: Layer

This pooling layer perform a reshape on the spacial dimensions: it take a (bs, h, w, c) ( if channels_last ) and reshape it to a (bs, h/p_h, w/p_w, c*p_w*p_h ), where p_w and p_h are the shape of the pool. By doing this the image size is reduced while the number of channels is increased.

References

Anil et al. https://arxiv.org/abs/1911.00937

Note

The image shape must be divisible by the pool shape.

Parameters:

• pool_size – tuple describing the pool shape

• data_format – can either be channels_last or channels_first

• name – name of the layer

• dtype – dtype of the layer

• **kwargs – params passed to the Layers constructor

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(inputs, **kwargs)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

class deel.lip.layers.InvertibleUpSampling(*args, **kwargs)

Bases: Layer

This Layer is the inverse of the InvertibleDownSampling layer. It take a (bs, h, w, c) ( if channels_last ) and reshape it to a (bs, h/p_h, w/p_w, c*p_w*p_h ), where p_w and p_h are the shape of the pool. By doing this the image size is reduced while the number of channels is increased.

References

Anil et al. https://arxiv.org/abs/1911.00937

Note

The input number of channels must be divisible by the p_w*p_h.

Parameters:

• pool_size – tuple describing the pool shape (p_h, p_w)

• data_format – can either be channels_last or channels_first

• name – name of the layer

• dtype – dtype of the layer

• **kwargs – params passed to the Layers constructor

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(inputs, **kwargs)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

class deel.lip.layers.LipschitzLayer

Bases: ABC

This class allow to set lipschitz factor of a layer. Lipschitz layer must inherit this class to allow user to set the lipschitz factor.

Warning

This class only regroup useful functions when developing new Lipschitz layers. But it does not ensure any property about the layer. This means that inheriting from this class won’t ensure anything about the lipschitz constant.

coef_lip = None

define correction coefficient (ie. Lipschitz bound ) of the layer ( multiply the output of the layer by this constant )

k_coef_lip = 1.0

variable used to store the lipschitz factor

set_klip_factor(klip_factor)

Allow to set the lipschitz factor of a layer.

Parameters:

klip_factor – the Lipschitz factor the user want to ensure.

Returns:

None

class deel.lip.layers.ScaledAveragePooling2D(*args, **kwargs)

Bases: AveragePooling2D, LipschitzLayer

Average pooling operation for spatial data, but with a lipschitz bound.

Parameters:

• pool_size – integer or tuple of 2 integers, factors by which to downscale (vertical, horizontal). (2, 2) will halve the input in both spatial dimension. If only one integer is specified, the same window length will be used for both dimensions.

• strides – Integer, tuple of 2 integers, or None. Strides values. If None, it will default to pool_size.

• padding – One of “valid” or “same” (case-insensitive).

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• k_coef_lip – the lipschitz factor to ensure

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, pooled_rows, pooled_cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, pooled_rows, pooled_cols).

This documentation reuse the body of the original keras.layers.AveragePooling2D doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

class deel.lip.layers.ScaledGlobalAveragePooling2D(*args, **kwargs)

Bases: GlobalAveragePooling2D, LipschitzLayer

Global average pooling operation for spatial data with Lipschitz bound.

Parameters:

data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape: 2D tensor with shape (batch_size, channels).

This documentation reuse the body of the original keras.layers.GlobalAveragePooling doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

class deel.lip.layers.ScaledGlobalL2NormPooling2D(*args, **kwargs)

Bases: GlobalAveragePooling2D, LipschitzLayer

Average pooling operation for spatial data, with a lipschitz bound. This pooling operation is norm preserving (aka gradient=1 almost everywhere).

[1]Y.-L.Boureau, J.Ponce, et Y.LeCun, « A Theoretical Analysis of Feature Pooling in Visual Recognition »,p.8.

Parameters:

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• k_coef_lip – the lipschitz factor to ensure

• eps_grad_sqrt – Epsilon value to avoid numerical instability due to non-defined gradient at 0 in the sqrt function

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, pooled_cols).

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

class deel.lip.layers.ScaledL2NormPooling2D(*args, **kwargs)

Bases: AveragePooling2D, LipschitzLayer

Average pooling operation for spatial data, with a lipschitz bound. This pooling operation is norm preserving (aka gradient=1 almost everywhere).

[1]Y.-L.Boureau, J.Ponce, et Y.LeCun, « A Theoretical Analysis of Feature Pooling in Visual Recognition »,p.8.

Parameters:

• pool_size – integer or tuple of 2 integers, factors by which to downscale (vertical, horizontal). (2, 2) will halve the input in both spatial dimension. If only one integer is specified, the same window length will be used for both dimensions.

• strides – Integer, tuple of 2 integers, or None. Strides values. If None, it will default to pool_size.

• padding – One of “valid” or “same” (case-insensitive).

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• k_coef_lip – the lipschitz factor to ensure

• eps_grad_sqrt – Epsilon value to avoid numerical instability due to non-defined gradient at 0 in the sqrt function

Input shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, rows, cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, rows, cols).

Output shape:

• If data_format=’channels_last’:

  4D tensor with shape (batch_size, pooled_rows, pooled_cols, channels).

• If data_format=’channels_first’:

  4D tensor with shape (batch_size, channels, pooled_rows, pooled_cols).

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

class deel.lip.layers.SpectralConv2D(*args, **kwargs)

Bases: Conv2D, LipschitzLayer, Condensable

This class is a Conv2D Layer constrained such that all singular of it’s kernel are 1. The computation based on Bjorck algorithm. As this is not enough to ensure 1 Lipschitzity a coertive coefficient is applied on the output. The computation is done in three steps:

1. reduce the largest singular value to 1, using iterated power method.

2. increase other singular values to 1, using Bjorck algorithm.

3. divide the output by the Lipschitz bound to ensure k Lipschitzity.

Parameters:

• filters – Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

• kernel_size – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.

• strides – An integer or tuple/list of 2 integers, specifying the strides of the convolution along the height and width. Can be a single integer to specify the same value for all spatial dimensions. Specifying any stride value != 1 is incompatible with specifying any dilation_rate value != 1.

• padding – one of “valid” or “same” (case-insensitive).

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• dilation_rate – an integer or tuple/list of 2 integers, specifying the dilation rate to use for dilated convolution. Can be a single integer to specify the same value for all spatial dimensions. Currently, specifying any dilation_rate value != 1 is incompatible with specifying any stride value != 1.

• activation – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

• use_bias – Boolean, whether the layer uses a bias vector.

• kernel_initializer – Initializer for the kernel weights matrix.

• bias_initializer – Initializer for the bias vector.

• kernel_regularizer – Regularizer function applied to the kernel weights matrix.

• bias_regularizer – Regularizer function applied to the bias vector.

• activity_regularizer – Regularizer function applied to the output of the layer (its “activation”)..

• kernel_constraint – Constraint function applied to the kernel matrix.

• bias_constraint – Constraint function applied to the bias vector.

• k_coef_lip – lipschitz constant to ensure

• eps_spectral – stopping criterion for the iterative power algorithm.

• eps_bjorck – stopping criterion Bjorck algorithm.

• beta_bjorck – beta parameter in bjorck algorithm.

• maxiter_spectral – maximum number of iterations for the power iteration.

• maxiter_bjorck – maximum number of iterations for bjorck algorithm.

This documentation reuse the body of the original keras.layers.Conv2D doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns:

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns:

self as super type

class deel.lip.layers.SpectralConv2DTranspose(*args, **kwargs)

Bases: Conv2DTranspose, LipschitzLayer, Condensable

This class is a Conv2DTranspose layer constrained such that all singular values of its kernel are 1. The computation is based on Björck orthogonalization algorithm.

The computation is done in three steps: 1. reduce the largest singular value to 1, using iterated power method. 2. increase other singular values to 1, using Björck algorithm. 3. divide the output by the Lipschitz target K to ensure K-Lipschitzity.

This documentation reuses the body of the original tf.keras.layers.Conv2DTranspose doc.

Parameters:

• filters – Integer, the dimensionality of the output space (i.e. the number of output filters in the convolution).

• kernel_size – An integer or tuple/list of 2 integers, specifying the height and width of the 2D convolution window. Can be a single integer to specify the same value for all spatial dimensions.

• strides – An integer or tuple/list of 2 integers, specifying the strides of the convolution along the height and width. Can be a single integer to specify the same value for all spatial dimensions.

• padding – only “same” padding is supported in this Lipschitz layer (case-insensitive).

• output_padding – if set to None (default), the output shape is inferred. Only None value is supported in this Lipschitz layer.

• data_format – A string, one of channels_last (default) or channels_first. The ordering of the dimensions in the inputs. channels_last corresponds to inputs with shape (batch, height, width, channels) while channels_first corresponds to inputs with shape (batch, channels, height, width). It defaults to the image_data_format value found in your Keras config file at ~/.keras/keras.json. If you never set it, then it will be “channels_last”.

• dilation_rate – an integer, specifying the dilation rate for all spatial dimensions for dilated convolution. This Lipschitz layer does not support dilation rate != 1.

• activation – Activation function to use. If you don’t specify anything, no activation is applied (see keras.activations).

• use_bias – Boolean, whether the layer uses a bias vector.

• kernel_initializer – Initializer for the kernel weights matrix (see keras.initializers). Defaults to SpectralInitializer.

• bias_initializer – Initializer for the bias vector (see keras.initializers). Defaults to ‘zeros’.

• kernel_regularizer – Regularizer function applied to the kernel weights matrix (see keras.regularizers).

• bias_regularizer – Regularizer function applied to the bias vector (see keras.regularizers).

• activity_regularizer – Regularizer function applied to the output of the layer (its “activation”) (see keras.regularizers).

• kernel_constraint – Constraint function applied to the kernel matrix (see keras.constraints).

• bias_constraint – Constraint function applied to the bias vector (see keras.constraints).

• k_coef_lip – Lipschitz constant to ensure

• eps_spectral – stopping criterion for the iterative power algorithm.

• eps_bjorck – stopping criterion Björck algorithm.

• beta_bjorck – beta parameter in Björck algorithm.

• maxiter_spectral – maximum number of iterations for the power iteration.

• maxiter_bjorck – maximum number of iterations for bjorck algorithm.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(inputs, training=True)

This is where the layer’s logic lives.

The call() method may not create state (except in its first invocation, wrapping the creation of variables or other resources in tf.init_scope()). It is recommended to create state, including tf.Variable instances and nested Layer instances,

in __init__(), or in the build() method that is

called automatically before call() executes for the first time.

Parameters:

• inputs –

  Input tensor, or dict/list/tuple of input tensors. The first positional inputs argument is subject to special rules: - inputs must be explicitly passed. A layer cannot have zero

  arguments, and inputs cannot be provided via the default value of a keyword argument.

  • NumPy array or Python scalar values in inputs get cast as tensors.

  • Keras mask metadata is only collected from inputs.

  • Layers are built (build(input_shape) method) using shape info from inputs only.

  • input_spec compatibility is only checked against inputs.

  • Mixed precision input casting is only applied to inputs. If a layer has tensor arguments in *args or **kwargs, their casting behavior in mixed precision should be handled manually.

  • The SavedModel input specification is generated using inputs only.

  • Integration with various ecosystem packages like TFMOT, TFLite, TF.js, etc is only supported for inputs and not for tensors in positional and keyword arguments.

• *args – Additional positional arguments. May contain tensors, although this is not recommended, for the reasons above.

• **kwargs –

  Additional keyword arguments. May contain tensors, although this is not recommended, for the reasons above. The following optional keyword arguments are reserved: - training: Boolean scalar tensor of Python boolean indicating

  whether the call is meant for training or inference.

  • mask: Boolean input mask. If the layer’s call() method takes a mask argument, its default value will be set to the mask generated for inputs by the previous layer (if input did come from a layer that generated a corresponding mask, i.e. if it came from a Keras layer with masking support).

Returns:

A tensor or list/tuple of tensors.

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns:

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns:

self as super type

class deel.lip.layers.SpectralDense(*args, **kwargs)

Bases: Dense, LipschitzLayer, Condensable

This class is a Dense Layer constrained such that all singular of it’s kernel are 1. The computation based on Bjorck algorithm. The computation is done in two steps:

1. reduce the larget singular value to 1, using iterated power method.

2. increase other singular values to 1, using Bjorck algorithm.

Parameters:

• units – Positive integer, dimensionality of the output space.

• activation – Activation function to use. If you don’t specify anything, no activation is applied (ie. “linear” activation: a(x) = x).

• use_bias – Boolean, whether the layer uses a bias vector.

• kernel_initializer – Initializer for the kernel weights matrix.

• bias_initializer – Initializer for the bias vector.

• kernel_regularizer – Regularizer function applied to the kernel weights matrix.

• bias_regularizer – Regularizer function applied to the bias vector.

• activity_regularizer – Regularizer function applied to the output of the layer (its “activation”)..

• kernel_constraint – Constraint function applied to the kernel weights matrix.

• bias_constraint – Constraint function applied to the bias vector.

• k_coef_lip – lipschitz constant to ensure

• eps_spectral – stopping criterion for the iterative power algorithm.

• eps_bjorck – stopping criterion Bjorck algorithm.

• beta_bjorck – beta parameter in bjorck algorithm.

• maxiter_spectral – maximum number of iterations for the power iteration.

• maxiter_bjorck – maximum number of iterations for bjorck algorithm.

Input shape:

N-D tensor with shape: (batch_size, …, input_dim). The most common situation would be a 2D input with shape (batch_size, input_dim).

Output shape:

N-D tensor with shape: (batch_size, …, units). For instance, for a 2D input with shape (batch_size, input_dim), the output would have shape (batch_size, units).

This documentation reuse the body of the original keras.layers.Dense doc.

build(input_shape)

Creates the variables of the layer (optional, for subclass implementers).

This is a method that implementers of subclasses of Layer or Model can override if they need a state-creation step in-between layer instantiation and layer call. It is invoked automatically before the first execution of call().

This is typically used to create the weights of Layer subclasses (at the discretion of the subclass implementer).

Parameters:

input_shape – Instance of TensorShape, or list of instances of TensorShape if the layer expects a list of inputs (one instance per input).

call(x, training=True)

condense()

The condense operation allow to overwrite the kernel and ensure that other variables are still consistent.

Returns:

None

get_config()

Returns the config of the layer.

A layer config is a Python dictionary (serializable) containing the configuration of a layer. The same layer can be reinstantiated later (without its trained weights) from this configuration.

The config of a layer does not include connectivity information, nor the layer class name. These are handled by Network (one layer of abstraction above).

Note that get_config() does not guarantee to return a fresh copy of dict every time it is called. The callers should make a copy of the returned dict if they want to modify it.

Returns:

Python dictionary.

vanilla_export()

This operation allow to turn this Layer to it’s super type, easing storage and serving.

Returns:

self as super type