dnnbrain.dnn.algo module

class dnnbrain.dnn.algo.Algorithm(dnn, layer=None, channel=None)

Bases: abc.ABC

An Abstract Base Classes class to define interface for dnn algorithm

get_layer()

Get layer or its channel

Parameters

  • layer (str) – Name of the layer where the algorithm performs on.

  • channel (int) – Number of the channel where the algorithm performs on.

set_layer(layer, channel)

Set layer or its channel.

Parameters

  • layer (str) – Name of the layer where the algorithm performs on.

  • channel (int) – Number of the channel where the algorithm performs on algorithm only support one channel operation at one time.

class dnnbrain.dnn.algo.EmpiricalReceptiveField(engine=None)

Bases: object

A Class to Estimate Empirical Receptive Field (RF) of a DNN Model.

compute(stimuli, save_path=None)

Compute empirical receptive field based on input stimulus.

Parameters

  • stimuli (Stimulus) – Input stimuli which loaded from files on the disk.

  • save_path (str) – Path to save single image’s receptive field. If None, it will not be saved.

Returns

emp_rf – Mean empirical receptive field of all the input images, its shape is equal to the theoretical rf size in specific layer.

Return type

ndarray

generate_rf(all_thresed_act)

Compute RF on Given Image for Target Layer and Channel.

Parameters

all_thresed_act (ndarray) – Shape must be (n_chn, dnn.img_size).

Returns

empirical_rf_size – Empirical rf size of specific image.

Return type

np.float64

set_params(engine)

Set engine to compute empirical receptive field.

Parameters

engine (UpsamplingActivationMapping, OccluderDiscrepancyMapping) – Must be an instance of UpsamplingActivationMapping or OccluderDiscrepancyMapping.

class dnnbrain.dnn.algo.GuidedSaliencyImage(dnn, from_layer=None, from_chn=None)

Bases: dnnbrain.dnn.algo.SaliencyImage

A class to compute Guided Backprob gradient for a image.

register_hooks()

Override the abstract method from BackPropGradient class to define a specific hook for guided backprop gradient.

class dnnbrain.dnn.algo.MaskedImage(dnn, layer=None, channel=None, unit=None, stdev_size_thr=1.0, filter_sigma=1.0, target_reduction_ratio=0.9)

Bases: dnnbrain.dnn.algo.Algorithm

Generate masked gray picture for images according to activation changes

prepare_test(masked_image)

Transfer pic to tenssor for dnn activation

Parameters

masked_image (ndarray) – Masked image waits for dnn activation

Returns

test_image – Pytorch tensor for dnn computation

Return type

tensor

put_mask(initial_image, maxiteration=100)

Put mask on image

Parameters

  • initial_image (ndarray) – Initial image waits for masking.

  • maxiteration (int) – The max number of iterations to stop.

Returns

masked_image – The masked image with shape as (n_chn, height, width).

Return type

ndarray

register_hooks()

Define register hook and register them to specific layer and channel.

set_parameters(unit=None, stdev_size_thr=1.0, filter_sigma=1.0, target_reduction_ratio=0.9)

Set parameters for mask

Parameters

  • unit (tuple) – Position of the target unit.

  • stdev_size_thr (float) – Fraction of standard dev threshold for size of blobs, default 1.0.

  • filter_sigma (float) – Sigma for final gaussian blur, default 1.0.

  • target_reduction_ratio (float) – Reduction ratio to achieve for tightening the mask,default 0.9.

class dnnbrain.dnn.algo.MinimalComponentImage(dnn, layer=None, channel=None)

Bases: dnnbrain.dnn.algo.Algorithm

A class to generate minmal image for a CNN model using a specific part decomposer and optimization criterion.

combine_component(index)

combine the indexed component into a image

generate_minimal_image()

Generate minimal image. We first sort the component by the activiton and then iterate to find the combination of the components which can maximally activate the target channel.

Note: before call this method, you should call xx_decompose method to decompose the image into parcels.

Parameters

stim (Stimulus) – Stimulus

ica_decompose()
pca_decompose()
set_params(meth='pca', criterion='max')

Set parameter for the estimator

sort_componet(order='descending')

sort the component according the activation of dnn. order : str

Sort order, ‘ascending’ or ‘descending’

class dnnbrain.dnn.algo.MinimalParcelImage(dnn, layer=None, channel=None, activaiton_criterion='max', search_criterion='max')

Bases: dnnbrain.dnn.algo.Algorithm

A class to generate minimal image for target channels from a DNN model.

combine_parcel(indices)

combine the indexed parcel into a image

Parameters

indices (list, slice) – Subscript indices.

Returns

image_container – Shape=(n_chn,height,width).

Return type

ndarray

felzenszwalb_decompose(image, scale=100, sigma=0.5, min_size=50)

Decompose image to multiple parcels using felzenszwalb method and put each parcel into a separated image with a black background.

Parameters

image (ndarray) – Shape=(height,width,n_chn).

Returns

parcel – Shape=(n_parcel,height,width,n_chn).

Return type

ndarray

generate_minimal_image()

Generate minimal image. We first sort the parcel by the activiton and then iterate to find the combination of the parcels which can maximally activate the target channel.

Note: before call this method, you should call xx_decompose method to decompose the image into parcels.

Returns

image_min – Final minimal images in shape (height,width,n_chn).

Return type

ndarray

quickshift_decompose(image, kernel_size=3, max_dist=6, ratio=0.5)

Decompose image to multiple parcels using quickshift method and put each parcel into a separated image with a black background.

Parameters

  • image (ndarray) – Shape (height,width,n_chn).

  • meth (str) – Method to decompose images.

Returns

parcel – Shape (n_parcel,height,width,n_chn).

Return type

ndarray

set_params(activaiton_criterion='max', search_criterion='max')

Set parameter for searching minmal image.

Parameters

  • activaiton_criterion (str) – The criterion of how to pooling activaiton, choices=(max, mean, median, L1, L2).

  • search_criterion (str) – The criterion of how to search minimal image, choices=(max, fitting curve).

slic_decompose(image, n_segments=250, compactness=10, sigma=1)

Decompose image to multiple parcels using slic method and put each parcel into a separated image with a black background.

Parameters

  • image (ndarray) – Shape (height,width,n_chn).

  • meth (str) – Method to decompose images.

Returns

parcel – Shape=(n_parcel,height,width,n_chn)

Return type

ndarray

sort_parcel(order='descending')

sort the parcel according the activation of dnn.

Parameters

order (str) – Ascending or descending.

Returns

parcel – Shape (n_parcel,height,width,n_chn) parcel after sorted.

Return type

ndarray

class dnnbrain.dnn.algo.OccluderDiscrepancyMapping(dnn, layer=None, channel=None, window=(11, 11), stride=(2, 2), metric='mean')

Bases: dnnbrain.dnn.algo.Algorithm

Slide a occluder window on an image, and calculate the change of the target channel’s activation after each step.

compute(image)

Compute discrepancy map of the image using a occluder window moving from top-left to bottom-right.

Parameters

image (ndarray, Tensor, PIL.Image) – An original image.

Returns

discrepancy_map – Discrepancy activation map.

Return type

ndarray

set_params(window, stride, metric)

Set parameter for occluder discrepancy mapping.

Parameters

  • window (tuple) – The size of sliding window - (width, height).

  • stride (tuple) – The step length of sliding window - (width_step, height_step).

  • metric (str) – The metric to summarize the target channel’s activation, ‘max’ or ‘mean’.

class dnnbrain.dnn.algo.SaliencyImage(dnn, from_layer=None, from_chn=None)

Bases: dnnbrain.dnn.algo.Algorithm

An Abstract Base Classes class to define interfaces for gradient back propagation. Note: the saliency image values are not applied with absolute operation.

backprop(image, to_layer=None)

Compute gradients of the to_layer corresponding to the from_layer and from_channel by back propagation algorithm.

Parameters

  • image (ndarray, Tensor, PIL.Image) – Image data.

  • to_layer (str) – Name of the layer where gradients back propagate to. If is None, get the first layer in the layers recorded in DNN.

Returns

gradient – Gradients of the to_layer with shape as (n_chn, n_row, n_col). If layer is the first layer of the model, its shape is (3, n_height, n_width).

Return type

ndarray

backprop_smooth(image, n_iter, sigma_multiplier=0.1, to_layer=None)

Compute smoothed gradient. It will use the gradient method to compute the gradient and then smooth it

Parameters

  • image (ndarray, Tensor, PIL.Image) – Image data

  • n_iter (int) – The number of noisy images to be generated before average.

  • sigma_multiplier (int) – Multiply when calculating std of noise.

  • to_layer (str) – Name of the layer where gradients back propagate to. If is None, get the first layer in the layers recorded in DNN.

Returns

gradient – Gradients of the to_layer with shape as (n_chn, n_row, n_col). If layer is the first layer of the model, its shape is (n_chn, n_height, n_width).

Return type

ndarray

abstract register_hooks()

Define register hooks and register them to specific layer and channel. As this a abstract method, it is needed to be override in every subclass.

class dnnbrain.dnn.algo.SynthesisImage(dnn, layer=None, channel=None, activ_metric='mean', regular_metric=None, regular_lambda=None, precondition_metric=None, GB_radius=None, smooth_metric=None, factor=None)

Bases: dnnbrain.dnn.algo.Algorithm

Generate a synthetic image that maximally activates a neuron.

register_hooks(unit=None)

Define register hook and register them to specific layer and channel.

Parameters

unit (tuple) – Determine unit position, None means channel, default None.

set_loss(activ_metric, regular_metric, regular_lambda)

This method is to set loss function for optimization. As the target usually is a 2-D feature map in convolutional layer with multiple units, ‘active_metric’ can make algorithm clear on how to omputue the loss value. Also there are some popular regularization to make synthesis more interpretable, ‘regular_metric’ can set one of them and ‘regular_lambda’ give the weights of this term.

Parameters

  • activ_metric (str) – The metric method to summarize activation.

  • regular_metric (str) – The metric method of regularization.

  • regular_lambda (float) – The lambda of the regularization.

set_precondition(precondition_metric, GB_radius)

This is the method to set whether a precondition metric will be used, precondition is one of the method to smooth the high frequency noise on synthesized image. It will applied on every interval image during the iteration.

Parameters

  • precondition_metric (str) – The metric method of preconditioning.

  • GB_radius (float) – Radius parameter for ‘GB’, gaussian blur.

set_smooth_gradient(smooth_metric, factor)

This method is to set smooth gradient metric, it’s a very effective way to prove synthesized image quality.

Parameters

  • smooth_metric (str) – The metric method of smoothing.

  • factor (float) – Factor parameter for ‘Fourier’, smooth fourier.

synthesize(init_image=None, unit=None, lr=0.1, n_iter=30, verbose=True, save_path=None, save_step=None)

Synthesize the image which maximally activates target layer and channel

Parameters

  • init_image (ndarray, Tensor, PIL.Image) – Initialized image.

  • unit (tuple) – Set target unit position.

  • lr (float) – Learning rate.

  • n_iter (int) – The number of iterations

  • verbose (bool) – print loss duration iteration or not

  • save_path (str) – The directory to save synthesized images.

  • save_step (int) – Save out synthesized images for every ‘save_step’ iterations. Only used when save_path is not None.

Returns

final_image – The synthesized image with shape as (n_chn, height, width).

Return type

ndarray

class dnnbrain.dnn.algo.TheoreticalReceptiveField(dnn, layer=None, channel=None)

Bases: dnnbrain.dnn.algo.Algorithm

A Class to Count Theoretical Receptive Field. Note: Currently only AlexNet, Vgg16, Vgg19 are supported. (All these net are linear structure.)

compute(batch_size=- 1, device='cuda', display=None)

Compute specific receptive field information for target dnn. Only support AlexNet, VGG11!

Parameters

  • batch_size (int) – The batch size used in computing.

  • device (str) – Input device, please specify ‘cuda’ or ‘cpu’.

  • display (bool) – If True, it will show the receptive field information in a table.

Returns

receptive field – Receptive field information which contains rf_size, feature_map_size, start, jump

Return type

OrderedDict

compute_size()
find_region(receptive_field)

Compute specific receptive field range for target dnn, layer and unit.

Parameters

field (receptive) – Receptive field information which contains rf_size, feature_map_size, start, jump.

Returns

rf_range – The theoretical receptive field region example:[(start_h, end_h), (start_w, end_w)].

Return type

list

set_parameters(unit)
Parameters

unit (tuple) – The unit location in its feature map.

class dnnbrain.dnn.algo.UpsamplingActivationMapping(dnn, layer=None, channel=None, interp_meth='bicubic', interp_threshold=None)

Bases: dnnbrain.dnn.algo.Algorithm

Resample the target channel’s feature map to the input size after threshold.

compute(image)

Resample the channel’s feature map to input size.

Parameters

image (ndarray, Tensor, PIL.Image) – An input image.

Returns

img_act – Image after resampling, Shape=(height, width).

Return type

ndarray

set_params(interp_meth, interp_threshold)

Set necessary parameters.

Parameters

  • interp_meth (str) – Algorithm used for resampling are ‘nearest’, ‘bilinear’, ‘bicubic’, ‘area’. Default: ‘bicubic’

  • interp_threshold (float) – value is in [0, 1]. The threshold used to filter the map after resampling. For example, if the threshold is 0.58, it means clip the feature map with the min as the minimum of the top 42% activation.

class dnnbrain.dnn.algo.VanillaSaliencyImage(dnn, from_layer=None, from_chn=None)

Bases: dnnbrain.dnn.algo.SaliencyImage

A class to compute vanila Backprob gradient for a image.

register_hooks()

Override the abstract method from BackPropGradient class to define a specific hook for vanila backprop gradient.