Plots¶

class deepdiagnostics.plots.plot.Display(model=None, data=None, run_id=None, save=True, show=False, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None, **kwargs)¶

Parameters used against all plots.

Parameters:

model (deepdiagnostics.models.model) – Model to calculate the metric for. Required.
data (deepdiagnostics.data.data) – Data to test against. Required.
out_dir (Optional[str], optional) – Directory to save a png ({plot_name}.png) to. Defaults to None.
run_id (str, optional) – Run ID to use for the plot name. Defaults to None.
save (bool, optional) – Save the output to png.
show (bool, optional) – Show the completed plot when finished.
use_progress_bar (Optional[bool], optional) – Show a progress bar when iteratively performing inference. Defaults to None.
samples_per_inference (Optional[int], optional) – Number of samples used in a single iteration of inference. Defaults to None.
percentiles (Optional[Sequence[int]], optional) – List of integer percentiles, for defining coverage regions. Defaults to None.
number_simulations (Optional[int], optional) – Number of different simulations to run. Often, this means that the number of inferences performed for a metric is samples_per_inference*number_simulations. Defaults to None.
parameter_names (Optional[Sequence], optional) – Name of each theta parameter to use for titling and labels. Corresponding with the dim=1 axis of theta given by data. Defaults to None.
parameter_colors (Optional[Sequence], optional) – Colors to use for each theta parameters when representing the parameters on the same plot. Defaults to None.
colorway (Optional[str], optional) – String colorway to use. Choose from matplotlib’s named colorways. Defaults to None.

abstract plot(data_display, **kwrgs)¶

For a given data display object, plot the data

If data_display is given as a dict, it will be converted into a dot object.

Return type:: tuple[figure, axes]
Parameters:: data_display (dict | data_display)

class deepdiagnostics.plots.CDFRanks(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

plot(data_display, **kwargs)¶

Make the CDF Ranks plot

Parameters:

display (data_display, str) – The data to plot, typically returned by the _data_setup. Must have the ‘ranks’ attribute.
data_display (DataDisplay | str)

Return type:

tuple[Figure, Axes]

class deepdiagnostics.plots.Ranks(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Adaptation of [TCBD+20].

A wrapper around SBI’s sbc_rank_plot function. More information can be found here Plots the histogram of each theta parameter’s rank.

from deepdiagnostics.plots import Ranks

Ranks(model, data, save=False, show=True)(num_bins=25)

plot(data_display, num_bins=20)¶

Parameters:

num_bins (int) – Number of histogram bins. Defaults to 20.
data_display (DataDisplay | dict)

Return type:

tuple[Figure, Axes]

class deepdiagnostics.plots.CoverageFraction(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Show posterior regions of confidence as a function of percentiles. Each parameter of theta is plotted against a coverage fraction for each given theta.

from deepdiagnostics.plots import CoverageFraction

CoverageFraction(model, data, show=True, save=False)(
    figure_alpha=0.8,
    legend_loc="upper left",
    reference_line_label="Ideal"
)

plot(data_display, figure_alpha=1.0, line_width=3, legend_loc=None, include_coverage_std=False, include_coverage_residual=False, include_coverage_residual_std=False, include_ideal_range=True, reference_line_label='Reference Line', reference_line_style='k--', x_label='Confidence Interval of the Posterior Volume', y_label='Coverage fraction within posterior volume', residual_y_label='Coverage Fraction Residual', title='')¶

Plot the coverage fraction and residuals if specified.

Parameters:

data_display (Union[DataDisplay, str]) – DataDisplay object or path to h5 file containing the data. If str, it will be loaded and requires the fields “coverage_fractions”, “coverage_percentiles”, and optionally “coverage_std”.
figure_alpha (float, optional) – Opacity of parameter lines. Defaults to 1.0.
line_width (int, optional) – Width of parameter lines. Defaults to 3.
legend_loc (str, optional) – Location of the legend. Defaults to matplotlib specified.
include_coverage_std (bool, optional) – Whether to include the standard deviation shading for coverage fractions . Defaults to False.
include_coverage_residual (bool, optional) – Whether to include the residual plot (coverage fraction - diagonal). Creates an additional subplot under the original plot. Defaults to False.
include_coverage_residual_std (bool, optional) – Whether to include the standard deviation shading for residuals. Defaults to False.
include_ideal_range (bool, optional) – Whether to include the ideal range shading (0.1/0.2 around the diagonal). Defaults to True.
reference_line_label (str, optional) – Label name for the diagonal ideal line. Defaults to “Reference Line”.
reference_line_style (str, optional) – Line style for the reference line. Defaults to “k–“.
x_label (str, optional) – y label. Defaults to “Confidence Interval of the Posterior Volume”.
y_label (str, optional) – y label. Defaults to “Fraction of Lenses within Posterior Volume”.
residual_y_label (str, optional) – y label for the residual plot. Defaults to “Coverage Fraction Residual”.
title (str, optional) – plot title. Defaults to “NPE”.

Return type:

tuple[Figure, Axes]

class deepdiagnostics.plots.TARP(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Produce a TARP plot as described in Lemos et. al. [LCHPL23]. Utilizes the implementation from here.

from deepdiagnostics.plots import TARP

TARP(models, data, show=True, save=False)(
    coverage_sigma=2,
    coverage_alpha=0.4,
    y_label="Credibility Level"
)

plot(data_display=None, coverage_sigma=3, reference_point='random', metric='euclidean', normalize=True, bootstrap_calculation=True, coverage_alpha=0.2, y_label='Expected Coverage', x_label='Expected Coverage', title='Test of Accuracy with Random Points')¶

Parameters:

coverage_sigma (int, optional) – Number of sigma to use for coverage. Defaults to 3.
reference_point (Union[str, np.ndarray], optional) – Reference points in the parameter space to test against. Defaults to “random”.
metric (bool, optional) – Distance metric ot use between reference points. Use “euclidean” or “manhattan”.Defaults to “euclidean”.
normalize (bool, optional) – Normalize input space to 1. Defaults to True.
bootstrap_calculation (bool, optional) – Estimate uncertainties using bootstrapped examples. Increases efficiency. Defaults to True.
coverage_alpha (float, optional) – Opacity of the difference coverage sigma. Defaults to 0.2.
y_label (str, optional) – Sup. label on the y axis. Defaults to “Expected Coverage”.
x_label (str, optional) – Sup. label on the x axis. Defaults to “Expected Coverage”.
title (str, optional) – Title of the entire figure. Defaults to “Test of Accuracy with Random Points”.
data_display (DataDisplay | dict)

Return type:

tuple[Figure, Axes]

deepdiagnostics.plots.LC2ST¶: alias of LocalTwoSampleTest

class deepdiagnostics.plots.local_two_sample.LocalTwoSampleTest(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Note

A simulator is required to run this plot.

Produce plots showing the local evaluation of a posterior estimator for a given observation. Adapted fom Linhart et. al. [LGR23].

Implements a pair plot, showing regions confidence regions of the CDF in comparison with the null hypothesis classifier results, and an intensity plot, showing the regions of accuracy for each parameter of theta.

Uses the following code as reference material:

github.com/JuliaLinhart/lc2st/graphical_diagnostics.py::pp_plot_lc2st.

github.com/JuliaLinhart/lc2st/graphical_diagnostics.py::eval_space_with_proba_intensity.

from deepdiagnostics.plots import LC2ST

LC2ST(model, data, save=False, show=True).plot(use_intensity_plot=True, n_alpha_samples=100, linear_classifier="MLP", n_null_hypothesis_trials=20)

plot(data_display=None, confidence_region_alpha=0.2, pairplot_y_label='', pairplot_x_label='', pairplot_title='Local Classifier PP-Plot', **kwargs)¶

Probability - Probability (P-P) plot for the classifier predicted: class probabilities in (L)C2ST to assess the validity of a (or several) density estimator(s).

References: https://github.com/JuliaLinhart/lc2st/blob/e221cc326480cb0daadfd2ba50df4eefd374793b/lc2st/graphical_diagnostics.py#L49

Parameters:

data_display (Union[dict, DataDisplay], optional) – DataDisplay object containing the necessary data for plotting (requires ‘probability’, ‘null_hypothesis_probability’, ‘cdf_alphas’). See metrics.local_two_sample.LocalTwoSampleTest for more details.
confidence_region_alpha (float, optional) – Opacity of the confidence region in the pair plot Defaults to 0.2.
pairplot_y_label (str, optional) – Y-axis label for the pair plot. Defaults to “”.
pairplot_x_label (str, optional) – X-axis label for the pair plot. Defaults to “”.
pairplot_title (str, optional) – Title for the pair plot. Defaults to “Local Classifier PP-Plot”.

Return type:

tuple[Figure, Axes]

plot_intensity(data_display=None, n_intensity_bins=20, intensity_plot_ylabel='', intensity_plot_xlabel='', intensity_plot_title='Local Classifier Intensity Distribution', **kwargs)¶

Plot 1d or 2d marginal histogram of samples of the density estimator: with probabilities as color intensity.

References: https://github.com/JuliaLinhart/lc2st/blob/e221cc326480cb0daadfd2ba50df4eefd374793b/lc2st/graphical_diagnostics.py#L133

Parameters:

data_display (Union[dict, DataDisplay], optional) – DataDisplay object containing the necessary data for plotting (requires ‘probability’, ‘evaluation_data’). See metrics.local_two_sample.LocalTwoSampleTest for more details.
n_intensity_bins (int, optional) – Number of bins to use when producing the intensity plots. Number of regions. Defaults to 20.
intensity_plot_ylabel (str, optional) – Y-axis label for the intensity plot. Defaults to “”.
intensity_plot_xlabel (str, optional) – X-axis label for the intensity plot. Defaults to “”.
intensity_plot_title (str, optional) – Title for the intensity plot. Defaults to “Local Classifier Intensity Distribution”.

Return type:

tuple[Figure, Axes]

class deepdiagnostics.plots.PPC(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Note

A simulator is required to run this plot.

Show the output of the model’s generated posterior against the true values for the same context. Can show either output vs input (in 1D) or examples of simulation output (in 2D).

from deepdiagnostics.plots import PPC

PPC(model, data, save=False, show=True)(n_unique_plots=5)  # Plot 5 examples

plot(data_display=None, n_coverage_sigma=3, theta_true_marker='^', include_axis_ticks=False, title='Predictive Posterior', y_label='Simulation Output', x_label='X')¶

Parameters:

n_coverage_sigma (Optional[int], optional) – Show the N different standard dev. sigma of the posterior results. Only used in 1D. Defaults to 3.
true_sigma (Optional[float], optional) – True std. of the known posterior. Used only if supplied. Defaults to None.
theta_true_marker (Optional[str], optional) – Marker to use for output of the true theta parameters. Only used in 1d. Defaults to ‘^’.
n_unique_plots (Optional[int], optional) – Number of samples of theta/x to use. Each one corresponds to a column. Defaults to 3.
include_axis_ticks (bool, optional) – _description_. Defaults to False.
title (str, optional) – Title of the plot. Defaults to “Predictive Posterior”.
y_label (str, optional) – Row label. Defaults to “Simulation Output”.
x_label (str, optional) – Column label. Defaults to “X”.
data_display (DataDisplay | dict)

Raises:

NotImplementedError – If trying to plot results of a simulation with more than 2 output dimensions.

Return type:

tuple[Figure, Axes]

class deepdiagnostics.plots.PriorPC(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Note

A simulator is required to run this plot.

Plot random samples of the simulator’s output from samples drawn from the prior

from deepdiagnostics.plots import PriorPC

PriorPC(model, data, show=True, save=False)(
    n_rows = 2,
    n_columns = 6, # Make 2x6 = 12 different samples
    row_parameter_index = 0,
    column_parameter_index = 1, # Include labels for theta parameters 0 and 1 from the prior
    round_parameters = True,
)

plot(data_display=None, row_parameter_index=0, column_parameter_index=1, round_parameters=True, sort_rows=True, label_samples='both', title='Simulated output from prior', y_label=None, x_label=None)¶

Parameters:

n_rows (Optional[int], optional) – Number of unique rows to make for priors. Defaults to 3.
n_columns (Optional[int], optional) – Number of unique columns for viewing prior predictions. Defaults to 3.
row_parameter_index (Optional[int], optional) – Index of the theta parameter to display as labels on rows. Defaults to 0.
column_parameter_index (Optional[int], optional) – Index of the theta parameter to display as labels on columns. Defaults to 1.
round_parameters (Optional[bool], optional) – In labels, round the theta parameters (recommended when thetas are float values). Defaults to True.
sort_columns (bool, optional) – Sort the plots by theta column value. Defaults to True.
label_samples (Optional[str], optional) – Label the prior values as a text box in each label. Row means using row_parameter_index as the title value. Choose from “rows”, “columns”, “both”. Defaults to ‘both’.
title (Optional[str], optional) – Title of the whole plot. Defaults to “Simulated output from prior”.
y_label (Optional[str], optional) – Column label, when None, label = theta_{column_index} = parameter_name. Defaults to None.
x_label (str, optional) – Row label, when None, label = theta_{row_index} = parameter_name. Defaults to None.
data_display (DataDisplay | None)
sort_rows (bool)

Return type:

tuple[Figure, Axes]

class deepdiagnostics.plots.Parity(model, data, run_id, save, show, out_dir=None, percentiles=None, use_progress_bar=None, samples_per_inference=None, number_simulations=None, parameter_names=None, parameter_colors=None, colorway=None)¶

Show plots directly comparing the posterior vs. true theta values. Make a plot that is (number of selected metrics) X dimensions of theta. Includes the option to show differences, residual, and percent residual as plots under the main parity plot.

from deepdiagnostics.plots import Parity

Parity(model, data, show=True, save=False)(
    n_samples=200 # 200 samples of the posterior
    include_residual = True # Add a plot showing the residual
)

plot(data_display=None, include_difference=False, include_residual=False, include_percentage=False, show_ideal=True, errorbar_color='black', title='Parity', y_label='$\\\\theta_{predicted}$', x_label='$\\\\theta_{true}$')¶

Parameters:

include_difference (bool, optional) – Include a plot that shows the difference between the posterior and true. Defaults to False.
include_residual (bool, optional) – Include a plot that shows the residual between posterior and true. Defaults to False.
include_percentage (bool, optional) – Include a plot that shows the residual as a percent between posterior and true. Defaults to False.
show_ideal (bool, optional) – Include a line showing where posterior=true. Defaults to True.
errorbar_color (str, optional) – _description_. Defaults to ‘black’.
title (str, optional) – Title of the plot. Defaults to “Parity”.
y_label (str, optional) – y axis label. Defaults to r”$ heta_{predicted}$”.
x_label (str, optional) – x axis label. Defaults to r”$ heta_{true}$”.
data_display (str | DataDisplay)

Return type:

tuple[Figure, Axes]

[LCHPL23]

Pablo Lemos, Adam Coogan, Yashar Hezaveh, and Laurence Perreault-Levasseur. Sampling-based accuracy testing of posterior estimators for general inference. 2023. arXiv:2302.03026.

[LGR23]

Julia Linhart, Alexandre Gramfort, and Pedro L. C. Rodrigues. L-c2st: local diagnostics for posterior approximations in simulation-based inference. 2023. arXiv:2306.03580.

[TCBD+20]

Alvaro Tejero-Cantero, Jan Boelts, Michael Deistler, Jan-Matthis Lueckmann, Conor Durkan, Pedro J. Gonçalves, David S. Greenberg, and Jakob H. Macke. Sbi: a toolkit for simulation-based inference. Journal of Open Source Software, 5(52):2505, 2020. URL: https://doi.org/10.21105/joss.02505, doi:10.21105/joss.02505.

Plots¶

Table of Contents

Previous topic

Next topic

This Page