lampe.diagnostics#

Diagnostics and reliability assessment.

Functions#

`expected_coverage_mc`	Estimates by Monte Carlo (MC) the expected coverages of a posterior estimator \(q(\theta \| x)\) over pairs \((\theta^, x^) \sim p(\theta, x)\).
`expected_coverage_ni`	Estimates by numerical integration (NI) the expected coverages of a posterior estimator \(q(\theta \| x)\) over pairs \((\theta^, x^) \sim p(\theta, x)\).

Descriptions#

lampe.diagnostics.expected_coverage_mc(posterior, pairs, n=1024, device=None)#

Estimates by Monte Carlo (MC) the expected coverages of a posterior estimator \(q(\theta | x)\) over pairs \((\theta^*, x^*) \sim p(\theta, x)\).

The expected coverage at credible level \(1 - \alpha\) is the probability of \(\theta^*\) to be included in the highest density region of total probability \(1 - \alpha\) of the posterior \(q(\theta | x^*)\), denoted \(\Theta_{q(\theta | x^*)}(1 - \alpha)\). To get the coverages, the proportion \(r^*\) of samples \(\theta \sim q(\theta | x^*)\) having a higher estimated density than \(\theta^*\) is computed for each pair \((\theta^*, x^*)\). Formally,

\[r^* = \mathbb{E}_{q(\theta | x^*)} \Big[ \mathbb{I} \big[ q(\theta | x^*) \geq q(\theta^* | x^*) \big] \Big] .\]

Then, the expected coverage at credible level \(1 - \alpha\) is the probability of \(r^*\) to be lower than \(1 - \alpha\),

\[P \big( \theta^* \in \Theta_{q(\theta | x^*)}(1 - \alpha) \big) = P(r^* \leq 1 - \alpha) .\]

In practice, Monte Carlo estimations of \(r^*\) are used.

References

Averting A Crisis In Simulation-Based Inference (Hermans et al., 2021)

https://arxiv.org/abs/2110.06581

Parameters

posterior (Callable[[Tensor], Distribution]) – A posterior estimator \(q(\theta | x)\).
pairs (Iterable[Tuple[Tensor, Tensor]]) – An iterable of pairs \((\theta^*, x^*) \sim p(\theta, x)\).
n (int) – The number of samples to draw from the posterior for each pair.
device (Optional[str]) – The device on which the computations are performed.

Returns

A vector of increasing credible levels and their respective expected coverages.

Return type

Tuple[Tensor, Tensor]

Example

>>> posterior = lampe.inference.NPE(3, 4)
>>> testset = lampe.data.H5Dataset('test.h5')
>>> levels, coverages = expected_coverage_mc(posterior.flow, testset)

lampe.diagnostics.expected_coverage_ni(log_p, pairs, domain, device=None, **kwargs)#

Estimates by numerical integration (NI) the expected coverages of a posterior estimator \(q(\theta | x)\) over pairs \((\theta^*, x^*) \sim p(\theta, x)\).

Equivalent to expected_coverage_mc but the proportions \(r^*\) are approximated by numerical integration over the domain, which is useful when the posterior estimator can be evaluated but not be sampled from.