Weighted observations

[cscherrer]

I'm kind of thinking the idea of weighting observations doesn't really make sense on its own, and when we say that we really mean a superposition of weighted Dirac measures.

Are there any concerns with representing it in this way?

[mschauer]

I could make sense of a weighted https://en.m.wikipedia.org/wiki/Empirical_measure

[cscherrer]

Thanks @mschauer . I think this is equivalent, and leads to some fine points on questions of how to best represent things.

Currently we have

struct WeightedMeasure{R,M} <: AbstractMeasure
    logweight :: R
    base :: M
end

So the weight (or logweight) is one value associated with the measure as a whole. To get to a discrete measure with weights on the components, we'd need to add a weights to each component and then superpose.

Another way to represent this is in terms of a DensityMeasure,

struct DensityMeasure{F,B,L} <: AbstractMeasure
    f    :: F
    base :: B
    log  :: L
end

And a special case of that is

∫(μ::AbstractMeasure, base::AbstractMeasure; log=true) = ∫(𝒹(μ,base), base; log=log)

This says, "represent μ as a weighted version of base". For example, we might have

m = ∫(Normal(), Cauchy())

which says to represent a normal in terms of weighted samples from a Cauchy, say if you want to be sure the tails are explored. It's just importance sampling, really.

The problem is, what is rand(m)? Currently in dev I'm trying out whether this should return something like

julia> rand(m)
3.18491e-27 * Dirac{Float64}(11.5004)

Concerns are

• If we do this, we have rand returning a value in some cases, and a measure in others. That doesn't feel right. We could also working in terms of a new combinator that always returns a measure, so we'd have lots of empirical measures.
• If we don't do this, how do we represent weighted samples? What does a weighted sample actually mean? What space does it live in?

[nignatiadis]

Another possible concern is that weights can have different semantics (cf. weight types in StatsBase.jl for an example). Another example that seems relevant for MeasureTheory.jl is that an importance sample weight may be normalized or unnormalized (i.e., are we using p(x)/q(x) exactly or only up to a constant). This may matter to downstream algorithms. How would such semantics be represented by the weighted Dirac approach?

Maybe one could use explicit types for weighted samples? Say,

abstract type WeightedSample{T} end

struct UnnormalizedImportanceWeightedSample{T,S} <: WeightedSample{T}
   sample::T
   weight::S
end

[cscherrer]

We already have (in dev)

struct WeightedMeasure{R,M} <: AbstractMeasure
    logweight :: R
    base :: M
end

So while I'm sure we'll want to make some changes for performance, we can already have a superposition of weighted Dirac measures. The thing I'm not sure of is, should a weighted sample be a measure? We often want to sample from it, so that suggests that it should. But whether it is or not, we need to think through the semantics and implications of the design choice.

As far as normalization, there are a few cases to consider. A measure can be finite or not. If it's finite, it may be a probability measure or not. Orthogonally, we may know the normalization factor of a finite measure, or not. So for example, we should be able to say "I know this is a probability measure, but I don't know the evidence (or normalization factor)"