Exact Common Information¶
The exact common information [KLEG14] is the entropy of the smallest variable \(V\) which renders all variables of interest independent:
\[\G{X_{0:n} | Y_{0:m}} = \min_{\ind X_{0:n} \mid Y_{0:m}, V} \H{V | Y_{0:m}}\]
Subadditivity of Independent Variables¶
Kumar et. al. [KLEG14] have shown that the exact common information of a pair of independent pairs of variables can be less than the sum of their individual exact common informations. Here we verify this claim:
In [1]: from dit.multivariate import exact_common_information as G
In [2]: d = dit.Distribution([(0,0), (0,1), (1,0)], [1/3]*3)
In [3]: d2 = d @ d # python 3.6 syntax for d.__matmul__(d)
File "<ipython-input-3-62b058140b08>", line 1
d2 = d @ d # python 3.6 syntax for d.__matmul__(d)
^
SyntaxError: invalid syntax
In [4]: print(d2)
Class: ScalarDistribution
Alphabet: (0, 1, 2)
Base: linear
x p(x)
0 1/3
1 1/3
2 1/3
In [5]: 2*G(d)
Out[5]: 1.8365493742565033
In [6]: G(d2, [[0, 2], [1, 3]])
ditExceptionTraceback (most recent call last)
<ipython-input-6-e847297a29df> in <module>()
----> 1 G(d2, [[0, 2], [1, 3]])
/home/docs/checkouts/readthedocs.org/user_builds/dit-dev/conda/latest/lib/python2.7/site-packages/dit/utils/units.pyc in wrapper(*args, **kwargs)
67 @wraps(f)
68 def wrapper(*args, **kwargs):
---> 69 value = f(*args, **kwargs)
70 return value
71
/home/docs/checkouts/readthedocs.org/user_builds/dit-dev/conda/latest/lib/python2.7/site-packages/dit/multivariate/common_informations/base_markov_optimizer.pyc in common_info(dist, rvs, crvs, niter, maxiter, polish, bound, rv_mode)
175 @unitful
176 def common_info(dist, rvs=None, crvs=None, niter=None, maxiter=1000, polish=1e-6, bound=None, rv_mode=None):
--> 177 dtc = dual_total_correlation(dist, rvs, crvs, rv_mode)
178 ent = entropy(dist, rvs, crvs, rv_mode)
179 if np.isclose(dtc, ent):
/home/docs/checkouts/readthedocs.org/user_builds/dit-dev/conda/latest/lib/python2.7/site-packages/dit/utils/units.pyc in wrapper(*args, **kwargs)
67 @wraps(f)
68 def wrapper(*args, **kwargs):
---> 69 value = f(*args, **kwargs)
70 return value
71
/home/docs/checkouts/readthedocs.org/user_builds/dit-dev/conda/latest/lib/python2.7/site-packages/dit/multivariate/dual_total_correlation.pyc in dual_total_correlation(dist, rvs, crvs, rv_mode)
51 contain non-existant random variables.
52 """
---> 53 rvs, crvs, rv_mode = normalize_rvs(dist, rvs, crvs, rv_mode)
54
55 others = lambda rv, rvs: set(set().union(*rvs)) - set(rv)
/home/docs/checkouts/readthedocs.org/user_builds/dit-dev/conda/latest/lib/python2.7/site-packages/dit/helpers.pyc in normalize_rvs(dist, rvs, crvs, rv_mode)
254 else:
255 msg = "The information measure requires a joint distribution."
--> 256 raise ditException(msg)
257
258 return rvs, crvs, rv_mode
ditException: The information measure requires a joint distribution.
API¶
-
exact_common_information(*args, **kwargs)¶ Computes the exact common information, min H[V] where V renders all rvs independent.
Parameters: - dist (Distribution) – The distribution for which the exact common information will be computed.
- rvs (list, None) – A list of lists. Each inner list specifies the indexes of the random variables used to calculate the exact common information. If None, then it calculated over all random variables, which is equivalent to passing rvs=dist.rvs.
- crvs (list, None) – A single list of indexes specifying the random variables to condition on. If None, then no variables are conditioned on.
- niter (int > 0) – Number of basin hoppings to perform during the optimization.
- maxiter (int > 0) – The number of iterations of the optimization subroutine to perform.
- polish (False, float) – Whether to polish the result or not. If a float, this will perform a second optimization seeded with the result of the first, but with smaller tolerances and probabilities below polish set to 0. If False, don’t polish.
- bound (int) – Bound the size of the Markov variable.
- rv_mode (str, None) – Specifies how to interpret rvs and crvs. Valid options are: {‘indices’, ‘names’}. If equal to ‘indices’, then the elements of crvs and rvs are interpreted as random variable indices. If equal to ‘names’, the the elements are interpreted as random variable names. If None, then the value of dist._rv_mode is consulted, which defaults to ‘indices’.
Returns: ci – The exact common information.
Return type: