Kullback-Leibler for Bayesian networks

import os

%matplotlib inline

from pylab import *
import matplotlib.pyplot as plt

import pyAgrum and pyAgrum.lib.notebook (for … notebooks :-) )

import pyAgrum as gum
import pyAgrum.lib.notebook as gnb

Create a first BN : bn

bn=gum.loadBN("res/asia.bif")
# randomly re-generate parameters for every Conditional Probability Table
bn.generateCPTs()
bn

Create a second BN : bn2

bn2=gum.loadBN("res/asia.bif")
bn2.generateCPTs()
bn2

bn vs bn2 : different parameters

gnb.flow.row(bn.cpt(3),bn2.cpt(3),
              captions=["a CPT in bn","same CPT in bn2 (with different parameters)"])

	positive_XraY
tuberculos_or_cancer	0	1
0	0.6672	0.3328
1	0.2073	0.7927

a CPT in bn

	positive_XraY
tuberculos_or_cancer	0	1
0	0.5097	0.4903
1	0.5226	0.4774

same CPT in bn2 (with different parameters)

Exact and (Gibbs) approximated KL-divergence

In order to compute KL-divergence, we just need to be sure that the 2 distributions are defined on the same domain (same variables, etc.)

Exact KL

g1=gum.ExactBNdistance(bn,bn2)
print(g1.compute())

{'klPQ': 9.32501615630327, 'errorPQ': 0, 'klQP': 7.018490844173645, 'errorQP': 0, 'hellinger': 1.272903811974082, 'bhattacharya': 1.6614791563033795, 'jensen-shannon': 0.8942220627718407}

If the models are not on the same domain :

bn_different_domain=gum.loadBN("res/alarm.dsl")

# g=gum.BruteForceKL(bn,bn_different_domain) # a KL-divergence between asia and alarm ... :(
#
# would cause
#---------------------------------------------------------------------------
#OperationNotAllowed                       Traceback (most recent call last)
#
#OperationNotAllowed: this operation is not allowed : KL : the 2 BNs are not compatible (not the same vars : visit_to_Asia?)

Gibbs-approximated KL

g=gum.GibbsBNdistance(bn,bn2)
g.setVerbosity(True)
g.setMaxTime(120)
g.setBurnIn(5000)
g.setEpsilon(1e-7)
g.setPeriodSize(500)

print(g.compute())
print("Computed in {0} s".format(g.currentTime()))

{'klPQ': 9.339677584823958, 'errorPQ': 0, 'klQP': 7.990166292911032, 'errorQP': 0, 'hellinger': 1.3016355326773732, 'bhattacharya': 1.6696562544345603, 'jensen-shannon': 0.9315331490653456}
Computed in 3.5710010000000003 s

print("--")

print(g.messageApproximationScheme())
print("--")

print("Temps de calcul : {0}".format(g.currentTime()))
print("Nombre d'itérations : {0}".format(g.nbrIterations()))

--
stopped with epsilon=1e-07
--
Temps de calcul : 3.5710010000000003
Nombre d'itérations : 966000

p=plot(g.history(), 'g')

Animation of Gibbs KL

Since it may be difficult to know what happens during approximation algorithm, pyAgrum allows to follow the iteration using animated matplotlib figure

g=gum.GibbsBNdistance(bn,bn2)
g.setMaxTime(60)
g.setBurnIn(500)
g.setEpsilon(1e-7)
g.setPeriodSize(5000)

gnb.animApproximationScheme(g) # logarithmique scale for Y
g.compute()

{'klPQ': 9.318774181864057,
 'errorPQ': 0,
 'klQP': 6.859180140067871,
 'errorQP': 0,
 'hellinger': 1.2677805516518006,
 'bhattacharya': 1.659557055530806,
 'jensen-shannon': 0.8875655276409298}