Hi, I have the above Bayesian Network and I need some help for my code.
s has 5 discrete states, and follows the distribution defined by alpha
t has 2 discrete states, with beta prior
q has 5 discrete states with theta as dirichlet prior
static void Main(string[] args)
{
Range two = new Range(2);
Range five = new Range(5);
Variable<double> beta = Variable.Beta(1, 1);
Variable<bool> t = Variable.Bernoulli(beta);
double[] prior = { 1, 1, 1, 1, 1 };
VariableArray2D<Vector> alpha = Variable.Array<Vector>(two, five);
Variable<Vector> theta = Variable.Dirichlet(prior);
Variable<int> q = Variable.Discrete(five, theta);
using (Variable.ForEach(two))
using (Variable.ForEach(five))
alpha[two, five] = Variable.Dirichlet(prior);
}
//Breaking Symmetry
Vector denseBeta = new Vector(5, 10.0);
Dirichlet dirich = new Dirichlet(denseBeta);
Dirichlet[,] initAlpha = new Dirichlet[2,5];
for (int i = 0; i < 2; i++) {
for(int j=0;j<5;j++)
initAlpha[i,j] = new Dirichlet(dirich.Sample());
alpha.InitialiseTo(Distribution<Vector>.Array(initAlpha));
Variable<int> s;// = Variable.New<int>();
using (Variable.If(t))
using (Variable.Switch(q))
//s.SetTo(Variable.Discrete( alpha[Variable.Constant<int>(1),q] ));
s = Variable.Discrete(alpha[Variable.Constant<int>(1), q]);
using (Variable.IfNot(t))
//s.SetTo(Variable.Discrete(alpha[Variable.Constant<int>(0), q]));
s = Variable.Discrete(alpha[Variable.Constant<int>(0), q]);
s.ObservedValue = 4;
InferenceEngine engine = new InferenceEngine();
//Console.WriteLine(engine.Infer(beta));
Console.WriteLine(engine.Infer(alpha));
Console.WriteLine(engine.Infer(theta));
Console.WriteLine(engine.Infer(t));
Console.WriteLine(engine.Infer(q));
Console.WriteLine("Press any key to exit.");
System.Console.ReadKey();
Are my codes correct? The parameters don't seem to change at all after inference
Firstly, you do need the Variable<int> s = Variable.New<int>() and the s.SetTo(Variable.Discrete(alpha[t, q])) if you are dealing with just a single data point (if you have an array, see below).
Secondly, you should use Variational Message passing for this inference rather than Expectation Propagation, as EP will tend to average over the modes.
Thirdly, a single data point seems to be not enough to draw the solution away from the symmetric fixed point. You can either add more data points, or make the priors on beta and theta spikier. The following code works for 10 data points (note that I have slightly simplified the code by encoding beta as an integer variable - though your Variable.If and Variable.IfNot are valid also):
Range two = new Range(2);Range five = new Range(5);Variable<Vector> beta = Variable.Dirichlet(two, new Vector(2, 1.0));Variable<Vector> theta = Variable.Dirichlet(five, new Vector(5, 1.0));VariableArray2D<Vector> alpha = Variable.Array<Vector>(two, five);alpha[two, five] = Variable.DirichletUniform(5).ForEach(two, five);// Breaking SymmetryVector denseBeta = new Vector(5, 10.0);Dirichlet dirich = new Dirichlet(denseBeta);Dirichlet[,] initAlpha = new Dirichlet[2, 5];for (int i = 0; i < 2; i++) for (int j=0; j<5; j++) initAlpha[i, j] = new Dirichlet(dirich.Sample());alpha.InitialiseTo(Distribution<Vector>.Array(initAlpha));var s = Variable.Observed(new int[] { 2, 4, 2, 1, 0, 0, 1, 3, 2, 4 });Range d = s.Range;VariableArray<int> tarr = Variable.Array<int>(d);VariableArray<int> qarr = Variable.Array<int>(d);using (Variable.ForEach(d)){ tarr[ d ] = Variable.Discrete(beta); qarr[ d ] = Variable.Discrete(theta); var t = tarr[ d ]; var q = qarr[ d ]; using (Variable.Switch(t)) using (Variable.Switch(q)) s[ d ] = Variable.Discrete(alpha[t, q]);}InferenceEngine engine = new InferenceEngine(new VariationalMessagePassing());Console.WriteLine(engine.Infer(beta));Console.WriteLine(engine.Infer(alpha));Console.WriteLine(engine.Infer(theta));Console.WriteLine(engine.Infer(tarr));Console.WriteLine(engine.Infer(qarr));
If you make s an array of length 1, you will see it fail again; however, you can deal with this by reducing the effect of the priors by changing the third and fourth lines to:
Variable<Vector> beta = Variable.Dirichlet(two, new Vector(2, 0.1));Variable<Vector> theta = Variable.Dirichlet(five, new Vector(5, 0.1));
you will see that symmetry is broken.
John G.
Thanks for the help. It is part of a larger system, hence, I only use 1 data point here. But I am not quite sure when to use VMP and EP. I have not read their theoretical papers. And for breaking the symmetry, if there are so many parameters in my model, which set of parameters do I need to break symmetry?