Yes, this will be supported in the next beta release.
Yes, there is no indeterminacy in the weights when you impose a proper prior.
Hi John G,
Thanks for the BernoulliFromLogOdds factor and the example. The Bayesian logistic regression model seems to work very well on a binary classification task. The weights inferred make sense.
I was wondering whether ti's straightforward to extend the model to a softmax or multiclass logistic regression model. I would imagine that instead of doing Variable.BernoulliFromLogOdds(w), I would have to impose pairwise constraints on a set of weights and the training vectors.
Any help (including code snippets) would be greatly appreciated.
Thanks,
Vincent
You can do multiclass logistic regression in a very similar way as the multiclass Bayes point machine. If there are C classes, you would have C weight vectors. Given an input vector, take inner products to get C scores. During training, if the correct class is 1 you create C-1 booleans indicating whether score 1 is greater than score j for all j > 1. In the Bayes point machine, these are explicit constraints of the form score[0] > score[j]. For logistic regression, these are soft constraints defined by BernoulliFromLogOdds(score[1] - score[j]). These booleans are all observed to be true. Mathematically, this model is similar to a softmax likelihood but not exactly the same. To do exact softmax, you would need the new factor that we are adding in the next version.
By the way, the code that John Guiver sent out for using the BernoulliFromLogOdds factor is not quite right. Because this factor is not completely integrated, what you need to say is:
Variable<bool> y = Variable<bool>.Factor(Factor.BernoulliFromLogOdds,w);instead of:
Variable<bool> y = Variable.BernoulliFromLogOdds(w);Tom
Infer.NET 2.3 now has built-in support for logistic/softmax regression and binomial/multinomial observations under VMP.
I’ll be sure to keep everyone posted.
Hi Tom,
Thanks for softmax regression factor. I tried to use it but somehow I get improper message exceptions or syntax problems. Can you please look at the relevant parts of my code. Thanks!
// The model
for (int c = 0; c < nClass; c++) { nItem[ c ] = Variable.New<int>().Named("nItem_" + c); item[ c ] = new Range(nItem[ c ]).Named("item_" + c); xValues[ c ] = Variable.Array<Vector>(item[ c ]).Named("xValues_" + c); using (Variable.ForEach(item[ c ])) { score = ComputeClassScores(weights, xValues[ c ][item[ c ]], rC); noisyScore = AddNoiseToScore(score, vPi, rC); ConstrainForLogisticRegression(c, noisyScore, rC); } }
private void ConstrainForLogisticRegression(int argmax, VariableArray<double> noisyScore, Range rC) { Vector v = new Vector(rC.SizeAsInt); for (int i = 0; i < rC.SizeAsInt; i++)v[ i ]=1.0; Variable<Vector> softProbs = Variable.Dirichlet(v); softProbs = Variable.Softmax(noisyScore); Vector hardVector = new Vector(rC.SizeAsInt); for (int i = 0; i < rC.SizeAsInt; i++) if (i == argmax) hardVector[ i ] = 1.0; Variable.ConstrainEqual(hardVector, softProbs); // If I do this, I get complains that the softmax factor needs a Dirichlet instead of vector //softProbs.ObservedValue = hardVector; // If I do this, I get improper message exceptions }
private VariableArray<double> ComputeClassScores(VariableArray<Vector> w, Variable<Vector> xValues, Range rC) { VariableArray<double> score = Variable.Array<double>(rC); score[rC] = Variable.InnerProduct(w[rC], xValues); return score; }
private VariableArray<double> AddNoiseToScore(VariableArray<double> score, Variable<double> prec, Range rC) { VariableArray<double> noisyScore = Variable.Array<double>(rC); noisyScore[rC] = Variable.GaussianFromMeanAndPrecision(score[rC], prec); return noisyScore; }
You want to use Variable.DiscreteFromLogProbs here, not Variable.Softmax. See the documentation for both functions.