Infer.NET Team Blog

Infer.NET team at PDC09

2009-11-12T23:10:00Z

Some lucky members of the Infer.NET team are being let loose from the lab to attend PDC09 and talk about the fun stuff you can do with Infer.NET.

You can find out full details of our session which is called 'Infer.NET: Building Software with Intelligence'.
Update: the video for our session is now up - so you can watch it here if you missed it.

Here's the blurb:

Would you like to write software that can adapt to the user, learn from examples or work with uncertain information? Infer.NET is a machine learning framework that lets you build these capabilities directly into your .NET application. The framework allows you to combine detailed domain knowledge with the latest machine learning algorithms to generate tailored code to solve your problem. An API based on random variables lets you call Infer.NET code from within your application. We provide examples of using Infer.NET in search and gaming.

If you're attending PDC, please do come along - we'll also be at the 'Ask the Experts' event to handle any particular questions you have about Infer.NET.

See you in LA!

The Infer.NET team

Parallel inference in Infer.NET 2.3

2009-08-13T13:55:00Z

Hello Infernauts!!

One of the many changes we made in the recently released Infer.NET 2.3 beta was allowing parallel inference on multiple cores. We are keen to make Infer.NET scale to ever larger datasets and supporting parallelism has always been an important part of that goal. Here I'll show how to use multi-core parallelism to speed up inference - of course, you'll only see the speedup on a multi-core machine.

The first thing you need to do is to install the Microsoft Parallel Extensions CTP. This is the library that Infer.NET uses to do multi-core parallelism. We have chosen to work with the CTP for the time being - the Parallel Extensions library is planned to be part of version 4.0 of the Microsoft .NET framework and we will move to using that version at a future date. You can read more about current and forthcoming Microsoft parallelism technologies at the MSDN Parallel Computing Developer Center.

OK, so you've installed the CTP. Now pick a model you want to parallelise. I'll use the click model which you can run from inside the Infer.NET examples browser, under 'Applications'. Make sure that inference on your model is working as you would expect. Now you can configure your inference engine to use parallel loops by adding the following line:


engine.Compiler.UseParallelForLoops = true;

The click model uses two engines, one for training and one for test so I added this line in both cases. You must configure the inference engine before any calls to Infer() for this to have an effect. The result of this change is that certain for loops in the generated inference code will be replaced with Parallel.For loops. To see how this affects the speed of inference, you can use the handy built-in ability to see timings for various stages of inference:


engine.ShowTimings = true;

Typical results looks on my 8-core machine are shown in the table below. For comparison, I have also shown results for running a Hidden Markov Model on a large data set.

	Time per iteration - Normal	Time per iteration - Parallel For
Click Model Training	218ms	177ms
Click Model Test	22ms	30ms
Hidden Markov Model	5341ms	1352ms

These results may be unexpected. The speed up for click model training is about 20% using parallel for loops but for click model testing, using parallel for loops actually slows things down. This is because parallel for loops introduce additional overhead compared to ordinary for loops, and so you will only get a benefit if the loop is large and a reasonable amount of work is being done inside the loop. In the test case, the iterations are quick and the time is dominated by the parallel overhead. This point is illustrated by the Hidden Markov Model results, where an almost 4x speedup is achieved using parallel for loops - quite a significant improvement for one line of code!!

So, in summary, parallel for loops can speed up inference, especially in larger models, and may speed it up considerably. The exact speedup you will get depends on your hardware, model and inference algorithm. The easiest way to determine it is just to try it out and see.

Whilst this post has discussed multi-core parallelism, it is also possible to distribute Infer.NET inference across a cluster, by dividing your model into chunks using shared variables. At the moment, you have to wire together together the chunks manually e.g. using MPI. We will be looking how to make this more automatic in a future version of Infer.NET and we'll be sure to blog about it when we do!

John W.

Infer.NET 2.3 Beta 1 released

2009-08-11T18:03:00Z

This new release is the first we have done since the January 2.2 Beta release; the version number change reflects the fact that an initial version of Gibbs sampling is now available. Just as importantly, there have been many practical improvements, particularly with respect to indexing and jagged arrays, which enable a larger class of models to be built. Also, some new factors are available including a logistic factor, and there are wrappers to make syntax a bit more natural for F# and for Iron Python. Finally, we have looked into what was preventing Infer.NET running on Linux under Mono (this was a combination of a bug in Mono, and some obfuscation issues), and we have worked around these; we have verified that the examples, at least, now run under Mono - let us know how you get along. There have been one or two minor API changes, but your models should mainly work without change, unless you have been using the InferAll method - more on this later.

The list of changes can be found in in the User Guide, and I will pick out and expand on a few of the changes below.

Gibbs Sampling

This represents the first release of Gibbs sampling within the Infer.NET framework. Gibbs sampling is a very different algorithm from Expectation Propagation and Variational Message Passing (which are approximate deterministic algorithms); it generates a chain of samples from the posterior distribution, and marginal distributions can estimated from the samples lists. Block Gibbs sampling is also supported as part of the implementation. This allows more efficient sampling for highly correlated variables, and is also required for dealing with deterministic factors and constraints.

In Infer.NET, you specify a model in a way that is completely independent of the inference algorithm; it is only when you want to run inference that you specify the algorithm, and your model code is then compiled for that algorithm. The examples browser installed with Infer.NET allows you to choose which algorithm to run on a given model (however, note that not all algorithms support all factors so you will get an error message for certain combinations of model and algorithm). As an example, take a look at the 'Learning a Gaussian with Ranges' example in the tutorial browser; you can run the same model with all three algorithms and compare results.

Here is a very simple example of Gibbs sampling where we infer the mean and precision of a Gaussian. Note how we can recover both the marginal and the chain of samples.

// The model
Variable<double> mean = Variable.GaussianFromMeanAndVariance(0, 100);
Variable<double> prec = Variable.GammaFromShapeAndScale(1, 1);
Range i = new Range(2);
VariableArray<double> data = Variable.Array<double>(i);
data[ i ] = Variable.GaussianFromMeanAndPrecision(mean, prec).ForEach(i);
// The observations
data.ObservedValue = new double[ ] { 5.0, 7.0 };
// The model
InferenceEngine engine = new InferenceEngine(new GibbsSampling());
var meanMarg = engine.Infer<Gaussian>(mean);
var meanSamples = engine.Infer<SampleList<double>>(mean, QueryTypes.Samples);
var precMarg = engine.Infer<Gamma>(prec);
var precSamples = engine.Infer<SampleList<double>>(prec, QueryTypes.Samples);

Note that Gibbs sampling is a work in progress, and some factors are not yet supported. In particular, Gates are not yet handled, so you will not be able to use Variable.Switch statements or Variable.If statements. Also, factors which take an array argument such as 'Sum' are not yet supported. Also on our to-do list are some speed improvements in the generated code, especially with respect to array variables.

Improvements in jagged arrays and indexing

Most real world problems have complex and irregular dependencies between variables. Support for such problems is now greatly improved, the generated code is more efficient, and several bugs have been fixed.

As an example, consider a social network. We might have various observations about the individuals in the social network, and the goal is to infer something about the link between two individuals. Variable ranges, which were available in Infer.NET 2.2 can be used to encode the graph structure - so we can range over all users, and for each user range over that user's friends. Improvements now allow edge indexing to be encoded via a 2D observed index array, and allow switching based on that edge index so that you can encode latent variables on the edges of the social network.

A later blog will look at these types of model in more detail.

API changes

API changes have occurred in two areas. First within the Infer and InferAll methods, and secondly in the evidence operators. You won't need to worry about the latter unless you've been writing your own factors and operators, so I'll concentrate on the former.

The Infer function (a method on the InferenceEngine) was previously a bit tricky to use with variable arrays (especially complex arrays such as jagged arrays of 2D arrays). The return types in these case were (and still are) quite difficult to understand and work with; each return type is a form of DistributionArray which can be thought of as a distribution over an array domain. It is more natural and less confusing to work with .NET arrays of distributions over non-array domains, and this is now the recommended usage (though the old will still work).

For example, suppose you have a jagged variable array of type

VariableArray<VariableArray<double>, double[][]>

where the individual items are drawn from Gaussians. You can now get the inference results as a jagged array of Gaussians as follows:

Gaussian[][] posterior = engine.Infer<Gaussian[][]>(w);

As part of the overhaul of the inference methods, the InferAll method no longer returns anything. You can call it as before, and it will have the same behaviour as before, and run the inference for the specified variables, but you will not be able to assign the results. To retrieve the results, just call individual Infers for each variable you are interested in; if this variable was specified in a previous call to InferAll, the marginal will be returned immediately without further computation.

Language Wrappers

The extensive use of generic methods in Infer.NET can make calling it from F# a bit cumbersome and unintuitive. A set of wrapper types for distribution arrays (for example GaussianArrayOfArray) and variable arrays (for example VariableGaussianArrayOfArray) are provided for commonly-used distributions and domain types. Also, the wrapper provides alternative operators for cases where Infer.NET operator overloads are not recognised; for example the "<" operator (though supported in Infer.NET for creating a boolean variable from two real variables) is not recognised when calling from F# - the wrapper provides "<<" as an alternative. Another part of the wrapper provides more F#-like syntax for working with Infer.NET if blocks and switch blocks. The F# wrapper is documented in the User Guide, and distributed as a DLL (Infer.FSharp.dll) and as a source file (FSharpWrapper.fs) - these can be found under the bin and source folders respectively in the installation folder.

There is also a wrapper for IronPython, which is distributed as IronPythonWrapper.py under the source folder under the installation folder. This wrapper is only useful/tested for IronPython 2.6 Beta 2 and above.

Well, that's a summary of some of the more visible changes. There have, of course, been many changes under the hood, fixing bugs, and making things more efficient. We plan to do some more focussed blogs on individual areas in the coming weeks, so stay tuned.

John G. and the Infer.NET team.

Bayesian PCA

2009-02-03T18:33:00Z

It’s been a couple of months now since we’ve released Infer.NET to the outside world, and a blog is overdue. I hope those of you have downloaded it are enjoying it and have had a chance at least to go through some of the tutorials to see the scope of what we’re providing. In the process of doing this release, one point of discussion has been the question of to what level we should provide and/or support standard black box models. Here I’m talking about the likes of Bayesian factor analysis, Bayesian PCA, HMMs, Bayesian neural nets, and the Bayes point machine. We have been reluctant to provide these as stock models because we have been keen that Infer.NET should be applied in a more customised way than just throwing data at a generic model. The click through model is an excellent example of a thoughtful customised model which tries to capture the behaviour of a user interacting with a page of results returned by a search engine.

That said, it is often useful in the early stages for a user to look at and employ some standard models just to get started. With the release, we have already provided a set of multi-class Bayes point machine examples which can be used for a variety of classification problems. In this blog I will look at Bayesian principal components analysis – see Chris Bishop’s book Pattern Recognition and Machine Learning for example. Most of you will be familiar with standard Principal Component Analysis which is a widely used technique for dimensionality reduction. Given an observation space of dimension D we would like to find a latent space of lower dimension M (the ‘principal space’) which captures most of the data. The mapping from principal space to observation space is via a linear mapping represented by a mixing matrix W. PCA can be formulated in probabilistic way (‘probabilistic PCA’) and this in itself gives many benefits as itemised in Bishop. If in addition we adopt a Bayesian approach to inferring W, we can also determine the optimal value for M using ‘automatic relevance determination’ (ARD).

A fully Bayesian treatment of probabilistic PCA (including ARD) is quite complex and it might take several weeks of work to implement an efficient deterministic inference algorithm. In Infer.NET, defining the model is 20 to 25 lines of code and a few hours’ work.

I am going to plunge straight in and show the factor graph for Bayesian PCA. Factor graphs are a great representation for probabilistic models, and especially useful if you are going to build models in Infer.NET. If you can draw a factor graph, you should be able to code up the model in Infer.NET (subject to the currently available distributions, factors, and message operators).

I’ll jointly explain aspects of the factor graph and Bayesian PCA as I go along. As a reminder, the first thing to note is that the variables in the problem are represented as circles/ovals, and the factors (functions, distributions, or constraints) are represented as squares/rectangles. Each factor is only ever attached to its variables and each variable is only ever attached to its factors. The big enclosing rectangles represent ‘plates’ and show replications of variables and factors across arrays. Factor graphs are important both because they provide a clarifying visualisation of complex probabilistic models and because of the close relationship with the underlying inference and sampling algorithms (particularly those based on local message passing).

The accompanying factor graph shows PCA as a generative model for randomly generating the N observations – these observations can be represented as an NxD array, and so are represented in the factor graph in the intersection of two plates of size N and D respectively. I have explicitly shown all indices in the factor graph to make the comparison with the code clearer. The generative process starts at the top left where, for each observation and for each component in latent space, we sample from a standard Gaussian to give an NxM array Z (with elements z_nm).

In the top right we have the matrix variable W (elements w_nm) which maps latent space to observation space – so this will be an MxD matrix which will post-multiply Z. We have specified a maximum M in advance, but what we really want to do is learn the ‘true’ number of components that explain the data, and we do this by drawing the rows of W from a zero-mean Gaussian whose precisions a (elements a_m) we are going to learn. If the precision on a certain row becomes very large (i.e. the variance becomes very small) then the row has negligible effect on the mapping and the effective dimension of the principal space is reduced by the number of such rows. The elements of a are drawn from a common Gamma prior.

The MatrixMultiply factor multiplies the Z with W; as this factor takes two double arrays of variables, it sits outside the plates. The result of this factor is an NxM matrix variable T (elements t_nm). We then add a bias vector variable m (elements m_d drawn from a common Gaussian prior) to each row of T to give matrix variable U (elements u_nm). Finally, the matrix X of observations vector is generated by drawing from Gaussians with mean U and precision given by the vector variable p (elements p_d drawn from a common Gamma prior). The elements in this final array of Gaussian factors are the likelihood factors and represent the likelihood of the observations conditional on the parameters of the model. These parameters are W, m and p which we want to infer.

This model is captured in the following few lines of C# code which, as you can hopefully see, directly matches the factor graph (I have omitted the variable and range declarations as these don’t add further insight, but the full code can be found here):

// Mixing matrix
vAlpha = Variable.Array<double>(rM).Named("Alpha");
vW = Variable.Array<double>(rM, rD).Named("W");
vAlpha[rM] = Variable.Random<double, Gamma>(priorAlpha).ForEach(rM);
vW[rM, rD] = Variable.GaussianFromMeanAndPrecision(0, vAlpha[rM]).ForEach(rD);

// Latent variables are drawn from a standard Gaussian
vZ = Variable.Array<double>(rN, rM).Named("Z");
vZ[rN, rM] = Variable.GaussianFromMeanAndPrecision(0.0, 1.0).ForEach(rN, rM);

// Multiply the latent variables with the mixing matrix...
vT = Variable.MatrixMultiply(vZ, vW).Named("T");
// ... add in a bias ...
vMu = Variable.Array<double>(rD).Named("mu");
vMu[rD] = Variable.Random<double, Gaussian>(priorMu).ForEach(rD);
vU = Variable.Array<double>(rN, rD).Named("U");
vU[rN, rD] = vT[rN, rD] + vMu[rD];
// ... and add in some observation noise ...
vPi = Variable.Array<double>(rD).Named("pi");
vPi[rD] = Variable.Random<double, Gamma>(priorPi).ForEach(rD);
// ... to give the likelihood of observing the data
vData[rN, rD] = Variable.GaussianFromMeanAndPrecision(vU[rN, rD], vPi[rD]);

In order to test out this model, I generated 1000 data points from a model with M=3, D= 10. However, when doing inference, I gave it M=6 to see if the inference could correctly determine the number of components. Here is the print-out from a run. As you can see, as expected, only three of the rows of W have significant non-zero entries, and the bias and noise vectors are successfully inferred.

Mean absolute means of rows of W: 0.05 0.24 0.20 0.02 0.21 0.01

True bias: -0.95 0.75 -0.20 0.20 0.30 -0.35 0.65 0.20 0.25 0.40
Inferred bias: -0.90 0.70 -0.22 0.22 0.31 -0.37 0.65 0.17 0.25 0.42
True noise: 1.00 2.00 4.00 2.00 1.00 2.00 3.00 4.00 2.00 1.00
Inferred noise: 0.98 1.99 4.09 2.00 1.08 2.00 2.70 3.51 2.16 1.01

The full C# code for this example can be found here. This code provides a class (BayesianPCA) for constructing a general Bayesian PCA model for which we can specify priors and data at inference time. There is also some example code for running inference and extracting marginals. One important aspect of running inference on this model is that we initially need to break symmetries in the model by initialising the W marginals to random values using the InitialiseTo() method – for another example of this see the Mixture of Gaussians tutorial.

I hope this provides a useful addition to your Infer.NET toolbox and also provides some insight into building models with Infer.NET. Keep the feedback and comments coming.

John Guiver and the Infer.NET team.

Infer.NET released!!

2008-12-09T02:33:00Z

It's a very exciting week for the Infer.NET team as we go live with our first public release, which you can get hold of right now!

This release represents many years of work and we believe it is a significant step forward in how automatic inference is performed. Here's a quick summary of what you get:

Rich modelling language: support for univariate and multivariate variables, both continuous and discrete. Models can be constructed from a broad range of factors including arithmetic operations, linear algebra, range and positivity constraints, Boolean operators, Dirichlet-Discrete, Gaussian,and mixtures or hierachical mixtures of any of these.
Multiple inference algorithms: built-in algorithms include expectation propagation, belief propagation (a special case of EP) and variational message passing (variational Bayes).
Designed for large scale inference: Infer.NET is architected as a compiler allowing for very efficient inference through generated code. In addition, the generated code can be viewed, stepped through, profiled or modified as needed, using standard development tools.
User-extendable: a plug-in architecture which makes Infer.NET open-ended and adaptable. Probability distributions, factors, message-passing operations and inference algorithms can all be added by the user.

We have written a lot of documentation on how the framework works and how you can add your own components to it. There are also lots of examples - ranging from simple explanatory tutorials to full applications - which should help you get into Infer.NET and then gain a deeper understanding of how it works.

Although we have spent many months of testing Infer.NET internally, it is inevitable that there will be some teething problems, with the system being used in ways which we have not anticipated. For this reason, we are initially releasing the system as a 'beta' and we are looking for lots of feedback from the community (i.e. you!) on how well it is working and what you are using it for. If you do encounter a problem, please raise it on the forum and we will aim to respond as quickly as we can.

We really hope that Infer.NET will prove to be a valuable platform for machine learning researchers and practitioners. Your comments and feedback will help us to make Infer.NET the machine learning solution of choice!

Thanks!

The Infer.NET team