I think the OP is referring to algorithmic probability. See
[1] R. J. Solomonoff. A formal theory of inductive inference: Parts 1 and 2. Information and Control, 7:1--22 and 224--254, 1964.
[2] R. J. Solomonoff. Complexity-based induction systems: Comparisons and convergence theorems. IEEE Transactions on Information Theory, IT-24:422-432, 1978.
[3] http://www.scholarpedia.org/article/Algorithmic_probability