Universal estimation of directed information

Jiantao Jiao, Haim H. Permuter, Lei Zhao, Young Han Kim, Tsachy Weissman

Research output: Contribution to journalArticlepeer-review


Four estimators of the directed information rate between a pair of jointly stationary ergodic finite-alphabet processes are proposed, based on universal probability assignments. The first one is a Shannon-McMillan-Breiman-type estimator, similar to those used by Verdú in 2005 and Cai in 2006 for estimation of other information measures. We show the almost sure and L 1 convergence properties of the estimator for any underlying universal probability assignment. The other three estimators map universal probability assignments to different functionals, each exhibiting relative merits such as smoothness, nonnegativity, and boundedness. We establish the consistency of these estimators in almost sure and L1 senses, and derive near-optimal rates of convergence in the minimax sense under mild conditions. These estimators carry over directly to estimating other information measures of stationary ergodic finite-alphabet processes, such as entropy rate and mutual information rate, with near-optimal performance and provide alternatives to classical approaches in the existing literature. Guided by these theoretical results, the proposed estimators are implemented using the context-tree weighting algorithm as the universal probability assignment. Experiments on synthetic and real data are presented, demonstrating the potential of the proposed schemes in practice and the utility of directed information estimation in detecting and measuring causal influence and delay.

Original languageEnglish
Article number6555871
Pages (from-to)6220-6242
Number of pages23
JournalIEEE Transactions on Information Theory
Issue number10
StatePublished - 25 Sep 2013


  • Causal influence
  • context-tree weighting (CTW)
  • directed information
  • rate of convergence
  • universal probability assignment

All Science Journal Classification (ASJC) codes

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences


Dive into the research topics of 'Universal estimation of directed information'. Together they form a unique fingerprint.

Cite this