Abstract

Dense datasets produced by high-throughput technologies allow for the objective determination of Bayesian priors when understood as solutions to Fredholm's integral equation. In order to select an appropriate methodology, we conducted a comprehensive review of alternative derivations for the Bayesian prior and posterior update rules comprising: classical iterative alternating algorithms, information theory-motivated variational approaches, and contemporary convex geometry methods. All of these share the key concepts of Kullback-Leibler (KL) relative entropy and of variational minimization. Since the KL relative entropy is a Bregman divergence allowing only linear exploration of tangent hyperplanes of the convex entropy functional, the resulting algorithms – reexpressed in terms of alternating Bregman projections – are necessarily iterative in nature. It has been recently argued that the prior can often only be understood in the context of the model likelihood/kernel, but iterative algorithms obscure the prior dependency on the kernel. In contrast, Jaynes maximal entropy principle yields data-constrained structural priors which make very explicit their kernel dependency, with the modelled density entropic convex dual – the convex analysis equivalent of coefficients from a Fourier decomposition – providing the model kernel weight function. Jaynesian-Bayesian modelling of dense datasets null hypothesis statistical testing (NHST) p-value distributions in terms of the Bayes factor BF(p) allows for a natural extension of the NHST framework. We illustrate all concepts in terms of newly derived analytical expressions for the Fisher-Student-Snedecor's t and F noncentral hypersphere distributions which afford informative representations on compact 2D spaces.

Author information

Contact details are reproduced from the original publication and may be historical.

Keywords

  • Fredholm equation
  • Kullback-Leibler relative entropy
  • Bregman divergence
  • Bregman projection
  • mirror update
  • Bayes-Jaynes prior

Mathematics Subject Classification

45B05, 94A17, 49N15, 62C10, 60E05

References

  1. R. C. Aster, B. Borchers, C. H. Clifford: Parameter Estimation and Inverse Problems, Elsevier, Amsterdam (2018).https://doi.org/10.1016/B978-0-12-804651-7.00015-8Google Scholar
  2. A. Banerjee, X. Guo, H. Wang: On the optimality of conditional expectation as a Bregman predictor, IEEE Trans. Inform. Theory 51/7 (2005) 2664–2669.https://doi.org/10.1109/TIT.2005.850145Google Scholar
  3. M. Basseville: Divergence measures for statistical data processing – an annotated bibliography, Signal Processing 93/4 (2013) 621–633.https://doi.org/10.1016/j.sigpro.2012.09.003Google Scholar
  4. H. H. Bauschke, A. S. Lewis: Dykstras algorithm with Bregman projections: a convergence proof, Optimization 48/4 (2000) 409–427.https://doi.org/10.1080/02331930008844513Google Scholar
  5. P. G. Bissiri, C. C. Holmes, S. G. Walker: A general framework for updating belief distributions, J. Royal Stat. Society, Ser. B 78/5 (2016) 1103–1130.https://doi.org/10.1111/rssb.12158Google Scholar
  6. J. M. Borwein: Maximum entropy and feasibility methods for convex and nonconvex inverse problems, Optimization 61/1 (2012) 1–33.https://doi.org/10.1080/02331934.2011.632502Google Scholar
  7. J. M. Borwein, D. R. Russell: Duality and convex programming, in: Handbook of Mathematical Methods in Imaging, O. Scherzer (ed.), Springer, New York (2011) 229–270.https://doi.org/10.1007/978-0-387-92920-0_7Google Scholar
  8. L. M. Bregman: The relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming, USSR Comput. Math. Math. Physics 7/3 (1967) 200–217.https://doi.org/10.1016/0041-5553(67)90040-7Google Scholar
  9. D. Calvetti, E. Somersalo: Inverse problems: From regularization to Bayesian inference, Wiley Interdisciplinary Reviews: Comp. Stat. 10/3 (2018) e1427.https://doi.org/10.1002/wics.1427Google Scholar
  10. A. Caticha, R. Preuss: Maximum entropy and Bayesian data analysis: entropic prior distributions, Phys. Review E 70/4 (2004) 046127.https://doi.org/10.1103/PhysRevE.70.046127Google Scholar
  11. Y. Censor, S. Reich: The Dykstra algorithm with Bregman projections, Comm. Appl. Analysis 2/3 (1998) 407–420.Google Scholar
  12. M. Chae, R. Martin, S. G. Walker: On an algorithm for solving Fredholm integrals of the first kind, Stat. Computing (2018) 1–10.https://doi.org/10.1007/s11222-018-9829-zGoogle Scholar
  13. I. Csiszár, G. Tusnády: Information geometry and alternating minimization procedures, Stat. Decisions 1 (1984) 205–237.Google Scholar
  14. S. Dudoit, J. P. Shaffer, J. C. Boldrick: Multiple hypothesis testing in microarray experiments, Stat. Science 18/1 (2003) 71–103.https://doi.org/10.1214/ss/1056397487Google Scholar
  15. R. L. Dykstra: An iterative procedure for obtaining I-projections onto the intersection of convex sets, Annals of Probability (1985) 975–984.https://doi.org/10.1214/aop/1176992918Google Scholar
  16. B. Efron: Size, power and false discovery rates, Annals of Statistics 35/4 (2007) 1351–1377.https://doi.org/10.1214/009053606000001460Google Scholar
  17. B. A. Frigyik, S. Srivastava, M. R. Gupta: Functional Bregman divergence and Bayesian estimation of distributions, IEEE Trans. Inform. Theory 54/11 (2008) 5130–5139.https://doi.org/10.1109/TIT.2008.929943Google Scholar
  18. A. Gelman, D. Simpson, M. Betancourt: The prior can often only be understood in the context of the likelihood, Entropy 19/10 (2017) 555.https://doi.org/10.3390/e19100555Google Scholar
  19. T. Gneiting, A. E. Raftery: Strictly proper scoring rules, prediction, and estimation, J. Amer. Stat. Assoc. 102/477 (2007) 359–378.https://doi.org/10.1198/016214506000001437Google Scholar
  20. P. D. Grünwald, A. P. Dawid: Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory, Annals of Statistics 32/4 (2004) 1367–1433.https://doi.org/10.1214/009053604000000553Google Scholar
  21. J. Hadamard: Sur les problèmes aux dérivées partielles et leur signification physique, Princeton Univ. Bulletin (1902) 49–52.Google Scholar
  22. D. Hestenes: Review of the book “E. T. Jaynes: Papers on Probability, Statistics, and Statistical Physics” by R. D. Rosenkrantz (ed.), Foundations of Physics 14/2 (1984) 187–191.Google Scholar
  23. E. T. Jaynes: On the rationale of maximum-entropy methods, Proc. IEEE 70/9 (1982) 939–952.https://doi.org/10.1109/PROC.1982.12425Google Scholar
  24. L. K. Jones, C. L. Byrne: General entropy criteria for inverse problems, with applications to data compression, pattern classification, and cluster analysis, IEEE Trans. Inform. Theory 36/1 (1990) 23–30.https://doi.org/10.1109/18.50370Google Scholar
  25. J. Khan, J. S. Wei, M. Ringner, L. H. Saal, M. Ladanyi, F. Westermann, F. Berthold, M. Schwab, C. R. Antonescu, C. Peterson, P. S. Meltzer: Classification and diagnostic prediction of cancers using gene expression profiling and artificial neural networks, Nature Medicine 7/6 (2001) 673–679.https://doi.org/10.1038/89044Google Scholar
  26. B. Kulis, M. A. Sustik, I. S. Dhillon: Low-rank kernel learning with Bregman matrix divergences, J. Machine Learning Res. 10 (2009) 341–376.Google Scholar
  27. S. Kullback, R. A. Leibler: On information and sufficiency, Ann. Math. Statistics 22/1 (1951) 79–86.https://doi.org/10.1214/aoms/1177729694Google Scholar
  28. G. Le Besnerais, J.-F. Bercher, G. Demoment: A new look at entropy for solving linear inverse problems, IEEE Trans. Inform. Theory 45/5 (1999) 1565–1578.https://doi.org/10.1109/18.771159Google Scholar
  29. R. Le Blanc: Bayesian analysis on a noncentral Fisher-Student’s hypersphere, Amer. Statistician 73/2 (2019) 126–140.https://doi.org/10.1080/00031305.2017.1377111Google Scholar
  30. Y. Lucet: What shape is your conjugate? A survey of computational convex analysis and its applications, SIAM Review 52/3 (2010) 505–542.https://doi.org/10.1137/100788458Google Scholar
  31. P. G. L. Mana: Geometry of maximum-entropy proofs: stationary points, convexity, Legendre transforms, exponential families, arXiv: 1707.00624 (2017).Google Scholar
  32. S. B. McGrayne: The Theory that Would Not Die: How Bayes’ Rule Cracked the Enigma Code, Hunted Down Russian Submarines, and Emerged Triumphant from Two Centuries of Controversy, Yale University Press, New Haven (2011).Google Scholar
  33. F. Nielsen: Legendre transformation and information geometry, https://www2.sonycsl.co.jp/person/nielsen/Note-LegendreTransformation.pdf (2010).Google Scholar
  34. V. C. Preda: The Student distribution and the principle of maximum entropy, Ann. Inst. Statist. Math. 34/1 (1982) 335–338.https://doi.org/10.1007/BF02481032Google Scholar
  35. M. D. Reid, R. M. Frongillo, R. C. Williamson, N. Mehta: Generalized mixability via entropic duality, arXiv: 1406.6130 (2014).Google Scholar
  36. C. E. Shannon: A mathematical theory of communication, Bell Sys. Tech. J. 27/3 (1948) 379–423.https://doi.org/10.1002/j.1538-7305.1948.tb01338.xGoogle Scholar
  37. A. M. Stuart: Inverse problems: a Bayesian perspective, Acta Numerica 19 (2010) 451–559.https://doi.org/10.1017/S0962492910000061Google Scholar
  38. B. Taskar, S. Lacoste-Julien, M. I. Jordan: Structured prediction, dual extragradient and Bregman projections, J. Machine Learning Res. 7 (2006) 1627–1653.Google Scholar
  39. A. N. Tikhonov, Y. Y. Arsenin: Solutions for Solving Ill-Posed Problems, John Wiley and Sons, New York (1977).Google Scholar
  40. B. C. Vemuri, M. Liu, S.-I. Amari, F. Nielsen: Total Bregman divergence and its applications to DTI analysis, IEEE Trans. Medical Imaging 30/2 (2011) 475–483.https://doi.org/10.1109/TMI.2010.2086464Google Scholar
  41. S. G. Walker: Bayesian inference via a minimization rule, Sankhyā: Indian J. Statistics (2006) 542–553.Google Scholar
  42. P. M. Williams: Bayesian conditionalisation and the principle of minimum information, British J. Phil. Science 31/2 (1980) 131–144.https://doi.org/10.1093/bjps/31.2.131Google Scholar
  43. A. Zellner: Optimal information processing and Bayes’s theorem, Amer. Statistician 42/4 (1988) 278–280.https://doi.org/10.1080/00031305.1988.10475585Google Scholar
  44. J. Zhu, N. Chen, E. P. Xing: Bayesian inference with posterior regularization and applications to infinite latent SVMs, J. Machine Learning Res. 15/1 (2014) 1799–1847.Google Scholar

Suggested citation

R. Le Blanc. “Entropic Convex Duality in the Determination of Data-Constrained Kernel-Based Bayes-Jaynes Priors.” Journal of Convex Analysis 29 (2022), No. 2, 623–647. https://doi.org/10.68381/jca29036

Download BibTeXPublished by Heldermann Verlag.

Published by Heldermann Verlag, 2022. Rights now held by Banach Press LLC.