In Shannon’s original paper and textbooks in information theory, the entropy of a discrete random variable is assumed or shown to be a continuous function. However, we found that all Shannon’s information measures including entropy and mutual information are discontinuous in the general case that the random variables take values in possibly countably infinite alphabets. This fundamental property explains why strong typicality and Fano’s inequality can only be applied on finite alphabets. Note that strong typicality and Fano’s inequality have wide applications in information theory so that it is important to extend them in full generality. In this talk, details about the discontinuity of Shannon’s information measures will be given. We will show how these results lead to a new definition of typicality and an inequality tighter than Fano’s inequality. Applications in network coding and information theoretic security will be discussed.
Raymond W. Yeung received the BS, MEng and PhD degrees in electrical engineering from Cornell University in 1984, 1985, and 1988, respectively. He was with AT&T Bell Laboratories from 1988 to 1991. Since 1999, he has been with Department of Information Engineering at The Chinese University of Hong Kong, where he is currently a chair professor. He is the author of the books A First Course in Information Theory (Kluwer 2002) and Information Theory and Network Coding (Springer 2008). His research interest is in information theory and network coding. He was a consultant in a project of Jet Propulsion Laboratory for salvaging the malfunctioning Galileo Spacecraft. Professor Yeung is a member of the Board of Governors of the IEEE Information Theory Society from 1999 to 2001. He has served on the committees of a number of information theory symposiums and workshops. He also has served on the editorial board of several academic journals. He was a recipient of the Croucher Senior Research Fellowship for 2000/01, the Best Paper Award (Communication Theory) of the 2004 International Conference on Communications, Circuits and System, the 2005 IEEE Information Theory Society Paper Award, and the Friedrich Wilhelm Bessel Research Award from the Alexander von Humboldt Foundation in 2007. He is a Distinguished Lecturer of IEEE Information Theory Society for 2011-12.
