Information theory for cognitive sciences

The course introduces the basic concepts and measures of information theory. It provides an overview of its practical applications within cognitive sciences and other related fields such as biology, linguistics and social sciences. Limitations of the information theory as applied to cognitive sciences will also be addressed, basing on current discussions of the semantic aspects of information and the link between information-theoretical and thermodynamic entropy.

The course introduces the basic concepts and measures of information theory. It provides an overview of its practical applications within cognitive sciences and other related fields such as biology, linguistics and social sciences.

The first part of the class will be devoted to the most important concepts of the theory (entropy, information measures, coding and compression of information, complexity). Next we will present applications of the measures designed within the theory of information in physics, biology, neuropsychology, linguistics. Finally, the limitations of information theory as applied to cognitive sciences will also be addressed, basing on current discussions of the semantic or pragmatic aspects of information and the link between information-theoretical and thermodynamic entropy.

1. Introduction. Range of problems for information theory. Communication and Information.

a. Gleick: Information: Prologue, Chapters 1,6,7; optional: Chapter 8,9,13.

b. Additional readings/tutorials on probability theory and logarithms.

2. Mathematical bases of information theory: entropy, conditional entropy, mutual information

Cover, T. M., Thomas, J. A. (2006). “Elements of Information Theory”, New York: Wiley. Chapter 2.

3. Mathematical bases of information theory: relative entropy, divergences & data processing inequality

a. Cover, T. M., Thomas, J. A. (2006). “Elements of Information Theory”, New York: Wiley; Chapter 1&2.

b. Kullback-Leibler divergence: formulation & applications (presenter, discussion leader)

c. Data Processing Inequality: principle & applications (presenter, discussion leader)

d. Sufficient statistics and Maximum Entropy Principle

4. Foundations of Information Theory

a. Weaver, W. Recent contributions to the mathematical theory of communication. & Shannon, C. “The Mathematical Theory of Communication” (selected fragments). In: Shannon, C. & Weaver, In: The Mathematical Theory of Communication. The University of Illinois Press: Urbana. (presenters, discussion leaders)

b. Brillouin, L. (1969). Nauka a Teoria Informacji. Rozdział 1.

5. Applications: Neurobiology

Ta. ononi, G.; Edelman, G.M.; Sporns, O. Complexity and coherency: Integrating information in the brain. Trends Cognitive Science 1998, 2, 474–484 (fragments - till p. 480, without “Reconciling information processing and information storage: matching complexity”)

(presenters, discussion leaders)

6. Applications: Language

a. Zipf, G. K. (1964). „The Psychobiology of Language: an introduction to dynamic Philology.” Rozdział II „The Form and Behavior of Words” (recommended)

b. Piantadosi, S.T. “2014 Zipf ’s word frequency law in natural language: A critical review and future directions” Psychon Bull Rev (presenter, discussion leader)

c. Coupé et al. (2019). Different languages, similar encoding efficiency: Comparable information rates across the human communicative niche. Science Advances, Vol. 5, no. 9, eaaw2594, DOI: 10.1126/sciadv.aaw2594 (presenter, discussion leader)

7. Applications: Networks & structure

a. Introduction to Networks

b. Klein, B., & Hoel, E. (2020). The Emergence of Informative Higher Scales in Complex Networks. Complexity, 2020, 1–12.

(presenter, discussion leader)

8. Algorithmic information theory

a. Cover, T. M., Thomas, J. A. (2006). “Elements of Information Theory”, Chapter 14 (fragments).

b. Intro to algorithmic information theory

9. Complexity Measures I

a. Soler-Toscano, F., Zenil, H., Delahaye, J.-P., & Gauvrit, N. (2014). Calculating Kolmogorov Complexity from the Output Frequency Distributions of Small Turing Machines. PLoS ONE, 9(5), e96223. https://doi.org/10.1371/journal.pone.0096223

(presenter, discussion leader)

Gauvrit, N., Zenil, H., Soler-Toscano, F., Delahaye, J.-P., & Brugger, P. (2017). Human behavioral complexity peaks at age 25. PLOS Computational Biology, 13(4), e1005408. https://doi.org/10.1371/journal.pcbi.1005408

(1 presenter, 1 discussion leader)

10. Complexity measures II: Information, energy, cognition

a. Lloyd & Pagels, 1988, Complexity as Thermodynamic Depth, ANNALS OF PHYSICS 188, str. 186-191 (recommended).

b. Klamut, Kutner & Struzik, 2020, Towards a universal measure of complexity, Entropy (recommended)

c. Deacon, T. & Koutroufinis, S. (2014). Complexity and Dynamical Depth. Information, 5, 404-423. (presenters, discussion leader)

11. Can Shannon information be a basis for semantic information?

a. Hasselman, F. (2022). Radical embodied computation: Emergence of meaning through the reproduction of similarity by analogy (...)) (recommended)

b. Isaac, A. (2019) The Semantics Latent in Shannon Information.

(presenters, discussion leader)

12 – 14. Project presentations @ mini conference

Students after completing the course will be able to:

Describe the history of the notion of the quantity of information, efforts to formalize it in relation to research on and modeling of cognitive systems. (K_W01, K_W02)

Define the most important concepts of the theory (information quantity, entropy, mutual and joint information) and use mathematical formulas to compute them. (K_W01, K_W08)

Indicate the main areas of application for the measures of information within cognitive sciences and related disciplines. (K_W02, K_U01, K_U03, K_K02)

Use terminology pertaining to information theory and its application. (K_W08)

Discuss information-theoretic problems with specialists from other fields. (K_U07, K_K07)

Formulate questions within cognitive science, which can be answered using information-theoretic measures. (K_U02; K_K06)

40% Project and its presentation

30% Short paper presentation and guiding the discussion

20% Homework(s)

10% Class presence and active participation

Attendance to the seminar is obligatory, 2 unexcused absences are allowed.

Students must respect the principles of academic integrity. Cheating and plagiarism (including copying work from other students, internet or other sources) are serious violations that are punishable and instructors are required to report all cases to the administration.

CMP I, philosophy of mind course, basic skills in math, probability