Undergraduate Teaching 2022-23

Engineering Tripos Part IIB, 4F5: Advanced Information Theory and Coding, 2018-19

Engineering Tripos Part IIB, 4F5: Advanced Information Theory and Coding, 2018-19

Not logged in. More information may be available... Login via Raven / direct.

PDF versionPDF version


Dr J Sayir


Dr J Sayir, Prof I Kontoyiannis

Timing and Structure

Lent term. 16 lectures. Assessment: 100% exam


3F7 assumed, 3F1, 3F4 useful but not necessary


The aims of the course are to:

  • Learn about applications of information theory to universal data compression, statistics and inference
  • Introduce students to the principles of algebraic coding and Reed Solomon coding in particular
  • Give students an overview of cryptology with example of techniques that share the same mathematical background as algebraic coding.


As specific objectives, by the end of the course students should be able to:

  • have gained an appreciation for the connection between information-theoretic concepts and fundamental problems in statistics
  • learnt some core information-theoretical tools that can be used in probability and statistics
  • have a good understanding of the foundations of the problem of universal data compression
  • know and be able to use the basic results in large deviations theory, especially as applied in information theory and communications
  • have gained a practical understanding of the algebraic fundamentals that underlie channel coding and cryptology
  • understand the properties of linear block codes over finite fields
  • be able to implement encoders and decoders for Reed Solomon codes
  • have gained an overview of methods and aims in cryptology (including cryptography, crypt- analysis, secrecy, authenticity)
  • be familiar with one example each of a block cipher and a stream cipher
  • be able to implement public key cryptosystems, in particular the Diffie-Hellman and Rivest- Shamir-Adleman (RSA) systems


  1. This course will introduce students to applications of information theory and coding theory in statistics, information storage, and cryptography.

    The first part of the course will discuss applications of information theory to universal data compression, statistics, and inference.

    The second part of the course will expand linear coding principles acquired in 3F7 to non-binary codes over finite fields. After establish the algebraic fundamentals, we will cover Reed-Solomon coding, a technique used in a wide range of communication and storage systems (hard disks, blu ray discs, QR codes, USB mass storage device class, DNA storage, and others.)

    The final part of the course will introduce the discipline of cryptology, which includes cryptography, the essential art of ensuring secrecy and authenticity, and cryptanalysis, the dark art of breaking that secrecy. The course will cover a number of methods to provide secrecy, ranging from mathematically provable secrecy to public key methods through which computationally secure communication links can be established over public channels.


Information theory and statistics (7-9L, Prof. Ioannis Kontoyiannis)

  • Source coding, probability of error, error exponents
  • Method of types, error rates in data compression and hypothesis testing
  • Fundamental limits of estimation and hypothesis testing: The Cram ́er-Rao bound, Chernoff information, Neyman-Pearson tests, Stein’s lemma, strong converses
  • Large deviations: Cram ́er’s theorem, Sanov’s theorem, the conditional limit theorem
  • Entropy and Poisson approximation
  • Universal source coding: The capacity-redundancy theorem, the price of universality, Rissanen’s lower bound


Introduction to practical number theory and algebra (2-3L, Dr Jossy Sayir)

  • Elementary number theory
  • Groups and fields, extension fields
  • 3 equivalent approaches to multiplication in extension fields
  • Matrix operations and the Discrete Fourier Transform


Algebraic Coding (3L, Dr Jossy Sayir)

  • Linear coding and the Singleton Bound
  • Distance profiles and MacWilliams Identities
  • Blahut’s theorem
  • Reed Solomon (RS) codes
  • Encoding and decoding of RS codes


Introduction to Cryptology (2-3L, Dr Jossy Sayir )

  • Overview of cryptology
  • Stream ciphers, examples
  • Block ciphers, examples
  • Public key cryptography, basic techniques


Further notes




Examples papers

3 examples papers:

  1. Information theory & data compression
  2. Number theory and algebra
  3. Coding & Cryptology





  • Information Theory:
    • Elements of Information Theory, T. M. Cover & J. A. Thomas, Wiley-Interscience, 2nd Ed, 2006.
    • Information Theory: Coding Theorems for Discrete Memoryless Systems, I. Csiszàr & J. Körner, Cambridge University Press, 2nd Ed. 2011.
  • Coding theory:
    • The Theory of Error-Correcting Codes, F. J. MacWilliams & N. J. A. Sloane, North Holland.
    • Algebraic Codes for Data Transmission, Richard E. Blahut, Cambridge University Press, 2003 (Online 2012)


Please see the Booklist for Group F Courses for library holdings.

Examination Guidelines

Please refer to Form & conduct of the examinations.


The UK Standard for Professional Engineering Competence (UK-SPEC) describes the requirements that have to be met in order to become a Chartered Engineer, and gives examples of ways of doing this.

UK-SPEC is published by the Engineering Council on behalf of the UK engineering profession. The standard has been developed, and is regularly updated, by panels representing professional engineering institutions, employers and engineering educators. Of particular relevance here is the 'Accreditation of Higher Education Programmes' (AHEP) document which sets out the standard for degree accreditation.

The Output Standards Matrices indicate where each of the Output Criteria as specified in the AHEP 3rd edition document is addressed within the Engineering and Manufacturing Engineering Triposes.

Last modified: 02/06/2018 00:47