Information Theory, Inference and Learning Algorithms

Front Cover
Cambridge University Press, Sep 25, 2003 - Computers - 628 pages
19 Reviews
Information theory and inference, often taught separately, are here united in one entertaining textbook. These topics lie at the heart of many exciting areas of contemporary science and engineering - communication, signal processing, data mining, machine learning, pattern recognition, computational neuroscience, bioinformatics, and cryptography. This textbook introduces theory in tandem with applications. Information theory is taught alongside practical communication systems, such as arithmetic coding for data compression and sparse-graph codes for error-correction. A toolbox of inference techniques, including message-passing algorithms, Monte Carlo methods, and variational approximations, are developed alongside applications of these tools to clustering, convolutional codes, independent component analysis, and neural networks. The final part of the book describes the state of the art in error-correcting codes, including low-density parity-check codes, turbo codes, and digital fountain codes -- the twenty-first century standards for satellite communications, disk drives, and data broadcast. Richly illustrated, filled with worked examples and over 400 exercises, some with detailed solutions, David MacKay's groundbreaking book is ideal for self-learning and for undergraduate or graduate courses. Interludes on crosswords, evolution, and sex provide entertainment along the way. In sum, this is a textbook on information, communication, and coding for a new generation of students, and an unparalleled entry point into these subjects for professionals in areas as diverse as computational biology, financial engineering, and machine learning.
 

What people are saying - Write a review

User ratings

5 stars
11
4 stars
7
3 stars
1
2 stars
0
1 star
0

Review: Information Theory, Inference and Learning Algorithms

User Review  - Jon Gauthier - Goodreads

NB: Both book and lectures are available for free online. (Check YouTube for lectures.) Read full review

Review: Information Theory, Inference and Learning Algorithms

User Review  - Goodreads

NB: Both book and lectures are available for free online. (Check YouTube for lectures.) Read full review

All 9 reviews »

Contents

Introduction to Information Theory
3
Probability Entropy and Inference
22
ful theoretical ideas of Shannon but also practical solutions to communica
34
More about Inference
48
Data Compression
65
The Source Coding Theorem
67
Symbol Codes
91
Stream Codes
110
Stream Codes 26 I Exact Margmalization in Graphs
346
Monte Carlo Methods
357
Efficient Monte Carlo Methods
387
Ising Models
400
Exact Monte Carlo Sampling
413
Variational Methods
422
Independent Component Analysis and Latent Variable Mod elling
437
Random Inference Topics
445

Codes for Integers
132
NoisyChannel Coding
137
Dependent Random Variables
138
Communication over a Noisy Channel
146
The NoisyChannel Coding Theorem
162
ErrorCorrecting Codes and Real Channels
177
Further Topics in Information Theory
191
Codes for Efficient Information Retrieval
193
Binary Codes
206
Very Good Linear Codes Exist
229
Further Exercises on Information Theory
233
Message Passing
241
Communication over Constrained Noiseless Channels
248
Crosswords and Codebreaking
260
Why have Sex? Information Acquisition and Evolution
269
Probabilities and Inference
281
Clustering
284
Exact Inference by Complete Enumeration
293
Maximum Likelihood and Clustering
300
Useful Probability Distributions
311
Exact Marginalization
319
Exact Marginalization in Trellises
324
Exact Marginalization in Graphs
334
Laplaces Method
341
Model Comparison and Occams Razor
343
Decision Theory
451
Bayesian Inference and Sampling Theory
457
Neural networks
467
Introduction to Neural Networks
468
The Single Neuron as a Classifier
471
Capacity of a Single Neuron
483
Learning as Inference
492
Hopfield Networks
505
Boltzmami Machines
522
Supervised Learning in Multilayer Networks
527
Gaussian Processes
535
Deconvolution
549
Sparse Graph Codes
555
LowDensity ParityCheck Codes
557
Convolutional Codes and Turbo Codes
574
Repeat Accumulate Codes
582
50 Digital Fountain Codes
588
Digital Fountain Codes
589
Appendices
597
A Notation
598
B Some Physics
601
Some Mathematics
605
Bibliography
613
Index
620
Copyright

Common terms and phrases

References to this book

All Book Search results »

Bibliographic information