Sparse and Redundant Representations


Book Description

A long long time ago, echoing philosophical and aesthetic principles that existed since antiquity, William of Ockham enounced the principle of parsimony, better known today as Ockham’s razor: “Entities should not be multiplied without neces sity. ” This principle enabled scientists to select the ”best” physical laws and theories to explain the workings of the Universe and continued to guide scienti?c research, leadingtobeautifulresultsliketheminimaldescriptionlength approachtostatistical inference and the related Kolmogorov complexity approach to pattern recognition. However, notions of complexity and description length are subjective concepts anddependonthelanguage“spoken”whenpresentingideasandresults. The?eldof sparse representations, that recently underwent a Big Bang like expansion, explic itly deals with the Yin Yang interplay between the parsimony of descriptions and the “language” or “dictionary” used in them, and it became an extremely exciting area of investigation. It already yielded a rich crop of mathematically pleasing, deep and beautiful results that quickly translated into a wealth of practical engineering applications. You are holding in your hands the ?rst guide book to Sparseland, and I am sure you’ll ?nd in it both familiar and new landscapes to see and admire, as well as ex cellent pointers that will help you ?nd further valuable treasures. Enjoy the journey to Sparseland! Haifa, Israel, December 2009 Alfred M. Bruckstein vii Preface This book was originally written to serve as the material for an advanced one semester (fourteen 2 hour lectures) graduate course for engineering students at the Technion, Israel.




A Wavelet Tour of Signal Processing


Book Description

This book is intended to serve as an invaluable reference for anyone concerned with the application of wavelets to signal processing. It has evolved from material used to teach "wavelet signal processing" courses in electrical engineering departments at Massachusetts Institute of Technology and Tel Aviv University, as well as applied mathematics departments at the Courant Institute of New York University and ÉcolePolytechnique in Paris. - Provides a broad perspective on the principles and applications of transient signal processing with wavelets - Emphasizes intuitive understanding, while providing the mathematical foundations and description of fast algorithms - Numerous examples of real applications to noise removal, deconvolution, audio and image compression, singularity and edge detection, multifractal analysis, and time-varying frequency measurements - Algorithms and numerical examples are implemented in Wavelab, which is a Matlab toolbox freely available over the Internet - Content is accessible on several level of complexity, depending on the individual reader's needs New to the Second Edition - Optical flow calculation and video compression algorithms - Image models with bounded variation functions - Bayes and Minimax theories for signal estimation - 200 pages rewritten and most illustrations redrawn - More problems and topics for a graduate course in wavelet signal processing, in engineering and applied mathematics




Sparse Image and Signal Processing


Book Description

Presents state-of-the-art sparse and multiscale image and signal processing with applications in astronomy, biology, MRI, media, and forensics.







Mathematics in Image Processing


Book Description

The theme of the 2010 PCMI Summer School was Mathematics in Image Processing in a broad sense, including mathematical theory, analysis, computation algorithms and applications. In image processing, information needs to be processed, extracted and analyzed from visual content, such as photographs or videos. These demands include standard tasks such as compression and denoising, as well as high-level understanding and analysis, such as recognition and classification. Centered on the theme of mathematics in image processing, the summer school covered quite a wide spectrum of topics in this field. The summer school is particularly timely and exciting due to the very recent advances and developments in the mathematical theory and computational methods for sparse representation. This volume collects three self-contained lecture series. The topics are multi-resolution based wavelet frames and applications to image processing, sparse and redundant representation modeling of images and simulation of elasticity, biomechanics, and virtual surgery. Recent advances in image processing, compressed sensing and sparse representation are discussed.




Independent Component Analysis and Blind Signal Separation


Book Description

In many situations found both in Nature and in human-built systems, a set of mixed signals is observed (frequently also with noise), and it is of great scientific and technological relevance to be able to isolate or separate them so that the information in each of the signals can be utilized. Blind source separation (BSS) research is one of the more interesting emerging fields now a days in the field of signal processing. It deals with the algorithms that allow the recovery of the original sources from a set of mixtures only. The adjective "blind" is applied because the purpose is to estimate the original sources without any a priori knowledge about either the sources or the mixing system. Most of the models employed in BSS assume the hypothesis about the independence of the original sources. Under this hypothesis, a BSS problem can be considered as a particular case of independent component analysis(ICA), a linear transformation technique that, starting from a multivariate representation of the data, minimizes the statistical dependence between the components of the representation. It can be claimed that most of the advances in ICA have been motivated by the search for solutions to the BSS problem and, the other way around, advances in ICA have been immediately applied to BSS. ICA and BSS algorithms start from a mixture model, whose parameters are estimated from the observed mixtures. Separation is achieved by applying the inverse mixture model to the observed signals(separating or unmixing model). Mixturem- els usually fall into three broad categories: instantaneous linear models, convolutive models and nonlinear models, the?rstone being the simplest but, in general, not near realistic applications. The development and test of the algorithms can be accomplished through synthetic data or with real-world data. Obviously, the most important aim(and most difficult) is the separation of real-world mixtures. BSS and ICA have strong relations also, apart from signal processing, with other fields such as statistics and artificial neural networks. As long as we can find a system that emits signals propagated through a mean, andthosesignalsarereceivedbyasetofsensorsandthereisaninterestinrecovering the original sources, we have a potential field of application for BSS and ICA. Inside that wide range of applications we can find, for instance: noise reduction applications, biomedical applications, audio systems, telecommunications, and many others. This volume comes out just 20 years after the first contributions in ICA and BSS 1 appeared . Therein after, the number of research groups working in ICA and BSS has been constantly growing, so that nowadays we can estimate that far more than 100 groups are researching in these fields. As proof of the recognition among the scientific community of ICA and BSS developments there have been numerous special sessions and special issues in several well- 1 J. Herault, B. Ans, "Circuits neuronaux à synapses modi?ables: décodage de messages composites para apprentissage non supervise", C.R. de l'Académie des Sciences, vol. 299, no. III-13,pp.525-528,1984




Sparse Modeling for Image and Vision Processing


Book Description

Sparse Modeling for Image and Vision Processing offers a self-contained view of sparse modeling for visual recognition and image processing. More specifically, it focuses on applications where the dictionary is learned and adapted to data, yielding a compact representation that has been successful in various contexts.




Graph Representation Learning


Book Description

Graph-structured data is ubiquitous throughout the natural and social sciences, from telecommunication networks to quantum chemistry. Building relational inductive biases into deep learning architectures is crucial for creating systems that can learn, reason, and generalize from this kind of data. Recent years have seen a surge in research on graph representation learning, including techniques for deep graph embeddings, generalizations of convolutional neural networks to graph-structured data, and neural message-passing approaches inspired by belief propagation. These advances in graph representation learning have led to new state-of-the-art results in numerous domains, including chemical synthesis, 3D vision, recommender systems, question answering, and social network analysis. This book provides a synthesis and overview of graph representation learning. It begins with a discussion of the goals of graph representation learning as well as key methodological foundations in graph theory and network analysis. Following this, the book introduces and reviews methods for learning node embeddings, including random-walk-based methods and applications to knowledge graphs. It then provides a technical synthesis and introduction to the highly successful graph neural network (GNN) formalism, which has become a dominant and fast-growing paradigm for deep learning with graph data. The book concludes with a synthesis of recent advancements in deep generative models for graphs—a nascent but quickly growing subset of graph representation learning.




Handbook of Mathematical Methods in Imaging


Book Description

The Handbook of Mathematical Methods in Imaging provides a comprehensive treatment of the mathematical techniques used in imaging science. The material is grouped into two central themes, namely, Inverse Problems (Algorithmic Reconstruction) and Signal and Image Processing. Each section within the themes covers applications (modeling), mathematics, numerical methods (using a case example) and open questions. Written by experts in the area, the presentation is mathematically rigorous. The entries are cross-referenced for easy navigation through connected topics. Available in both print and electronic forms, the handbook is enhanced by more than 150 illustrations and an extended bibliography. It will benefit students, scientists and researchers in applied mathematics. Engineers and computer scientists working in imaging will also find this handbook useful.




Dictionary Learning Algorithms and Applications


Book Description

This book covers all the relevant dictionary learning algorithms, presenting them in full detail and showing their distinct characteristics while also revealing the similarities. It gives implementation tricks that are often ignored but that are crucial for a successful program. Besides MOD, K-SVD, and other standard algorithms, it provides the significant dictionary learning problem variations, such as regularization, incoherence enforcing, finding an economical size, or learning adapted to specific problems like classification. Several types of dictionary structures are treated, including shift invariant; orthogonal blocks or factored dictionaries; and separable dictionaries for multidimensional signals. Nonlinear extensions such as kernel dictionary learning can also be found in the book. The discussion of all these dictionary types and algorithms is enriched with a thorough numerical comparison on several classic problems, thus showing the strengths and weaknesses of each algorithm. A few selected applications, related to classification, denoising and compression, complete the view on the capabilities of the presented dictionary learning algorithms. The book is accompanied by code for all algorithms and for reproducing most tables and figures. Presents all relevant dictionary learning algorithms - for the standard problem and its main variations - in detail and ready for implementation; Covers all dictionary structures that are meaningful in applications; Examines the numerical properties of the algorithms and shows how to choose the appropriate dictionary learning algorithm.