Reproducing kernel Hilbert space

In functional analysis, a reproducing kernel Hilbert space (RKHS) is a Hilbert space of functions in which point evaluation is a continuous linear functional. Specifically, a Hilbert space ${\displaystyle H}$ of functions from a set ${\displaystyle X}$ (to ${\displaystyle \mathbb {R} }$ or ${\displaystyle \mathbb {C} }$ ) is an RKHS if, for each ${\displaystyle x\in X}$ , there exists a function ${\displaystyle K_{x}\in H}$ such that for all ${\displaystyle f\in H}$ ,

${\displaystyle \langle f,K_{x}\rangle =f(x).}$

The function ${\displaystyle K_{x}}$ is called the reproducing kernel, and it reproduces the value of ${\displaystyle f}$ at ${\displaystyle x}$ via the inner product.

An immediate consequence of this property is that if two functions ${\displaystyle f}$ and ${\displaystyle g}$ in the RKHS are close in norm (i.e., ${\displaystyle \|f-g\|}$ is small), then ${\displaystyle f}$ and ${\displaystyle g}$ are also pointwise close (i.e., ${\displaystyle \sup |f(x)-g(x)|}$ is small). This follows from the fact that the inner product induces pointwise evaluation control. Roughly speaking, this means that closeness in the RKHS norm implies pointwise closeness, but the converse does not necessarily hold.

For example, consider the sequence of functions ${\displaystyle \sin ^{2n}(x)}$. These functions converge pointwise to 0 as ${\displaystyle n\to \infty }$ , but they do not converge uniformly (i.e., they do not converge with respect to the supremum norm). This illustrates that pointwise convergence does not imply convergence in norm. It is important to note that the supremum norm does not arise from any inner product, as it does not satisfy the parallelogram law.

It is not entirely straightforward to construct natural examples of a Hilbert space which are not an RKHS in a non-trivial fashion.^[1] Some examples, however, have been found.^[2][3]

While L² spaces is usually defined as a Hilbert space whose elements are equivalence classes of functions it can be trivially redefined as a Hilbert space of functions by using choice to select a (total) function as a representative for each equivalence class. However, no choice of representatives can make this space an RKHS (${\displaystyle K_{0}}$ would need to be the non-existent Dirac delta function). However, there are RKHSs in which the norm is an L²-norm, such as the space of band-limited functions (see the example below).

An RKHS is associated with a kernel that reproduces every function in the space in the sense that for every ${\displaystyle x}$ in the set on which the functions are defined, "evaluation at ${\displaystyle x}$" can be performed by taking an inner product with a function determined by the kernel. Such a reproducing kernel exists if and only if every evaluation functional is continuous.

The reproducing kernel was first introduced in the 1907 work of Stanisław Zaremba concerning boundary value problems for harmonic and biharmonic functions. James Mercer simultaneously examined functions which satisfy the reproducing property in the theory of integral equations. The idea of the reproducing kernel remained untouched for nearly twenty years until it appeared in the dissertations of Gábor Szegő, Stefan Bergman, and Salomon Bochner. The subject was eventually systematically developed in the early 1950s by Nachman Aronszajn and Stefan Bergman.^[4]

These spaces have wide applications, including complex analysis, harmonic analysis, and quantum mechanics. Reproducing kernel Hilbert spaces are particularly important in the field of statistical learning theory because of the celebrated representer theorem which states that every function in an RKHS that minimises an empirical risk functional can be written as a linear combination of the kernel function evaluated at the training points. This is a practically useful result as it effectively simplifies the empirical risk minimization problem from an infinite dimensional to a finite dimensional optimization problem.

For ease of understanding, we provide the framework for real-valued Hilbert spaces. The theory can be easily extended to spaces of complex-valued functions and hence include the many important examples of reproducing kernel Hilbert spaces that are spaces of analytic functions.^[5]