Random feature

Random features (RF) are a technique used in machine learning to approximate kernel methods, introduced by Ali Rahimi and Ben Recht in their 2007 paper "Random Features for Large-Scale Kernel Machines",^[1] and extended by ^[2]^[3]. RF uses a Monte Carlo approximation to kernel functions by randomly sampled feature maps. It is used for datasets that are too large for traditional kernel methods like support vector machine, kernel ridge regression, and gaussian process.

Mathematics

Kernel method

Given a feature map ${\textstyle \phi :\mathbb {R} ^{d}\to V}$ , where ${\textstyle V}$ is a Hilbert space (more specifically, a reproducing kernel Hilbert space), the kernel trick replaces inner products in feature space $\langle \phi (x_{i}),\phi (x_{j})\rangle _{V}$ by a kernel function $k(x_{i},x_{j}):\mathbb {R} ^{d}\times \mathbb {R} ^{d}\to \mathbb {R}$ Kernel methods replaces linear operations in high-dimensional space by operations on the kernel matrix: $K_{X}:=[k(x_{i},x_{j})]_{i,j\in 1:N}$ where ${\textstyle N}$ is the number of data points.

Random kernel method

The problem with kernel methods is that the kernel matrix ${\textstyle K_{X}}$ has size ${\textstyle N\times N}$ . This becomes computationally infeasible when ${\textstyle N}$ reaches the order of a million. The random kernel method replaces the kernel function ${\textstyle k}$ by an inner product in low-dimensional feature space ${\textstyle \mathbb {R} ^{D}}$ : $k(x,y)\approx \langle z(x),z(y)\rangle$ where ${\textstyle z}$ is a randomly sampled feature map ${\textstyle z:\mathbb {R} ^{d}\to \mathbb {R} ^{D}}$ .

This converts kernel linear regression into linear regression in feature space, kernel SVM into SVM in feature space, etc. Since we have $K_{X}\approx Z_{X}^{T}Z_{X}$ where $Z_{X}=[z(x_{1}),\dots ,z(x_{N})]$ , these methods no longer involve matrices of size ${\textstyle O(N^{2})}$ , but only random feature matrices of size ${\textstyle O(DN)}$ .

Random Fourier feature

Radial basis function kernel

The radial basis function (RBF) kernel on two samples $x_{i},x_{j}\in \mathbb {R} ^{d}$ is defined as^[4]

k(x_{i},x_{j})=\exp \left(-{\frac {\|x_{i}-x_{j}\|^{2}}{2\sigma ^{2}}}\right)

where $\|x_{i}-x_{j}\|^{2}$ is the squared Euclidean distance and $\sigma$ is a free parameter defining the shape of the kernel. It can be approximated by a random Fourier feature map $z:\mathbb {R} ^{d}\to \mathbb {R} ^{2D}$ : $z(x):={\frac {1}{\sqrt {D}}}[\cos \langle \omega _{1},x\rangle ,\sin \langle \omega _{1},x\rangle ,\ldots ,\cos \langle \omega _{D},x\rangle ,\sin \langle \omega _{D},x\rangle ]^{T}$ where $\omega _{1},...,\omega _{D}$ are IID samples from the multidimensional normal distribution $N(0,\sigma ^{-2}I)$ .

Theorem — -

(Unbiased estimation) $\operatorname {E} [\langle z(x),z(y)\rangle ]=e^{\|x-y\|^{2}/(2\sigma ^{2})}.$
(Variance bound) $\operatorname {Var} [\langle z(x),z(y)\rangle ]=O(D^{-1})$
(Convergence) As $D\to \infty$ , the approximation converges in probability to the true kernel.

Proof

(Unbiased estimation) By independence of $\omega _{1},...,\omega _{D}$ , it suffices to prove the case of $D=1$ . By the trigonometric identity $\cos(a-b)=\cos(a)\cos(b)+\sin(a)\sin(b)$ , $\langle z(x),z(y)\rangle ={\frac {1}{D}}\sum _{i=1}^{D}\cos \langle \omega _{i},x-y\rangle$ Apply the spherical symmetry of normal distribution, then evaluate the integral: $\int _{-\infty }^{\infty }{\frac {\cos(kx)e^{-x^{2}/2}}{\sqrt {2\pi }}}dx=e^{-k^{2}/2}.$

(Variance bound) Since $\omega _{1},...,\omega _{D}$ are IID, it suffices to prove that the variance of $\cos \langle \omega _{1},x-y\rangle$ is finite, which is true since it is bounded within $[-1,+1]$ .

(Convergence) By Chebyshev's inequality.

Since $\cos ,\sin$ are bounded, there is a stronger convergence guarantee by Hoeffding's inequality.^[1]^{: Claim 1}

Random Fourier features

By Bochner's theorem, the above construction can be generalized to arbitrary positive definite shift-invariant kernel $k(x,y)=k(x-y)$ .

Define its Fourier transform $p(\omega )={\frac {1}{2\pi }}\int _{\mathbb {R} ^{d}}e^{-j\langle \omega ,\Delta \rangle }k(\Delta )d\Delta$ then $\omega _{1},...,\omega _{D}$ are sampled IID from the probability distribution with probability density $p$ . This applies for other kernels like the Laplace kernel and the Cauchy kernel.

Neural network interpretation

Given a random Fourier feature map $z$ , training the feature on a dataset by featurized linear regression is equivalent to fitting complex parameters $\theta _{1},\dots ,\theta _{D}\in \mathbb {C}$ such that $f_{\theta }(x)=\mathrm {Re} \left(\sum _{k}\theta _{k}e^{i\langle \omega _{k},x\rangle }\right)$ which is a neural network with a single hidden layer, with activation function $t\mapsto e^{it}$ , zero bias, and the parameters in the first layer frozen.

In the overparameterized case, when $2D\geq N$ , the network linearly interpolates the dataset $\{(x_{i},y_{i})\}_{i\in 1:N}$ , and the network parameters is the least-norm solution: ${\hat {\theta }}=\arg \min _{\theta \in \mathbb {C} ^{D},f_{\theta }(x_{k})=y_{k}\forall k\in 1:N}\|\theta \|$ At the limit of $D\to \infty$ , the L2 norm $\|{\hat {\theta }}\|\to \|f_{K}\|_{H}$ where $f_{K}$ is the interpolating function obtained by the kernel regression with the original kernel, and $\|\cdot \|_{H}$ is the norm in the reproducing kernel Hilbert space for the kernel.^[5]

Other examples

Random binning features

A random binning features map partitions the input space using randomly shifted grids at randomly chosen resolutions and assigns to an input point a binary bit string that corresponds to the bins in which it falls. The grids are constructed so that the probability that two points $x_{i},x_{j}\in \mathbb {R} ^{d}$ are assigned to the same bin is proportional to $K(x_{i},x_{j})$ . The inner product between a pair of transformed points is proportional to the number of times the two points are binned together, and is therefore an unbiased estimate of $K(x_{i},x_{j})$ . Since this mapping is not smooth and uses the proximity between input points, Random Binning Features works well for approximating kernels that depend only on the $L_{1}$ distance between datapoints.

Orthogonal random features

Orthogonal random features^[6] uses a random orthogonal matrix instead of a random Fourier matrix.

Historical context

In NIPS 2006, deep learning had just become competitive with linear models like PCA and linear SVMs for large datasets, and people speculated about whether it could compete with kernel SVMs. However, there was no way to train kernel SVM on large datasets. The two authors developed the random feature method to train those.

It was then found that the $O(1/D)$ variance bound did not match practice: The variance bound predicts that approximation to within $0.01$ requires $D\sim 10^{4}$ , but in practice required only $\sim 10^{2}$ . Attempting to discover what caused this led to the subsequent two papers ^[2]^[3].^[7]

References

^ ^a ^b Rahimi, Ali; Recht, Benjamin (2007). "Random Features for Large-Scale Kernel Machines". Advances in Neural Information Processing Systems. 20.
^ ^a ^b Rahimi, Ali; Recht, Benjamin (September 2008). "Uniform approximation of functions with random bases". 2008 46th Annual Allerton Conference on Communication, Control, and Computing. IEEE. doi:10.1109/allerton.2008.4797607.
^ ^a ^b Rahimi, Ali; Recht, Benjamin (2008). "Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning". Advances in Neural Information Processing Systems. 21. Curran Associates, Inc.
^ Jean-Philippe Vert, Koji Tsuda, and Bernhard Schölkopf (2004). "A primer on kernel methods". Kernel Methods in Computational Biology.
^ Belkin, Mikhail (May 2021). "Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation". Acta Numerica. 30: 203–248. doi:10.1017/S0962492921000039. ISSN 0962-4929.
^ Yu, Felix Xinnan X; Suresh, Ananda Theertha; Choromanski, Krzysztof M; Holtmann-Rice, Daniel N; Kumar, Sanjiv (2016). "Orthogonal Random Features". Advances in Neural Information Processing Systems. 29. Curran Associates, Inc.
^ Recht, Benjamin. "Reflections on Random Kitchen Sinks". arg min blog. Retrieved 2024-09-29.

External links

Random Walks - Random Fourier features

[Rahimi2007-1] Rahimi, Ali; Recht, Benjamin (2007). "Random Features for Large-Scale Kernel Machines". Advances in Neural Information Processing Systems. 20.

[:0-2] Rahimi, Ali; Recht, Benjamin (September 2008). "Uniform approximation of functions with random bases". 2008 46th Annual Allerton Conference on Communication, Control, and Computing. IEEE. doi:10.1109/allerton.2008.4797607.

[:1-3] Rahimi, Ali; Recht, Benjamin (2008). "Weighted Sums of Random Kitchen Sinks: Replacing minimization with randomization in learning". Advances in Neural Information Processing Systems. 21. Curran Associates, Inc.

[primer-4] Jean-Philippe Vert, Koji Tsuda, and Bernhard Schölkopf (2004). "A primer on kernel methods". Kernel Methods in Computational Biology.

[5] Belkin, Mikhail (May 2021). "Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation". Acta Numerica. 30: 203–248. doi:10.1017/S0962492921000039. ISSN 0962-4929.

[6] Yu, Felix Xinnan X; Suresh, Ananda Theertha; Choromanski, Krzysztof M; Holtmann-Rice, Daniel N; Kumar, Sanjiv (2016). "Orthogonal Random Features". Advances in Neural Information Processing Systems. 29. Curran Associates, Inc.

[7] Recht, Benjamin. "Reflections on Random Kitchen Sinks". arg min blog. Retrieved 2024-09-29.

[1]

[2]

[3]

[4]

[5]

[6]

[7]