You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
@@ -3,34 +3,174 @@ title: "Perspectives into Tensors, Signals, and Kernel Methods"
3
3
category: technical
4
4
date: 2025-09-08
5
5
math: true
6
-
draft: true
6
+
draft: true
7
7
---
8
8
9
9
{{< toc >}}
10
10
11
11
## Abstract
12
12
13
-
[Linear algebra](https://linear.axler.net/LADR4e.pdf), [signal processing](https://en.wikipedia.org/wiki/Signal_processing), and [machine learning](https://en.wikipedia.org/wiki/Machine_learning) are (one of many groups of) topics that enjoy a beautiful relationship protected by a nutshell of mathematics. Inside we can find a kernel of perspectives which are simply a pleasure to entertain. Here, I provide the [linear-operator](https://en.wikipedia.org/wiki/Linear_map) perspective into [systems of many dimensions](https://en.wikipedia.org/wiki/Multidimensional_system), characterizing properties like [time-invariance](https://en.wikipedia.org/wiki/Time-invariant_system) and [causality](https://en.wikipedia.org/wiki/Causal_system) via [tensor](https://en.wikipedia.org/wiki/Tensor) representations. I finish with an original perspective that connects the [convolution kernel](https://en.wikipedia.org/wiki/Convolution) to the [reproducing kernel](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space#:~:text=then%20called%20the-,reproducing%20kernel,-%2C%20and%20it%20reproduces).
13
+
Linear algebra, signal processing, and machine learning methods are (one of many groups of) topics that enjoy beautiful relationships enclosed in a dense shell of mathematics. Within the shell, one finds a kernel of surprisingly diverse perspectives which are simply a pleasure to entertain. This article hopes to give the reader a romantic and thematic glimpse of the truths in this particular group of topics.
14
14
15
15
---
16
16
17
-
## Background
17
+
## Overview
18
18
19
-
This section is no more than a refresher on select topics. I will not make an attempt to explain them completely. If this is what you need, I recommend you explore these (completely free) resources:
19
+
I follow an essay-like structure including an introduction, body, and conclusion. The introduction sets the stage building from "class-style" knowledge, assuming a first course in linear algebra, signal processing, and machine learning. For readers lacking this background, I leave pointers to free resources.
20
20
21
-
1.**Vector spaces.**[_Linear Algebra Done Right_](https://linear.axler.net/) by Axler.
22
-
2.**Signals and Systems.**[_Signals & Systems: Theory and Applications_](https://ss2-2e.eecs.umich.edu/) by Ulaby and
23
-
Yagle.
24
-
3.**Kernel methods.**[_Foundations of Machine Learning_](https://cs.nyu.edu/~mohri/mlbook/) by Mohri, Rostamizadeh, and Talwalkar.
21
+
The introduction arrives at the [linear operator](https://en.wikipedia.org/wiki/Linear_map) perspective of [systems of many dimensions](https://en.wikipedia.org/wiki/Multidimensional_system), defining system properties like [time-invariance](https://en.wikipedia.org/wiki/Time-invariant_system) and [causality](https://en.wikipedia.org/wiki/Causal_system) in terms of [tensor](https://en.wikipedia.org/wiki/Tensor) representations. This results in the marriage of many linear algebra and signal processing concepts. These perspectives then help the body, where I present a cross-cutting perspective of the [convolution kernel](https://en.wikipedia.org/wiki/Convolution) and the [reproducing kernel](https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space#:~:text=then%20called%20the-,reproducing%20kernel,-%2C%20and%20it%20reproduces).
25
22
26
-
### Vector Spaces
23
+
Finally, the conclusion provides some subjective thematization to the concepts emphasized in the rest of the piece, summarizing mathematical details and indulging in a little bit of sensationalism. An attempt is made of providing pointers to further reading on adjacent concepts.
27
24
28
25
26
+
{{% hint title="Clarification" %}}
29
27
30
-
### Signals and Systems
28
+
The word "kernel" is criminally polysemous in mathematics, and it will be used a lot in this piece. I mostly use the word under two semantics: The convolution kernel from signal processing and the reproducing kernel from machine learning. All other instances of the word should simply refer to its english meaning.
31
29
32
-
### Kernel Methods
30
+
However, a primary objective of this piece is to reach a perspective of the convolution and reproducing kernels that allows them to be seen through a common lense. So an acute reader may interpret this as an explanation of why they are both called "kernel," assuming that they were both named that way because they represent the same kind of mathematical object in some profound way.
31
+
32
+
What is interesting is that they both received their names for a superficial reason -- because the symbols that represent them show up inside other symbols. One could imagine that people started calling them "kernel" independently just to avoid saying the phrase "that term in in the middle" while pointing at a blackboard.
33
+
34
+
As such, the connecting view of the convolution and reproducing kernels that we will work towards only applies to these two kernels (convolution and reproducing). If these were the only two kernels in mathematics, maybe one could now say that they are named the same for a profound reason. But there are numerous other kinds of kernels for which the constructions in this piece simply do not apply.
35
+
36
+
{{% /hint %}}
33
37
34
38
---
35
39
40
+
## Introduction
41
+
42
+
This section provides not much more than a definition-based refresher on select topics from first courses in linear algebra, signal processing, and machine learning. For readers in need of comprehensive review or first-time coverage, I leave these free resources on said topics:
43
+
44
+
***Kernel methods.**[_Foundations of Machine Learning_](https://cs.nyu.edu/~mohri/mlbook/) by Mohri, Rostamizadeh, and Talwalkar.
45
+
***Signals and Systems.**[_Signals & Systems: Theory and Applications_](https://ss2-2e.eecs.umich.edu/) by Ulaby and Yagle.
46
+
***Linear algebra.**[_Linear Algebra Done Right_](https://linear.axler.net/) by Axler.
47
+
48
+
### Bases of Vector Spaces
49
+
50
+
In what is nowadays close to being a canon of linear algebra education, Sheldon Axler opens with the statement below to set the stage for the rest of _Linear Algebra Done Right_:
51
+
52
+
> Linear algebra is the study of linear maps on finite-dimensional vector spaces.
53
+
54
+
Here, the restriction of vector spaces to the finite-dimensional case was one the most mathematically respectful ways to negotiate generality with practical pedagogy. However, the spirit of linear algebra is alive way beyond the finite-dimensional case.
55
+
56
+
#### Hamel Bases
57
+
58
+
Most engineers are familiar with the concept of a (Hamel) basis of a vector space. If we have a vector space $V$ over a field $\mathbb{F}$ and a Hamel basis $\mathcal{B}$, then "$\mathcal{B}$ spans $V$" translates to
59
+
60
+
$$
61
+
\begin{equation}
62
+
\forall v \in V, \; v = \sum_{i \, = \, 0}^{k} c_i b_i \;\; \text{s.t.} \;\; c_i \in \mathbb{F}, \, b_i \in \mathcal{B}, \, k \in \mathbb{N}.
63
+
\end{equation}
64
+
$$
65
+
66
+
Importantly, for $B$ to be a Hamel basis, the sum in $(1)$ must have finite terms. Note that this is allowed even in cases where $\mathcal{B}$ is infinite (or in other words, where $V$ is infinite-dimensional), as one does not necessarily assign nonzero coefficients $c_i$ to each element of $\mathcal{B}$.
67
+
68
+
{{% hint title="Example" %}}
69
+
70
+
The vector space of polynomials (of finite terms) with coefficients in a field $\mathbb{F}$,
has the infinite basis $\mathcal{B}_{\mathbb{F}[x]} = \\{1, x, x^2, x^3, \dots \\}$. Each of its elements, however, is the linear combination of a finite number of basis elements. For example, the polynomial
77
+
78
+
$$
79
+
p(x) = 3 + 4x^2 + x^3
80
+
$$
81
+
82
+
can be expressed as a (finite) linear combination of basis elements,
83
+
84
+
$$
85
+
p(x) = 3
86
+
\begin{bmatrix}
87
+
1 \\
88
+
0 \\
89
+
0 \\
90
+
0 \\
91
+
\vdots
92
+
\end{bmatrix} + 4
93
+
\begin{bmatrix}
94
+
0 \\
95
+
0 \\
96
+
1 \\
97
+
0 \\
98
+
\vdots
99
+
\end{bmatrix} + 1
100
+
\begin{bmatrix}
101
+
0 \\
102
+
0 \\
103
+
0 \\
104
+
1 \\
105
+
\vdots
106
+
\end{bmatrix}.
107
+
$$
108
+
109
+
Here we imposed a (canonical) representation such that, for example, $x^2 = \left[ 0, \\, 0, \\, 1, \\, 0, \\, {\dots} \right]^\top$. We see that, despite each basis vector being infinite-dimensional, all polynomials are determined by a finite number of them.
110
+
111
+
{{% /hint %}}
112
+
113
+
#### Schauder Bases
114
+
115
+
Interpreting Axler strictly, $\mathbb{F}[x]$ is already beyond linear algebra because it is of [semi-infinite](https://en.wikipedia.org/wiki/Semi-infinite) dimension. But definitionally, it is a perfectly valid vector space. Just as finite dimensionality is not necessary in order to access the theorems of linear algebra, having a countable Hamel basis is also not necessary; all vector spaces do have a Hamel basis[^axiom-choice], but not all of them have a countable one.
116
+
117
+
{{% hint title="Note" %}}
118
+
119
+
Countable bases are desireable not for being countable per se, but rather that, in most cases where a vector space does not have a countable Hamel basis, the uncountable Hamel basis is unconstructive and unutterable. Put another way, the most useful fact about an uncountable Hamel basis, in many cases, is that it exists.
120
+
121
+
{{% /hint %}}
122
+
123
+
For some vector spaces that do not have a countable Hamel basis, one can relax the definition of a basis itself to obtain one that is countable. Specifically, we redefine the phrase "$\mathcal{B}$ spans $V$" to
If the above is true for a vector space $V$ over $\mathbb{F}$, then $\mathcal{B}$ is a Schauder basis of said space. The critical difference to a Hamel basis is of course the generality afforded by the possiblity of infinite terms for the sum in $(2)$, giving us a new countably infinite flavor of linear combination.
has no Hamel basis because, no matter how you define one, you can come up with an element of $\ell^2$ which requires a decomposition into an infinite number of basis elements (which is not allowed). However, it does have the countably infinite Schauder basis
142
+
143
+
$$
144
+
\mathcal{B}_{\ell^2} = \left\{
145
+
\left(
146
+
1, \,
147
+
0, \,
148
+
0, \,
149
+
\dots
150
+
\right), \,
151
+
\left(
152
+
0, \,
153
+
1, \,
154
+
0, \,
155
+
\dots
156
+
\right), \,
157
+
\left(
158
+
0, \,
159
+
0, \,
160
+
1, \,
161
+
\dots
162
+
\right), \,
163
+
{\dots}
164
+
\right\}.
165
+
$$
166
+
167
+
This is quite remarkable, as the uncountably infinite $\ell^2$ is spanned by the countably infinite $\mathcal{B}_{\ell^2}$. Allowing infinite linear combinations bridges this countability barrier via convergence.
168
+
169
+
{{% /hint %}}
170
+
171
+
### Signals and Systems
172
+
173
+
### Kernel Methods
174
+
36
175
176
+
[^axiom-choice]: When considering infinite-dimensional vector spaces, this statement is true if and only if one admits the axiom of choice. Perhaps this was another motivation of Axler's restriction to finite-dimensional vector spaces.
<p><ahref="https://linear.axler.net/LADR4e.pdf">Linear algebra</a>, <ahref="https://en.wikipedia.org/wiki/Signal_processing">signal processing</a>, and <ahref="https://en.wikipedia.org/wiki/Machine_learning">machine learning</a> are (one of many groups of) topics that enjoy a beautiful relationship protected by a nutshell of mathematics. Inside we can find a kernel of perspectives which are simply a pleasure to entertain. Here, I provide the <ahref="https://en.wikipedia.org/wiki/Linear_map">linear-operator</a> perspective into <ahref="https://en.wikipedia.org/wiki/Multidimensional_system">systems of many dimensions</a>, characterizing properties like <ahref="https://en.wikipedia.org/wiki/Time-invariant_system">time-invariance</a> and <ahref="https://en.wikipedia.org/wiki/Causal_system">causality</a> via <ahref="https://en.wikipedia.org/wiki/Tensor">tensor</a> representations. I finish with an original perspective that connects the <ahref="https://en.wikipedia.org/wiki/Convolution">convolution kernel</a> to the <ahref="https://en.wikipedia.org/wiki/Reproducing_kernel_Hilbert_space#:~:text=then%20called%20the-,reproducing%20kernel,-%2C%20and%20it%20reproduces">reproducing kernel</a>.</p>
89
+
<hr>
90
+
<h2id="background">Background</h2>
91
+
<p>This section is no more than a refresher on select topics. I will not make an attempt to explain them completely. If this is what you need, I recommend you explore these (completely free) resources:</p>
92
+
<ol>
93
+
<li><strong>Vector spaces.</strong><ahref="https://linear.axler.net/"><em>Linear Algebra Done Right</em></a> by Axler.</li>
94
+
<li><strong>Signals and Systems.</strong><ahref="https://ss2-2e.eecs.umich.edu/"><em>Signals & Systems: Theory and Applications</em></a> by Ulaby and
95
+
Yagle.</li>
96
+
<li><strong>Kernel methods.</strong><ahref="https://cs.nyu.edu/~mohri/mlbook/"><em>Foundations of Machine Learning</em></a> by Mohri, Rostamizadeh, and Talwalkar.</li>
97
+
</ol>
98
+
<h3id="vector-spaces">Vector Spaces</h3>
99
+
<h3id="signals-and-systems">Signals and Systems</h3>
0 commit comments