How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

How does meaning work? What do we mean when we say knife?

Semantic Features

A knife is [+SHARP EDGE] [+HAND SIZED HANDLE] [-LONG BLADE] [-DOUBLE EDGE] [-HEAVY EDGE]

Similar to linear regression techniques.

How does meaning work? What do we mean when we say knife?

Prototype Theory

A knife is something similar to

More similarity imbues more "knifeness"

A candidate procedure: pick a number of prototypes to locate and then

1. Pick random candidates
2. Improve each candidate

A candidate procedure: pick a number of prototypes to locate and then

1. Pick random candidates
2. Improve each candidate with respect to a quality measure

Iterative greedy descent

Within cluster point scatter: \[ W(C) = \frac12\sum_{k=1}^K\sum_{x\in C_k}\sum_{y\in C_k} d(x,y) \] Sum distances between points that occupy the same cluster.

Between cluster point scatter: \[ B(C) = \frac12\sum_{k=1}^K\sum_{x\in C_k}\sum_{y\not\in C_k} d(x,y) \]

Total scatter: \[ T = \frac12\sum_x\sum_y d(x,y) = W(C) + B(C) \]

Iterative greedy descent algorithm for clustering.

• All variables quantitative
• Squared Euclidean distance

Note that \[ \sum_{x,x'}(x-x')^2 = \sum_{x,x'}(x-\overline x + \overline x-x')^2 = \\ \sum_{x,x'}(x-\overline x)^2 + \sum_{x,x'}(x'-\overline x)^2 - 2\sum_{x,x'}(x-\overline x)(x'-\overline x) = \\ 2\sum_{x}(x-\overline x)^2 -2\sum_{x'}(x'-\overline x)\sum_{x}(x-\overline x) \]

And that \(\sum_{x}(x-\overline x) = \left(\sum x - N\overline x\right) = (N\overline x- N\overline x) = 0\).

Using \(\sum_{x,x'}(x-x')^2 = 2\sum_x(x-\overline x)^2\) it follows that \[ W(C) = \frac12\sum_k\sum_{x\in C_k}\sum_{x'\in C_k}\|x-x'\|^2 = \sum_k|C_k|\sum_{x\in C_k}\|x-\overline x_k\|^2 \]

It follows that for a particular cluster assignment \(C\), the within group scatter is completely determined by the means or centroids of each cluster.

1. For each chosen mean \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the centroids \(x_k=\frac{1}{|C_k|}\sum_{x\in C_k}x\).
3. If no cluster assignment changes, the algorithm terminates.

Once the means are fixed, minimizing distances to assigned mean corresponds to picking the closest mean for each point.

Once the clusters are fixed, the within group scatter is determined by the centroid.

Data matrix is X. Number of clusters is k.

kct = kmeans(X,k)
kct$cluster # holds cluster index for points in X
kct$centers # holds coordinates for cluster centers
kct$tot.withinss # holds within-group scatter

Data matrix is X. Number of clusters is k.

from sklearn import cluster
model = cluster.KMeans(k).fit(X)
model.labels_          # holds cluster indices
model.cluster_centers_ # holds coordinates for cluster centers

Data matrix is X. Number of clusters is k.

from sklearn import cluster
model = cluster.KMeans(k)
labels = model.fit_predict(X) # holds cluster indices
model.cluster_centers_

Data matrix is X. Number of clusters is k.

from sklearn import cluster
model = cluster.KMeans(k)
clusterspace = model.fit_transform(X)
model.labels_
model.cluster_centers_

The scikit-learn implementation additionally provides the possibility to change coordinates into cluster-distance space:

The point \(x\) is assigned the new coordinate \[ \hat x = (d(x,x_1), \dots, d(x,x_k)) \] consisting of distances to all cluster centers.

One way to model multimodal distributions is to add two or more densities together, weighted so that the weights sum to 1. \[ p_{\mathcal M}(x) = \lambda_1p_1(x) + \dots + \lambda_kp_k(x) \]

Most common setup: mixed Gaussians \[ p_{\mathcal M}(x) = \lambda_1p_{\mathcal N(\mu_1,\sigma_1^2)}(x) + \dots + \lambda_kp_{\mathcal N(\mu_k,\sigma_k^2)}(x) \]

Fitting a mixture model is difficult. The solution is the EM-algorithm.

Consider a two-sample mixture model, determined by \(\lambda_1,\mu_1,\mu_2,\sigma_1^2,\sigma_2^2\). Posit latent variables \(\Delta_i\) that takes the value \(1\) if \(X_i\) comes from model 1, \(0\) otherwise.

Instead of \(\Delta_i\), which is inaccessible, we work with \[ \gamma_i(\theta) = \mathbb{E}(\Delta_i|\theta,X) = \mathbb{P}(\Delta_i=1|\theta,X) \]

We call \(\gamma_i\) the responsibility of model 2 for observation i.

The Expectation Maximization algorithm alternates two steps:

Expectation Calculate a soft assignment of responsibilities to the observations, using current estimates of all parameters.

Maximization Use the responsibility estimates for a maximum likelihood estimation of the parameters.

1. Guess at parameters: Means - pick observations at random; variances - overall sample variance; weights \(\hat\lambda_j\) - \(1/k\)
2. Compute responsibilities \[ \hat\gamma^j_i = \frac {\hat\lambda_j\mathcal L_{\mathcal N(\hat\mu_j,\hat\sigma_j^2)}(x_i)} {\sum_k\hat\lambda_k\mathcal L_{\mathcal N(\hat\mu_k,\hat\sigma_k^2)}(x_i)} \]
3. Recompute weighted sample means and variances \[ \hat\mu_j = \frac {\sum_i\hat\gamma^j_ix_i} {\sum_i\hat\gamma^j_i} \qquad \hat\sigma_j^2 = \frac {\sum_i\hat\gamma_i^j(x_i-\hat\mu_j)^2} {\sum_i\hat\gamma_i^j} \\ \hat\lambda_j=\sum\hat\gamma_i^j/n \]

We had two main requirements for \(k\)-means:

1. Sum of squares dissimilarity measure.
2. All continuous numeric variables

We had two main requirements for \(k\)-means:

1. Sum of squares dissimilarity measure.
2. All continuous numeric variables

We can relax these by:

1. Picking any dissimilarity measure, changing the task of minimizing \(W(C)\).

We had two main requirements for \(k\)-means:

1. Sum of squares dissimilarity measure.
2. All continuous numeric variables

We can relax these by:

1. Picking any dissimilarity measure, changing the task of minimizing \(W(C)\).
2. Picking observations instead of synthesizing centroids.

Recall the algorithm

1. For each chosen mean \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the centroids \(x_k=\frac{1}{|C_k|}\sum_{x\in C_k}x\).
3. If no cluster assignment changes, the algorithm terminates.

Recall the algorithm

1. For each chosen mean \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the centroids \(x_k=\frac{1}{|C_k|}\sum_{x\in C_k}x\).
3. If no cluster assignment changes, the algorithm terminates.

Recall the algorithm

1. For each chosen medioid \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the centroids \(x_k=\frac{1}{|C_k|}\sum_{x\in C_k}x\).
3. If no cluster assignment changes, the algorithm terminates.

Recall the algorithm

1. For each chosen medioid \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the centroids \(x_k=\frac{1}{|C_k|}\sum_{x\in C_k}x\).
3. If no cluster assignment changes, the algorithm terminates.

Recall the algorithm

1. For each chosen medioid \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the medioids \(x_k=\text{argmin}_{x\in C_k}\sum_{x'\in C_k}d(x,x')\).
3. If no cluster assignment changes, the algorithm terminates.

Recall the algorithm

1. For each chosen medioid \(x_k\), create the clusters \(C_k=\{x:d(x,x_k) < d(x,x_{k'}) \forall k'\}\).
2. For each chosen cluster \(C_k\), calculate the medioids \(x_k=\text{argmin}_{x\in C_k}\sum_{x'\in C_k}d(x,x')\).
3. If no cluster assignment changes, the algorithm terminates.

Since the data within each \(x\) is never referenced, only the dissimilarities, the \(k\)-medioids algorithm can operate directly on dissimilarity matrices without reference to the data.

The \(k\)-means algorithm is NP-hard if either the number of clusters or the dimensionality of the data are not fixed.

For fixed \(k\) and \(d\), \(k\)-means can be solved in \(O(n^{dk+1})\).

There are several improvements: Lloyd's algorithm runs in \(O(nkdi)\). We will not describe this algorithm here.

\(k\)-medioids increases the computational complexity of the optimization step from \(O(kn)\) to \(O(kn^2)\)

One proposed strategy to improve \(k\)-medioids is to iteratively changing each medioid for another one that decreases \[ \min_{C, \{x_k\}}\sum_k\sum_{x\in C_k}d(x,x_k) \]

When no more improvement can be found, the algorithm terminates.

\(k\)-means produces a compression scheme. We'll explain on an image:

1. Subdivide the image into \(d\times d\) pixel blocks.
2. Create \(k\)-means clusters of the pixel blocks.
3. Use the list of means together with cluster indices to produce a compressed image.

Usually through examining \(W(C)\). Balance complexity of model (high \(k\)) against explanatory power (low \(W(C)\)).

• Plot \(W\) for different choices of \(k\), look for a sharp inflection point
  Splitting separated groups brings a larger drop in \(W(C)\) than splitting within a group.

• The gap statistic compares the \(\log W(C)\) curve to the correpsonding curve from uniform data and picks the point with the largest distance between the two curves.
• Pick the first value close enough (using simulated std.dev.) to a local (or global) maximum of the gap statistic.

• Pick random data points
• Pick random \(x_1\), then pick each new data point to minimize \[ \text{argmin}_{x_k}\sum_k\sum_{x\in C_k} d(x,x_k) \]
• Pick starting centers using domain knowledge