Transformations
Processes that extract or compute information.
Kernel Density Estimation
Dimensionality Reduction
A very common problem in machine learning is the presence of numerous features. This lead to a problem called the Curse of Dimensionality.
The curse of dimentionality makes it difficult to visualise the datasets with many dimensions. It also results in data sparsity, where information important for training a model is spread across many features.
This is where dimension reduction helps by reduction the number of features while retaining much of the information.
PCA
PCA summarises multiple fields of data into principal components, usually just 2 so that it is easier to visualise in a 2-dimensional plot. The 1st component will show the most variance of the entire dataset in the hyperplane, while the 2nd shows the 2nd shows the most variance at a right angle to the 1st.
Because of the strong variance between data points, patterns tend to be teased out from a high dimension to even when there’s just two dimensions. These 2 components can serve as new features for a supervised analysis.
In short, PCA finds the best possible characteristics, that summarises the classes of a feature. Two excellent sites elaborate more: setosa, quora. The most challenging part of PCA is interpreting the components.
from sklearn.decomposition import PCA
def pca_explained(X, threshold):
"""
prints optimal principal components based on threshold of PCA's explained variance
Args
----
X: (df, array) of features
threshold: (float) % of explained variance as cut off point
"""
features = X.shape[1]
for i in range(2, features):
pca = PCA(n_components = i).fit(X)
sum_ = pca.explained_variance_ratio_
percent = sum(sum_)
print('{} components at {:.2f}% explained variance'.format(i, percent*100))
if percent > threshold:
break
pca_explained(X, 0.85)
# 2 components at 61.64% explained variance
# 3 components at 77.41% explained variance
# 4 components at 86.63% explained variance
MDS
Multi-Dimensional Scaling (MDS) is a type of manifold learning algorithm that to visualize a high dimensional dataset and project it onto a lower dimensional space - in most cases, a two-dimensional page. PCA is weak in this aspect.
sklearn gives a good overview of various manifold techniques.
from adspy_shared_utilities import plot_labelled_scatter
from sklearn.preprocessing import StandardScaler
from sklearn.manifold import MDS
# each feature should be centered (zero mean) and with unit variance
X_fruits_normalized = StandardScaler().fit(X_fruits).transform(X_fruits)
mds = MDS(n_components = 2)
X_fruits_mds = mds.fit_transform(X_fruits_normalized)
t-SNE
t-Distributed Stochastic Neighbor Embedding (t-SNE) is a powerful manifold learning algorithm for visualizing clusters. It finds a two-dimensional representation of your data, such that the distances between points in the 2D scatterplot match as closely as possible the distances between the same points in the original high dimensional dataset.
In particular, t-SNE gives much more weight to preserving information about distances between points that are neighbors.
More behind this algorithm.
from sklearn.manifold import TSNE
tsne = TSNE(random_state = 0)
X_tsne = tsne.fit_transform(X_fruits_normalized)
plot_labelled_scatter(X_tsne, y_fruits,
['apple', 'mandarin', 'orange', 'lemon'])
plt.xlabel('First t-SNE feature')
plt.ylabel('Second t-SNE feature')
plt.title('Fruits dataset t-SNE');
LDA
Latent Dirichlet Allocation is another dimension reduction method, but unlike PCA, it is a supervised method.
It attempts to find a feature subspace or decision boundary that maximizes class separability. It then projects the data points to new dimensions in a way that the clusters are as separate from each other as possible and the individual elements within a cluster are as close to the centroid of the cluster as possible.
More from sebastianraschka.com and stackabuse.com.
from sklearn.decomposition import LatentDirichletAllocation
from sklearn.datasets import make_multilabel_classification
# This produces a feature matrix of token counts,
# similar to what CountVectorizer would produce on text.
X, _ = make_multilabel_classification(random_state=0)
lda = LatentDirichletAllocation(n_components=5, random_state=0)
X_lda = lda.fit_transform(X, y)
# check the explained variance
percent = lda.explained_variance_ratio_
print(percent)
print(sum(percent))