Applied Unsupervised Learning with R

In the previous lesson, we have learnt about the most basic algorithms of unsupervised learning: k-means clustering and k-medoids clustering. These are not only important for practical use but are also important for understanding clustering itself. In this lesson, we will study some other advanced clustering algorithms. k-means is a general-purpose clustering algorithm that is enough for most cases, but in some special cases, depending on the type of data, advanced clustering algorithms can produce better results.

All the clustering types that we have studied so far are based on a distance metric. What if we get a dataset in which it's not possible to measure the distance between variables in a traditional sense, as in the case of categorical variables? In such cases, we use k-modes clustering. k-modes clustering is an extension of k-means clustering, dealing with modes instead of means. One of the major applications of k-modes clustering is analyzing categorical data such as survey results.

Density based clustering or DBSCAN is one of the most intuitive forms of clustering. It is easy to find naturally occurring clusters and outliers in data with this type of clustering. It doesn’t require you to define number of clusters.

Hierarchical clustering merges clusters sequentially. We can see that more clearly with help of the output of a hierarchical clustering algorithm called dendrogram.

Hierarchical clustering is of two types:

• Agglomerative

• Divisive

Let us learn about these types in detail in the video.

Let us quickly recap our learning from this lesson.

Another form of unsupervised learning called probability distribution is a part of classical statistics in mathematics text courses. With the advent of big data, the use of probability distributions in exploratory data analysis and other modelling applications has increased. Let us study the use of probability distributions as a form of unsupervised learning in this lesson.

There are two families of methods in statistics: parametric and non-parametric methods. Non-parametric methods are meant to deal with data that could take any shape. Parametric methods, by contrast, make assumptions about the shape that data takes on. These assumptions are often encoded as parameters. Let us learn about the different types in this video.

So far, we've studied parametric distributions in this lesson, but in real life, all distributions are either approximations of parametric distributions or don't resemble any parametric distributions at all. In such cases, we use a technique called Kernel Density Estimation, or KDE, to estimate their probability distributions.

Now that we’ve learned how to generate the probability density functions of datasets that don't closely resemble standard distributions, we’ll learn how to perform some tests to distinguish these nonstandard distributions from each other.

Let us quickly recap our learning from this lesson.

This lesson presents techniques for unsupervised learning that accomplish something called dimension reduction. First, we will discuss what a dimension is, why we want to avoid having too many dimensions, and the basic idea of dimension reduction.

The dimensions of a dataset are nothing more than the collection of distinct numbers that are required to describe observations in it. Let us learn about the various dimension reduction techniques.

Market basket analysis or Affinity Analysis is a method that allows us to take high-dimensional data and reduce it to something that is simple and manageable without losing too much information along the way. In market basket analysis, our goal is to generate rules that govern the data. Let us understand this better through an example of a grocery store in the upcoming video.

Now that we have prepared our data for the main steps of market basket analysis, let us consider the factors that will determine the parameters we will use in our algorithm.

The next type of dimension reduction method we will cover is called PCA. This is a very common technique used by researchers in a wide variety of fields. Let us learn about PCA and its implementation in this video.

Let us quickly recap our learning from this lesson.

Unsupervised learning is concerned with analyzing the structure of data to draw useful conclusions. In this lesson, we will examine methods that enable us to use the structure of data to compare datasets. The major methods we will look at are hash functions, analytic signatures, and latent variable models.

Imagine that you want to send an R script to your friend. However, you and your friend have been having technical problems with your files – maybe your computers have been infected by malware, or maybe a hacker is tampering with your files. So, you need a way to ensure that your script is sent intact to your friend, without being corrupted or changed. Let us understand the different means to overcome this situation

Each time you sign your name, your signature should look the same. To verify whether a signature matches, we need to check for substantial similarity but not perfect identity. The method for encoding image data that enables quick and approximate comparisons between datasets is referred as an analytic signature. Before we create an analytic signature, let us do some data preparation, as outlined in the next exercise.

Now that we have learnt about Analytic signatures, as next step, we can compare these two signatures, to see whether they have mapped our different images to different signature values.

So far, we have only used hashes and analytic signatures to compare two images or four short strings. However, there is no limit to the unsupervised learning methods that can be applied to hashes or analytic signatures. Let’s see how.

Latent variable model’s express data in terms of a small number of variables that are hidden or latent. By finding the latent variables that correspond to a dataset, we can better understand the data and potentially even understand where it came from or how it was generated. Let us learn this better through different theories of intelligence.

Let us quickly recap our learning from this lesson.

Data analysis often begins with an implicit assumption that all observations are valid, accurate, and trustworthy. Unsupervised learning is used in various instances to detect anomalous behavior, for example, to identify fraudulent transactions in a bank, to find defective products by manufactures and so on. Let us study several anomaly detection techniques in the following sections.

Anomaly detection is simplest in the univariate case, that is, when each observation is only one number. For univariate data, anomaly detection consists of looking for outliers. R's built-in boxplot function makes an initial exploratory check for outliers quite easy. Let us learn this better through the following exercise.

Parametric anomaly detection, as explained in this video, is common practice. However, its applicability depends on whether the parametric assumptions behind it are valid.

Collective anomalies are a collection of related data instances that are anomalous to the entire data set. Contextual anomalies are observations that are considered anomalies only because of their neighbors. Let us learn these concepts better through an example in this video.

Kernel density estimation provides a natural way to test whether these new results are anomalous, even without having specialized knowledge of the blood test being used or even of medicine in general. To calculate Kernel density, we need to specify the bandwidth which is a measure of the variance of the normal distributions that we are using to model our data. Let us learn this in detail through the following video.

Let us quickly recap our learning from this lesson.