February 1, 2023

# Rethinking linear algebra part two: ellipsoids in data science

1 Our exploration of eigenvectors still continues

This article is still going to have to do with eigenvectors and PCA, and this post still will not cover LDA (linear discriminant analysis). Hereby I would like you to have more natural links of the data science concepts with eigenvectors.

In the 2nd short article, we have actually covered the following points:

You can imagine linear improvements with matrices by determining displacement vectors, and they usually look like vectors swirling.
Diagonalization is finding an instructions in which the displacement vectors do not swirl, which is equivalent to finding brand-new axis/basis where you can explain its direct improvements more straightforwardly. But we need to consider diagonalizability of the matrices.
In direct measurement reduction such as PCA or LDA, we mainly use types of matrices called favorable guaranteed or positive semidefinite matrices.

In the last short article we have seen the following points:

PCA is an algorithm of computing orthogonal axes along which data “swell” the a lot of.
PCA is equivalent to determining a new orthonormal basis for the information where the covariance between parts is no.
You can minimized the dimension of the information in the new coordinate system by neglecting the axes representing small eigenvalues.
Covariance matrices enable direct change of rotation and expansion and contraction of vectors.

* Let me first present some mathematical facts and how I signify them throughout this article beforehand. If you are allergic to mathematics, relax or please go back to my former posts.

* In the last post, I represented the covariance of information as, based upon Pattern Recognition and Machine Learning by C. M. Bishop.

I believe they are all crucial when you learn linear algebra for information science of maker knowing. And you should be mindful that, in the context of maker learning or information science, just an extremely restricted type of matrices are crucial, which I have actually been explaining throughout this post.

Covariance matrices are genuine symmetric matrices, and likewise they are favorable semidefinite. That suggests you can constantly diagonalize covariance matrices, and their eigenvalues are all equal or greater than 0.
PCA is equivalent to discovering axes of quadratic curves in which gradients are most significant. The values of quadratic curves increases the most in those directions, and that means the instructions explain terrific offer of info of information distribution.
Intuitively measurement decrease by PCA amounts to fitting a high dimensional ellipsoid on data and cutting off the axes corresponding to little eigenvalues.

I emphasized that the axes are more crucial than the surface area of the high dimensional ellipsoids, but in this short article lets focus more on the surface of ellipsoids, or I would rather state basic quadratic curves. After likewise seeing how to draw ellipsoids on data, you would see the following points about PCA or eigenvectors.

Even if you currently understand PCA to some degree, I hope this short article offers you with deeper insight into PCA, and a minimum of after reading this article, I think you would be more or less able to visually control eigenvectors and ellipsoids with the Numpy and Maplotlib libraries.

2 Rotation or forecast?

You initially make a basic ellipsoid symmetric about xyz axis utilizing polar coordinates, and you can turn the whole ellipsoid with rotation matrices. If you put in a rotation matrix which diagonalize the covariance matrix of information and a list of 3 radiuses, you can rotate the original ellipsoid so that it fits the information well.

The discussion above can be generalized to spaces with dimensions higher than 3. When is an orthonormal matrix and a vector, you can project to or turn it to, where and. To put it simply, which means you can turn back to the initial point with the rotation matrix.

Lets consider a function, where is a real symmetric matrix. You can always diagonalize real symmetric matrices, so the formula suggests that the shapes of quadratic curves largely depend on eigenvectors.

* represents an inner product of and.

In the last post, I mentioned that if is a real symmetric matrix, you can diagonalize with a rotation matrix, such that, where. I likewise explained that PCA is a case where, that is, is the covariance matrix of specific information.

Mathematically, you need to think about the determinant of the rotation matrix. You can do a “cube rotation” when, and in the case above was, and you required to flip one axis to make the factor. In the example in the figure listed below, you can match the basis. This likewise can be generalized to greater measurements, but that is also beyond the scope of this article series. If you are truly interested, you need to prepare some coffee and treats and books on linear algebra, and some weekends.

To be specific, you can not naively multiply or for rotation. Lets take a part of information I revealed in the last post as an example. In the figure listed below, I forecasted data on the basis.

In the preliminary position, the edges of the cube are aligned with the 3 orthogonal black axes, with one corner of the cube located at the origin point of those axes. The purple dot signifies the corner of the cube directly opposite the origin corner. The cube is turned in three measurements, with the origin corner staying fixed in place. After the rotation with a pivot at the origin, the edges of the cube are now lined up with a new set of orthogonal axes, displayed in red. You might understand that more plainly with an equation:. In short this rotation suggests you keep relative position of, I indicate its collaborates, in the brand-new orthonormal basis. In this post, let me call this a “cube rotation.”

* We are visiting information of the shapes of quadratic “curves” or “functions” in the next section.

You might have seen that you can refrain from doing a “cube rotation” in this case. If you make the coordinate system with your left hand, like you may have performed in science classes in school to discover Flemings guideline, you would soon understand that the coordinate systems in the figure above do not match. You require to flip the instructions of one axis to match them.

Assume that you have got an orthonormal rotation matrix which diagonalizes. In the last article I stated diagonalization is equivalent to discovering new orthogonal axes formed by eigenvectors, and in the case of this area you got brand-new orthonoramal basis which are in red in the figure below. When you replace the orginal orthonormal basis with as in the best side of the figure below, you can understand the forecast as a rotation from to by a rotation matrix.

Next, lets see what rotation is. In case of rotation, you must envision that you turn the point in the exact same coordinate system, rather than projecting to other coordinate system. You can turn by multiplying it with. This rotation looks like the figure listed below.

I think you at least saw that rotation and forecast are generally the very same, which is only a matter of how you take a look at the coordinate systems. I would state the concept of forecast is more essential through out this article.

* This article may look like a mathematical writing, however I would state this is more about computer science. I offered top priority to visualizing necessary mathematical concepts in my article series.

* You have to keep it in mind that are all discrepancies.

Even if this section has actually been confusing to you, you just need to keep one point in your mind: we have actually been talking about general quadratic curves, however in PCA, you just need to consider a case where is a covariance matrix, that is. PCA represents the case where you shift and rotate the curve (a) into (a)”. Deducting the mean of information from each point of data represents moving quadratic curve (a) to (a). Determining eigenvectors of represents determining a rotation matrix such that the curve (a) comes to (a)” after using the rotation, or forecasting curves on eigenvectors of. Notably we are only talking about the covariance of certain data, not the distribution of the data itself.

You can shift these quadratic curves so that their center points concern the origin, without rotation, and the resulting curves are as follows The curves can be all signified as.

In linear measurement reduction, or at least in this post series you generally need to think about ellipsoids. However ellipsoids are simply one type of quadratic curves. In the last short article, I discussed that when the center of a D dimensional ellipsoid is the origin point of a typical coordinate system, the formula of the surface area of the ellipsoid is as follows:, where satisfies particular conditions. To be concrete, when is the surface of a ellipsoid, needs to be positive and diagonalizable certain.

In this post mainly (a)”, (g)”, (h)”, and (i)” are essential. General equations for the curves is as follows.

* You do not have to think excessive about what the “semi” of the term “favorable semi-definite” implies fow now.

As you can see, is a genuine symmetric matrix. As I have actually pointed out consistently, when all the components of a symmetric matrix are real values and its eigen values are, there exist orthogonal/orthonormal matrices such that, where. Let be an orthogonal matrix such that.

( i)”:.

* Just in case you have an interest in a little more mathematical sides: it is known that if you turn all the points on the curve with the rotation matrix, those points are mapped into a brand-new quadratic curve. That means the rotation of the original quadratic curve with (or rather turning axes) allows getting rid of the terms. It is known that when, with proper translations and rotations, the quadratic curve can be mapped into one of the types of quadratic curves in the figure listed below, depending on coefficients of the initial quadratic curve. And the conversation up until now can be generalized to higher dimensional areas, but that is beyond the scope of this post series. Please consult decent books on linear algebra around you for further details.

, where.

Let be, then the quadratic curves can be simply signified with a matrix and a 3-dimensional vector as follows:, where,. General quadratic curves are roughly categorized into the 9 types listed below.

( a)”:.

* Real symmetric matrices are diagonalizable, and positive guaranteed matrices have just positive eigenvalues. Covariance matrices, whose displacement vectors I visualized in the last 2 short articles, are known to be symmetric genuine matrices and favorable semi-defintie. The surface area of an ellipsoid which fit the information is, not.

( g)”:.

( h)”:.

4 Eigenvectors are gradients and often variances.

* But as I have repeatedly mentioned, ellipsoid which fit data well is.

The quadratic curves in the figure above are all “curves” in my term, which can be represented as or. If you replace of (g)”, (h)”, and (i)” with, you can translate the “curves” as “functions” which are denoted as. This may sounds too obvious to you, and my point is you can picture how values of “functions” modification only when the inputs are 2 dimensional.

You can understand what I have discussed in another way: eigenvectors, to be precise eigenvectors of real symmetric matrices, are gradients. And in case of PCA, I indicate when eigenvalues are likewise variations. Prior to explaining what that implies, let me describe a little of the completely typical facts on mathematics. I believe you can comprehend functions in 2 methods if you have variables. One is a regular “functions”, and the others are “curves”. “Functions” get an input and provides an output, just as well as typical functions you would think of. “Curves” are rather sets of such that.

The formulas of (g)”, (h)”, and (i)” represent each type of, and thier curves appear like the three graphs listed below.

The 2nd article: I focused on what sort of linear transformations convariance matrices allow, by visualizing displacement vectors. And those vectors appear like extending and swirling into directions of eigenvectors of.
The third post: We straight found instructions where certain information circulation “swell” the most, to find that information swell the most in directions of eigenvectors.
In this article, we have seen PCA represents just one case of quadratic functions, where the matrix is a covariance matrix. The quadratic function increases the many when you go in the instructions of eigenvectors corresponding to big eigenvalues. That implies data samples have bigger variations when projected on the eigenvectors. Therefore you can cut off eigenvectors corresponding to small eigenvectors because they maintain little info about data, and that is comparable to fitting an ellipsoid on information and cutting off axes with small radiuses.

* You may have seen the curve above in the context of optimization with stochastic gradient descent. The origin of the curve above is an infamous saddle point, where gradients are all in any directions however not a local optimum or minimum. Points can be stuck in this point throughout optimization.

Particularly in case of PCA, is a covariance matrix, hence. Eigenvalues of are all equal to or higher than. And it is understood that in this case is the variance of information forecasted on its corresponding eigenvector. Thus, if you project, quadratic curves formed by a covariance matrix, on eigenvectors of, you get. This reveals that you can re-weight, the collaborates of information predicted on eigenvectors of, with, which are variations. As I pointed out in an example of data of test scores in the last post, the bigger a variation is, the more the feature described by vary from sample to sample. Simply put, you can neglect eigenvectors representing small eigenvalues.

.

I have described PCA in three different ways over 3 posts.

That is a great tip why primary components corresponding to large eigenvectors contain much information of the information circulation. And you can also interpret PCA as a “climbing” a bowl of, as I have visualized when it comes to (g) type curve in the figure above.

As we have actually seen,, the eigenvalues of the covariance matrix of information are differences or information when forecasted on it eigenvectors. At the very same time, when you fit an ellipsoid on the information, is the radius of the ellipsoid representing. Therefore disregarding information forecasted on eigenvectors representing small eigenvalues is comparable to cutting of the axes of the ellipsoid with little radiusses.

In case of (h), the same realities hold. In this case, you can also come down the curve.

* Please assume that the terms “functions” and “curves” are my original words. I use them simply in case I stop working to use functions and curves properly.

If you get information like the left side of the figure below, a lot of description on PCA would just fit an oval on this information distribution. After reading this posts series so far, you would have discovered to see PCA from different perspectives like at the best side of the figure listed below

* Let be a covariance matrix, and you can diagonalize it with an orthogonal matrix as follow:, where. At the end enables the reverse rotation.

* You need to be cautious that even if you slice a type (h) curve with a place the resulting cross area does not fit the initial information well since the equation of the cross section is The figure below is an example of slicing the same as the one above with, and the resulting cross section.

When a symmetric genuine matrices have 2 eigenvalues, the circulation of quadratic curves can be approximately categorized to the following 3 types.

In the second section I explained that you can express quadratic functions in an extremely simple way by forecasting on eigenvectors of.

And in reality, when start from the origin and go in the instructions of an eigenvector, is the gradient of the instructions. When you limit the distribution of to a system circle, you can see that more plainly. Like in the figure below, in case, which is categorized to (g), the distribution appears like the left side, and if you restrict the distribution in the unit circle, the distribution appears like a bowl like the middle and the right side. When you move in the direction of, you can climb the bowl as high as, in as high as.

5 Ellipsoids in Gaussian distributions.

.

I made some modules so that you can see the grape bunch from several angles. This may look very simple to you, but the locations of berries are organized thoroughly so that it looks like they are positioned around a stem which the berries are not too near each other

Yasuto Tamura.

I have explained that if the covariance of a data circulation is, the ellipsoid which fits the circulation the very best is. You may have seen the part elsewhere. It is the exponent of basic Gaussian distributions:. It is known that the eigenvalues of are, and eigenvectors corresponding to each eigenvalue are also respectively. Thus simply as well as what we have actually seen, if you project on each eigenvector of, we can transform the exponent of the Gaussian circulation.

* To be mathematically precise about changing variations of normal distributions, you have to consider for instance Jacobian matrices.

The programs code I developed for this post is completly available here.

Let be and be, where. Just as we have seen,. Hence.

[2] 理工系新課程 線形代数 基礎から応用まで, 培風館 、( 2017 ).

[4] これなら分かる 応用数学教室 最小二乗法からウェーブレットまで, 金谷健一著 、 共立出版, (2019 ), pp.165-208.

In the last article, I mentioned that if is a genuine symmetric matrix, you can diagonalize with a rotation matrix, such that, where. If you put in a rotation matrix which diagonalize the covariance matrix of information and a list of three radiuses, you can turn the initial ellipsoid so that it fits the information well.

[3] これなら分かる 最適化数学 基礎原理から計算手法まで, 金谷健一著 、 共立出版, (2019 ), pp. 17-49.

[5] サボテンパイソン https://sabopy.com/

In truth our exploration of ellipsoids, or PCA still continues, just as Star Wars series still continues. Particularly if I need to explain an algorithm named probabilistic PCA, I require to discuss the “Bayesian world” of artificial intelligence. Most machine learning algorithms covered by major introductory textbooks tend to be dependent and too deterministic on the size of data. Many of those algorithms have another “parallel world,” where you can handle error in much better ways. I hope I can likewise blog about them, and I may prepare another trilogy for such PCA. But I will not disappoint you, like “The Phantom Menace.”.

This results above demonstrate that, by predicting data on the eigenvectors of its covariance matrix, you can factorize the initial multi-dimensional Gaussian circulation into a product of Gaussian circulations which are unimportant to each other. At the same time, that is the potential limit of estimating information with PCA. This idea is going to be more crucial when you believe about more probabilistic ways to deal with PCA, which is more robust to lack of data.

If you can manage quadratic curves, improving and rotating them, you can make a design of a grape of olive lot on Matplotlib. I made a program of making a design of a bunch of berries on Matplotlib using the module to draw ellipsoids which I introduced earlier. You can examine the codes in this page.

[1] C. M. Bishop, “Pattern Recognition and Machine Learning,” (2006 ), Springer, pp. 78-83, 559-577.

.

Appendix: making a model of a lot of grape with ellipsoid berries.

[Refereces]

Covariance matrices, whose displacement vectors I visualized in the last 2 posts, are known to be symmetric genuine matrices and positive semi-defintie. In this article, we have actually seen PCA corresponds to just one case of quadratic functions, where the matrix is a covariance matrix. * Let be a covariance matrix, and you can diagonalize it with an orthogonal matrix as follow:, where.

* I have no concept how numerous people on this earth need making such models.

I have actually described PCA over 3 posts from various viewpoints. If you have actually been patient enough to read my article series, I think you have actually gained some deeper insight into not only PCA, however likewise direct algebra, and that should be valuable when you find out or teach data science.

Data Science Intern at DATANOMIQ.
Majoring in computer system science. Currently studying mathematical sides of deep learning, such as largely connected layers, CNN, RNN, autoencoders, and making study products on them. Begun intending at Bayesian deep learning algorithms.