Why you should understand Mathematics in Deep Learning?
Over the years I have observed two things in the prospective students of deep learning. One being that understanding mathematics is not that important when dealing with a complex subject like deep learning, I fail to understand where this notion stems from. Another one is that they fail to connect the abstract mathematics to the real world. In this blog we will talk about these problems.
Let us take up the first problem. "Mathematics is not that important to learn deep learning". This notion often grows in those who are just starting out with deep learning. It is actually understandable assuming that the person never did abstract mathematics. When you are starting out and take your first steps in the world of deep learning then I recommend to stay away from mathematics and build the intuitions first. It would make little sense if you are thrown into the pool of words like bernoulli distribution before understanding dropouts(a regularization technique used in neural networks).
It is encouraged that you understand dropouts first like the following -
"Dropouts are a technique which avoids overfitting and helps neural network to generalize. We simply switch off neurons during training(I know dropouts can also be used in test phase, but let's not get into that for now) based on a probability. This allows us to hide some data from some neurons during training phase so that asymmetry is maintained throughout the neural network and every neuron contributes during inference based on their unique learning experience. Since we are switching some neuron off, architecture of neural network changes and during test phase when we switch all the neurons on, we get average(ensemble) of all those neural network architecture."
If someone who is starting out in deep learning can understand the above text then applaud for them because they have done a commendable job. Now the next step should be to read the paper on dropouts and understand how exactly dropouts work. Here is the link to the paper - http://jmlr.org/papers/volume15/srivastava14a.old/srivastava14a.pdf . When you will start reading the paper you will need to have an understanding of mathematics and statistics to question why bernoulli distribution is being used. Why not Gaussian? Rather you need to ask yourself what exactly is a distribution in the first place?
Note - A distribution in statistics is simply a function which maps the state of a random variable to number of time that state can occur.
Also Mathematics becomes important when you realize that clubbing dropouts with batch normalization creates trouble. Check this paper out - https://arxiv.org/pdf/1801.05134.pdf . Also look up Fixup initialization.
Without thorough understanding of mathematics answering fundamental questions will become troublesome. If you really want to realize something in AI, mathematics is necessary. If you want to formulate a problem well, mathematics will help you out. I know a lot of people preach that Mathematics is not necessary to understand deep learning or Artificial Intelligence don't believe them. Don't even believe me!
Here is what Swami Vivekananda said "Do not believe a thing because you have read about it in a book. Do not believe a thing because another man has said it was true. Do not believe in words because they are hallowed by tradition. Find out the truth for yourself. Reason it out. That is realization."
Once you truly understand entropy in information theory the integrals in KL divergence will look trivial!
The second problem is what I can relate to and I empathize. The people who have the problem of being able to understand mathematics but not able to realize it, are those who have understood the importance of mathematics in the first place.
I still remember the days when I was studying pure mathematics in my college and I used to sit in the garden or even on football ground and tried to see mathematics. Those walks which I used to take to the top of small hill(which is inside our college campus) and sit and just wonder the beauty of mathematics.
I spent most of my time in college in realizing mathematics rather than doing rote learning of formulae and spit it out during examinations. I genuinely believe that mathematics should be taught in nature. Sometimes it gives me goosebumps imagining how Ramanujan might have felt when he used to do mathematics with chalk on the floor and walls of a small temple while ocean waves create a soothing sound in background mixed with melody of bells from the temple. Divinity!
I can give you a few examples from my intuitions but honestly it will require practice and patience to develop an eye for mathematics. It is similar for photographers, they see the world differently.
Here is a question I ask, "Can God create a triangle with more than 180 degrees?" - an omnipotent paradox. Think about it. The answer is no in the cartesian space but yes in curved spaces. Take up a ball and draw a triangle on top of it. The sum of the angles will be more than 180 degrees.
The subjects like number theory, topology etc may seem too abstract in the first place because they talk about n-dimensional spaces. However if we pause and notice that in the world of deep learning or machine learning, we are constantly dealing with multidimensional spaces and data. Images are inherently multidimensional(width x height x number of color channels) for example. Hence the sooner we are comfortable with mathematics the better we will be able to understand complex subjects like deep learning. There was a paper in which the authors were using topology to study how the weights evolve during training of neural network - https://www.semanticscholar.org/paper/Topology-of-Learning-in-Artificial-Neural-Networks-Gabella-Afambo/e9e306f2a288065061f854aa976525d97716190b
If you want to understand aesthetics of mathematics in detail and see how mathematicians think they I strongly recommend you to read an essay - "A mathematician's apology" by G.H.Hardy.
Following are some topics which you should learn being a deep learning researcher -
1. Linear Algebra
2. Basic Topology
3. Information Theory
4. Various Statistical Distributions
5. Probability - Frequentist as well as Bayesian
I hope that I was able to throw some light on the importance of understanding mathematics and how it will affect you positively.
Feel free to share or critique it.