Mathspp: mathematics and programming

Pt En This post is going to follow up what was discussed here , and I will show you how changing the value of the discount factor may affect what the optimal policy is. For that, consider the MDP defined below: This MDP is inspired in the example I have been using; I added an extra state and stripped the MDP of the majority of the transitions so we can focus on what is essential here. From the MDP above there is a clear contender for the title of optimal policy: $$\begin{align} &\pi(H) = E\\ &\pi(T) = D\\ &\pi(TU) = D\end{align}$$ Assume that when we enter the MDP there is a $50\%$ chance we start Hungry and a $50\%$ chance we start Thirsty. If we use the policy $\pi$ then the expected reward is $$E[R | \pi] = 0.5E[R_T | \pi] + 0.5E[R_H | \pi]$$ where $E[R_s | \pi]$ is the expected reward we get by following policy $\pi$ starting from state $s$. From my previous post we know that $E[R_T | \pi] = E[R_H | \pi] = \frac{1}{1-\gamma}$ and hence $$E[R | \pi] ...

Pt En In this previous post I defined a Markov Decision Process and explained all of its components; now, we will be exploring what the discount factor $\gamma$ really is and how it influences the MDP. Let us start with the complete example of last post: In this MDP the states are Hungry and Thirsty (which we will represent with $H$ and $T$) and the actions are Eat and Drink (which we will represent with $E$ and $D$). The transition probabilities are specified by the numbers on top of the arrows. In the previous post we put forward that the best policy for this MDP was defined as $$\begin{cases} \pi(H) = E\\ \pi(T) = D\end{cases}$$ but I didn't really prove that. I will do that in a second, but first what are all the other possible policies? Well, recall that the policy $\pi$ is the "best strategy" to be followed, and $\pi$ is formally seen as a function from the states to the actions, i.e. $\pi: S \to A$. Because of that, we must know what $\pi(H)...

Pt En In this post we will see why it is not helpful to have two consecutive layers of neurons with linear activation functions in neural networks. With just a bit of maths we can conclude that $n$ consecutive linear layers compute the exact same functions as $1$ single layer. Claim: having two fully connected layers of neurons using linear activation functions is the same as having just one layer with a linear activation function. We just have to lay down some notation in order for the maths to be doable. Assume the two consecutive layers of linear neurons are preceded by a layer with $n_0$ neurons, whose outpus are $o_1, o_2, \cdots, o_{n_0}$. Let us say that after that layer, there is a layer of $n_1$ neurons with linear activation functions $f_i(x) = a_ix + b_i$; the weight from neuron $t$ of the previous layer to the neuron $i$ of this layer is $w_{t,i}$. The second layer of neurons has $n_2$ neurons, with linear activation functions $f_i'(x) = a_i'x+b_...

Pt En It is a shame but I kind of dropped this when I was $41\%$ done... I hope I man up and finish this in the near future. After watching this video from Siraj Raval, I decided to jump right on board of the #100daysofMLcode initiative! (even though I am something like 73 days late...) The goal here is to devote (at least) 1h every day, for the next 100 days, to studying ML or writing code! According to the rules posted by Siraj, I must: Make a public pledge for this, which this post is; Make a log of everything, which this post will also be; Whenever I see something related to this #100DaysofMLCode, be supportive! Progress log For the day $0$ I wrote this post and spent quite some time thinking about what I will do throughout. I am thinking of studying several topics about ML and then writing educative posts here, for the blog. For today I wrote this twitter proof , tackling a mathematical property of neural networks with linear activation functions. ...

Pt En < change language In this post I want to showcase the beginning of what I think will be a really cool project. After reading a really nice book (that Bill Gates himself recommends) on machine learning, I decided to experiment with genetic algorithms. For that matter, the main goal here will be to develop a genetic algorithm that teaches a vacuum cleaner how it should move in a dirty room to clean it in the best way possible (that is what is happening in the animation above). The first step is to define what I mean by room : a room will be a rectangular grid where each cell has a number from $0$ to $1$. A cell with a $0$ is perfectly clean and a cell with a number $1$ is as dirty as a piece of floor can get. Below we see a $5\times 5$ room with a robot (in red) already in the middle of the room: As I read in The Master Algorithm (and frankly I found it very enlightening), when one is going to use genetic algorithms we must previously define the structu...