2022-04-25 Skolotāju konference LiepU

https://playground.tensorflow.org

Derivative rules

Constant rule

Power rule

Exponent rule

Chain rule

Product rule

Quotient Rule

Reciprocal rule

SGD

MAE derivative

${\mathcal{L}}_{MAE}=\sum |y^{\prime }-y|$$\mathcal{L}_{MAE} = \sum |y'-y|$

${\mathcal{L}}_{MAE}=|a|=\sqrt{a^2}=(a^2{)}^{\frac{1}{2}}$$\mathcal{L}_{MAE} = |a| = \sqrt{a^2} = (a^2)^{\frac{1}{2}}$

$\frac{{\mathcal{L}}_{MAE}}{\partial y^{\prime }}=?$$\frac{\mathcal{L}_{MAE}}{\partial y'} = ?$

$\frac{1}{n}=n^{-1}$$\frac{1}{n} = n^{-1}$

$\frac{1}{n^{10}}=n^{-10}$$\frac{1}{n^{10}} = n^{-10}$

$\begin{array}{c}\frac{{\mathcal{L}}_{MAE}}{\partial a}=\\ \frac{1}{2}(a^2{)}^{\frac{1}{2}-1}\cdot \frac{a^2}{\partial a}=\\ \frac{1}{2}(a^2{)}^{\frac{1}{2}-1}\cdot 2a=\\ a\cdot (a^2{)}^{-\frac{1}{2}}=\\ \frac{a}{(a^2{)}^{\frac{1}{2}}}=\\ \frac{a}{\sqrt{(a^2)}}=\\ \frac{a}{|a|+ϵ}\end{array}$$\frac{\mathcal{L}_{MAE}}{\partial a} =\\ \frac{1}{2}(a^2)^{\frac{1}{2}-1} \cdot \frac{a^2}{\partial a} =\\ \frac{1}{2}(a^2)^{\frac{1}{2}-1} \cdot 2a =\\ a \cdot (a^2)^{-\frac{1}{2}} =\\ \frac{a}{(a^2)^{\frac{1}{2}}} =\\ \frac{a}{\sqrt{(a^2)}} =\\ \frac{a}{|a| + \epsilon}$

Linear function

$Linear(x,W,b)=W\cdot x+b$$Linear(x, W, b) = W \cdot x + b$

$\frac{Linear(x,W,b)}{\partial W}=x$$\frac{Linear(x, W, b)}{\partial W} = x$

$\frac{Linear(x,W,b)}{\partial x}=W$$\frac{Linear(x, W, b)}{\partial x} = W$

$\frac{Linear(x,W,b)}{\partial b}=1\cdot b^0=1$$\frac{Linear(x, W, b)}{\partial b} = 1 \cdot b^0 = 1$

Sigmoid function

$\sigma (x)=\frac{1}{1+e^{-x}}$$\sigma(x) = \frac{1}{1 + e^{-x}}$

$\begin{array}{c}\frac{\sigma (x)}{\partial x}=\frac{1}{1+e^{-x}}=(1+e^{-x}{)}^{-1}=\\ \frac{(1+e^{-x})}{\partial x}\cdot -1(1+e^{-x}{)}^{-2}=\\ e^{-x}\cdot \frac{-x}{\partial x}\cdot -1(1+e^{-x}{)}^{-2}=\\ \frac{e^{-x}}{(1+e^{-x}{)}^2}=\sigma (x)(1-\sigma (x))\end{array}$$\frac{\sigma(x)}{\partial x} = \frac{1}{1 + e^{-x}} = (1+e^{-x})^{-1} =\\ \frac{(1+e^{-x})}{\partial x} \cdot -1(1+e^{-x})^{-2}=\\ e^{-x}\cdot \frac{-x}{\partial x} \cdot -1(1+e^{-x})^{-2} =\\ \frac{e^{-x}}{(1+ e^{-x})^2} = \sigma(x) (1 - \sigma(x))$

reciprocal rule = chain & power rule $dx\frac{1}{f(x)}=f(x{)}^{-1}=-f(x{)}^{-2}\cdot \frac{f(x)}{dx}$$dx\; \frac{1}{f(x)} = f(x)^{-1} = -f(x)^{-2} \cdot \frac{f(x)}{dx}$

exponent rule $\frac{e^{f(x)}}{dx}=e^{f(x)}\cdot \frac{f(x)}{dx}$$\frac{e^{f(x)}}{dx} = e^{f(x)} \cdot \frac{f(x)}{dx}$

import numpy as np
import matplotlib.pyplot as plt

x = np.array(
    [
     [2002, 300_000],
     [2012, 100_000],
     [2018, 1_000],
     ])

y = np.array(
    [
     [2000],
     [10_000],
     [30_000]
    ]
)

x_mean = np.mean(x, axis=0)
x_std = np.std(x, axis=0)
x  = (x - x_mean) / x_std
print(x.shape)

y_mean = np.mean(y, axis=0)
y_std = np.std(y, axis=0)
y = (y - y_mean) / y_std

W_1 = np.ones((2, 4))
b_1 = np.zeros((4, ))

W_2 = np.ones((4, 1))
b_2 = np.zeros((1, ))

alpha = 0.1

losses = []
for epoch in range(10):
    print(x)
    linear_1_w = W_1.T @ x[:, :, np.newaxis]    
    linear_1 = linear_1_w[:, :, 0] + b_1 
    sigmoid_2 = 1 / (1 + np.exp(-linear_1))    
    linear_2_w = W_2.T @ sigmoid_2[:, :, np.newaxis]
    y_prim = linear_2_w[:, :, 0] + b_2

    print(y)
    print(y_prim)

    loss = np.mean(np.abs(y_prim - y))
    #loss = np.mean((y - y_prim)** 2)
    losses.append(loss)

    d_loss = (y_prim - y)/(np.abs(y_prim - y) + 1e-8)
    #d_loss = -2*(y - y_prim)

    d_W_2 = sigmoid_2[:, :, np.newaxis] @ d_loss[:, :, np.newaxis]
    d_b_2  = 1 * d_loss

    
    d_x_2 = W_2 @ d_loss[:, :, np.newaxis]    
    d_x_2 = d_x_2[:, :, 0]
    

    d_x_sigmoid = (np.exp(-linear_1)/(1 + np.exp(-linear_1))**2) 
    d_x_sigmoid_chain = d_x_sigmoid * d_x_2

    d_W_1 = x[:, :, np.newaxis] @ d_x_sigmoid_chain[:, np.newaxis, :]
    d_b_1 = 1 * d_x_sigmoid_chain.shape

    W_1 = W_1 - np.mean(d_W_1, axis=0) * alpha
    b_1 = b_1 - np.mean(d_b_1, axis=0) * alpha

    W_2 = W_2 - np.mean(d_W_2, axis=0) * alpha
    b_2 = b_2 - np.mean(d_b_2, axis=0) * alpha


plt.plot(losses)