Missing sign in deriving sigmoid function

Question

I'm working through Andrej Karpathy's awesome introduction to neural networks and the backpropagation algorithm, and am trying to differentiate the sigmoid function:

$$ \sigma(x) = \frac{1}{1+e^{-x}} $$

My understanding is the quotient rule holds that given an equation $y=\frac{t}{b}$ the derivative of the function should be:

$$y' = \frac{t'b - tb'}{b^{2}}$$

After applying the quotient rule to the sigmoid function, I thought the unsimplified result would be:

$$ \frac{-e^{-x}}{(1 + e^{-x})^{2}} $$

Because $t'$ is 0, which I thought would yield $ 0 - tb' $ in the numerator, or $-e^{-x}$. However, Karpathy's unsimplified derivative looks like:

$$ \frac{e^{-x}}{(1 + e^{-x})^{2}} $$

Does anyone know why the negative sign in the numerator gets dropped? I'd be very grateful for any help others can offer with this question!

the derivative of $e^{-x}$ is $-e^{-x}$, together with the minus sign from your formula, they cancel out — barmanthewise, Dec 24 '17 at 18:08
@barmanthewise I thought the derivative of $e^{-x}$ is just $e^{-x}$, but is $e^{-x}$ really $-e^{-x}$? — duhaime, Dec 24 '17 at 18:13
no, thats the derivative if $e^x$, look up a thing called 'chain rule' — barmanthewise, Dec 24 '17 at 18:14
Ah yes, you're quite right, I was using the exponent rule to handle $e^{-x}$ and my book didn't show negative exponent examples. This is exactly what I was overlooking. If you make this an answer I'll mark it right. — duhaime, Dec 24 '17 at 18:16
Related: https://math.stackexchange.com/questions/78575/derivative-of-sigmoid-function-sigma-x-frac11e-x — Hans Lundmark, Dec 24 '17 at 20:38

score 1 · Accepted Answer · answered Dec 24 '17 at 18:18

1

$$\sigma(x) = \frac{1}{1 + e^{-x}}$$

Use the quotient rule, don't forget about the MINUS sign from the rule, and the MINUS sign due to the derivative of $e^{-x}$.

$$\sigma'(x) = \frac{-(-e^{-x})}{(1 + e^{-x})^2} = \frac{e^{-x}}{(1+e^{-x})^2}$$

answered Dec 24 '17 at 18:18

Enrico M.

26,114

1

I had missed the second minus sign, thank you for spelling it out. – duhaime Dec 24 '17 at 18:21

score 0 · Answer 2 · edited Dec 24 '17 at 18:19

0

better you write $$\sigma(x)=\frac{e^x}{e^x+1}$$ and then the first derivative is given by $$\sigma'(x)=\frac{e^x\cdot (e^x+1)-e^x\cdot e^x}{(e^x+1)^2}$$

edited Dec 24 '17 at 18:19

duhaime

207
2
11

answered Dec 24 '17 at 18:11

Dr. Sonnhard Graubner

95,283

@Dr.SonhardtGraubner I think that's clever. This is exactly the rule I wanted to apply, and multiplying the numerator and den by $e^{x}$ does simplify things a bit. – duhaime Dec 24 '17 at 18:20
1

and you can not lose a minus sign – Dr. Sonnhard Graubner Dec 24 '17 at 18:22

Missing sign in deriving sigmoid function

2 Answers2