What is the Sigmoid Function? How it is implemented in Logistic Regression?

According to Wikipedia, the activation function of a node defines the output of that node given an input or set of inputs in terms of an artificial neural network. A standard integrated circuit can be seen as a digital network of activation functions that can be “ON” (1) or “OFF” (0), depending on the input.

There are different types of activation functions

· Binary Step · Linear · Sigmoid · Tanh · ReLU · Leaky ReLU · Parameterised ReLU· Exponential Linear Unit· Swish · Softmax

Today, we will discuss more on the common type of activation, sigmoid function.

What is the Sigmoid Function?

The sigmoid function is a mathematical function having a characteristic “S” — shaped curve, which transforms the values between the range 0 and 1. The sigmoid function also called the sigmoidal curve or logistic function. It is one of the most widely used non- linear activation function.

The mathematical expression for sigmoid:

Image for post
Figure1. Derivative and Integral of Sigmoid Function

Graph

Image for post
Figure 2. Graph of Sigmoid Function

In the above graph, if the value of x goes to positive infinity then the predicted value of y will become 1 and if it goes to negative infinity then the predicted value of y will become 0.

I assume you know the logistic regression, which is the common algorithm used for binary classification or when the value of the target variable is categorical in nature. Logit function or sigmoid is used to predict the probabilities of a binary outcome. For example, we use logistic regression for classification in spam detection, fraud detection etc.

Let’s walk through one of the examples of implementing Logistic Regression in Python.

First, we have to import the required libraries.

Image for post

let’s load the dataset, today we are going to use with some doctors on heart attack patients. We copied the data from here to the excel sheet.

Our target is to predict whether the patient will have had a second heart attack within 1 year (yes = 1). We have two independent variables; one is whether the patient completed a treatment consisting of anger control practices (yes=1). The other IV is a score on a trait anxiety scale (a higher score means more anxious).

Image for post

I assume you do have a basic understanding of training the datasets. Now, we will use the “train_test_split” function in Scikit-learn to split the data into training and testing sections.

We have used the StandardScaler from Scikit-Learn to rescale the data which brings values that may have extremely different ranges or units.

Image for post

Now, we’ll import the logistic regression algorithm from sci-kit-learn and feed into the logistic regression and then create an instance of the classifier and fit it to the training data.

Image for post

Once we train and fit the data, we will be able to see the coefficient and intercept for the respective variable.

Image for post
Image for post

Below is the z (equation) in the sigmoid function with respective coefficient and intercept.

Image for post

Calculation of z for the different variable for the test data using an equation from above.

Image for post

Even, we can get the decision function or z, we calculated above with the decision function from the logistic_clf variable.

Image for post

We will define the sigmoid function as below.

Image for post

We already know that sigmoid function will convert real value between 0 and 1. We can test that scenario with this function with our calculated z or decision function. For example, we calculated z1 as -1.061 which is not between 0 and 1. Below we can see it was converted to 0.052.

Image for post

Else, we can use predict_prob class:

Image for post

Here, we use the sigmoid or logit function to map predicted values to probabilities. The functions will map any real value into another value which will be between 0 and 1 or in other word predictions to probabilities.

Decision boundary

What after the conversion to probabilities value, from our sigmoid function? To map the binary class, we will decide some threshold value, which varies on the use case, to classify the values into two different categories. The common threshold is 0.5.

Image for post
Image for post
Figure 3. Decision Boundary for the prediction

Making Predictions

Using our knowledge of sigmoid functions and decision boundaries, we can now write a prediction function. A prediction function in logistic regression returns the probability of our observation being positive, True, or “Yes”.

For example, if our threshold was .5 and our prediction function returned .7, we would classify the observation as a person will have a 2nd heart attack with a 70% chance. If our prediction was .2 we would classify the observation as a person will have a 2nd heart attack with a 20% chance. As the probability gets closer to 1, our model is more confident that the observation being positive.

Leave a Comment

Your email address will not be published. Required fields are marked *