Neural Network Workshop – Lab 4 Data Shaping and Scaling

 

Cleanup

Cleanup from Last Lab

  1. To find Correlations, we label encoded the column ‘Route To Market’, which we need to one hot encode, comment out the label encoding for ‘Route To Market’ and the code to find correlations from the previous lab.

Reference Material

https://stackoverflow.com/questions/36644144/shortcut-key-for-commenting-out-lines-of-python-code-in-spyder

Full Solution 
#Encode columns using label encoding 
from sklearn.preprocessing import LabelEncoder

responsetimeencoder = LabelEncoder()
dataset['Opportunity Result'] = responsetimeencoder.fit_transform(dataset['Opportunity Result'])

suppliesgroupencoder = LabelEncoder()
dataset['Supplies Group'] = suppliesgroupencoder.fit_transform(dataset['Supplies Group'])

suppliessubgroupencoder = LabelEncoder()
dataset['Supplies Subgroup'] = suppliessubgroupencoder.fit_transform(dataset['Supplies Subgroup'])

regionencoder = LabelEncoder()
dataset['Region'] = regionencoder.fit_transform(dataset['Region'])

competitortypeencoder = LabelEncoder()
dataset['Competitor Type'] = competitortypeencoder.fit_transform(dataset['Competitor Type'])

#routetomarketencoder = LabelEncoder()
#dataset['Route To Market'] = routetomarketencoder.fit_transform(dataset['Route To Market'])
#
##What are the correlations between columns and target
#correlations = dataset.corr()['Opportunity Result'].sort_values()

[collapse]

[collapse]
Drop Poorly Correlated Columns

  1. In lab 3 we selected columns to throw out, drop those columns.

Reference Material

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html

Hint 1

The following rows were selected to be dropped:

[collapse]
Hint 2

Call this function to drop the columns:

dataset = dataset.drop()

[collapse]
Full Solution 
#Throw out unneeded columns 
dataset = dataset.drop('Client Size By Employee Count', axis=1)
dataset = dataset.drop('Client Size By Revenue', axis=1)
dataset = dataset.drop('Elapsed Days In Sales Stage', axis=1)
dataset = dataset.drop('Opportunity Number', axis=1)
dataset = dataset.drop('Opportunity Amount USD', axis=1)
dataset = dataset.drop('Competitor Type', axis=1)
dataset = dataset.drop('Supplies Group', axis=1) 
dataset = dataset.drop('Supplies Subgroup', axis=1) 
dataset = dataset.drop('Region', axis=1)

[collapse]

[collapse]

 

 

Encoding and Scaling

One Hot Encoding

  1. Use One Hot Encoding from the pandas library to encode the Route To Market column.  Don’t forget to avoid the dummy variable trap, you should have one less column than the number of values in the Route to Market column.  Prefix the new columns with Route To Market.
  2. Drop the original Route to Market column after it has been encoded.

Reference Material

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html

https://pandas.pydata.org/pandas-docs/version/0.22/generated/pandas.get_dummies.html

https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.concat.html

Hint 1

Start with the encoding:

pd.get_dummies(dataset['Route To Market'])

[collapse]
Hint 2

Add the prefix to the new columns:

pd.get_dummies(dataset['Route To Market'], prefix='Route To Market')

[collapse]
Hint 3

Drop the first column to avoid the dummy variable trap:

pd.get_dummies(dataset['Route To Market'], prefix='Route To Market', drop_first=True)

[collapse]
Hint 4

Now concat the new columns to the front of the dataset:

dataset = pd.concat([pd.get_dummies(dataset['Route To Market'], prefix='Route To Market', drop_first=True),dataset], axis=1)

[collapse]
Hint 5

Drop Route To Market Column:

dataset = dataset.drop('Route To Market', axis=1)

[collapse]
Full Solution 
#One Hot Encode columns that are more than binary
# avoid the dummy variable trap
dataset = pd.concat([pd.get_dummies(dataset['Route To Market'], prefix='Route To Market', drop_first=True),dataset], axis=1)
dataset = dataset.drop('Route To Market', axis=1)

[collapse]

[collapse]
Seperate the X (inputs) from the y (result)

  1. Set X to contain all columns except Opportunity Result.
  2. Set y to contain only Opportunity result.
  3. Array slicing is the easiest way to accomplish both, pay special attention to the section “Slicing” in the arrays indexing reference below.

Reference Material

https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.drop.html

https://pandas.pydata.org/pandas-docs/version/0.17.0/generated/pandas.DataFrame.iloc.html

https://pandas.pydata.org/pandas-docs/version/0.23/generated/pandas.DataFrame.values.html

https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.indexing.html

Hint 1

Start by creating y:

y = dataset.iloc[???].values

[collapse]
Hint 2

Here is the full answer to create y including the array slicing:

y = dataset.iloc[:, dataset.columns.get_loc('Opportunity Result')].values

[collapse]
Hint 3

To create X first drop the column Opportunity result:

X = dataset.drop('Opportunity Result', axis=1)

[collapse]
Hint 4

Use slicing again to convert the rest of the dataframe into an array:

X = dataset.drop('Opportunity Result', axis=1).iloc[:, 0:dataset.shape[1] - 1].values

[collapse]
Full Solution

Use slicing again to convert the rest of the dataframe into an array:

#Create the input data set (X) and the outcome (y)
X = dataset.drop('Opportunity Result', axis=1).iloc[:, 0:dataset.shape[1] - 1].values
y = dataset.iloc[:, dataset.columns.get_loc('Opportunity Result')].values

[collapse]
  1. Open up y in Variable explorer, note that this is a simple array containing the encoded opportunity result.
  2. Open up X in Variable explorer, also open dataset at the same time and compare.  Notice that it is the same thing missing the opportunity result column!

[collapse]
Scale the Inputs

  1. Import StandardScaler from sklearn.preprocessing.
  2. Use both fit and transform to standardize X by scaling to unit variance.

Reference Material

http://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html

Hint 1

Import the class:

from sklearn.preprocessing import StandardScaler

[collapse]
Hint 2

Instantiate a StandardScaler:

sc = StandardScaler()

[collapse]
Hint 3

Use fit_transform to rescale X:

X = sc.fit_transform(X)

[collapse]
Full Solution 
# Feature Scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit_transform(X)

[collapse]
  1. Open up X in Variable explorer, also open dataset at the same time and compare.  Notice how dramatically the data has been rescaled.

[collapse]

 

Lab Complete!

 

 

Extra Credit – The Effect of Different Scalers on Data with Outliers

Read More

http://scikit-learn.org/stable/auto_examples/preprocessing/plot_all_scaling.html#sphx-glr-auto-examples-preprocessing-plot-all-scaling-py

[collapse]