Before we can train any machine learning models, it's important to be able to inspect and understand
our data. Our models will only be as good as our data.
And so this class will focus on data science, specifically the key terminology, some simple
approaches to loading datasets and then to visualize to understand relationships in the data.
To begin with, let's go over some key terms so that we can work with our data.
First, what is data? It's units of information across different observations.
This could be as simple as a poll or a survey taken in the school, to something as complex as financial
information taken by banks. In data, there are features and features are descriptors of different qualities
or information that you want to know about your data.
For example, in medical records it could be your height, it could be your gender, it could be whether
you have a certain blood sugar level. Now, within data there is often a format called rows and columns,
rows indicate different examples. So a particular instance or observation within your data and they
can be small hundreds to thousands or they could be massive, even up to millions depending on what
you're working with. And then there are columns and columns are the different features which
are different descriptors of what you're doing with your data.
Within data science, there are also different key terms.
For example, if you've taken a statistics course, you might be familiar with the context of an outlier.
In the context of data science, an outlier has multiple meanings.
One of them could be a true statistical deviance.
So something very strange or weird about the data point that it behaves from the bulk average, but
it could be also a missing number or an incorrect value.
And it's up to you to be able to introspect your data to ask what kind of outlier this might be
and whether it should be included in your dataset.
Finally, there's pre-processing, and pre-processing is an approach to take where you prepare your data
so that machine learning models can be able to take the most information from them.
This might include dropping, missing values, imputing them, which is figuring out what these values
should be or preparing as we will learn later to scale them in a way that a model can be actually able
to compute numerical values. So let's go over an example of dataset to really anchor these principles.
In the following dataset, you're looking at various different patients and their medical histories,
specifically to ask whether or not someone might be indicative of having diabetes, a row index from 0-4.
So five examples indicate each different patient and it's an example of one patient across this data
set. The target which not all datasets have indicate a quality that you might be looking to observe.
So, for example, in the 01 case here indicates whether someone does not have diabetes which is a zero
or if someone does, which is a one. Not every dataset has a target, meaning that you might not know
a priori or beforehand what you're trying to predict.
Lastly, there are columns and columns are the features of interest.
For example, you're looking at things like blood sugar which is glucose, blood pressure.
These values can be very different depending on different patients.
And likewise they try to compare between different observations or different rows in your dataset.
The target is always a type of column in your dataset and you would like to be able to use this to be
able to build predictive models. Now to analyze different features,
let's talk about different data types. The most popular ones you will work with our numerical and categorical.
A numerical data type is a feature that should be a number.
It's discreet or an integer, which is a count value.
For example, how many coins I might have in my pocket versus a numerical number
that's a float point which is a continuous value number
like the temperature. On the other hand, there's categorical variables.
Categorical are features that are labeled. For example, we could use a mapping like a one hot encoder
which we will learn shortly to be able to represent these labels as a form of a number.
Some examples could be what kind of shape is this toy?
A circle or a square? More complex data types include images and text.
These types of data often require very complex pre-processing and so we won't touch on these, but machine
learning models can work with this kind of data. And lastly, one last thing I wanted to cover was a
NaN or not a number. This is often a missing value and corrupted value or something that is strange
in the dataset. So it's up to you to figure out whether a NaN is because it's missing or if there's
some issue with the dataset. Before we get started, I'd like to give an example of how we can convert
categorical variables into some numerical input so that a machine learning model can take it.
Imagine I'm trying to predict the type of weather I'll see.
So in the next five days I log what I see weather it's sunny, whether it's rainy, cloudy or even snowy
and so for each day I write down what I see outside and I'm trying to build a predictor that predicts
what the next day is going to be. However, I can't very well past the term sunny or rainy to a model
and so I need to convert this into some sort of number.
So I could create an arbitrary numbering scheme. So sunny could be represented by one, rainy could
be represented as, two cloudy is three and snowy is four. And thus those labels from the prior slide that
you saw can be converted into ones, threes, twos and fours depending on what kind of weather it is.
However, there is one other approach we can take and this is called the one hot encoder and the goal
of a one hot encoder is to create a vector or a way to represent a question asked of each dataset.
So in the following we still use the labelling 1, 2, 3 and 4.
But instead of specifically pointing out the 1, 2, 3 or 4 label, we're going to use a vector to represent
the weather. So in the following X we see four dimensions, X-one through X-four, and it effectively asked
the question for example, for X-one, is it sunny or not?
For X-two, is it rainy or not and so on and thus forth.
And so if we wanted to represent something like day two was sunny and day three was cloudy, we can
create a one hot encoding by replacing the answer of one whenever the weather type is correctly assessed.
So for X day two, as you can see in this following figure, We know that it was sunny outside.
So the first element is going to be one and the rest are zeros. And likewise for day three except
with respect to cloudy. Now with the one hot vector only one of these elements can be one and the rest
must be zero. So make sure that you have all of the instances or labels that you want to be able to
represent. Next we're going to work with pandas.
A very powerful Python package to be able to load and understand our dataset.
First, we're going to use something called a CSV or a comma separated value file.
There are many different ways of representing data, but many of them especially those found on cargo
can be represented as CSVs. Pandas has many different types of read functions to be able to access
information and automatically infer what the rows and columns and the features of your data
such should be. So for example I can use the term read_CSV which you will see in the challenge questions
to load this diabetes CSV dataset. I can additionally get a sneak peek of the dataset by using the
command "head" on specifically the data frame or what I load into my data frame which
is through the read_CSV command. Now the head command allows us to inspect to see if pandas
did a good job actually making the right inferences.
So as you can see here I have the pregnancy, the glucose, all the values of the medical information
and the outcome properly read in. So I feel comfortable that pandas did a good job inferring what I needed
from this dataset. Now if you're unsure what kind of way pandas is reading each of the columns of the
features, you can use a command called "dtypes" and dtypes tries to infer what different data types
are. Now it's not going to be in the same format as we talked about earlier with numerical and categorical
information, but you might see something along the lines of int, float or object.
It's up to you to decide exactly how to categorize it.
For example, is it a numerical int or is it a categorical representation of an int. And you have to
sit there and think through exactly how you want to represent your data.
But the types can give you a great way of getting a first pass understanding of whether pandas is doing
a good job at representing the information properly.
The next step is to be able to visualize your data. Prior to machine learning models
and even after you've trained your models, it's important to understand the relationship that either
models predict between the features and their outputs or even features with each other.
And so we're going to go over some strategies to be able to plot and understand datasets.
The first example I'll give is a histogram. And the goal of a histogram is to come up with
a distribution of your data. You can think of a distribution as a way of trying to figure out what
the most frequent values within your dataset are for a given column or a given feature.
This, however, requires you to bend your data as you're seeing from the figure here where I'm plotting
a very specific example of the age of the patients that I have.
Now let me give you an anchoring example to help you walk through this approach.
Now for histogram imagine I wanted to know how many coins my friends had in their pockets.
So I collect some data. I asked the question how many coins you have in your pocket and I come up with
a couple of answers. Zero represents no coins in their pockets.
One represents 1 coin. Some of them even have 10 coins in their pockets.
I can create bins where I justify the lower and the upper bounds and I consider someone in
a bin. If they are between the lower and the upper bound. For example, one of the bins could ask
do you have a "no coins" in your pocket? And it turns out in this dataset that I have here represented
by X, there's only one person with no coins in their pocket.
The next bin could be one coin and I see that there are two people that had that situation occur.
Bins do not have to be equal lengths as you can see in the very last bin where we look at more than
four coins between four and 11. So that accounts for people who have 4, 5, 6 all the way up through 11
coins in their pockets. So in this way we can choose the bin values either by providing a number and
equally spacing them or by providing the explicit edges of these bins in which to count our data in.
The next example is going to be scatter plots and while histograms can generally be very good at
getting a single distribution and in more advanced cases multiple distributions, scatter plots are
extraordinarily good at finding multidimensional relationships.
In fact, the relationship between two different features by plotting them on the X
axis horizontal and the Y axis which is vertical.
So on the following figure, we're looking at the blood pressure versus the age of the patient and
what we notice is something very strange. In fact, there are many people with zero blood pressure.
Now this is medically a very bad thing. And so therefore it must be a mistake in the filing of this
dataset. And so because of that, scatter plots can be very useful to quickly assess anomalies in
your data, particularly with regards to a very specific feature.
Now they can also help you detect trends. For example, you might expect lower blood pressure for people
who are younger and you might see some vague trend like that in the dataset right here.
To walk through exactly how to read a scatter plot, I'm going to use the following example.
So in this figure you see an X and a Y. The X is often an input feature and the Y can be either
a different feature in your dataset, but more often than not an output target.
And you'd like to understand, how does that input feature vary with regards to that output target?
Sometimes people will use the connotation, XI and YJ, which represents the input feature for an arbitrary
"Ith" data point or the arbitrary Y output point.
This basically is trying to say an arbitrary point as opposed to specifying the 1st, 2nd or last.
And so this allows us to generalize some of the mathematical tools we would like to a particular data
set. So in summary, before we wrap up, we've learned that pandas can help us work with datasets
to help us load, visualize and understand them. Of the visualisation techniques that we learned we learned
about histograms, which lets us see distributions of our data, particularly with regards to a
single feature or column. We have additionally looked at scatter plots which allow us to study the
relationship between multiple features, specifically two of them, either to features in our dataset
or even particularly an input feature and an output target. In the next class,
we're going to be talking about types of machine learning algorithms and how to best deploy them.