Okay, we're just about at the project for this course.
If you reflect on this specialization as a whole,
you'll realize that you started with
probably little or no understanding of Python,
progressed through the basic control structures
and libraries included with the language.
With the help of a digital textbook,
moved on to more high level representations
of data and functions with objects,
and now started to explore
third-party libraries that exist for Python,
which allow you to manipulate and display images.
This is really quite an achievement.
You've also no doubt found that as you progress,
that demands on you to engage in
more self-discovery have also increased.
While the first assignments were maybe straight forward,
the ones in this week require you to struggle a bit more
with planning and debugging code as you develop.
But you persisted.
I'd like to share with you just one more set of
features before we head over to the project.
The OpenCV library contains
mechanisms to do face detection on images.
The technique used is based on Haar cascades,
which is a machine-learning approach.
Now, we're not going to go into
the machine learning bits.
We have another specialization on that called
The Applied Data Science with Python specialization.
You can take that after this one if you're interested.
But here we're going to treat
OpenCV more like a black box.
OpenCV comes with train models for detecting faces,
eyes, and smiles which we'll be using.
You can train models for detecting other things,
like hot dogs or flutes.
If you're interested in that,
I'd recommend you check out the OpenCV docs
on how to train a cascade classifier,
and here's a URL.
However, in this lecture,
we just want to use the current classifiers to see if
we can detect portions of an image which are interesting.
So the first step is to load
OpenCV and the XML-based classifiers.
So import cv2 as cv,
and then we'll create a face_cascade classifiers.
So cv.CascadeClassifier, we'll load that from read only.
I put it in the Haarcascade_frontalface_default.xml.
With the classifiers loaded,
we now want to try and detect a face.
Let's pull on the picture we played with the last time.
So image equals cv.imread
and will bring in the floyd picture.
We'll convert it to grayscale using the cvtColor image.
So gray equals cv.cvtColor,
and we convert this to grayscale.
The next step is to use the face_cascade classifier.
I'll let you go explore the docs if you'd like to.
But the norm is to use the detectMultiscaleFunction.
This function returns a list of objects as rectangles.
The first parameter is an ndarray of the image.
So faces equals face_cascade.detectMultiScale,
and we just pass it in ndarray or image gray.
Now let's just print those phases out to the screen.
We'll print them out of the list.
The resulting rectangles are in the format of x, y, w,
h. Where the x and y denote
the upper left-hand corner point for the image,
and the width and height represent the bounding box.
We know how to handle this already in PIL.
So from PIL import image,
let's create a pill image objects.
So PIL image equals image.fromarray.
We pass in gray and we set the mode to luminosity.
Now let's bring in our drawing objects.
So from PIL import image draw,
and let's create a context.
So drawing equals ImageDraw.Draw
and pass in the PIL image.
Now, let's pull the rectangle out of the faces object.
So we'll take a rectangle and faces to the list.
We know there's one item in there,
so sub-zero, and now we're
just going to draw a rectangle around the bounds.
So drawing.rectangle,
passing the rectangle we're interested in,
set the outline to white,
and display the image in line,
so display PIL image.
So not quite what we're looking for,
what do you think went wrong?
Well, a quick double check of the docs and its apparent
that OpenCV is returning the coordinates as x,
y, w, h, while PIL.ImageDraw is looking for x1,
y1 and x2, y2.
So it looks like this is an easy fix.
So let's wipe our old image.
So PIL image and we'll just reload our image,
set up our drawing context,
and draw the new box.
So drawing.rectangle, and so we'll take
rec sub zero and rec sub one.
Then we'll add to that rec sub two
and rec sub three as appropriate and set outline,
and display in line.
We see the face detection
works pretty good on this image.
Note that it's apparent that this is not head detection,
but that the haarcascade file that we're
using is for eyes and a mouth.
Let's try something a little bit more complex.
Let's read in that MSI recruitment image.
So image equals cv.Imread,
and we'll bring in the msi_recruitment.gif.
Let's take a look at that image,
to remind ourselves what it looks like.
So display image.fromarray.
Remember, we have to do this,
but we don't have to pass them luminosity values
because it's in full color.
Well, what's that error about?
Looks like there's an error on a line deep
within the PIL Image.py file,
and it is trying to call
an internal private member called,
dander array underscore interface dander,
on the image object.
But this object is none.
It turns out that the root of this error is that
OpenCV can't work with gif images.
This is kind of a pain and unfortunate.
But we know how to fix this, right?
One way is that we can just open this in PIL and
then save it as a png and then open that in OpenCV.
So let's use PIL to open our image.
So PIL image equals Image.open and bring in our gif.
Now let's convert it to grayscale
for OpenCV and get the by test.
So OpenCV version is
equal to pilimage.convert luminosity,
and that is going to return to us an array.
Now, let's just write that out to a file.
So open_cv_version.save, msirecruitment.png.
Now that the conversion of format is done,
let's try reading this back into OpenCV.
So OpenCV image equals cv.imread and bring out the png.
We don't need to color convert this,
because we saved it as grayscale.
Let's try and detect faces in that image.
So faces equal face_cascaded.detectMultiScale,
and passing the mdarrays cv image.
Now, we still have our PIL color version in a gif.
So PIL image equals Image.open the gif.
So we'll set a drawing contexts.
So drawing equals ImageDraw.Draw.
Now, for each line in faces,
let's surround it with a red box.
So for x, y,
w, and h in faces.
So this might actually be new syntax for you.
Recall that faces is a list of
rectangles in the x y width and height format.
That is a list of lists.
Instead of having to do an iteration and
then manually pull out each item,
we can use something called tuple unpacking to pull out
individual items in the sub-list directly to variables.
This is really nice Python feature.
All right. So now we just need to draw our box.
So drawing.rectangle x, y
and then x plus width, can't forget this,
and y plus height and set
the outline to white, and display Pil.image.
Well, what happened here?
We see that we've detected faces,
so there's some white boxes and
then we'd drawn boxes around them.
But the colors have gone all weird.
This it turns out has to do with
the color limitations for gif images.
In short, a gif image has
a very limited number of colors.
This is called a color pallette,
after the palette artists use to mix paints.
For gifs, the pallettes can only be 256 colors.
But they can be any 256 colors.
When a new color is introduced,
it has to take the space of an old color.
In this case, PIL adds white to the pallette,
but doesn't know which color to represent,
and thus messes up the image.
Who knew there was so much to learn about image formats?
We can see what mode
an image is with the dot mode attribute.
So if we do pil_img.mode,
we can see that there's a list of
modes in the Pillow documentation,
and they correspond with
color spaces that we've been using.
For the moment though, let's just change back to RGB,
which represents color as
a three byte tuple instead of in a pallette.
So let's read in our image,
bring back our gif image, nice and clean.
Let's convert it to RGB mode. So that's pretty easy.
We just do pil_image.converts with RGB,
and let's print out this mode.
Okay. We're pretty convinced that we've changed it.
Now let's go back to drawing rectangles.
Let's get our drawing object.
Let's iterate through the face sequence again.
We'll tuple and pack as we go.
So x, y width and height in faces.
Remember again, width and
height so we have to add these appropriately.
So drawing rectangle x, y,
x plus width, y plus height,
set the outline to white.
Finally, let's display that image to the screen.
Awesome. We managed to detect
a bunch of faces in that image.
It looks like we've missed four faces.
In the Machine Learning world,
we would call these false negatives.
Something which the machine thought was not a face,
so a negative, but then it was incorrect on.
Consequently, we would call it actual faces
that were detected as true positives.
Something that the machine thought was
a face and it was correct on.
This leaves us with false positives,
something the machine thought was faced but it wasn't.
We can see that there's two of these in the image,
picking up the shadow patterns or textures in
the shirts and matching them with hard cascades.
Finally, we have a class of
true negatives where the set
of all possible rectangles at
the Machine Learning classifier could consider
where it correctly indicated
that the result was not a face.
In this case, there's many,
many, many true negatives.
There's a few ways we could try and improve this,
and really it requires a lot of
experimentation to find good values for a given image.
First, let's create a function which we'll
plot out rectangles for us in the image.
So deaf show_rects will have that pass and faces.
Let's read in our gif and convert it.
So we'll read it in and we'll convert it to
RGB in the same line.
We'll set our drawing context,
and we'll plot all of
the faces in all of the rectangles in faces.
So for x, y, width and height and faces,
and we've seen this before withdrawing.rectangles,
and we'll set the outline to white,
and finally we'll display this.
All right. So now we have a function show_faces.
So first step, we can try and binarize this image,
and it turns out that OpenCV has a built in
binarization function called threshold.
You simply pass in the image,
the midpoint, and the maximum value,
as well as a flag which
indicates whether the threshold should
be a binary or something else. So let's try this.
So we'll do cv_image_bin equals cv.thresholds.
So we pass in the image we're interested in.
I'll choose 124 our threshold,
255 for our top-end, and then cv.threshbinary.
We're just going to pull from this actually sub one,
because this function returns
a list and we want the second value.
Now, let's do the actual face detection on this.
So faces equals face_cascade.detectMultiscale.
We'll just pass in this new CV image
after it's been binarized,
and then let's call our show_recs
faces to see the results.
So that's interesting. Not better,
but we do see that there is one
false positive towards the bottom where
the classifier detected the sunglasses as
eyes and the dark shadow line as a mouth.
If you're following in the Notebook with this video,
why don't you pause things and try
a few different parameters for the threshold value?
The detect multiscale function from
OpenCV also is a couple of parameters.
The first of these is the scale factor.
The scale factor changes the size of
rectangles which are considered against the model.
That is, the hard cascades XML file.
You'd think of it as if it were changing
the size of the rectangle which are on the screen.
Let's experiment with the scale factor.
Usually, it's a small value.
So let's try 1.05.
So faces equals face_cascade.detectMulti-scale,
and we'll pass in our image and 1.05.
Now, let's render those results
to the screen through show racks.
Now, let's also try this on 1.15.
So I'll just put that in there quickly.
Then finally, let's try and do this on 1.25 as well.
So we'll put that in there, and let's give it a run.
We can see that as we change the scale factor,
we changed the number of
true and false positives and negatives.
With the scale set to 1.05,
we have seven true positives
which are correctly identified faces,
and three false negatives which are
faces which are there but not detected.
We have three false positives which are known
faces which OpenCV think are faces.
When we change this to 1.15,
we lose the false positives,
but also lose one of the true positives,
the person to the right wearing a hat.
When we change this to 1.25,
we lost more true positives as well.
This is actually a really interesting phenomena
in Machine Learning and Artificial Intelligence.
There's a trade-off between
not only how accurate the model is,
but how the inaccuracy actually happens.
So which of these three models do you think are best?
Well, the answer to that question is really, it depends.
It depends why you're trying to
detect faces and what you're going to do with them.
If you think these issues are interesting,
you might want to check out the Applied Data Science with
Python Specialization Michigan offers here on Coursera.
Beyond an opportunity to advertise,
did you notice anything else that
happened when we changed the scale factor?
It's subtle, but the speed at which
the processing ran took longer at smaller scale factors.
This is because more sub-images
are being considered for those scales.
This could also affect which method we might use.
Jupyter has nice support for timing commands.
You might have seen this before.
A line that starts with a percentage sign in
Jupyter is called a Magic function.
This isn't normal Python,
it's actually a shorthand way of writing
a function which Jupyter has pre-defined.
It looks a lot like the decorators
we've talked about in a previous lecture,
but the Magic functions were around long before
decorators were part of the Python language.
One of the built-in Magic functions
in Jupyter is called Timeit.
This repeats a piece of Python 10 times by default,
and tells you the average speed it took to complete.
Let's find the speed of detectMultiscale
when using a scale of 1.05.
So percentage Timeit, call the Magic function,
and then we just write our normal Python code.
So phase cascade.detectMultiscale(cv img and
1.05). Now, let's compare that to
the speed at scale 1.15.
So, the same thing,
timeit face_cascade.detectMultiscale(cv img 1.15).
So you can see that this is a dramatic difference,
roughly 2.5 times slower when using the smaller scale.
This wraps up our discussion of detecting
faces in OpenCV. You'll see that.
Like OCR, this is not a foolproof process,
but we can build on the work others have done in
Machine Learning and leverage powerful libraries to bring
us closer to building a turnkey Python-based solution.
Remember that the detection mechanism
isn't specific to faces,
and that's just the hat cascade's training data we used.
On the web, you'll be able to find
other training data to detect
other objects including eyes,
animals, and so forth.