Welcome back for this way of the programmer segment on
how to approach a big complicated nested data structure.
The answer is to take it one step at a time.
I gave you a little preview of my approach previously which I
call understand, extract, repeat.
To illustrate this, we'll walk through extracting information from
data formatted in a way that's returned by the Twitter API.
This nested dictionary results from querying Twitter,
asking for three tweets,
matching University of Michigan.
As you'll see, it's quite a daunting data structure
even when printed with nice indentation.
Here we'll just take a little tour through it.
You can see just to get the information about three tweets,
there's a lot of stuff and there's a lot of indentation.
How many levels of nesting do we have here?
Maybe five or six.
So, this can be pretty daunting,
but we're going to take it one step at a time.
The mantra is understand, extract, repeat.
So, first we will understand it.
To understand it, you might want to just print out the first few characters of it.
I'm using that json.dumps to
pretty print it and I'm printing only the first 100 characters here,
so that it doesn't take up too much space.
We can see at least from this that it's a dictionary
and that one of its keys is called search_metadata.
If you don't want to depend on looking at it and seeing the curly brace,
here I've printed out type of res and it is a dictionary.
Whenever I have a dictionary,
the first thing to do in order to understand that
dictionary is to ask for what are the keys.
In this case there are only two keys.
There's one that's search_metadata and there's other statuses.
When I printed out the first 100 characters you could see search_metadata,
but you couldn't see statuses because that was way more than
a 100 characters in before that second key was going to show up,
because it was first going to show all of
the nested data that's part of the value of the search_metadata key.
Once we've done our first level of understanding,
we're going to discern one level we will extract.
But I want to show you another tool that can be pretty helpful
for getting an overview of what the whole data structure is going to be,
which is an outline or view of the whole nested data structure.
If I dump this data into JSON format,
I can copy it to an external site that lets me look at it in outline view.
So, I'm going to get rid of these other print statements and I'm
just going to print out the whole thing in JSON format.
Instead of printing out just the first 100 characters,
I'm going to print out the whole thing.
Now I'm going to copy the whole contents which is quite a long thing.
This is all data just for three tweets believe it or not.
Finally, I've got it all. I'm going to copy it.
We're now visiting a site called jsoneditoronline.org
and on the left side they let you paste in a string that's in JSON format.
I've copied that big long string and I'm now pasting it.
All 600 and some lines of it and I can click on
this arrow here and it's now going to give me an outline or view of it.
Remember there were two keys search_metadata and statuses.
It's telling me that search_metadata is a dictionary with nine keys.
One of them's called count completed in max_id_str and so on.
Under statuses, it's got a square brackets three which is
telling me that it's a list with three items in it.
I can see that the first element in that list is a dictionary with 24 keys in it,
and I can keep descending down there.
My goal is actually going to be to get the authors of the tweets.
So, I'm going to guess that that's in
the user key and sure enough there's a screen_name or a name.
So, if I'm very careful about this,
I could just write a very complicated expression to go
in four levels of nesting and grab a screen_name of 31brooks.
But I'm going to do this one step at a time.
I've gotten myself oriented and I might come back here to help me stay oriented.
But then I'm going to work with the code and I'm going to build up
my code one step at a time where at each step I understand what I
have at the current level of nesting and then I
extract to get something one more level of nesting in.
Let's go back to our code.
I figured out that there were two keys and that the information that I wanted to extract,
the author names were in the second of those.
They were in the value associated with the statuses.
So, I've just done something to extract the value associated with the statuses key.
This is my first extract.
I did an understand and then an extract and then I'm going to repeat.
I have extracted something at level two and now I
want to print some stuff out to help me understand what I've got.
So, I'm printing out the word level two and checking what is the type? It's a list.
Well, if it's a list, the first thing that I always want to do
is check its length and it has three items.
Now, from what we looked at in the JSON Online editor,
we also would have gotten that same information.
You can tell that we've descended to that level of the data.
Since I know that this was a query that return three tweets,
I'm going to guess that each of the items in this list
is representing one tweet from Twitter.
I've now understood at this level it's a list with
three items and so I'm now ready to extract.
I could either extract a single item or I could iterate through all the items.
What I'm going to want to do is extract all of the authors. So, I'm going to iterate.
For each thing in red is two,
I'm going to do something with it.
So, that's why I'm iterating.
here on line eight.
But as I'm developing my code,
I don't want to have to have lots of stuff showing up in the output window.
So, I actually wanna deal initially with just one item at a time.
That's why I've done what I've done here on line eight.
I'm building up a template for my code that I'm going to iterate, but in fact,
I'm only taking a slice containing one item from the result at level two.
So, I'm only going to execute lines 9 and 10 one time.
At level three, we're printing out that we've got some information about a tweet and
then I'm just getting the first 30 characters of whatever the thing
is I dumped it to be JSON and I've asked for the first 30 characters,
it looks like it's a dictionary.
So, to help in the understand phase,
I'm printing out the type of res3 and its type is that it's a dictionary.
Because it's a dictionary,
I'd like to print out the keys.
In this case, there's quite a few keys and I got to look through these and try to guess
which of these is going to have the author of the tweet and it's the user key.
Therefore, I'm going to extract the user key.
I'm going to extract the user key from
the res3 dictionary and that's going to be my level four result.
We've extracted into the variable res4 and now
it's time to make a few print statements to figure out what's in res4.
Turns out that what's in res4 is
a dictionary and it
has even more keys,
but we're going to have to look in here and decide maybe we want to extract screen_name.
Our next extract operation is going to take screen_name from res4.
Here I've got res4 and I'm extracting the screen_name.
I also decided to just find out the time when the tweet was created.
I've commented out a bunch of
the other print statements so that we're not going to have quite as busy of a printout.
Now we are printing
out 31brooks and the time at which that Tweet was created.
So, at this point if I get rid of some of the print statements that
are commented out and are distracting in the code,
I would start to have something that's reasonably compact that prints
out the screen_name and the creation time for the first tweet.
My next step once I've got enough of
the code to do this is that now I'm ready to generalize and say,
"Hey, I don't want just the one item,
I'd really like to get all of them."
So, here the change is that instead of only getting one item from res2,
we're going to get all of them.
Then I get 31brooks,
but I also get froyoho and MDuncan,
because there were three different tweets.
Once I've gotten this far,
now I really have built up my code one step at a time,
I can now simplify the code.
We could actually do something as simple as this,
where we have combined some things into more complex expressions.
So, instead of making a new variable name for each level that we descend,
we're just going to say res[statuses] and we're going to iterate through that.
Our loop variable becomes res3 and with
each of those res3 is still a pretty complicated dictionary,
but we can do res3[user] [screen_name] to get the screen_name.
And we can do [user] [created_at] to get the time that it was published.
I can run this I get the same results as the more complicated code.
Then you could try just writing
these two lines of code like this from the very beginning,
but if it doesn't work out it would be really hard to debug.
So, I really recommend building it up one step at a time,
where you just descend one level at a time into the data.
In summary, my suggestion is that if you need to extract
something from a complicated deeply nested structure,
develop your code one layer at a time.
At each step, print out what you have,
through the keys of the dictionary or
the first few characters in the printed representation of the first item.
Then extract a little more.
At the end you can remove all the print statements and collapse some of the code into
more complex expressions that give you something compact like we
see on the screen now. See you next time.