1
00:00:07,910 --> 00:00:13,545
Welcome back. Let's take a look at the code for the request with caching module.

2
00:00:13,545 --> 00:00:16,950
It's not very complicated and it's pretty instructive.

3
00:00:16,950 --> 00:00:18,600
Plus, it's good practice to read

4
00:00:18,600 --> 00:00:21,825
slightly longer code chunks and try to make sense of them.

5
00:00:21,825 --> 00:00:25,525
Let's start here with the definition of the get function.

6
00:00:25,525 --> 00:00:29,075
It takes the same parameters as request.get,

7
00:00:29,075 --> 00:00:33,740
a baseurl, and the optional params dictionary.

8
00:00:33,740 --> 00:00:38,605
But it also has a few extra optional parameters like private_keys_to_ignore,

9
00:00:38,605 --> 00:00:40,970
we'll come back to those.

10
00:00:41,170 --> 00:00:43,485
In the previous video,

11
00:00:43,485 --> 00:00:46,735
I described conceptually how a cache works.

12
00:00:46,735 --> 00:00:50,870
Now we've got a datastore where we're going

13
00:00:50,870 --> 00:00:56,350
to store the results of our previous invocations of request.get.

14
00:00:56,780 --> 00:00:59,980
But how are we actually going to implement that in Python?

15
00:00:59,980 --> 00:01:04,320
The answer is we're going to do it as a dictionary,

16
00:01:04,720 --> 00:01:12,280
where each key will represent one invocation of request.get.

17
00:01:12,280 --> 00:01:15,730
The key has to somehow encode what

18
00:01:15,730 --> 00:01:20,320
the baseurl and what the parameters were of that invocation of requests.get,

19
00:01:20,320 --> 00:01:26,185
and then the results will be some text

20
00:01:26,185 --> 00:01:31,225
that was what we got back from running request.get.

21
00:01:31,225 --> 00:01:33,730
So, in our code here,

22
00:01:33,730 --> 00:01:38,295
we're going to compute a cache key.

23
00:01:38,295 --> 00:01:42,960
We're going to make this value K by calling a function,

24
00:01:42,960 --> 00:01:46,750
a little helper function that I'll show you in a minute called make_cache_key.

25
00:01:46,750 --> 00:01:54,155
It's going to take the baseurl and the params and turn it into a string.

26
00:01:54,155 --> 00:01:58,070
We'll take a look at that make_cache_key function in a minute.

27
00:01:58,070 --> 00:02:00,950
But just think of it for now is that it's producing

28
00:02:00,950 --> 00:02:05,675
some string that we'll use as a key in the cache dictionary.

29
00:02:05,675 --> 00:02:11,465
Now the heart of this get function is this if, elif, else.

30
00:02:11,465 --> 00:02:17,839
So, we're going to check whether this cache key is in either of our two caches,

31
00:02:17,839 --> 00:02:22,975
the page specific cache or the permanent cache.

32
00:02:22,975 --> 00:02:26,820
So, if it finds it in the temporary cache,

33
00:02:26,890 --> 00:02:29,315
let's say it's here,

34
00:02:29,315 --> 00:02:33,455
then we're going to take these results and return them.

35
00:02:33,455 --> 00:02:35,680
So, if we found them in temp_cache,

36
00:02:35,680 --> 00:02:40,670
that is the cache key that we're looking for is in the temp_cache,

37
00:02:40,670 --> 00:02:43,805
then we go get the value associated with that cache key,

38
00:02:43,805 --> 00:02:45,760
and that's what we're going to return.

39
00:02:45,760 --> 00:02:48,740
You may recall that we're trying to produce the same kind of

40
00:02:48,740 --> 00:02:52,610
object that request.get produces.

41
00:02:52,610 --> 00:02:59,560
That was a response object that has not only the text but also the URL.

42
00:02:59,560 --> 00:03:02,240
Now, if we actually call request.get,

43
00:03:02,240 --> 00:03:05,930
this would be the URL that we fetched from the remote server.

44
00:03:05,930 --> 00:03:08,540
In this case, it's going to be the URL that we would have

45
00:03:08,540 --> 00:03:13,300
fetched if we had actually gone to the remote server,

46
00:03:13,300 --> 00:03:15,710
but instead, we'll produce that URL that we would

47
00:03:15,710 --> 00:03:18,440
have used if we had to go out to the remote server.

48
00:03:18,440 --> 00:03:24,815
So, that was computed up here using another little helper function.

49
00:03:24,815 --> 00:03:28,620
If it didn't find it in the temporary cache,

50
00:03:28,620 --> 00:03:29,790
it'll do the same thing,

51
00:03:29,790 --> 00:03:31,610
it'll check in the permanent cache.

52
00:03:31,610 --> 00:03:34,010
If it was in the permanent cash,

53
00:03:34,010 --> 00:03:37,820
return the text and the URL.

54
00:03:37,820 --> 00:03:41,835
If it didn't find it in either of the two caches,

55
00:03:41,835 --> 00:03:46,770
then we will actually call request.get,

56
00:03:46,770 --> 00:03:49,335
passing the baseurl in the params.

57
00:03:49,335 --> 00:03:51,690
We will add those,

58
00:03:51,690 --> 00:03:53,865
whatever the result was,

59
00:03:53,865 --> 00:03:56,900
gets added into the cache, and again,

60
00:03:56,900 --> 00:04:00,800
this is another helper function that we'll look at in a minute called add_to_cache.

61
00:04:00,800 --> 00:04:04,940
So, the basic structure is we calculate what

62
00:04:04,940 --> 00:04:09,650
the cache key ought to be and then we check in the two caches is that key there?

63
00:04:09,650 --> 00:04:11,370
Is it in either spot?

64
00:04:11,370 --> 00:04:15,585
If it's in either spot, we return the results from that cache dictionary,

65
00:04:15,585 --> 00:04:17,670
and if it's not there,

66
00:04:17,670 --> 00:04:20,240
then we call request.get and we add

67
00:04:20,240 --> 00:04:23,790
it to the cache so that it will be there the next time.

68
00:04:24,700 --> 00:04:31,400
The only other little tricky part in here and I glossed over is that we

69
00:04:31,400 --> 00:04:38,335
want these caches to live beyond the current run of the active code window.

70
00:04:38,335 --> 00:04:41,665
So to make it be a little more permanent,

71
00:04:41,665 --> 00:04:44,140
at least to live a little longer than the current run,

72
00:04:44,140 --> 00:04:46,010
we want to save it in a file.

73
00:04:46,010 --> 00:04:49,385
I've shown the cache as if it's just a dictionary,

74
00:04:49,385 --> 00:04:53,190
but actually this dictionary we're going to store in a file.

75
00:04:53,330 --> 00:04:57,200
So, we have to read this dictionary from a file,

76
00:04:57,200 --> 00:05:02,520
and I'll show you this _read_from_file as another little helper function.

77
00:05:04,600 --> 00:05:08,150
So, let's take a look at those helper functions now,

78
00:05:08,150 --> 00:05:18,000
now that we understand the basic structure The first one is this, make_cache_key.

79
00:05:18,290 --> 00:05:23,025
So, it's taking the baseurl and the params dictionary,

80
00:05:23,025 --> 00:05:25,980
and it has to make a string out of that request.

81
00:05:25,980 --> 00:05:30,935
A text string that we can use as a key in the cache dictionary.

82
00:05:30,935 --> 00:05:32,795
Now, the URL itself,

83
00:05:32,795 --> 00:05:37,819
we could just make the full URL out of this base URL and the parameters dictionary,

84
00:05:37,819 --> 00:05:41,060
but we're making something slightly different in

85
00:05:41,060 --> 00:05:44,950
our case because there are some keys that we want to exclude.

86
00:05:44,950 --> 00:05:49,355
In particular, they may be private data that's used for authentication,

87
00:05:49,355 --> 00:05:53,195
and we'll see more about this when we get to the Flickr API.

88
00:05:53,195 --> 00:05:56,330
The other thing that we want to do here is to make sure that

89
00:05:56,330 --> 00:05:59,005
we always get the parameters in the same order.

90
00:05:59,005 --> 00:06:05,430
If there is a couple of different parameters remember this dictionary, the params_d.

91
00:06:07,390 --> 00:06:10,685
The keys could come out in any order,

92
00:06:10,685 --> 00:06:13,130
and if one time we call it we get

93
00:06:13,130 --> 00:06:16,850
the keys in one order and another time we get them in a different order,

94
00:06:16,850 --> 00:06:20,990
we might fail in our lookup trying to find something in the cache.

95
00:06:20,990 --> 00:06:23,905
So, we've created a canonical order.

96
00:06:23,905 --> 00:06:27,035
We alphabetize the keys by sorting them,

97
00:06:27,035 --> 00:06:30,440
and the default order is to sort them in alphabetical order.

98
00:06:30,440 --> 00:06:33,380
Then we go through each of the keys and we just create

99
00:06:33,380 --> 00:06:37,825
a little string that puts together the key and the value.

100
00:06:37,825 --> 00:06:40,895
So, we're getting all of the keys and values from params_d,

101
00:06:40,895 --> 00:06:43,670
and we're sticking them together into

102
00:06:43,670 --> 00:06:48,814
one giant string separated by underscore characters.

103
00:06:48,814 --> 00:06:52,615
So, what we're going to get is something that looks a lot like the full URL,

104
00:06:52,615 --> 00:06:55,450
but might be a little bit different.

105
00:06:55,880 --> 00:06:59,850
Let's look at the other helper functions.

106
00:06:59,860 --> 00:07:04,020
First, let's look at this read_from_file.

107
00:07:04,820 --> 00:07:07,655
Basically, we're just taking

108
00:07:07,655 --> 00:07:11,300
the contents of a file and reading them into a Python dictionary.

109
00:07:11,300 --> 00:07:13,460
So, most of the action is right here,

110
00:07:13,460 --> 00:07:16,900
we open the file and then we read it,

111
00:07:16,900 --> 00:07:19,760
and we call json.loads(res) which takes the text

112
00:07:19,760 --> 00:07:22,955
of the file and turns it into a Python object.

113
00:07:22,955 --> 00:07:25,835
In our case, it's going to be a dictionary.

114
00:07:25,835 --> 00:07:29,260
I've wrapped this in something called try and except.

115
00:07:29,260 --> 00:07:30,860
If you haven't seen that before,

116
00:07:30,860 --> 00:07:32,120
don't worry about it too much,

117
00:07:32,120 --> 00:07:34,850
it's covered in the next course in the specialization.

118
00:07:34,850 --> 00:07:37,195
It just lets the code fail gracefully.

119
00:07:37,195 --> 00:07:42,755
So, if it can't find this file or if it can't load it as JSON,

120
00:07:42,755 --> 00:07:45,755
then we'll just return an empty dictionary.

121
00:07:45,755 --> 00:07:47,975
So it'll fail gracefully.

122
00:07:47,975 --> 00:07:53,780
So, that's reading it from the file and turning it into a Python dictionary.

123
00:07:53,780 --> 00:07:57,170
The other helper function is this add_to_cache.

124
00:07:57,170 --> 00:08:00,800
So, we've got some cash filename and we've got some new

125
00:08:00,800 --> 00:08:04,340
key and results that we want to save into the cache.

126
00:08:04,340 --> 00:08:09,545
So, we first read that dictionary using read_from_file,

127
00:08:09,545 --> 00:08:12,020
we add a key to it,

128
00:08:12,020 --> 00:08:15,455
with the value that has been provided,

129
00:08:15,455 --> 00:08:18,280
and then we just write that back out to the file.

130
00:08:18,280 --> 00:08:22,040
And write_to_file, you can see up here is another helper function

131
00:08:22,040 --> 00:08:26,315
that just opens the file and writes the dictionary out.

132
00:08:26,315 --> 00:08:29,455
So, that's a simple caching pattern that we've implemented.

133
00:08:29,455 --> 00:08:35,120
It's a wrapper around the expensive and unreliable operation request.get.

134
00:08:35,120 --> 00:08:38,225
You can just call request with caching.get,

135
00:08:38,225 --> 00:08:40,635
it returns the same thing as request.get.

136
00:08:40,635 --> 00:08:45,380
If it has a saved previous version of the request it returns those contents,

137
00:08:45,380 --> 00:08:48,600
and if not, then it calls request.get.

138
00:08:48,600 --> 00:08:52,820
It's useful because it helps you avoid problems with rate limits when you're debugging

139
00:08:52,820 --> 00:08:58,830
your code that's working with external rest APIs. I'll see you next time