Calling Web APIs with Requests
Send the request, check it worked, use the data
In this lesson
requests is the library almost everyone uses to call an API from Python. This lesson goes through a whole call end to end: sending parameters and headers, checking the status before trusting anything, surviving a network that misbehaves, turning a nested response into records you control, and collecting results that arrive one page at a time.
Explain it like I’m 5
requests knocks on a web service’s door for you, waits a sensible amount of time, and hands you whatever answer comes back — including a bad one, which is why you check before you read.
A first call, and the response object
requests is not part of the standard library, so it gets installed with pip install requests (inside a virtual environment, as in Unit 5). One requests.get() gives you a response object carrying the status code, the headers, the raw text, and a .json() helper that parses the body for you.
import requests
response = requests.get("https://api.example.com/current")
print(response.status_code) # 200
print(response.ok) # True when the status is below 400
data = response.json() # parsed into dicts and lists
Query parameters and headers
Never build a URL by gluing strings together. Pass a dict as params= and requests assembles the query string properly, including escaping characters that would otherwise break the URL.
Headers carry metadata about the request rather than the question itself: who you are, what format you want back, which client is calling.
import requests
response = requests.get(
"https://api.example.com/search",
params={"q": "python books", "page": 2},
headers={"Accept": "application/json", "User-Agent": "pyli5-demo/1.0"},
)
print(response.url)
# https://api.example.com/search?q=python+books&page=2
Practice the shape without the network. Finish build_url so it joins the parameters into name=value pairs separated by &, after a ?.
def build_url(base, params):
pairs = []
for key, value in params.items():
pass # TODO: append "key=value" to pairs
return base + "?" + "&".join(pairs)
print(build_url("https://api.example.com/search", {"q": "python", "page": 2}))Inside the loop, build each pair with an f-string: f"{key}={value}", then pairs.append(...) it.
def build_url(base, params):
pairs = []
for key, value in params.items():
pairs.append(f"{key}={value}")
return base + "?" + "&".join(pairs)
print(build_url("https://api.example.com/search", {"q": "python", "page": 2}))Check the status, and survive the network
Two things go wrong with a network call, and they fail differently. The server can answer with a bad status (it replied, but with 404 or 500), or the request can never complete (the host is unreachable, or it hangs forever).
raise_for_status() turns a bad status into an exception. A timeout= stops a hung request waiting for eternity, and without one your script can block indefinitely. Both belong in every real call.
import requests
try:
response = requests.get("https://api.example.com/current", timeout=10)
response.raise_for_status()
except requests.exceptions.Timeout:
print("The API took too long to answer.")
except requests.exceptions.HTTPError as err:
print(f"The API returned an error status: {err}")
except requests.exceptions.RequestException as err:
print(f"The request could not be made at all: {err}")
else:
data = response.json()
print("Got", len(data), "fields")
Turn a nested response into clean records
An API gives you the shape that suits the provider: deeply nested, with keys you didn’t choose and extras you don’t need. Do not scatter data["results"][0]["fields"]["title"] through your program. Flatten it once, at the edge, into simple records you control.
That one step is what stops an API redesign from breaking twenty places at once. If their shape changes, you fix the flattening function and nothing else.
payload = {
"results": [
{"id": 1, "fields": {"title": "Dune", "year": 1965}},
{"id": 2, "fields": {"title": "Neuromancer", "year": 1984}},
]
}
rows = []
for item in payload["results"]:
rows.append({
"id": item["id"],
"title": item["fields"]["title"],
"year": item["fields"]["year"],
})
print(rows[0])
{'id': 1, 'title': 'Dune', 'year': 1965}
Flatten the response. Build one dict per result with the keys id, title, and year, then print each on its own line as shown.
payload = {
"results": [
{"id": 1, "fields": {"title": "Dune", "year": 1965}},
{"id": 2, "fields": {"title": "Neuromancer", "year": 1984}},
]
}
rows = []
# TODO: fill rows with flat dicts
for row in rows:
print(row["id"], row["title"], row["year"])Loop over payload["results"]. id is on the item itself; title and year live one level down inside item["fields"].
payload = {
"results": [
{"id": 1, "fields": {"title": "Dune", "year": 1965}},
{"id": 2, "fields": {"title": "Neuromancer", "year": 1984}},
]
}
rows = []
for item in payload["results"]:
rows.append({
"id": item["id"],
"title": item["fields"]["title"],
"year": item["fields"]["year"],
})
for row in rows:
print(row["id"], row["title"], row["year"])Pagination: the data arrives in slices
An API will not hand you fifty thousand records at once. It returns one page and tells you how to get the next: either a page= number you increment, or a cursor in the response you pass back.
So you loop. Ask, collect, check whether there is more, and stop when there isn’t. The one rule that saves you: always have a way for the loop to end. A pagination loop with no exit is how you accidentally send ten thousand requests.
import requests
all_items = []
page = 1
while True:
response = requests.get(
"https://api.example.com/search",
params={"q": "python", "page": page},
timeout=10,
)
response.raise_for_status()
batch = response.json()["results"]
if not batch: # an empty page means we are done
break
all_items.extend(batch)
print(f"page {page}: {len(batch)} items, {len(all_items)} total")
page += 1
if page > 50: # a safety net, never loop forever
break
Practice the loop against a fake pager, no network needed. Keep calling fetch_page and collecting items until it returns an empty list.
PAGES = {1: ["a", "b"], 2: ["c", "d"], 3: []}
def fetch_page(page):
return PAGES.get(page, [])
all_items = []
page = 1
# TODO: loop until fetch_page returns an empty list
print(all_items)
print("pages fetched:", page)Use while True:. Fetch the page, break if the batch is empty, otherwise all_items.extend(batch) and add one to page.
PAGES = {1: ["a", "b"], 2: ["c", "d"], 3: []}
def fetch_page(page):
return PAGES.get(page, [])
all_items = []
page = 1
while True:
batch = fetch_page(page)
if not batch:
break
all_items.extend(batch)
page += 1
print(all_items)
print("pages fetched:", page)Common mistake: Building the URL by gluing strings together
An f-string looks like the shortest path from parameters to a URL.
Pass params={...} and let requests build and escape the query string. Hand-built URLs break on spaces, ampersands, and any non-ASCII character.
Common mistake: Calling .json() before checking the status
The body is the interesting bit, so it’s tempting to go straight for it.
Call raise_for_status() first. An error response has a body too, and parsing it produces a confusing KeyError instead of the real problem.
Common mistake: Leaving out timeout=
It works fine on a fast connection, so nothing prompts you to add it.
Always pass timeout=. Without one, requests waits indefinitely, and an unattended script hangs silently instead of failing.
Common mistake: A pagination loop with no way out
The exit condition depends on the API behaving exactly as documented.
Break on an empty page and add a maximum-page guard. Two exits mean a surprise from the API costs you one wasted run, not ten thousand requests.
Why pass params={...} instead of building the URL with an f-string?
Characters like spaces and & must be escaped to be legal in a URL. params= handles that; string concatenation does not.
What does response.raise_for_status() do?
It converts an error status into an exception so a failed call cannot quietly continue as if it worked.
Why add a maximum-page guard to a pagination loop?
The empty-page check is the normal exit; the guard is insurance for when the API does something you didn’t expect.
Mini exercise (medium)
Write collect(fetch, limit) that walks a paginated source. Call fetch(page) starting at page 1, collect every item into one list, stop when a page comes back empty, and never fetch more than limit pages.
Have a go. Finish the code and press Run to see the result immediately, right on this page.
PAGES = {1: ["a", "b"], 2: ["c"], 3: []}
def fetch(page):
return PAGES.get(page, [])
def collect(fetch, limit):
items = []
# TODO: page from 1, stop on an empty batch or when limit is passed
return items
print(collect(fetch, 10))A while True: with two breaks: one when the batch is empty, one when the page number passes limit. Use extend, not append, so you add the items rather than the list.
PAGES = {1: ["a", "b"], 2: ["c"], 3: []}
def fetch(page):
return PAGES.get(page, [])
def collect(fetch, limit):
items = []
page = 1
while page <= limit:
batch = fetch(page)
if not batch:
break
items.extend(batch)
page += 1
return items
print(collect(fetch, 10))