Python for Data: The Big Picture
Asking a pile of facts a careful question
In this lesson
You can now fetch data, store it, and process it without falling over. This unit is about the step after that: working out what the data says.
Before any library appears, it is worth being clear about what data analysis is. It is not math. It is asking a careful question, checking that the data can honestly answer it, and then being disciplined about the answer.
Explain it like I’m 5
Data analysis is asking a pile of facts a careful question — and then checking whether the pile really answers it.
Rows and columns
Almost all analysis starts with a table. A table has rows and columns, and mixing them up is the single most common early confusion.
A row is one thing: one snack, one student, one sale, one day. A column is one fact about every thing: price, score, date. If you can say “each row is one ___”, you understand the table. If you cannot, stop and work that out before writing any code.
A table like this is called a dataset. Here is one small enough to hold in your head, written with the tools you already have.
snacks = [
{"name": "crisps", "price": 1.20, "rating": 4},
{"name": "chocolate", "price": 2.50, "rating": 5},
{"name": "apple", "price": 0.60, "rating": 3},
{"name": "biscuits", "price": 1.80, "rating": 5},
]
total = sum(snack["price"] for snack in snacks)
best = max(snacks, key=lambda snack: snack["rating"])
cheap = [snack["name"] for snack in snacks if snack["price"] < 1.50]
print(f"rows: {len(snacks)}, columns: {len(snacks[0])}")
print(f"total price: {total:.2f}")
print("highest rated:", best["name"])
print("under 1.50:", cheap)
rows: 4, columns: 3 total price: 6.10 highest rated: chocolate under 1.50: ['crisps', 'apple']
Which questions can this table answer?
This is the discipline that separates analysis from guessing. A dataset can only answer questions about the columns it actually contains.
The snack table can tell you the average price, the highest-rated item, and how many cost under $1.50. It cannot tell you which snack sells best, whether the ratings are fair, or whether chocolate is worth the money. There is no sales column, no rater column, and “worth it” is not a fact at all.
Asking a question the data cannot answer is not a small mistake. It is how confident, well-formatted, completely wrong conclusions get made.
Answer two questions the table genuinely supports. Work out the average price across all four snacks, and collect the names of every snack rated exactly 5. Plain Python, no libraries needed yet.
snacks = [
{"name": "crisps", "price": 1.20, "rating": 4},
{"name": "chocolate", "price": 2.50, "rating": 5},
{"name": "apple", "price": 0.60, "rating": 3},
{"name": "biscuits", "price": 1.80, "rating": 5},
]
# TODO: the mean price across every snack
average_price = 0
# TODO: the names of the snacks rated exactly 5
top_rated = []
print(f"average price: {average_price:.2f}")
print("rated 5:", top_rated)The average is sum(s["price"] for s in snacks) / len(snacks). For the names, filter with a comprehension: [s["name"] for s in snacks if s["rating"] == 5].
snacks = [
{"name": "crisps", "price": 1.20, "rating": 4},
{"name": "chocolate", "price": 2.50, "rating": 5},
{"name": "apple", "price": 0.60, "rating": 3},
{"name": "biscuits", "price": 1.80, "rating": 5},
]
average_price = sum(s["price"] for s in snacks) / len(snacks)
top_rated = [s["name"] for s in snacks if s["rating"] == 5]
print(f"average price: {average_price:.2f}")
print("rated 5:", top_rated)Cleaning comes first, always
The snack table above is fake, which is why it is tidy. Real data is not. It arrives with blank cells, "London" and "london" treated as different places, numbers stored as text, stray spaces, and duplicated rows.
This matters more than it sounds. An average computed over a column where a third of the values are missing is not a slightly-off average — it is an answer to a different question than the one you asked. Cleaning is not tidying-up before the real work. It is part of the real work, and it gets a whole lesson later in this unit.
Why the libraries exist
You just answered three questions with plain Python, so a fair question is why anyone needs anything else.
Try scaling what you wrote. Four rows and three columns fitted on screen. At fifty thousand rows the loop still works but is slow. At forty columns the dictionary access becomes unreadable. And “average price per category, for last month only, ignoring the missing rows” turns into a page of nested loops that you will not want to re-read tomorrow.
NumPy makes math over many numbers fast and short. pandas gives you the whole table as one object you can filter, group, and summarize in a line each. Matplotlib turns the result into a picture. Those three are the rest of this unit, and each one replaces a loop you could write yourself but would rather not.
Common mistake: Assuming the data is clean
It loads without an error, so it looks fine.
Loading successfully says nothing about the contents. Print the first rows and count the blanks before you calculate anything.
Common mistake: Confusing a row with a column
Both are just “a line of the table” when you picture a spreadsheet.
A row is one thing; a column is one fact about every thing. Say “each row is one ___” out loud before starting.
Common mistake: Answering a question the columns cannot support
The question is interesting, and the data is right there.
Check that every part of the question maps to a column. If it does not, you need different data, not cleverer code.
Common mistake: Treating one number or chart as proof
A result that took effort to produce feels like it has earned trust.
Ask what else could produce the same number. Small samples, missing rows, and outliers all fake patterns convincingly.
In a table of students and their scores, what is one row?
A row is a single thing. A column is one fact recorded for every thing.
Why clean data before analyzing it?
An average over a column that is a third empty answers a different question than the one you asked, with no error to warn you.
What is a dataset?
Rows are the things being recorded, columns are the facts recorded about them.
Mini exercise (easy)
Being able to tell an answerable question from an unanswerable one is the skill this lesson is really teaching. Given a table with the columns name, price, and rating, mark each of six questions "yes" (the columns can answer it) or "no" (they cannot).
Try it yourself. Complete the snippet and hit Run; it executes in your browser, no setup required.
# The table has these columns only: name, price, rating.
# Mark each question "yes" if those columns can answer it, "no" if they cannot.
QUESTIONS = {
"What is the average price?": "?",
"Which snack sells the most?": "?",
"Which snack has the highest rating?": "?",
"How many snacks cost under 1.50?": "?",
"Who gave chocolate its rating?": "?",
"When was the apple bought?": "?",
}
answerable = [q for q, verdict in QUESTIONS.items() if verdict == "yes"]
print("answerable:", len(answerable))
for question, verdict in QUESTIONS.items():
print(f"{verdict}: {question}")For each question, list the columns it needs. If any needed column is missing from name, price, rating, the answer is "no", however reasonable the question sounds.
# The table has these columns only: name, price, rating.
# Mark each question "yes" if those columns can answer it, "no" if they cannot.
QUESTIONS = {
"What is the average price?": "yes",
"Which snack sells the most?": "no",
"Which snack has the highest rating?": "yes",
"How many snacks cost under 1.50?": "yes",
"Who gave chocolate its rating?": "no",
"When was the apple bought?": "no",
}
answerable = [q for q, verdict in QUESTIONS.items() if verdict == "yes"]
print("answerable:", len(answerable))
for question, verdict in QUESTIONS.items():
print(f"{verdict}: {question}")