Wiki Python for data

Python for data: lists, dictionaries and comprehensions

The basics that show up in every data script: comprehensions, counting by key, stable sorting and None.

The Python examples in this wiki do not run in the browser (it has no Python), but the output written here is not from memory: the site test runs each one with python3 and requires exactly this text.

Comprehensions

A comprehension builds a list (or dictionary, or set) from another one, with an optional filter.

PythonOutput checked by running Python 3

orders = [
    {"id": 101, "customer": "Ana", "amount": 230.0, "status": "delivered"},
    {"id": 103, "customer": "Ana", "amount": 80.0, "status": "canceled"},
    {"id": 104, "customer": "Carla", "amount": 180.0, "status": "delivered"},
    {"id": 106, "customer": "Carla", "amount": None, "status": "pending"},
]
delivered = [o["id"] for o in orders if o["status"] == "delivered"]
by_id = {o["id"]: o["amount"] for o in orders}
customers = {o["customer"] for o in orders}
print(delivered)
print(by_id[106])
print(sorted(customers))

Output (checked)

[101, 104]
None
['Ana', 'Carla']

Counting and summing by key

collections.Counter counts; defaultdict avoids the "if the key does not exist, create it".

PythonOutput checked by running Python 3

from collections import Counter, defaultdict

statuses = ["delivered", "canceled", "delivered", "pending", "delivered"]
print(Counter(statuses).most_common(2))

total = defaultdict(float)
for customer, amount in [("Ana", 230.0), ("Carla", 180.0), ("Ana", 900.0)]:
    total[customer] += amount
print(dict(total))

Output (checked)

[('delivered', 3), ('canceled', 1)]
{'Ana': 1130.0, 'Carla': 180.0}

Sorting by a key

sorted accepts key; for descending order use reverse=True or a minus sign on the key. Python's sort is stable: ties keep the input order.

PythonOutput checked by running Python 3

rows = [("Bruno", 1050.0), ("Ana", 1130.0), ("Diego", 1050.0)]
print(sorted(rows, key=lambda x: x[1], reverse=True))
print(sorted(rows, key=lambda x: (-x[1], x[0])))

Output (checked)

[('Ana', 1130.0), ('Bruno', 1050.0), ('Diego', 1050.0)]
[('Ana', 1130.0), ('Bruno', 1050.0), ('Diego', 1050.0)]

None is not zero

None is Python's NULL, but the semantics are different: adding None raises an error instead of propagating.

PythonOutput checked by running Python 3

values = [230.0, None, 180.0]
try:
    print(sum(values))
except TypeError as error:
    print("TypeError:", error)
print(sum(v for v in values if v is not None))

Output (checked)

TypeError: unsupported operand type(s) for +: 'float' and 'NoneType'
410.0

Compare with is None, not == None (PEP 8).
At volume, replace the Python loop with a vectorized operation (pandas, Spark): the row-by-row loop is what costs the most.

Sources

Shared knowledge from Genesis Innovation. Not a claim about the person behind this site.

PreviousSubqueries, IN and EXISTS NextGenerators and lazy evaluation