Strings & Text Processing¶
What this lesson gives you
Strings are immutable sequences of Unicode characters. Mastering f-strings, common methods, and the immutability gotcha is daily-driver knowledge.
Estimated time: 20 min read · Part: Data Structures That Carry Real Work
Learning objectives
- Understand Python's Unicode string model and why encoding matters in production
- Master the complete set of essential string methods for data cleaning and parsing
- Write f-strings confidently including format spec, alignment, and debug
=form - Avoid the O(n²) concatenation bug and use
"".join()correctly - Apply regex for pattern matching and extraction when string methods fall short
A Python string is an immutable sequence of Unicode code points. Python 3 stores strings internally in one of three encodings (Latin-1, UCS-2, or UCS-4) depending on the highest code point used, but this is transparent to you — you work with a uniform sequence of characters regardless. Immutability means every "modification" operation creates a new string object; the original is never changed.
Most text work involves chaining the rich built-in string methods. The full method list is large, but a focused subset of around 15 methods handles 90% of real production text processing: parsing logs, cleaning API responses, building URIs, formatting output for humans.
The Essential String Methods¶
# Cleaning and normalizing
s = " Ada Lovelace "
s.strip() # "Ada Lovelace" remove leading/trailing whitespace
s.lstrip() # "Ada Lovelace " left only
s.rstrip() # " Ada Lovelace" right only
s.strip().lower() # "ada lovelace" chain methods freely
# Splitting and joining
"a,b,c".split(",") # ["a", "b", "c"]
" a b c ".split() # ["a", "b", "c"] no arg: split on ANY whitespace
"-".join(["2026", "06", "27"]) # "2026-06-27"
"\n".join(["line1", "line2"]) # "line1\nline2"
# Searching and checking
"hello world".find("world") # 6 (-1 if not found)
"hello world".index("world") # 6 (raises ValueError if not found)
"hello world".count("l") # 3
"report.pdf".endswith(".pdf") # True
"config.yaml".startswith("con") # True
"hello world".replace("l", "L") # "heLLo worLd"
# Testing string content
"abc123".isalpha() # False (has digits)
"abc".isalpha() # True
"123".isdigit() # True
"abc123".isalnum() # True
" ".isspace() # True
# Case manipulation
"ada lovelace".title() # "Ada Lovelace"
"Ada".upper() # "ADA"
"ADA".lower() # "ada"
"Ada".swapcase() # "aDA"
# Padding and alignment
"42".zfill(5) # "00042" zero-pad on left
"left".ljust(10) # "left "
"right".rjust(10) # " right"
"center".center(12, "-") # "---center---"
| Method | Purpose | Returns | Notes |
|---|---|---|---|
strip(chars) | Remove leading/trailing whitespace or chars | str | Without arg: all whitespace |
split(sep, maxsplit) | Split on separator | list[str] | No arg: split on any whitespace |
join(iterable) | Join sequence with separator | str | Called on separator: ",".join([...]) |
replace(old, new) | Replace all occurrences | str | Add count arg to limit replacements |
find(sub) | First index of substring | int | Returns -1 if not found |
startswith(prefix) | Test beginning | bool | Accepts tuple of prefixes |
endswith(suffix) | Test ending | bool | Accepts tuple of suffixes |
lower() / upper() | Case conversion | str | Use for case-insensitive comparison |
encode(encoding) | str → bytes | bytes | Default "utf-8" |
partition(sep) | Split into 3-tuple at first sep | tuple | Always returns 3 parts |
f-strings: The Modern Formatting Standard¶
Introduced in Python 3.6 (PEP 498), f-strings are the fastest, most readable formatting option. They evaluate expressions at runtime inside {} delimiters embedded directly in the string literal. The format spec mini-language after : controls number formatting, alignment, padding, and precision — the same spec used by format().
name = "Ada"
qty = 3
price = 19.5
score = 0.9234
# Basic expression embedding — any valid Python expression
f"Hello, {name}!" # "Hello, Ada!"
f"{qty} × {price} = {qty * price}" # "3 × 19.5 = 58.5"
f"{name.upper()!r}" # "'ADA'" (!r adds repr())
f"{name!s} {name!a}" # str() and ascii() conversion
# Number formatting — the format spec after :
f"{price:.2f}" # "19.50" 2 decimal places
f"{price:08.2f}" # "00019.50" width 8, zero-padded
f"{score:.1%}" # "92.3%" percentage
f"{1_000_000:,}" # "1,000,000" thousands separator
f"{255:#010x}" # "0x000000ff" hex with prefix, padded
# Alignment
f"{'left':<10}|" # "left |" left-align in width 10
f"{'right':>10}|" # " right|" right-align
f"{'center':^10}|" # " center |" center
f"{'fill':*^10}|" # "**fill****|" fill with *
# Debug form (Python 3.8+): variable=value
f"{price=}" # "price=19.5"
f"{qty * price = :.2f}" # "qty * price = 58.50"
# Multi-line f-string
report = (
f"Customer: {name}\n"
f" Items: {qty}\n"
f" Total: ${qty * price:.2f}\n"
)
# Conditional expression inside f-string
f"Status: {'active' if score > 0.9 else 'inactive'}"
# "Status: active"
"19.50"
f"{price:.2f}"¶
"92.3%"
f"{score:.1%}"¶
"price=19.5"
f"{price=}"¶
Format spec mini-language reference
The format spec syntax is [[fill]align][sign][#][0][width][grouping][.precision][type]. Common types: f (float), d (int), s (string), x (hex), o (octal), b (binary), e (scientific), % (percentage). The same spec works in format(value, spec) and str.format() — once you know it, it transfers everywhere.
Encoding, Bytes, and the Unicode Model¶
Python 3 maintains a strict separation between text (str, Unicode characters) and binary data (bytes, raw octets). Every time text crosses a system boundary — file I/O, network, subprocess — it must be encoded (str → bytes) or decoded (bytes → str). Failing to specify an encoding explicitly is the source of countless UnicodeDecodeError and garbled-text bugs in production.
# str (text) vs bytes (binary)
text = "Hello, 世界" # str: Unicode, abstract characters
raw = text.encode("utf-8") # bytes: b'Hello, \xe4\xb8\x96\xe7\x95\x8c'
back = raw.decode("utf-8") # str again: "Hello, 世界"
len(text) # 9 characters
len(raw) # 13 bytes (CJK characters take 3 bytes each in UTF-8)
# ALWAYS specify encoding when opening files
with open("data.txt", "w", encoding="utf-8") as f:
f.write("Hello, 世界\n")
with open("data.txt", encoding="utf-8") as f:
content = f.read()
# Common encoding pitfall: platform default != utf-8 on Windows
# open("file.txt") on Windows might use cp1252 — always be explicit
# Practical: check and normalize encoding in a data pipeline
def safe_decode(data: bytes, fallback="replace") -> str:
"""Try UTF-8 first, fall back gracefully."""
try:
return data.decode("utf-8")
except UnicodeDecodeError:
return data.decode("latin-1", errors=fallback)
Building strings in a loop with + is O(n²)
Because strings are immutable, result += piece inside a loop creates a brand-new string each iteration, copying all prior characters. For a loop of n iterations with growing strings, that's 1+2+3+...+n = O(n²) total work. The canonical fix: collect pieces in a list and call "".join(parts) once at the end — O(n). CPython does have a limited optimization for simple += patterns, but it is fragile and unreliable in non-trivial loops. Always use join.
import timeit
# SLOW: O(n²) — each += copies the whole string built so far
def slow_concat(n):
result = ""
for i in range(n):
result += str(i) + ","
return result
# FAST: O(n) — collect, then join once
def fast_concat(n):
parts = []
for i in range(n):
parts.append(str(i))
return ",".join(parts)
# FASTEST: generator expression into join
def fastest_concat(n):
return ",".join(str(i) for i in range(n))
n = 10_000
slow = timeit.timeit(lambda: slow_concat(n), number=100)
fast = timeit.timeit(lambda: fast_concat(n), number=100)
print(f"slow: {slow:.3f}s fast: {fast:.3f}s speedup: {slow/fast:.1f}x")
# typical: slow: 1.8s fast: 0.12s speedup: 15x
slow: 1.842s fast: 0.118s speedup: 15.6x
Regular Expressions: When String Methods Aren't Enough¶
String methods handle fixed patterns cleanly. When the pattern is variable — "find a date in any of three formats," "extract all URLs," "validate an email address" — regular expressions (the re module) are the right tool. A regex is a compact pattern language that describes a set of strings. The trade-off: they're powerful and concise, but cryptic. Use them when you need them; don't reach for them when split() or replace() suffices.
import re
log_line = "2026-06-27 14:32:01 ERROR user_id=1042 msg='Payment failed'"
# re.search: find first match anywhere in string
match = re.search(r"\d{4}-\d{2}-\d{2}", log_line)
if match:
print(match.group()) # "2026-06-27"
# re.findall: return all non-overlapping matches as list
re.findall(r"\d+", log_line) # ["2026", "06", "27", "14", "32", "01", "1042"]
# Named groups: extract structured data
pattern = re.compile(
r"(?P<date>\d{4}-\d{2}-\d{2}) "
r"(?P<time>\d{2}:\d{2}:\d{2}) "
r"(?P<level>\w+) "
r"user_id=(?P<uid>\d+)"
)
m = pattern.search(log_line)
m.group("date") # "2026-06-27"
m.group("level") # "ERROR"
m.group("uid") # "1042"
m.groupdict() # {'date': '2026-06-27', 'time': '14:32:01', 'level': 'ERROR', 'uid': '1042'}
# re.sub: replace with regex
cleaned = re.sub(r"\s+", " ", " too many spaces ").strip()
# "too many spaces"
# Compile patterns you reuse — avoids recompiling on every call
EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
emails = EMAIL_RE.findall("contact ada@x.io or bob@y.com for help")
# ['ada@x.io', 'bob@y.com']
"2026-06-27" ['ada@x.io', 'bob@y.com'] WHY Why are Python strings immutable? Immutability enables safe string interning (CPython caches common strings), makes strings hashable (usable as dict keys), and simplifies reasoning about identity vs equality. When two variables hold the same short string, CPython may intern them to the same object. The cost is that mutation requires new objects — but most text operations return new strings anyway, so the practical impact is mainly the concatenation-in-loop gotcha.
HOW How does "".join() beat += ? join() receives the complete list of pieces upfront. It makes two passes: first to compute the total output length, then to allocate exactly one buffer and copy all pieces into it — two O(n) passes = O(n). Repeated += allocates a new buffer on every iteration, copying all prior content each time — O(n²) total. The same "collect then batch" principle applies whenever you know the full input before building the output.
WHERE Where does encoding fail most in production? Most commonly: reading CSV files written on Windows (cp1252) on a Linux server without specifying encoding. Or scraping web pages and not decoding with the charset from the HTTP Content-Type header. Always set encoding="utf-8" on file opens, use response.text (not response.content) with requests, and validate encoding at ingestion boundaries rather than letting errors surface mid-pipeline.
Knowledge check
InterviewYou are concatenating tens of thousands of substrings in a loop and it's slow. What is the idiomatic, efficient fix?
- Use a tuple instead — tuples concatenate faster.
- Append each piece to a list and call
"".join(parts)once — O(n) instead of O(n²). - Switch the loop to
whileinstead offor. - Use
+=but add strings in reverse order.
Answer
Append each piece to a list and call "".join(parts) once — O(n) instead of O(n²).
Strings are immutable, so += in a loop rebuilds the entire string each iteration (quadratic). str.join() receives the complete list of parts, pre-allocates one buffer sized for the total, and copies everything in a single linear pass. The pattern is canonical Python: collect fragments, join once. The same approach works in any language where string concatenation is linear in the result length.
Knowledge check
Concept CheckWhat is the difference between str.find("x") and str.index("x")?
- They are identical; both raise ValueError if not found.
find()returns -1 if not found;index()raisesValueError. Usefind()when absence is normal;index()when absence is a bug.find()searches from the right;index()from the left.find()is case-insensitive;index()is case-sensitive.
Answer
find() returns -1 if not found; index() raises ValueError. Use find() when absence is normal; index() when absence is a bug.
Both return the index of the first occurrence of the substring. The only difference is failure behavior: find() returns -1 (safe for checking); index() raises ValueError (useful when absence is unexpected and you want to fail loudly). For simple "is it there?" checks, the idiomatic Python is just "x" in s , which returns a bool directly.
Knowledge check
ProductionYou open a CSV file with open("data.csv") and get a UnicodeDecodeError in production on Linux but not on your Mac. What is the likely cause?
- Linux doesn't support Unicode at all.
- The CSV has a syntax error that Linux is stricter about.
- The file uses a Windows encoding (cp1252 or latin-1) and
open()withoutencoding=picks the OS default — UTF-8 on Linux, often cp1252 on Windows where the file was created. - Python 3 on Linux is an older version without Unicode support.
Answer
The file uses a Windows encoding (cp1252 or latin-1) and open() without encoding= picks the OS default — UTF-8 on Linux, often cp1252 on Windows where the file was created.
When you don't specify encoding= , Python uses the locale's default: UTF-8 on modern Linux/macOS, but often cp1252 (Windows-1252) on Windows. A file written on Windows with cp1252 contains bytes (like \x92 for a curly apostrophe) that are invalid UTF-8. On Mac it may "work" if the locale default happens to be latin-1-compatible. Always specify encoding="utf-8" (or the correct encoding) explicitly.
Exercise: Parse and Clean an Application Log
Given a list of raw log strings (some malformed), write a function parse_logs(lines) that returns a list of dicts with keys timestamp, level, and message. Skip lines that don't match the expected format. Normalize levels to uppercase. Return only ERROR and WARNING entries. Example input: "2026-06-27 14:32:01 error payment failed".
Show solution
import re
from typing import Optional
LOG_PATTERN = re.compile(
r"^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+"
r"(?P<level>\w+)\s+"
r"(?P<msg>.+)$"
)
ALERT_LEVELS = {"ERROR", "WARNING", "WARN"}
def parse_logs(lines: list[str]) -> list[dict]:
results = []
for line in lines:
line = line.strip()
if not line:
continue
m = LOG_PATTERN.match(line)
if not m:
continue # skip malformed lines
level = m.group("level").upper()
if level not in ALERT_LEVELS:
continue # only errors and warnings
results.append({
"timestamp": m.group("ts"),
"level": level,
"message": m.group("msg").strip(),
})
return results
# Test
logs = [
"2026-06-27 14:32:01 error payment failed",
"2026-06-27 14:32:02 info user logged in",
"2026-06-27 14:32:03 WARNING disk at 90%",
"malformed line without timestamp",
"",
"2026-06-27 14:32:05 ERROR db connection timeout",
]
for entry in parse_logs(logs):
print(f"[{entry['level']}] {entry['timestamp']} — {entry['message']}")
# [ERROR] 2026-06-27 14:32:01 — payment failed
# [WARNING] 2026-06-27 14:32:03 — disk at 90%
# [ERROR] 2026-06-27 14:32:05 — db connection timeout
Key takeaways
- Strings are immutable Unicode sequences; every "modification" returns a new string.
- Core methods:
strip(),split(),join(),replace(),startswith()/endswith(),find(). - f-strings are the modern, fast formatting tool — use
f"{value:.2f}",f"{price=}", and alignment specs. - Use
"".join(parts)— never+=in a loop — for O(n) concatenation. - Always specify
encoding="utf-8"on file opens to avoid platform-dependent failures. - Reach for
rewhen string methods can't express the pattern; compile repeated patterns.