Skip to content

Strings & Text Processing

What this lesson gives you

Strings are immutable sequences of Unicode characters. Mastering f-strings, common methods, and the immutability gotcha is daily-driver knowledge.

Estimated time: 20 min read · Part: Data Structures That Carry Real Work

Learning objectives

  • Understand Python's Unicode string model and why encoding matters in production
  • Master the complete set of essential string methods for data cleaning and parsing
  • Write f-strings confidently including format spec, alignment, and debug = form
  • Avoid the O(n²) concatenation bug and use "".join() correctly
  • Apply regex for pattern matching and extraction when string methods fall short

A Python string is an immutable sequence of Unicode code points. Python 3 stores strings internally in one of three encodings (Latin-1, UCS-2, or UCS-4) depending on the highest code point used, but this is transparent to you — you work with a uniform sequence of characters regardless. Immutability means every "modification" operation creates a new string object; the original is never changed.

Most text work involves chaining the rich built-in string methods. The full method list is large, but a focused subset of around 15 methods handles 90% of real production text processing: parsing logs, cleaning API responses, building URIs, formatting output for humans.

The Essential String Methods

string_methods.py
# Cleaning and normalizing
s = "  Ada Lovelace  "
s.strip()           # "Ada Lovelace"    remove leading/trailing whitespace
s.lstrip()          # "Ada Lovelace  "  left only
s.rstrip()          # "  Ada Lovelace"  right only
s.strip().lower()   # "ada lovelace"    chain methods freely

# Splitting and joining
"a,b,c".split(",")          # ["a", "b", "c"]
"  a  b  c  ".split()       # ["a", "b", "c"]  no arg: split on ANY whitespace
"-".join(["2026", "06", "27"])   # "2026-06-27"
"\n".join(["line1", "line2"])    # "line1\nline2"

# Searching and checking
"hello world".find("world")   # 6      (-1 if not found)
"hello world".index("world")  # 6      (raises ValueError if not found)
"hello world".count("l")      # 3
"report.pdf".endswith(".pdf")  # True
"config.yaml".startswith("con")  # True
"hello world".replace("l", "L")  # "heLLo worLd"

# Testing string content
"abc123".isalpha()    # False  (has digits)
"abc".isalpha()       # True
"123".isdigit()       # True
"abc123".isalnum()    # True
"  ".isspace()        # True

# Case manipulation
"ada lovelace".title()   # "Ada Lovelace"
"Ada".upper()            # "ADA"
"ADA".lower()            # "ada"
"Ada".swapcase()         # "aDA"

# Padding and alignment
"42".zfill(5)                 # "00042"   zero-pad on left
"left".ljust(10)              # "left      "
"right".rjust(10)             # "     right"
"center".center(12, "-")      # "---center---"
Method Purpose Returns Notes
strip(chars) Remove leading/trailing whitespace or chars str Without arg: all whitespace
split(sep, maxsplit) Split on separator list[str] No arg: split on any whitespace
join(iterable) Join sequence with separator str Called on separator: ",".join([...])
replace(old, new) Replace all occurrences str Add count arg to limit replacements
find(sub) First index of substring int Returns -1 if not found
startswith(prefix) Test beginning bool Accepts tuple of prefixes
endswith(suffix) Test ending bool Accepts tuple of suffixes
lower() / upper() Case conversion str Use for case-insensitive comparison
encode(encoding) str → bytes bytes Default "utf-8"
partition(sep) Split into 3-tuple at first sep tuple Always returns 3 parts

f-strings: The Modern Formatting Standard

Introduced in Python 3.6 (PEP 498), f-strings are the fastest, most readable formatting option. They evaluate expressions at runtime inside {} delimiters embedded directly in the string literal. The format spec mini-language after : controls number formatting, alignment, padding, and precision — the same spec used by format().

fstrings.py
name  = "Ada"
qty   = 3
price = 19.5
score = 0.9234

# Basic expression embedding — any valid Python expression
f"Hello, {name}!"                      # "Hello, Ada!"
f"{qty} × {price} = {qty * price}"     # "3 × 19.5 = 58.5"
f"{name.upper()!r}"                    # "'ADA'"  (!r adds repr())
f"{name!s} {name!a}"                   # str() and ascii() conversion

# Number formatting — the format spec after :
f"{price:.2f}"           # "19.50"   2 decimal places
f"{price:08.2f}"         # "00019.50"  width 8, zero-padded
f"{score:.1%}"           # "92.3%"   percentage
f"{1_000_000:,}"         # "1,000,000"  thousands separator
f"{255:#010x}"           # "0x000000ff"  hex with prefix, padded

# Alignment
f"{'left':<10}|"         # "left      |"   left-align in width 10
f"{'right':>10}|"        # "     right|"   right-align
f"{'center':^10}|"       # "  center  |"   center
f"{'fill':*^10}|"        # "**fill****|"   fill with *

# Debug form (Python 3.8+): variable=value
f"{price=}"              # "price=19.5"
f"{qty * price = :.2f}"  # "qty * price = 58.50"

# Multi-line f-string
report = (
    f"Customer: {name}\n"
    f"  Items:  {qty}\n"
    f"  Total:  ${qty * price:.2f}\n"
)

# Conditional expression inside f-string
f"Status: {'active' if score > 0.9 else 'inactive'}"
# "Status: active"

"19.50"

f"{price:.2f}"

"92.3%"

f"{score:.1%}"

"price=19.5"

f"{price=}"

Format spec mini-language reference

The format spec syntax is [[fill]align][sign][#][0][width][grouping][.precision][type]. Common types: f (float), d (int), s (string), x (hex), o (octal), b (binary), e (scientific), % (percentage). The same spec works in format(value, spec) and str.format() — once you know it, it transfers everywhere.

Encoding, Bytes, and the Unicode Model

Python 3 maintains a strict separation between text (str, Unicode characters) and binary data (bytes, raw octets). Every time text crosses a system boundary — file I/O, network, subprocess — it must be encoded (str → bytes) or decoded (bytes → str). Failing to specify an encoding explicitly is the source of countless UnicodeDecodeError and garbled-text bugs in production.

encoding.py
# str (text) vs bytes (binary)
text  = "Hello, 世界"           # str: Unicode, abstract characters
raw   = text.encode("utf-8")    # bytes: b'Hello, \xe4\xb8\x96\xe7\x95\x8c'
back  = raw.decode("utf-8")     # str again: "Hello, 世界"

len(text)   # 9  characters
len(raw)    # 13  bytes (CJK characters take 3 bytes each in UTF-8)

# ALWAYS specify encoding when opening files
with open("data.txt", "w", encoding="utf-8") as f:
    f.write("Hello, 世界\n")

with open("data.txt", encoding="utf-8") as f:
    content = f.read()

# Common encoding pitfall: platform default != utf-8 on Windows
# open("file.txt") on Windows might use cp1252 — always be explicit

# Practical: check and normalize encoding in a data pipeline
def safe_decode(data: bytes, fallback="replace") -> str:
    """Try UTF-8 first, fall back gracefully."""
    try:
        return data.decode("utf-8")
    except UnicodeDecodeError:
        return data.decode("latin-1", errors=fallback)

Building strings in a loop with + is O(n²)

Because strings are immutable, result += piece inside a loop creates a brand-new string each iteration, copying all prior characters. For a loop of n iterations with growing strings, that's 1+2+3+...+n = O(n²) total work. The canonical fix: collect pieces in a list and call "".join(parts) once at the end — O(n). CPython does have a limited optimization for simple += patterns, but it is fragile and unreliable in non-trivial loops. Always use join.

concat_pattern.py
import timeit

# SLOW: O(n²) — each += copies the whole string built so far
def slow_concat(n):
    result = ""
    for i in range(n):
        result += str(i) + ","
    return result

# FAST: O(n) — collect, then join once
def fast_concat(n):
    parts = []
    for i in range(n):
        parts.append(str(i))
    return ",".join(parts)

# FASTEST: generator expression into join
def fastest_concat(n):
    return ",".join(str(i) for i in range(n))

n = 10_000
slow = timeit.timeit(lambda: slow_concat(n), number=100)
fast = timeit.timeit(lambda: fast_concat(n), number=100)

print(f"slow: {slow:.3f}s   fast: {fast:.3f}s   speedup: {slow/fast:.1f}x")
# typical: slow: 1.8s   fast: 0.12s   speedup: 15x

slow: 1.842s fast: 0.118s speedup: 15.6x

Regular Expressions: When String Methods Aren't Enough

String methods handle fixed patterns cleanly. When the pattern is variable — "find a date in any of three formats," "extract all URLs," "validate an email address" — regular expressions (the re module) are the right tool. A regex is a compact pattern language that describes a set of strings. The trade-off: they're powerful and concise, but cryptic. Use them when you need them; don't reach for them when split() or replace() suffices.

regex_basics.py
import re

log_line = "2026-06-27 14:32:01 ERROR user_id=1042 msg='Payment failed'"

# re.search: find first match anywhere in string
match = re.search(r"\d{4}-\d{2}-\d{2}", log_line)
if match:
    print(match.group())    # "2026-06-27"

# re.findall: return all non-overlapping matches as list
re.findall(r"\d+", log_line)    # ["2026", "06", "27", "14", "32", "01", "1042"]

# Named groups: extract structured data
pattern = re.compile(
    r"(?P<date>\d{4}-\d{2}-\d{2}) "
    r"(?P<time>\d{2}:\d{2}:\d{2}) "
    r"(?P<level>\w+) "
    r"user_id=(?P<uid>\d+)"
)
m = pattern.search(log_line)
m.group("date")     # "2026-06-27"
m.group("level")    # "ERROR"
m.group("uid")      # "1042"
m.groupdict()       # {'date': '2026-06-27', 'time': '14:32:01', 'level': 'ERROR', 'uid': '1042'}

# re.sub: replace with regex
cleaned = re.sub(r"\s+", " ", "  too   many   spaces  ").strip()
# "too many spaces"

# Compile patterns you reuse — avoids recompiling on every call
EMAIL_RE = re.compile(r"[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}")
emails = EMAIL_RE.findall("contact ada@x.io or bob@y.com for help")
# ['ada@x.io', 'bob@y.com']

"2026-06-27" ['ada@x.io', 'bob@y.com'] WHY Why are Python strings immutable? Immutability enables safe string interning (CPython caches common strings), makes strings hashable (usable as dict keys), and simplifies reasoning about identity vs equality. When two variables hold the same short string, CPython may intern them to the same object. The cost is that mutation requires new objects — but most text operations return new strings anyway, so the practical impact is mainly the concatenation-in-loop gotcha.

HOW How does "".join() beat += ? join() receives the complete list of pieces upfront. It makes two passes: first to compute the total output length, then to allocate exactly one buffer and copy all pieces into it — two O(n) passes = O(n). Repeated += allocates a new buffer on every iteration, copying all prior content each time — O(n²) total. The same "collect then batch" principle applies whenever you know the full input before building the output.

WHERE Where does encoding fail most in production? Most commonly: reading CSV files written on Windows (cp1252) on a Linux server without specifying encoding. Or scraping web pages and not decoding with the charset from the HTTP Content-Type header. Always set encoding="utf-8" on file opens, use response.text (not response.content) with requests, and validate encoding at ingestion boundaries rather than letting errors surface mid-pipeline.

Knowledge check

InterviewYou are concatenating tens of thousands of substrings in a loop and it's slow. What is the idiomatic, efficient fix?

  • Use a tuple instead — tuples concatenate faster.
  • Append each piece to a list and call "".join(parts) once — O(n) instead of O(n²).
  • Switch the loop to while instead of for.
  • Use += but add strings in reverse order.
Answer

Append each piece to a list and call "".join(parts) once — O(n) instead of O(n²).

Strings are immutable, so += in a loop rebuilds the entire string each iteration (quadratic). str.join() receives the complete list of parts, pre-allocates one buffer sized for the total, and copies everything in a single linear pass. The pattern is canonical Python: collect fragments, join once. The same approach works in any language where string concatenation is linear in the result length.

Knowledge check

Concept CheckWhat is the difference between str.find("x") and str.index("x")?

  • They are identical; both raise ValueError if not found.
  • find() returns -1 if not found; index() raises ValueError. Use find() when absence is normal; index() when absence is a bug.
  • find() searches from the right; index() from the left.
  • find() is case-insensitive; index() is case-sensitive.
Answer

find() returns -1 if not found; index() raises ValueError. Use find() when absence is normal; index() when absence is a bug.

Both return the index of the first occurrence of the substring. The only difference is failure behavior: find() returns -1 (safe for checking); index() raises ValueError (useful when absence is unexpected and you want to fail loudly). For simple "is it there?" checks, the idiomatic Python is just "x" in s , which returns a bool directly.

Knowledge check

ProductionYou open a CSV file with open("data.csv") and get a UnicodeDecodeError in production on Linux but not on your Mac. What is the likely cause?

  • Linux doesn't support Unicode at all.
  • The CSV has a syntax error that Linux is stricter about.
  • The file uses a Windows encoding (cp1252 or latin-1) and open() without encoding= picks the OS default — UTF-8 on Linux, often cp1252 on Windows where the file was created.
  • Python 3 on Linux is an older version without Unicode support.
Answer

The file uses a Windows encoding (cp1252 or latin-1) and open() without encoding= picks the OS default — UTF-8 on Linux, often cp1252 on Windows where the file was created.

When you don't specify encoding= , Python uses the locale's default: UTF-8 on modern Linux/macOS, but often cp1252 (Windows-1252) on Windows. A file written on Windows with cp1252 contains bytes (like \x92 for a curly apostrophe) that are invalid UTF-8. On Mac it may "work" if the locale default happens to be latin-1-compatible. Always specify encoding="utf-8" (or the correct encoding) explicitly.

Exercise: Parse and Clean an Application Log

Given a list of raw log strings (some malformed), write a function parse_logs(lines) that returns a list of dicts with keys timestamp, level, and message. Skip lines that don't match the expected format. Normalize levels to uppercase. Return only ERROR and WARNING entries. Example input: "2026-06-27 14:32:01 error payment failed".

Show solution
import re
from typing import Optional

LOG_PATTERN = re.compile(
    r"^(?P<ts>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+"
    r"(?P<level>\w+)\s+"
    r"(?P<msg>.+)$"
)
ALERT_LEVELS = {"ERROR", "WARNING", "WARN"}

def parse_logs(lines: list[str]) -> list[dict]:
    results = []
    for line in lines:
        line = line.strip()
        if not line:
            continue
        m = LOG_PATTERN.match(line)
        if not m:
            continue                         # skip malformed lines
        level = m.group("level").upper()
        if level not in ALERT_LEVELS:
            continue                         # only errors and warnings
        results.append({
            "timestamp": m.group("ts"),
            "level":     level,
            "message":   m.group("msg").strip(),
        })
    return results

# Test
logs = [
    "2026-06-27 14:32:01 error payment failed",
    "2026-06-27 14:32:02 info user logged in",
    "2026-06-27 14:32:03 WARNING disk at 90%",
    "malformed line without timestamp",
    "",
    "2026-06-27 14:32:05 ERROR db connection timeout",
]

for entry in parse_logs(logs):
    print(f"[{entry['level']}] {entry['timestamp']} — {entry['message']}")

# [ERROR] 2026-06-27 14:32:01 — payment failed
# [WARNING] 2026-06-27 14:32:03 — disk at 90%
# [ERROR] 2026-06-27 14:32:05 — db connection timeout

Key takeaways

  • Strings are immutable Unicode sequences; every "modification" returns a new string.
  • Core methods: strip(), split(), join(), replace(), startswith()/endswith(), find().
  • f-strings are the modern, fast formatting tool — use f"{value:.2f}", f"{price=}", and alignment specs.
  • Use "".join(parts) — never += in a loop — for O(n) concatenation.
  • Always specify encoding="utf-8" on file opens to avoid platform-dependent failures.
  • Reach for re when string methods can't express the pattern; compile repeated patterns.