Advanced SQL
Window Functions Deep Dive
Section titled “Window Functions Deep Dive”Window functions compute values across a set of rows related to the current row without collapsing The result set. This section covers the framing mechanics, exclusion clauses, window groups, and Window chains that give window functions their full power.
Window Function Anatomy
Section titled “Window Function Anatomy”function_name([arguments]) OVER ( [window_name] [PARTITION BY partition_expr, ...] [ORDER BY sort_expr [ASC|DESC] [NULLS {FIRST|LAST}], ...] [frame_clause])The three optional components — partitioning, ordering, and framing — work together to define the Set of rows visible to the function.
Framing Clauses
Section titled “Framing Clauses”The frame clause defines the subset of rows within the partition that the function sees. It is only Meaningful when ORDER BY is present (without ORDER BYThe default frame is the entire Partition).
-- Frame boundariesROWS BETWEEN start AND endRANGE BETWEEN start AND endGROUPS BETWEEN start AND end
-- Start/end boundary options:-- UNBOUNDED PRECEDING -- first row of partition-- UNBOUNDED FOLLOWING -- last row of partition-- n PRECEDING -- n rows before current row-- n FOLLOWING -- n rows after current row-- CURRENT ROW -- current rowROWS counts physical rows. RANGE counts logical peers (rows with the same ORDER BY value). GROUPS counts distinct peer groups.
-- Running total (ROWS: 3-row moving sum)SELECT order_date, amount, SUM(amount) OVER ( ORDER BY order_date ROWS BETWEEN 2 PRECEDING AND CURRENT ROW ) AS rolling_3dayFROM daily_sales;
-- Cumulative total (ROWS: all rows from start)SELECT order_date, amount, SUM(amount) OVER ( ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW ) AS cumulativeFROM daily_sales;ROWS vs RANGE vs GROUPS
Section titled “ROWS vs RANGE vs GROUPS”The distinction matters when there are ties in the ORDER BY column:
-- Given data with duplicate dates:-- 2024-01-01 | 100-- 2024-01-01 | 200-- 2024-01-02 | 150-- 2024-01-03 | 300
-- ROWS BETWEEN 1 PRECEDING AND CURRENT ROW-- For row 2 (2024-01-01, 200): sees rows 1 and 2 → SUM = 300-- For row 3 (2024-01-02, 150): sees rows 2 and 3 → SUM = 350
-- RANGE BETWEEN 1 PRECEDING AND CURRENT ROW-- "1 PRECEDING" in RANGE means "ORDER BY value - 1"-- For row 2 (date=Jan 1): sees all rows where date >= Jan 0 → sees rows 1, 2 → SUM = 300-- For row 3 (date=Jan 2): sees all rows where date >= Jan 1 → sees rows 1, 2, 3 → SUM = 650
-- GROUPS BETWEEN 1 PRECEDING AND CURRENT ROW-- For row 2 (date=Jan 1): sees current group (Jan 1) and 1 group before (none) → SUM = 300-- For row 3 (date=Jan 2): sees current group (Jan 2) and 1 group before (Jan 1) → SUM = 650EXCLUDE Clause
Section titled “EXCLUDE Clause”PostgreSQL 11+ supports EXCLUDE within the frame clause to omit specific rows:
-- Exclude the current row from the frameSUM(amount) OVER ( ORDER BY order_date ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE CURRENT ROW) AS sum_excluding_current
-- Exclude other rows with the same ORDER BY value (peers)SUM(amount) OVER ( ORDER BY order_date RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE GROUP) AS sum_excluding_peers
-- Exclude both current row and its tiesSUM(amount) OVER ( ORDER BY order_date RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW EXCLUDE TIES) AS sum_excluding_ties_and_current| EXCLUDE Option | What It Removes |
|---|---|
CURRENT ROW | Only the current row |
GROUP | Current row and all peers (same ORDER BY value) |
TIES | Only the peers, keeps the current row |
NO OTHERS | Nothing (default) |
WINDOW Clause (Window Chains)
Section titled “WINDOW Clause (Window Chains)”The WINDOW clause defines named windows that can be reused, avoiding repetition:
SELECT department_id, emp_id, salary, ROW_NUMBER() OVER w AS row_num, RANK() OVER w AS rank, DENSE_RANK() OVER w AS dense_rank, SUM(salary) OVER (w ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS running_totalFROM employeesWINDOW w AS (PARTITION BY department_id ORDER BY salary DESC);Named windows can be composed by extending a base window:
SELECT department_id, emp_id, salary, AVG(salary) OVER dept_avg AS dept_avg_salary, salary - AVG(salary) OVER dept_avg AS delta_from_avg, ROW_NUMBER() OVER dept_order AS salary_rankFROM employeesWINDOW dept AS (PARTITION BY department_id), dept_avg AS (dept), dept_order AS (dept ORDER BY salary DESC);Advanced Ranking: NTILE, PERCENT_RANK, CUME_DIST
Section titled “Advanced Ranking: NTILE, PERCENT_RANK, CUME_DIST”-- NTILE divides rows into n roughly equal bucketsSELECT emp_id, salary, NTILE(4) OVER (ORDER BY salary DESC) AS salary_quartileFROM employees;
-- PERCENT_RANK: (rank - 1) / (total_rows - 1), range [0, 1]SELECT emp_id, salary, PERCENT_RANK() OVER (ORDER BY salary DESC) AS pct_rankFROM employees;
-- CUME_DIST: proportion of rows with value <= current row, range (0, 1]SELECT emp_id, salary, CUME_DIST() OVER (ORDER BY salary DESC) AS cumulative_distFROM employees;| Function | Ties at 150k, 150k, 140k, 130k | Range |
|---|---|---|
ROW_NUMBER | 1, 2, 3, 4 | N/A |
RANK | 1, 1, 3, 4 | 1 to N |
DENSE_RANK | 1, 1, 2, 3 | 1 to N |
NTILE(2) | 1, 1, 2, 2 | 1 to n |
PERCENT_RANK | 0.0, 0.0, 0.667, 1.0 | 0.0 to 1.0 |
CUME_DIST | 0.5, 0.5, 0.75, 1.0 | 0.0 to 1.0 |
Advanced Common Table Expressions
Section titled “Advanced Common Table Expressions”Multiple CTEs with Data Modification
Section titled “Multiple CTEs with Data Modification”PostgreSQL allows mixing reads and writes in a single CTE chain:
WITH new_orders AS ( INSERT INTO orders (customer_id, total, status) VALUES (42, 250.00, "pending') RETURNING order_id, customer_id, total),inventory_update AS ( UPDATE inventory i SET quantity = i.quantity - 1 FROM new_orders o JOIN order_items oi ON oi.order_id = o.order_id WHERE i.product_id = oi.product_id RETURNING i.product_id, i.quantity),audit_entry AS ( INSERT INTO audit_log (action, details) SELECT 'order_created', json_build_object( 'order_id', o.order_id, 'customer_id', o.customer_id, 'total', o.total ) FROM new_orders o RETURNING log_id)SELECT o.order_id, o.total, a.log_id AS audit_idFROM new_orders oJOIN audit_entry a ON TRUE;Execution order within a CTE chain is not guaranteed to follow the textual order. The optimizer May reorder data-modifying CTEs. If you need ordering, use triggers or application-level Orchestration.
CTE Materialization (PostgreSQL 12+)
Section titled “CTE Materialization (PostgreSQL 12+)”Before PostgreSQL 12, every CTE was materialized (executed once, stored as a temporary result). PostgreSQL 12+ allows the optimizer to inline CTEs (fold them into the outer query like subqueries) When the CTE is referenced once and is non-recursive.
-- Inlined (faster for single-reference CTEs):WITH active_users AS ( SELECT user_id, email FROM users WHERE is_active = TRUE)SELECT * FROM active_users WHERE email LIKE '%@company.com';
-- Force materialization (useful when referenced multiple times):WITH active_users AS MATERIALIZED ( SELECT user_id, email FROM users WHERE is_active = TRUE)SELECT (SELECT COUNT(*) FROM active_users) AS total_active, (SELECT COUNT(*) FROM active_users WHERE email LIKE '%@company.com') AS company_users;| Strategy | When to Use | Trade-off |
|---|---|---|
| Inlined | CTE referenced once, simple filter | Planner can push predicates, use indexes |
| Materialized | CTE referenced multiple times | Computed once but cannot use outer indexes |
MATERIALIZED keyword | Explicit control over inlining | Overrides the planner’s decision |
Recursive CTEs for Tree Traversal
Section titled “Recursive CTEs for Tree Traversal”-- Org chart with full pathWITH RECURSIVE org_tree AS ( SELECT emp_id, first_name, manager_id, 1 AS depth, ARRAY[first_name] AS path_names FROM employees WHERE manager_id IS NULL
UNION ALL
SELECT e.emp_id, e.first_name, e.manager_id, t.depth + 1, t.path_names || e.first_name FROM employees e JOIN org_tree t ON e.manager_id = t.emp_id)SELECT emp_id, first_name, depth, array_to_string(path_names, ' -> ') AS reporting_chainFROM org_treeORDER BY path_names;Recursive CTEs for Graph Traversal (BFS)
Section titled “Recursive CTEs for Graph Traversal (BFS)”-- Find all reachable nodes from a starting nodeWITH RECURSIVE bfs AS ( SELECT from_node, to_node, 0 AS hops, ARRAY[from_node] AS visited FROM edges WHERE from_node = 'A'
UNION ALL
SELECT e.from_node, e.to_node, b.hops + 1, b.visited || e.to_node FROM edges e JOIN bfs b ON e.from_node = b.to_node WHERE NOT (e.to_node = ANY(b.visited)) AND b.hops < 10)SELECT DISTINCT to_node, MIN(hops) AS shortest_pathFROM bfsGROUP BY to_nodeORDER BY shortest_path;LATERAL is implicitly applied for function calls in the FROM list (e.g., FROM generate_series(1, 10)). You only need the explicit keyword when the subquery references Outer columns.
Full-Text Search
Section titled “Full-Text Search”PostgreSQL’s built-in full-text search provides ranked text search without external dependencies Like Elasticsearch.
tsvector and tsquery
Section titled “tsvector and tsquery”-- Create a tsvector from textSELECT to_tsvector('english', 'The quick brown fox jumps over the lazy dog');-- Result: "brown'':3 "dog':9 'fox':4 'jump':5 'lazi':8 'quick':2
-- Create a tsquery from a search stringSELECT to_tsquery('english', 'quick & fox');-- Result: "quick'' & "fox'
-- Match rowsSELECT title, bodyFROM documentsWHERE to_tsvector('english', body) @@ to_tsquery('english', 'database & performance');Indexed Full-Text Search
Section titled “Indexed Full-Text Search”-- Add a pre-computed tsvector columnALTER TABLE documents ADD COLUMN search_vector tsvector GENERATED ALWAYS AS (to_tsvector('english', coalesce(title, '') || ' ' || coalesce(body, ''))) STORED;
-- Create a GIN index on the tsvectorCREATE INDEX idx_documents_search ON documents USING GIN(search_vector);
-- Query using the pre-computed columnSELECT title, ts_rank(search_vector, query) AS rankFROM documents, to_tsquery('english', 'postgresql & performance') queryWHERE search_vector @@ queryORDER BY rank DESC;Ranking Functions
Section titled “Ranking Functions”-- ts_rank: standard ranking based on term frequency and document lengthSELECT title, ts_rank(search_vector, query) AS rankFROM documents, to_tsquery('english', 'database') queryWHERE search_vector @@ queryORDER BY rank DESC;
-- ts_rank_cd: cover density ranking (considers proximity of terms)SELECT title, ts_rank_cd(search_vector, query) AS rankFROM documents, to_tsquery('english', 'database performance') queryWHERE search_vector @@ queryORDER BY rank DESC;
-- rank with normalization by document lengthSELECT title, ts_rank(search_vector, query, 2) AS normalized_rankFROM documents, to_tsquery('english', 'database') queryWHERE search_vector @@ queryORDER BY normalized_rank DESC;The normalization flags for ts_rank and ts_rank_cd:
| Flag | Meaning |
|---|---|
| 0 | No normalization |
| 1 | Normalize by document length |
| 2 | Normalize by unique terms |
| 4 | Normalize by both length and unique terms |
| 8 | Normalize by document length + unique terms |
| 16 | Normalize by log of document length |
Highlighting Results
Section titled “Highlighting Results”SELECT title, ts_headline('english', body, query, 'MaxWords=50, MinWords=25, ShortWord=3') AS snippetFROM documents, to_tsquery('english', 'performance & tuning') queryWHERE search_vector @@ query;Trigram Search (pg_trgm)
Section titled “Trigram Search (pg_trgm)”For substring and fuzzy matching, trigram indexes are often more practical than full-word search:
CREATE EXTENSION IF NOT EXISTS pg_trgm;
-- GIN trigram index for LIKE/ILIKECREATE INDEX idx_documents_body_trgm ON documents USING GIN(body gin_trgm_ops);
-- Fast LIKE queriesSELECT title FROM documents WHERE body ILIKE '%performance tun%';
-- Fast similarity searchSELECT title, similarity(body, 'database performance tuning') AS simFROM documentsWHERE body % 'database performance tuning'ORDER BY sim DESC;| Method | Best For | Index Type |
|---|---|---|
| tsvector/tsquery | Full-word search, ranking | GIN |
| pg_trgm | Substring, fuzzy, ILIKE | GIN or GiST |
Materialized Views
Section titled “Materialized Views”A materialized view caches the result of a query physically on disk, allowing fast reads at the cost Of stale data that must be refreshed.
Creating and Refreshing
Section titled “Creating and Refreshing”-- Create a materialized viewCREATE MATERIALIZED VIEW mv_daily_sales_summary ASSELECT order_date, COUNT(DISTINCT order_id) AS order_count, SUM(total) AS revenue, AVG(total) AS avg_order_valueFROM ordersGROUP BY order_dateWITH DATA;
-- Refresh the entire materialized viewREFRESH MATERIALIZED VIEW mv_daily_sales_summary;
-- Refresh concurrently (does not lock the view for reads)REFRESH MATERIALIZED VIEW CONCURRENTLY mv_daily_sales_summary;PostgreSQL currently only supports STORED generated columns. VIRTUAL (computed on read) is in The SQL standard but not yet implemented. Other databases like MySQL and SQL Server support both.
Domains and Custom Types
Section titled “Domains and Custom Types”Domains
Section titled “Domains”Domains add constraints to existing types:
CREATE DOMAIN email_address AS TEXT CHECK (VALUE ~ '^[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}$');
CREATE DOMAIN positive_amount AS NUMERIC(12,2) CHECK (VALUE > 0) DEFAULT 0;
CREATE DOMAIN iso_country_code AS CHAR(2) CHECK (VALUE ~ '^[A-Z]{2}$');
CREATE TABLE customers ( customer_id SERIAL PRIMARY KEY, email email_address NOT NULL, credit_limit positive_amount DEFAULT 1000.00, country iso_country_code);ENUM Types
Section titled “ENUM Types”CREATE TYPE order_status AS ENUM ( 'pending', 'confirmed', 'processing', 'shipped', 'delivered', 'cancelled');
CREATE TABLE orders ( order_id SERIAL PRIMARY KEY, status order_status NOT NULL DEFAULT 'pending', created_at TIMESTAMPTZ DEFAULT NOW());
-- ENUM types support comparison and orderingSELECT * FROM orders WHERE status >= 'shipped';