290 lines
8.0 KiB
Markdown
290 lines
8.0 KiB
Markdown
# psql Tips — Advanced Debugging, Performance & Safety
|
|
|
|
Part of the psql tips reference. See also: tips-workflows.md
|
|
|
|
Advanced techniques for debugging, performance tuning, and safe psql usage.
|
|
|
|
## Table of Contents
|
|
|
|
- [Performance Tips](#performance-tips)
|
|
- [Debugging and Introspection](#debugging-and-introspection)
|
|
- [Safety and Best Practices](#safety-and-best-practices)
|
|
- [Gotchas and Common Mistakes](#gotchas-and-common-mistakes)
|
|
|
|
---
|
|
|
|
## Performance Tips
|
|
|
|
### Large Result Sets
|
|
|
|
```sql
|
|
-- Don't load entire result into memory
|
|
\set FETCH_COUNT 1000
|
|
SELECT * FROM billion_row_table;
|
|
|
|
-- Use \copy instead of COPY for client-side operations
|
|
\copy huge_table TO '/data/export.csv' WITH (FORMAT csv)
|
|
```
|
|
|
|
### Pipeline Mode for Batch Operations
|
|
|
|
Pipeline mode batches multiple queries into single network round trips:
|
|
|
|
```sql
|
|
\startpipeline
|
|
INSERT INTO logs (msg) VALUES ($1)
|
|
\bind 'entry 1' \sendpipeline
|
|
INSERT INTO logs (msg) VALUES ($1)
|
|
\bind 'entry 2' \sendpipeline
|
|
INSERT INTO logs (msg) VALUES ($1)
|
|
\bind 'entry 3' \sendpipeline
|
|
\getresults
|
|
\endpipeline
|
|
```
|
|
|
|
This sends all three INSERTs in one network round trip instead of three.
|
|
|
|
### Script Execution
|
|
|
|
```bash
|
|
# Run in a single transaction (faster, and all-or-nothing)
|
|
psql -1 -f migration.sql mydb
|
|
|
|
# Multiple files sequentially
|
|
psql -1 -f 001.sql -f 002.sql -f 003.sql mydb
|
|
```
|
|
|
|
---
|
|
|
|
## Debugging and Introspection
|
|
|
|
### See What psql Sends to the Server
|
|
|
|
```sql
|
|
-- Echo all SQL commands
|
|
\set ECHO queries
|
|
|
|
-- See the SQL behind \d commands (incredibly useful for learning)
|
|
\set ECHO_HIDDEN on
|
|
|
|
-- Or in noexec mode (show but don't execute)
|
|
\set ECHO_HIDDEN noexec
|
|
|
|
-- Now run any \d command to see its SQL
|
|
\dt
|
|
\d users
|
|
```
|
|
|
|
### Check Query Plans
|
|
|
|
```sql
|
|
-- Basic plan
|
|
EXPLAIN SELECT * FROM users WHERE email = 'test@example.com';
|
|
|
|
-- With actual execution time
|
|
EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com';
|
|
|
|
-- Detailed with buffer info
|
|
EXPLAIN (ANALYZE, BUFFERS, VERBOSE) SELECT ...;
|
|
|
|
-- JSON format for tooling
|
|
EXPLAIN (FORMAT JSON) SELECT ...;
|
|
```
|
|
|
|
### Lock Analysis
|
|
|
|
```sql
|
|
-- Current locks waiting to be granted
|
|
SELECT * FROM pg_locks WHERE NOT granted;
|
|
|
|
-- Blocked sessions with their blockers (PostgreSQL 9.6+)
|
|
SELECT blocked.pid,
|
|
blocked.query,
|
|
pg_blocking_pids(blocked.pid) AS blocked_by_pids
|
|
FROM pg_stat_activity blocked
|
|
WHERE cardinality(pg_blocking_pids(blocked.pid)) > 0;
|
|
```
|
|
|
|
---
|
|
|
|
## Safety and Best Practices
|
|
|
|
### Always Set ON_ERROR_STOP in Scripts
|
|
|
|
Without `ON_ERROR_STOP`, a script continues even after errors, potentially leaving the database in an inconsistent state:
|
|
|
|
```sql
|
|
-- Top of every script
|
|
\set ON_ERROR_STOP on
|
|
```
|
|
|
|
### Use Single-Transaction Mode for Migrations
|
|
|
|
```bash
|
|
# -1 wraps everything in BEGIN...COMMIT
|
|
# On error, the entire migration rolls back
|
|
psql -1 -f migration.sql mydb
|
|
```
|
|
|
|
### Never Use PGPASSWORD in Scripts
|
|
|
|
```bash
|
|
# BAD: Password visible in process list, env vars
|
|
PGPASSWORD=secret psql -c "SELECT 1" mydb# GOOD: Use ~/.pgpass (manually edit to avoid shell history)
|
|
touch ~/.pgpass && chmod 600 ~/.pgpass
|
|
# Then edit ~/.pgpass and your add:
|
|
# hostname:port:database:username:password
|
|
# Example: localhost:5432:mydb:myuser:mysecret
|
|
psql -c "SELECT 1" mydb
|
|
```
|
|
|
|
### Preview Before Executing
|
|
|
|
```sql
|
|
-- Dry-run a script to see what commands will execute (shows SQL, does NOT execute)
|
|
\set ECHO all
|
|
BEGIN;
|
|
-- Paste or review migration SQL here, then ROLLBACK instead of COMMIT
|
|
\i migration.sql
|
|
ROLLBACK;
|
|
|
|
-- Or use \gdesc to check result columns without executing
|
|
SELECT * FROM complex_view \gdesc
|
|
```
|
|
|
|
### Use \copy Over COPY
|
|
|
|
`\copy` uses client permissions and filesystem. SQL `COPY` runs on the server and requires superuser or `pg_read_server_files`/`pg_write_server_files` roles. `\copy` transfers all data through the client/server connection, which is less efficient than SQL `COPY` for very large datasets. For bulk data transfer, prefer SQL `COPY` when server-side file access is available.
|
|
|
|
```sql
|
|
-- BAD (requires server-side file access)
|
|
COPY users TO '/tmp/users.csv' WITH CSV HEADER;
|
|
|
|
-- GOOD (uses client-side file access)
|
|
\copy users TO '/tmp/users.csv' WITH CSV HEADER
|
|
```
|
|
|
|
### search_path Safety for Untrusted Users
|
|
|
|
If untrusted users have access to the database, remove publicly-writable schemas from `search_path` at session start:
|
|
|
|
```sql
|
|
SELECT pg_catalog.set_config('search_path', '', false);
|
|
```
|
|
|
|
### Automatic LISTEN/NOTIFY Polling
|
|
|
|
Whenever a command is executed, psql automatically polls for asynchronous notification events generated by `LISTEN`/`NOTIFY`. This happens without any special configuration.
|
|
|
|
---
|
|
|
|
## Gotchas and Common Mistakes
|
|
|
|
### Semicolons in \copy
|
|
|
|
`\copy` does NOT end with a semicolon. It's a meta-command:
|
|
|
|
```sql
|
|
-- CORRECT
|
|
\copy users TO '/tmp/users.csv' WITH CSV HEADER
|
|
|
|
-- WRONG (psql interprets the semicolon oddly)
|
|
\copy users TO '/tmp/users.csv' WITH CSV HEADER;
|
|
```
|
|
|
|
### Variable Substitution and SQL Injection
|
|
|
|
psql variables are simple text substitution. They are NOT parameterized queries:
|
|
|
|
```sql
|
|
-- The :'varname' form (single-quoted) escapes embedded single quotes, making it
|
|
-- safe against SQL injection for STRING VALUES in WHERE clauses:
|
|
\set name "Robert'); DROP TABLE students;--"
|
|
SELECT * FROM users WHERE name = :'name';
|
|
-- Expands to: WHERE name = 'Robert''); DROP TABLE students;--'
|
|
-- The '' is an escaped quote, so the entire value is a single string literal — NOT injected.
|
|
|
|
-- However, :'varname' is NOT safe for identifiers or unquoted contexts.
|
|
-- For identifiers (table/column names), use :"varname" (double-quoted form).
|
|
-- For the safest parameterized queries, use \bind:
|
|
SELECT * FROM users WHERE name = $1;
|
|
\bind 'Robert' \g
|
|
```
|
|
|
|
### Transaction State After Error
|
|
|
|
After an error in a transaction block, all subsequent commands fail until ROLLBACK:
|
|
|
|
```sql
|
|
BEGIN;
|
|
INSERT INTO users (id) VALUES (1);
|
|
INSERT INTO users (id) VALUES ('bad'); -- ERROR
|
|
INSERT INTO users (id) VALUES (2); -- Also fails!
|
|
COMMIT; -- Also fails!
|
|
```
|
|
|
|
Use `ON_ERROR_ROLLBACK` to auto-savepoint:
|
|
|
|
```sql
|
|
\set ON_ERROR_ROLLBACK on
|
|
BEGIN;
|
|
INSERT INTO users (id) VALUES (1);
|
|
INSERT INTO users (id) VALUES ('bad'); -- ERROR, auto-rollback to savepoint
|
|
INSERT INTO users (id) VALUES (2); -- This works now
|
|
COMMIT;
|
|
```
|
|
|
|
### Pattern Matching Uses Regex
|
|
|
|
The `*` and `?` in \d commands are converted to regex (`.*` and `.`). Advanced regex like `[0-9]` works. Important differences from standard regex:
|
|
|
|
- `.` is a schema/object separator, not any-char (use `?` for any single char)
|
|
- `$` is matched literally, not as an anchor (pattern must match whole name anyway)
|
|
- Within double quotes, all special characters (`*`, `?`, regex chars) are literal
|
|
|
|
```sql
|
|
-- These work as expected
|
|
\dt user* -- matches users, user_accounts, etc.
|
|
\dt user? -- matches users, user1, etc.
|
|
\dt user[0-9]* -- matches user1, user2, user123
|
|
|
|
-- If your table name contains special chars, use double quotes
|
|
\dt "table.with.dots" -- matches literally, dots not treated as separator
|
|
```
|
|
|
|
### Connection String vs CLI Arguments
|
|
|
|
```bash
|
|
# These are equivalent
|
|
psql -h localhost -p 5432 -U admin -d mydb
|
|
psql "postgresql://admin@localhost:5432/mydb"
|
|
|
|
# But you can't mix freely — URI overrides individual flags
|
|
psql -h otherhost "postgresql://admin@localhost:5432/mydb" -- uses localhost, not otherhost
|
|
```
|
|
|
|
### \i vs \ir
|
|
|
|
- `\i filename` — resolves relative to the **current working directory** (where psql was started)
|
|
- `\ir filename` — resolves relative to the **currently executing script's directory**
|
|
|
|
For script portability, prefer `\ir`:
|
|
|
|
```sql
|
|
-- In /scripts/migrations/run_all.sql:
|
|
\ir 001_schema.sql -- resolves to /scripts/migrations/001_schema.sql
|
|
\ir 002_data.sql -- resolves to /scripts/migrations/002_data.sql
|
|
```
|
|
|
|
### \o and Query Output
|
|
|
|
`\o` redirects query output, not meta-command output:
|
|
|
|
```sql
|
|
\o /tmp/output.txt
|
|
SELECT * FROM users; -- goes to file
|
|
\d users -- also goes to file
|
|
\echo 'hello' -- goes to STDOUT (not affected by \o)
|
|
\qecho 'hello' -- goes to /tmp/output.txt (affected by \o)
|
|
```
|