# psql Tips — Advanced Debugging, Performance & Safety

Part of the psql tips reference. See also: tips-workflows.md

Advanced techniques for debugging, performance tuning, and safe psql usage.

## Table of Contents

- [Performance Tips](#performance-tips)
- [Debugging and Introspection](#debugging-and-introspection)
- [Safety and Best Practices](#safety-and-best-practices)
- [Gotchas and Common Mistakes](#gotchas-and-common-mistakes)

---

## Performance Tips

### Large Result Sets

```sql
-- Don't load entire result into memory
\set FETCH_COUNT 1000
SELECT * FROM billion_row_table;

-- Use \copy instead of COPY for client-side operations
\copy huge_table TO '/data/export.csv' WITH (FORMAT csv)
```

### Pipeline Mode for Batch Operations

Pipeline mode batches multiple queries into single network round trips:

```sql
\startpipeline
  INSERT INTO logs (msg) VALUES ($1)
  \bind 'entry 1' \sendpipeline
  INSERT INTO logs (msg) VALUES ($1)
  \bind 'entry 2' \sendpipeline
  INSERT INTO logs (msg) VALUES ($1)
  \bind 'entry 3' \sendpipeline
  \getresults
\endpipeline
```

This sends all three INSERTs in one network round trip instead of three.

### Script Execution

```bash
# Run in a single transaction (faster, and all-or-nothing)
psql -1 -f migration.sql mydb

# Multiple files sequentially
psql -1 -f 001.sql -f 002.sql -f 003.sql mydb
```

---

## Debugging and Introspection

### See What psql Sends to the Server

```sql
-- Echo all SQL commands
\set ECHO queries

-- See the SQL behind \d commands (incredibly useful for learning)
\set ECHO_HIDDEN on

-- Or in noexec mode (show but don't execute)
\set ECHO_HIDDEN noexec

-- Now run any \d command to see its SQL
\dt
\d users
```

### Check Query Plans

```sql
-- Basic plan
EXPLAIN SELECT * FROM users WHERE email = 'test@example.com';

-- With actual execution time
EXPLAIN ANALYZE SELECT * FROM users WHERE email = 'test@example.com';

-- Detailed with buffer info
EXPLAIN (ANALYZE, BUFFERS, VERBOSE) SELECT ...;

-- JSON format for tooling
EXPLAIN (FORMAT JSON) SELECT ...;
```

### Lock Analysis

```sql
-- Current locks waiting to be granted
SELECT * FROM pg_locks WHERE NOT granted;

-- Blocked sessions with their blockers (PostgreSQL 9.6+)
SELECT blocked.pid,
       blocked.query,
       pg_blocking_pids(blocked.pid) AS blocked_by_pids
FROM pg_stat_activity blocked
WHERE cardinality(pg_blocking_pids(blocked.pid)) > 0;
```

---

## Safety and Best Practices

### Always Set ON_ERROR_STOP in Scripts

Without `ON_ERROR_STOP`, a script continues even after errors, potentially leaving the database in an inconsistent state:

```sql
-- Top of every script
\set ON_ERROR_STOP on
```

### Use Single-Transaction Mode for Migrations

```bash
# -1 wraps everything in BEGIN...COMMIT
# On error, the entire migration rolls back
psql -1 -f migration.sql mydb
```

### Never Use PGPASSWORD in Scripts

```bash
# BAD: Password visible in process list, env vars
PGPASSWORD=secret psql -c "SELECT 1" mydb# GOOD: Use ~/.pgpass (manually edit to avoid shell history)
touch ~/.pgpass && chmod 600 ~/.pgpass
# Then edit ~/.pgpass and your add:
# hostname:port:database:username:password
# Example: localhost:5432:mydb:myuser:mysecret
psql -c "SELECT 1" mydb
```

### Preview Before Executing

```sql
-- Dry-run a script to see what commands will execute (shows SQL, does NOT execute)
\set ECHO all
BEGIN;
-- Paste or review migration SQL here, then ROLLBACK instead of COMMIT
\i migration.sql
ROLLBACK;

-- Or use \gdesc to check result columns without executing
SELECT * FROM complex_view \gdesc
```

### Use \copy Over COPY

`\copy` uses client permissions and filesystem. SQL `COPY` runs on the server and requires superuser or `pg_read_server_files`/`pg_write_server_files` roles. `\copy` transfers all data through the client/server connection, which is less efficient than SQL `COPY` for very large datasets. For bulk data transfer, prefer SQL `COPY` when server-side file access is available.

```sql
-- BAD (requires server-side file access)
COPY users TO '/tmp/users.csv' WITH CSV HEADER;

-- GOOD (uses client-side file access)
\copy users TO '/tmp/users.csv' WITH CSV HEADER
```

### search_path Safety for Untrusted Users

If untrusted users have access to the database, remove publicly-writable schemas from `search_path` at session start:

```sql
SELECT pg_catalog.set_config('search_path', '', false);
```

### Automatic LISTEN/NOTIFY Polling

Whenever a command is executed, psql automatically polls for asynchronous notification events generated by `LISTEN`/`NOTIFY`. This happens without any special configuration.

---

## Gotchas and Common Mistakes

### Semicolons in \copy

`\copy` does NOT end with a semicolon. It's a meta-command:

```sql
-- CORRECT
\copy users TO '/tmp/users.csv' WITH CSV HEADER

-- WRONG (psql interprets the semicolon oddly)
\copy users TO '/tmp/users.csv' WITH CSV HEADER;
```

### Variable Substitution and SQL Injection

psql variables are simple text substitution. They are NOT parameterized queries:

```sql
-- The :'varname' form (single-quoted) escapes embedded single quotes, making it
-- safe against SQL injection for STRING VALUES in WHERE clauses:
\set name "Robert'); DROP TABLE students;--"
SELECT * FROM users WHERE name = :'name';
-- Expands to: WHERE name = 'Robert''); DROP TABLE students;--'
-- The '' is an escaped quote, so the entire value is a single string literal — NOT injected.

-- However, :'varname' is NOT safe for identifiers or unquoted contexts.
-- For identifiers (table/column names), use :"varname" (double-quoted form).
-- For the safest parameterized queries, use \bind:
SELECT * FROM users WHERE name = $1;
\bind 'Robert' \g
```

### Transaction State After Error

After an error in a transaction block, all subsequent commands fail until ROLLBACK:

```sql
BEGIN;
INSERT INTO users (id) VALUES (1);
INSERT INTO users (id) VALUES ('bad');  -- ERROR
INSERT INTO users (id) VALUES (2);      -- Also fails!
COMMIT;                                  -- Also fails!
```

Use `ON_ERROR_ROLLBACK` to auto-savepoint:

```sql
\set ON_ERROR_ROLLBACK on
BEGIN;
INSERT INTO users (id) VALUES (1);
INSERT INTO users (id) VALUES ('bad');  -- ERROR, auto-rollback to savepoint
INSERT INTO users (id) VALUES (2);      -- This works now
COMMIT;
```

### Pattern Matching Uses Regex

The `*` and `?` in \d commands are converted to regex (`.*` and `.`). Advanced regex like `[0-9]` works. Important differences from standard regex:

- `.` is a schema/object separator, not any-char (use `?` for any single char)
- `$` is matched literally, not as an anchor (pattern must match whole name anyway)
- Within double quotes, all special characters (`*`, `?`, regex chars) are literal

```sql
-- These work as expected
\dt user*              -- matches users, user_accounts, etc.
\dt user?              -- matches users, user1, etc.
\dt user[0-9]*         -- matches user1, user2, user123

-- If your table name contains special chars, use double quotes
\dt "table.with.dots"  -- matches literally, dots not treated as separator
```

### Connection String vs CLI Arguments

```bash
# These are equivalent
psql -h localhost -p 5432 -U admin -d mydb
psql "postgresql://admin@localhost:5432/mydb"

# But you can't mix freely — URI overrides individual flags
psql -h otherhost "postgresql://admin@localhost:5432/mydb"  -- uses localhost, not otherhost
```

### \i vs \ir

- `\i filename` — resolves relative to the **current working directory** (where psql was started)
- `\ir filename` — resolves relative to the **currently executing script's directory**

For script portability, prefer `\ir`:

```sql
-- In /scripts/migrations/run_all.sql:
\ir 001_schema.sql    -- resolves to /scripts/migrations/001_schema.sql
\ir 002_data.sql      -- resolves to /scripts/migrations/002_data.sql
```

### \o and Query Output

`\o` redirects query output, not meta-command output:

```sql
\o /tmp/output.txt
SELECT * FROM users;     -- goes to file
\d users                 -- also goes to file
\echo 'hello'            -- goes to STDOUT (not affected by \o)
\qecho 'hello'           -- goes to /tmp/output.txt (affected by \o)
```