Skip to content

docs: Document database-backed query analyzer #2904

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Oct 24, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
79 changes: 79 additions & 0 deletions docs/howto/generate.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# `generate` - Generating code

`sqlc generate` parses SQL, analyzes the results, and outputs code. Your schema and queries are stored in separate SQL files. The paths to these files live in a `sqlc.yaml` configuration file.

```yaml
version: "2"
sql:
- engine: "postgresql"
queries: "query.sql"
schema: "schema.sql"
gen:
go:
package: "tutorial"
out: "tutorial"
sql_package: "pgx/v5"
```

We've written extensive docs on [retrieving](select.md), [inserting](insert.md),
[updating](update.md), and [deleting](delete.md) rows.

By default, sqlc runs its analysis using a built-in query analysis engine. While fast, this engine can't handle some complex queries and type-inference.

You can configure sqlc to use a database connection for enhanced analysis using metadata from that database.

The database-backed analyzer currently supports PostgreSQL, with [MySQL](https://github.com/sqlc-dev/sqlc/issues/2902) and [SQLite](https://github.com/sqlc-dev/sqlc/issues/2903)
support planned in the future.

## Enhanced analysis with managed databases

```{note}
Managed databases are powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
```

With [managed databases](managed-databases.md) configured, `generate` will automatically create a hosted ephemeral database with your
schema and use that database to improve its query analysis. And sqlc will cache its analysis locally
on a per-query basis to speed up future `generate` runs. This saves you the trouble of running and maintaining a database with
an up-to-date schema. Here's a minimal working configuration:

```yaml
version: "2"
cloud:
project: "<PROJECT_ID>"
sql:
- engine: "postgresql"
queries: "query.sql"
schema: "schema.sql"
database:
managed: true
gen:
go:
out: "db"
sql_package: "pgx/v5"
```

## Enhanced analysis using your own database

You can opt-in to database-backed analysis using your own database, by providing a `uri` in your sqlc
[database](../reference/config.md#database) configuration.

The `uri` string can contain references to environment variables using the `${...}`
syntax. In the following example, the connection string will have the value of
the `PG_PASSWORD` environment variable set as its password.

```yaml
version: "2"
sql:
- engine: "postgresql"
queries: "query.sql"
schema: "schema.sql"
database:
uri: "postgres://postgres:${PG_PASSWORD}@localhost:5432/postgres"
gen:
go:
out: "db"
sql_package: "pgx/v5"
```

Databases configured with a `uri` must have an up-to-date schema for query analysis to work correctly, and `sqlc` does not apply schema migrations your database. Use your migration tool of choice to create the necessary
tables and objects before running `sqlc generate`.
4 changes: 2 additions & 2 deletions docs/howto/upload.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Uploading projects
# `upload` - Uploading projects

```{note}
Project uploads are powered by [sqlc Cloud](https://dashboard.sqlc.dev). Sign up for [free](https://dashboard.sqlc.dev) today.
Expand All @@ -18,7 +18,7 @@ After creating a project, add the project ID to your sqlc configuration file.
```yaml
version: "2"
cloud:
project: "<PROJECT-ID>"
project: "<PROJECT_ID>"
```

You'll also need to create an auth token and make it available via the
Expand Down
35 changes: 24 additions & 11 deletions docs/howto/vet.md
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Linting queries
# `vet` - Linting queries

*Added in v1.19.0*

Expand Down Expand Up @@ -43,7 +43,7 @@ message Parameter
```

In addition to this basic information, when you have a PostgreSQL or MySQL
[database connection configured](../reference/config.html#database)
[database connection configured](../reference/config.md#database)
each CEL expression has access to the output from running `EXPLAIN ...` on your query
via the `postgresql.explain` and `mysql.explain` variables.
This output is quite complex and depends on the structure of your query but sqlc attempts
Expand Down Expand Up @@ -95,7 +95,7 @@ rules:
The CEL expression environment has two variables containing `EXPLAIN ...` output,
`postgresql.explain` and `mysql.explain`. `sqlc` only populates the variable associated with
your configured database engine, and only when you have a
[database connection configured](../reference/config.html#database).
[database connection configured](../reference/config.md#database).

For the `postgresql` engine, `sqlc` runs

Expand Down Expand Up @@ -163,22 +163,27 @@ rules:
rule: "!has(postgresql.explain)" # A dummy rule to trigger explain
```

Please note that `sqlc` does not manage or migrate your database. Use your
migration tool of choice to create the necessary database tables and objects
before running `sqlc vet` with rules that depend on `EXPLAIN ...` output.
Please note that databases configured with a `uri` must have an up-to-date
schema for `vet` to work correctly, and `sqlc` does not apply schema migrations
to your database. Use your migration tool of choice to create the necessary
tables and objects before running `sqlc vet` with rules that depend on
`EXPLAIN ...` output.

Alternatively, configure [managed databases](managed-databases.md) to have
`sqlc` create hosted ephemeral databases with the correct schema automatically.

## Built-in rules

### sqlc/db-prepare

When a [database](../reference/config.html#database) connection is configured, you can
When a [database](../reference/config.md#database) connection is configured, you can
run the built-in `sqlc/db-prepare` rule. This rule will attempt to prepare
each of your queries against the connected database and report any failures.

```yaml
version: 2
sql:
- schema: "query.sql"
- schema: "schema.sql"
queries: "query.sql"
engine: "postgresql"
gen:
Expand All @@ -191,12 +196,20 @@ sql:
- sqlc/db-prepare
```

Databases configured with a `uri` must have an up-to-date schema, and `sqlc` does not apply schema migrations your database. You can configure [managed databases](managed-databases.md) instead to have `sqlc` create and migrate databases automatically.
Please note that databases configured with a `uri` must have an up-to-date
schema for `vet` to work correctly, and `sqlc` does not apply schema migrations
to your database. Use your migration tool of choice to create the necessary
tables and objects before running `sqlc vet` with the `sqlc/db-prepare` rule.

Alternatively, configure [managed databases](managed-databases.md) to have
`sqlc` create hosted ephemeral databases with the correct schema automatically.

```yaml
version: 2
cloud:
project: "<PROJECT_ID>"
sql:
- schema: "query.sql"
- schema: "schema.sql"
queries: "query.sql"
engine: "postgresql"
gen:
Expand All @@ -215,7 +228,7 @@ example](https://github.com/sqlc-dev/sqlc/blob/main/examples/authors/sqlc.yaml).
## Running lint rules

When you add the name of a defined rule to the rules list
for a [sql package](https://docs.sqlc.dev/en/stable/reference/config.html#sql),
for a [sql package](../reference/config.md#sql),
`sqlc vet` will evaluate that rule against every query in the package.

In the example below, two rules are defined but only one is enabled.
Expand Down
20 changes: 16 additions & 4 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,15 @@ code ever again.
tutorials/getting-started-postgresql.md
tutorials/getting-started-sqlite.md

.. toctree::
:maxdepth: 2
:caption: Commands
:hidden:

howto/generate.md
howto/vet.md
howto/upload.md

.. toctree::
:maxdepth: 2
:caption: How-to Guides
Expand All @@ -57,10 +66,12 @@ code ever again.
howto/overrides.md
howto/rename.md

howto/vet.md
.. toctree::
:maxdepth: 3
:caption: sqlc Cloud
:hidden:

howto/managed-databases.md
howto/ci-cd.md
howto/upload.md

.. toctree::
:maxdepth: 3
Expand All @@ -81,7 +92,8 @@ code ever again.
:caption: Conceptual Guides
:hidden:

howto/ci-cd.md
guides/using-go-and-pgx.rst
guides/development.md
guides/plugins.md
guides/development.md
guides/privacy.md
17 changes: 15 additions & 2 deletions docs/reference/config.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ file must be in the directory where the `sqlc` command is run.

```yaml
version: "2"
cloud:
project: "<PROJECT_ID>"
sql:
- schema: "postgresql/schema.sql"
queries: "postgresql/query.sql"
Expand All @@ -16,7 +18,7 @@ sql:
package: "authors"
out: "postgresql"
database:
uri: "postgresql://postgres:postgres@localhost:5432/postgres"
managed: true
rules:
- sqlc/db-prepare
- schema: "mysql/schema.sql"
Expand Down Expand Up @@ -46,6 +48,8 @@ Each mapping in the `sql` collection has the following keys:
- A mapping to configure database connections. See [database](#database) for the supported keys.
- `rules`:
- A collection of rule names to run via `sqlc vet`. See [rules](#rules) for configuration options.
- `analzyer`:
- A mapping to configure query analysis. See [analyzer](#analyzer) for the supported keys.
- `strict_function_checks`
- If true, return an error if a called SQL function does not exist. Defaults to `false`.

Expand Down Expand Up @@ -85,6 +89,8 @@ sql:

The `database` mapping supports the following keys:

- `managed`:
- If true, connect to a [managed database](../howto/managed-databases.md). Defaults to `false`.
- `uri`:
- Database connection URI

Expand All @@ -105,7 +111,14 @@ sql:
package: authors
out: postgresql
```


### analyzer

The `analyzer` mapping supports the following keys:

- `database`:
- If false, do not use the configured database for query analysis. Defaults to `true`.

### gen

The `gen` mapping supports the following keys:
Expand Down
2 changes: 1 addition & 1 deletion docs/tutorials/getting-started-postgresql.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ following contents:
```yaml
version: "2"
cloud:
project: "<your project id>"
project: "<PROJECT_ID>"
sql:
- engine: "postgresql"
queries: "query.sql"
Expand Down