Document pgbouncer-related prepared statement breakage

Fixes: #121.
Fixes: #149.

Author: elprans

.github/ISSUE_TEMPLATE.md

@@ -12,6 +12,8 @@ Thank you!
 
 * **asyncpg version**:
 * **PostgreSQL version**:
+* **Do you use a PostgreSQL SaaS?  If so, which?  Can you reproduce
+  the issue with a local PostgreSQL install?**:
 * **Python version**:
 * **Platform**:
 * **Do you use pgbouncer?**:

asyncpg/exceptions/_base.py

@@ -7,6 +7,7 @@
 
 import asyncpg
 import sys
+import textwrap
 
 
 __all__ = ('PostgresError', 'FatalPostgresError', 'UnknownPostgresError',
@@ -121,6 +122,31 @@ def _make_constructor(cls, fields, query=None):
             message = ('cached statement plan is invalid due to a database '
                        'schema or configuration change')
 
+        is_prepared_stmt_error = (
+            exccls.__name__ in ('DuplicatePreparedStatementError',
+                                'InvalidSQLStatementNameError') and
+            _is_asyncpg_class(exccls)
+        )
+
+        if is_prepared_stmt_error:
+            hint = dct.get('hint', '')
+            hint += textwrap.dedent("""\
+
+                NOTE: pgbouncer with pool_mode set to "transaction" or
+                "statement" does not support prepared statements properly.
+                You have two options:
+
+                * if you are using pgbouncer for connection pooling to a
+                  single server, switch to the connection pool functionality
+                  provided by asyncpg, it is a much better option for this
+                  purpose;
+
+                * if you have no option of avoiding the use of pgbouncer,
+                  then you must switch pgbouncer's pool_mode to "session".
+            """)
+
+            dct['hint'] = hint
+
         return exccls, message, dct
 
     def as_dict(self):

asyncpg/transaction.py

@@ -117,7 +117,7 @@ async def start(self):
 
         try:
             await self._connection.execute(query)
-        except:
+        except BaseException:
             self._state = TransactionState.FAILED
             raise
         else:
@@ -158,7 +158,7 @@ async def __commit(self):
 
         try:
             await self._connection.execute(query)
-        except:
+        except BaseException:
             self._state = TransactionState.FAILED
             raise
         else:
@@ -177,7 +177,7 @@ async def __rollback(self):
 
         try:
             await self._connection.execute(query)
-        except:
+        except BaseException:
             self._state = TransactionState.FAILED
             raise
         else:

docs/api/index.rst

@@ -53,6 +53,12 @@ a need to run the same query again.
    during calls to the :meth:`~Connection.fetch`, :meth:`~Connection.fetchrow`,
    or :meth:`~Connection.fetchval` methods.
 
+.. warning::
+
+   If you are using pgbouncer with ``pool_mode`` set to ``transaction`` or
+   ``statement``, prepared statements will not work correctly.  See
+   :ref:`asyncpg-prepared-stmt-errors` for more information.
+
 
 .. autoclass:: asyncpg.prepared_stmt.PreparedStatement()
    :members:

docs/faq.rst

@@ -5,37 +5,73 @@ Frequently Asked Questions
 ==========================
 
 Does asyncpg support DB-API?
-    No.  DB-API is a synchronous API, while asyncpg is based
-    around an asynchronous I/O model.  Thus, full drop-in compatibility
-    with DB-API is not possible and we decided to design asyncpg API
-    in a way that is better aligned with PostgreSQL architecture and
-    terminology.  We will release a synchronous DB-API-compatible version
-    of asyncpg at some point in the future.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+No.  DB-API is a synchronous API, while asyncpg is based
+around an asynchronous I/O model.  Thus, full drop-in compatibility
+with DB-API is not possible and we decided to design asyncpg API
+in a way that is better aligned with PostgreSQL architecture and
+terminology.  We will release a synchronous DB-API-compatible version
+of asyncpg at some point in the future.
+
 
 Can I use asyncpg with SQLAlchemy ORM?
-    Short answer: no.  asyncpg uses asynchronous execution model
-    and API, which is fundamentally incompatible with SQLAlchemy.
-    However, it is possible to use asyncpg and SQLAlchemy Core
-    with the help of a third-party adapter, such as asyncpgsa_.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Short answer: no.  asyncpg uses asynchronous execution model
+and API, which is fundamentally incompatible with SQLAlchemy.
+However, it is possible to use asyncpg and SQLAlchemy Core
+with the help of a third-party adapter, such as asyncpgsa_.
+
 
 Can I use dot-notation with :class:`asyncpg.Record`?  It looks cleaner.
-    We decided against making :class:`asyncpg.Record` a named tuple
-    because we want to keep the ``Record`` method namespace separate
-    from the column namespace.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We decided against making :class:`asyncpg.Record` a named tuple
+because we want to keep the ``Record`` method namespace separate
+from the column namespace.
+
 
 Why can't I use a :ref:`cursor <asyncpg-api-cursor>` outside of a transaction?
-    Cursors created by a call to
-    :meth:`Connection.cursor() <asyncpg.connection.Connection.cursor>` or
-    :meth:`PreparedStatement.cursor() \
-    <asyncpg.prepared_stmt.PreparedStatement.cursor>`
-    cannot be used outside of a transaction.  Any such attempt will result in
-    ``InterfaceError``.
-    To create a cursor usable outside of a transaction, use the
-    ``DECLARE ... CURSOR WITH HOLD`` SQL statement directly.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Cursors created by a call to
+:meth:`Connection.cursor() <asyncpg.connection.Connection.cursor>` or
+:meth:`PreparedStatement.cursor() \
+<asyncpg.prepared_stmt.PreparedStatement.cursor>`
+cannot be used outside of a transaction.  Any such attempt will result in
+``InterfaceError``.
+To create a cursor usable outside of a transaction, use the
+``DECLARE ... CURSOR WITH HOLD`` SQL statement directly.
+
+
+.. _asyncpg-prepared-stmt-errors:
+
+Why am I getting prepared statement errors?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are getting intermittent ``prepared statement "__asyncpg_stmt_xx__"
+does not exist`` or ``prepared statement “__asyncpg_stmt_xx__”
+already exists`` errors, you are most likely not connecting to the
+PostgreSQL server directly, but via
+`pgbouncer <https://pgbouncer.github.io/>`_.  pgbouncer, when
+in the ``"transaction"`` or ``"statement"`` pooling mode, does not support
+prepared statements.  You have two options:
+
+* if you are using pgbouncer for connection pooling to a single server,
+  switch to the :ref:`connection pool <asyncpg-connection-pool>`
+  functionality provided by asyncpg, it is a much better option for this
+  purpose;
+
+* if you have no option of avoiding the use of pgbouncer, then you need to
+  switch pgbouncer's ``pool_mode`` to ``session``.
+
 
 Why do I get ``PostgresSyntaxError`` when using ``expression IN $1``?
-    ``expression IN $1`` is not a valid PostgreSQL syntax.  To check
-    a value against a sequence use ``expression = any($1::mytype[])``,
-    where ``mytype`` is the array element type.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+``expression IN $1`` is not a valid PostgreSQL syntax.  To check
+a value against a sequence use ``expression = any($1::mytype[])``,
+where ``mytype`` is the array element type.
 
 .. _asyncpgsa: https://github.com/CanopyTax/asyncpgsa

docs/usage.rst

@@ -222,6 +222,8 @@ The most common way to use transactions is through an ``async with`` statement:
 See the :ref:`asyncpg-api-transaction` API documentation for more information.
 
 
+.. _asyncpg-connection-pool:
+
 Connection Pools
 ----------------
 
@@ -232,7 +234,7 @@ pool implementation, which eliminates the need to use an external connection
 pooler such as PgBouncer.
 
 To create a connection pool, use the
-:func:`asyncpg.create_pool <asyncpg.pool.create_pool>` function.
+:func:`asyncpg.create_pool() <asyncpg.pool.create_pool>` function.
 The resulting :class:`Pool <asyncpg.pool.Pool>` object can then be used
 to borrow connections from the pool.
 

tests/test_prepare.py

@@ -569,3 +569,28 @@ async def test_prepare_30_invalid_arg_count(self):
                 exceptions.InterfaceError,
                 'the server expects 0 arguments for this query, 1 was passed'):
             await self.con.fetchval('SELECT 1', 1)
+
+    async def test_prepare_31_pgbouncer_note(self):
+        try:
+            await self.con.execute("""
+                DO $$ BEGIN
+                    RAISE EXCEPTION
+                        'duplicate statement' USING ERRCODE = '42P05';
+                END; $$ LANGUAGE plpgsql;
+            """)
+        except asyncpg.DuplicatePreparedStatementError as e:
+            self.assertTrue('pgbouncer' in e.hint)
+        else:
+            self.fail('DuplicatePreparedStatementError not raised')
+
+        try:
+            await self.con.execute("""
+                DO $$ BEGIN
+                    RAISE EXCEPTION
+                        'invalid statement' USING ERRCODE = '26000';
+                END; $$ LANGUAGE plpgsql;
+            """)
+        except asyncpg.InvalidSQLStatementNameError as e:
+            self.assertTrue('pgbouncer' in e.hint)
+        else:
+            self.fail('InvalidSQLStatementNameError not raised')