Add support for direct path load in thin mode.

Author: anthony-tuininga

README.md

@@ -1,16 +1,16 @@
 # python-oracledb
 
-Python-oracledb is an open-source [Python][python] extension module allowing
-Python programs to connect to [Oracle Database][oracledb]. The module conforms
-to the [Python Database API 2.0 specification][pep249] with a considerable
-number of additions and a couple of minor exclusions, see the [feature
-list][features]. It is maintained by Oracle.
-
-Python-oracledb is used for executing SQL and PL/SQL; for calling NoSQL-style
-document APIs; for working with data frames; for receiving database
+Python-oracledb is the widely used, open-source [Python][python] extension
+module allowing Python programs to connect to [Oracle Database][oracledb]. The
+module conforms to the [Python Database API 2.0 specification][pep249] with a
+considerable number of additions and a couple of minor exclusions, see the
+[feature list][features]. It is maintained by Oracle.
+
+Python-oracledb is used for executing SQL and PL/SQL; for working with data
+frames; for calling NoSQL-style document APIs; for receiving database
 notifications and messages; and for starting and stopping the database. It has
-features for high availability and security. It is used by many Python
-Frameworks, SQL Generators, ORMs, and libraries.
+features for fast data loading, high availability, and security. It is used by
+many Python frameworks, SQL generators, ORMs, and libraries.
 
 Synchronous and [concurrent][concurrent] coding styles are supported. Database
 operations can optionally be [pipelined][pipelining].

doc/src/api_manual/async_connection.rst

@@ -77,6 +77,14 @@ AsyncConnection Methods
 
     .. versionadded:: 2.1.0
 
+.. automethod:: AsyncConnection.direct_path_load
+
+    See :ref:`directpathloads`.
+
+    .. versionadded:: 3.4.0
+
+    .. dbapimethodextension::
+
 .. automethod:: AsyncConnection.encode_oson
 
     .. versionadded:: 2.1.0

doc/src/api_manual/connection.rst

@@ -73,6 +73,14 @@ Connection Methods
 
     .. dbapimethodextension::
 
+.. automethod:: Connection.direct_path_load
+
+    See :ref:`directpathloads`.
+
+    .. versionadded:: 3.4.0
+
+    .. dbapimethodextension::
+
 .. automethod:: Connection.encode_oson
 
     .. versionadded:: 2.1.0

doc/src/release_notes.rst

@@ -19,6 +19,9 @@ oracledb `3.4.0 <https://github.com/oracle/python-oracledb/compare/v3.3.0...v3.4
 Thin Mode Changes
 +++++++++++++++++
 
+#)  Added support for Oracle Database's :ref:`Direct Path Load
+    <directpathloads>` functionality which is very efficient for loading large
+    datasets into a database.
 #)  Fixed bug when setting values of type ``datetime.date`` on variables (such
     as created by :meth:`Cursor.var()` or implicitly by
     :meth:`Cursor.setinputsizes()`) of types

doc/src/user_guide/appendix_a.rst

@@ -245,6 +245,10 @@ For more details see :ref:`driverdiff` and :ref:`upgrading83`.
       - No
       - Yes
       - Yes
+    * - Direct Path Loads (see :ref:`directpathloads`)
+      - Yes
+      - No
+      - No
     * - Oracle Database 23ai JSON-Relational Duality Views (see :ref:`jsondualityviews`)
       - Yes
       - Yes

doc/src/user_guide/batch_statement.rst

@@ -14,6 +14,11 @@ easily optimize batch insertion, and also allows "noisy" data (values not in a
 suitable format) to be filtered for review while other, correct, values are
 inserted.
 
+In addition to Oracle Database "Array DML" batch loading,
+:ref:`directpathloads` can be used for very fast loading of large data sets if
+certain schema criteria can be met. Another option for frequent, small inserts
+is to load data using the Oracle Database :ref:`memoptimized`.
+
 Related topics include :ref:`tuning` and :ref:`dataframeformat`.
 
 Batch Statement Execution
@@ -618,3 +623,155 @@ B19E-449D-9968-1121AF06D793>`__ between the databases and using
 INSERT INTO SELECT or CREATE AS SELECT.
 
 You can control the data transfer by changing your SELECT statement.
+
+.. _directpathloads:
+
+Direct Path Loads
+=================
+
+Direct Path Loads allows data being inserted into Oracle Database to bypass
+code layers such as the database buffer cache. Also there are no INSERT
+statements used. This can be very efficient for ingestion of huge amounts of
+data but, as a consequence of the architecture, there are restrictions on when
+Direct Path Loads can be used. For more information see Oracle Database
+documentation such as on SQL*Loader `Direct Path Loads
+<https://www.oracle.com/pls/topic/lookup?ctx=dblatest&id=
+GUID-0D576DEF-7918-4DD2-A184-754D217C021F>`__ and on the Oracle Call Interface
+`Direct Path Load Interface
+<https://www.oracle.com/pls/topic/lookup?ctx=dblatest&id=
+GUID-596F5F9B-47A1-48DB-8702-FEED7BE038B9>`__.
+
+The end-to-end insertion time when using Direct Path Loads for smaller data
+sets may not be faster than using :meth:`Cursor.executemany()`, however there
+can still be reduced load on the database.
+
+.. note::
+
+    Direct Path Loads are only supported in python-oracledb Thin mode.
+
+Direct Path Loading is performed by the :meth:`Connection.direct_path_load()`
+method. For example, if you have the table::
+
+    create table TestDirectPathLoad (
+        id    number(9),
+        name  varchar2(20)
+    );
+
+Then you can load data into it using the code:
+
+.. code-block:: python
+
+    SCHEMA_NAME = "HR"
+    TABLE_NAME = "TESTDIRECTPATHLOAD"
+    COLUMN_NAMES = ["ID", "NAME"]
+    DATA = [
+        (1, "A first row"),
+        (2, "A second row"),
+        (3, "A third row"),
+    ]
+
+    connection.direct_path_load(
+        schema_name=SCHEMA_NAME,
+        table_name=TABLE_NAME,
+        column_names=COLUMN_NAMES,
+        data=DATA
+    )
+
+The records are always implicitly committed.
+
+The ``data`` parameter can be a list of sequences, a :ref:`DataFrame
+<oracledataframeobj>` object, or a third-party DataFrame instance that supports
+the Apache Arrow PyCapsule Interface, see :ref:`dfppl`.
+
+To load into VECTOR columns, pass an appropriate `Python array.array()
+<https://docs.python.org/3/library/array.html>`__ value, or a list of values.
+For example, if you have the table::
+
+    create table TestDirectPathLoad (
+        id    number(9),
+        name  varchar2(20),
+        v64   vector(3, float64)
+    );
+
+Then you can load data into it using the code:
+
+.. code-block:: python
+
+    SCHEMA_NAME = "HR"
+    TABLE_NAME = "TESTDIRECTPATHLOAD"
+    COLUMN_NAMES = ["ID", "NAME", "V64"]
+    DATA = [
+        (1, "A first row", array.array("d", [1, 2, 3])),
+        (2, "A second row", [4, 5, 6]),
+        (3, "A third row", array.array("d", [7, 8, 9])),
+    ]
+
+    connection.direct_path_load(
+        schema_name=SCHEMA_NAME,
+        table_name=TABLE_NAME,
+        column_names=COLUMN_NAMES,
+        data=DATA
+    )
+
+
+For more on vectors, see :ref:`vectors`.
+
+Runnable Direct Path Load examples are in the `GitHub examples
+<https://github.com/oracle/python-oracledb/tree/main/samples>`__ directory.
+
+**Notes on Direct Path Loads**
+
+- Data is implicitly committed.
+- Data being inserted into CLOB or BLOB columns must be strings or bytes, not
+  python-oracledb :ref:`LOB Objects <lobobj>`.
+- Insertion of python-oracledb :ref:`DbObjectType Objects <dbobjecttype>` is
+  not supported
+
+Review Oracle Database documentation for database requirements and
+restrictions.
+
+Batching of Direct Path Loads
+-----------------------------
+
+If buffer, network, or database limits make it desirable to process smaller
+sets of records, you can either make repeated calls to
+:meth:`Connection.direct_path_load()` or you can use the ``batch_size``
+parameter. For example:
+
+.. code-block:: python
+
+    SCHEMA_NAME = "HR"
+    TABLE_NAME = "TESTDIRECTPATHLOAD"
+    COLUMN_NAMES = ["ID", "NAME"]
+    DATA = [
+        (1, "A first row"),
+        (2, "A second row"),
+        . . .
+        (10_000_000, "Ten millionth row"),
+    ]
+
+    connection.direct_path_load(
+        schema_name=SCHEMA_NAME,
+        table_name=TABLE_NAME,
+        column_names=COLUMN_NAMES,
+        data=DATA,
+        batch_size=1_000_000
+    )
+
+This will send the data to the database in batches of 1,000,000 records until
+all 10,000,000 records have been inserted.
+
+.. _memoptimized:
+
+Memoptimized Rowstore
+=====================
+
+The Memoptimized Rowstore is another Oracle Database feature for data
+ingestion, particularly for frequent single row inserts. It can also aid query
+performance. Configuration and control is handled by database configuration and
+the use of specific SQL statements. As a result, there is no specific
+python-oracledb requirement or API needed to take advantage of the feature.
+
+To use the Memoptimized Rowstore see Oracle Database documentation `Enabling
+High Performance Data Streaming with the Memoptimized Rowstore
+<https://www.oracle.com/pls/topic/lookup?ctx=dblatest&id=GUID-9752E93D-55A7-4584-B09B-9623B33B5CCF>`__.

doc/src/user_guide/dataframes.rst

@@ -647,7 +647,12 @@ Inserting Data Frames
 Python-oracledb :ref:`DataFrame <oracledataframeobj>` instances, or third-party
 DataFrame instances that support the Apache Arrow PyCapsule Interface, can be
 inserted into Oracle Database by passing them directly to
-:meth:`Cursor.executemany()` or :meth:`AsyncCursor.executemany()`.
+:meth:`Cursor.executemany()` or :meth:`AsyncCursor.executemany()`.  They can
+also be passed to :meth:`Connection.direct_path_load()` and
+:meth:`AsyncConnection.direct_path_load()`.
+
+Inserting Data Frames with executemany()
+----------------------------------------
 
 For example, with the table::
 
@@ -686,6 +691,45 @@ For general information about fast data ingestion, and discussion of
 :meth:`Cursor.executemany()` and :meth:`AsyncCursor.executemany()` options, see
 :ref:`batchstmnt`.
 
+.. _dfppl:
+
+Inserting Data Frames with Direct Path Loads
+--------------------------------------------
+
+Very large :ref:`DataFrame <oracledataframeobj>` objects can be efficiently
+inserted using Oracle Database Direct Path Loading by passing them to
+:meth:`Connection.direct_path_load()`. You can also pass third-party DataFrame
+instances that support the Apache Arrow PyCapsule Interface.
+
+See :ref:`directpathloads` for general information about Direct Path Loads.
+
+For example, if the user "HR" has the table::
+
+    create table mytab (
+                   id   number(9),
+                   name varchar2(100));
+
+The following code will insert a Pandas DataFrame:
+
+.. code-block:: python
+
+    import pandas
+
+    d = [
+        (1, "Abigail"),
+        (2, "Anna"),
+        (3, "Janey"),
+        (4, "Jessica"),
+    ]
+    pdf = pandas.DataFrame(data=d)
+
+    connection.direct_path_load(
+        schema_name="hr",
+        table_name="mytab",
+        column_names=["id", "name"],
+        data=pdf
+    )
+
 Explicit Conversion to DataFrame or ArrowArray
 ==============================================
 

doc/src/user_guide/sql_execution.rst

@@ -1114,8 +1114,8 @@ easily be executed with python-oracledb.  For example:
 Do not concatenate or interpolate user data into SQL statements.  See
 :ref:`bind` instead.
 
-When handling multiple data values, use :meth:`Cursor.executemany()` for
-performance.  See :ref:`batchstmnt`
+When handling multiple data values, use :meth:`Cursor.executemany()` or
+:meth:`Connection.direct_path_load()` for performance. See :ref:`batchstmnt`
 
 By default data is not committed to the database and other users will not be
 able to see your changes until your connection commits them by calling

doc/src/user_guide/tuning.rst

@@ -21,9 +21,11 @@ Some general tuning tips are:
 
   Make use of efficient python-oracledb functions. For example, to insert
   multiple rows use :meth:`Cursor.executemany()` instead of
-  :meth:`Cursor.execute()`. Another example is to fetch data directly as
-  :ref:`data frames <dataframeformat>` when working with packages like Pandas
-  and NumPy.
+  :meth:`Cursor.execute()`. Alternatively use
+  :meth:`Connection.direct_path_load()` for inserting very large
+  datasets. Another example is to fetch data directly as :ref:`data frames
+  <dataframeformat>` instead of using the traditional query code path when
+  working with packages like Pandas and NumPy.
 
 * Tune your SQL statements.  See the `SQL Tuning Guide
   <https://www.oracle.com/pls/topic/lookup?ctx=dblatest&id=TGSQL>`__.