Improved stored array documentation

Author: davidbrochart

docs/source/stored_arrays.rst

@@ -7,6 +7,9 @@
 Stored Arrays
 =============
 
+File arrays
+-----------
+
 Arrays can be stored on a file system using ``xfile_array``, enabling
 persistence of data. This type of array is a file-backed cached ``xarray``,
 meaning that you can use it as a normal array, and it will be flushed to the
@@ -16,7 +19,7 @@ file system or Google Cloud Storage, and data can be stored in various formats,
 e.g. GZip or Blosc.
 
 File Mode
----------
+^^^^^^^^^
 
 A file array can be created using one of the three following file modes:
 
@@ -30,9 +33,35 @@ A file array can be created using one of the three following file modes:
 
 The default mode is ``load``.
 
-Example : on-disk file array
-----------------------------
+IO handler
+^^^^^^^^^^
+
+Stored arrays can be read and written to various file systems using an IO
+handler, which is a template parameter of ``xfile_array``. The following IO
+handlers are currently supported:
+
+- ``xio_disk_handler``: for accessing the local file system.
+- ``xio_gcs_handler``: for accessing Google Cloud Storage.
+
+The IO handler is itself templated by a file format.
+
+File format
+^^^^^^^^^^^
 
+An array is stored in a file system using a file format, which usually performs
+some kind of compression. A file format has the ability to store the data, and
+optionally the shape of the array. It can also optionally be configured. The
+following file formats are currently supported:
+
+- ``xio_binary_config``: raw binary format.
+- ``xio_gzip_config``: GZip format.
+- ``xio_blosc_config``: Blosc format.
+
+These formats currently only store the data, not the shape. GZip and Blosc
+formats are configurable, but not the binary format.
+
+Example : on-disk file array
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code:: cpp
 
@@ -64,13 +93,13 @@ Example : on-disk file array
         a2.resize(shape);
 
         // a1 and a2 are equal
-        assert(xt:all(xt::equal(a1, a2)));
+        assert(xt::all(xt::equal(a1, a2)));
 
         return 0;
     }
 
-Stored Chunked Arrays
----------------------
+Chunked File Arrays
+-------------------
 
 As for a "normal" array, a chunked array can be stored on a file system. Under
 the hood, it will use ``xfile_array`` to store the chunks. But rather than
@@ -79,7 +108,10 @@ only a limited number of file arrays are used at the same time in a chunk pool.
 The container which is responsible for managing the chunk pool (i.e. map
 logical chunks in the array to physical chunks in the pool) is the
 ``xchunk_store_manager``, but you should not use it directly. Instead, we
-provide factory functions to create a stored chunked array, as shown below:
+provide factory functions to create a chunked file array, as shown below:
+
+Example : on-disk chunked file array
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: cpp
 
@@ -105,5 +137,6 @@ provide factory functions to create a stored chunked array, as shown below:
         a1(0, 0) = 5.6;  // because the pool is full, this saves chunk (1, 0) to disk
                          // and assigns to chunk (0, 0) in memory
         // when a1 is destroyed, all the modified chunks are saved to disk
-        // this can be forced with a1.chunks().flush()
+        // here, only chunks (0, 1) and (0, 0) are saved, since chunk (1, 0) was not changed
+        // flushing can be triggered manually by calling a1.chunks().flush()
     }

include/xtensor-io/xchunk_store_manager.hpp

@@ -77,11 +77,12 @@ namespace xt
      *
      * The xchunk_store_manager class implements a multidimensional chunk container.
      * Chunks are managed in a pool, allowing for a limited number of chunks
-     * that can simultaneously be hold in memory.
+     * that can simultaneously be hold in memory, by swapping chunks when the pool
+     * is full. Should not be used directly, instead use chunked_file_array factory
+     * functions.
      *
      * @tparam EC The type of a chunk (e.g. xfile_array)
      * @tparam IP The type of the index-to-path transformer (default: xindex_path)
-     * @sa xfile_array, xchunked_array, chunked_array, chunked_file_array
      */
     template <class EC, class IP = xindex_path>
     class xchunk_store_manager: public xaccessible<xchunk_store_manager<EC, IP>>,
@@ -200,7 +201,7 @@ namespace xt
     };
 
     /**
-     * Creates a stored chunked array.
+     * Creates a chunked file array.
      * This function returns an uninitialized ``xchunked_array<xchunk_store_manager<xfile_array<T, IOH>>>``.
      *
      * @tparam T The type of the elements (e.g. double)
@@ -226,8 +227,8 @@ namespace xt
                        layout_type chunk_memory_layout = XTENSOR_DEFAULT_LAYOUT);
 
     /**
-     * Creates a stored chunked array.
-     * This function returns a uninitialized ``xchunked_array<xchunk_store_manager<xfile_array<T, IOH>>>``.
+     * Creates a chunked file array.
+     * This function returns an uninitialized ``xchunked_array<xchunk_store_manager<xfile_array<T, IOH>>>``.
      *
      * @tparam T The type of the elements (e.g. double)
      * @tparam IOH The type of the IO handler (e.g. xio_disk_handler)
@@ -254,7 +255,7 @@ namespace xt
                        layout_type chunk_memory_layout = XTENSOR_DEFAULT_LAYOUT);
 
     /**
-     * Creates a stored chunked array.
+     * Creates a chunked file array.
      * This function returns a ``xchunked_array<xchunk_store_manager<xfile_array<T, IOH>>>`` initialized from an expression.
      *
      * @tparam T The type of the elements (e.g. double)
@@ -280,7 +281,7 @@ namespace xt
                        layout_type chunk_memory_layout = XTENSOR_DEFAULT_LAYOUT);
 
     /**
-     * Creates a stored chunked array.
+     * Creates a chunked file array.
      * This function returns a ``xchunked_array<xchunk_store_manager<xfile_array<T, IOH>>>`` initialized from an expression.
      *
      * @tparam IOH The type of the IO handler (e.g. xio_disk_handler)

include/xtensor-io/xfile_array.hpp

@@ -113,7 +113,7 @@ namespace xt
      *
      * @tparam E The type of the container holding the elements
      * @tparam IOH The type of the IO handler (e.g. xio_disk_handler)
-     * @sa chunked_file_array, xchunk_store_manager
+     * @sa xchunk_store_manager
      */
     template <class E, class IOH>
     class xfile_array_container : public xaccessible<xfile_array_container<E, IOH>>,