Merge pull request #458 from DivanteLtd/release/v1.12.0-rc.1

Release/v1.12.0

Author: Tomasz Kostuch

.eslintignore

@@ -0,0 +1 @@
+node_modules

CHANGELOG.md

@@ -4,6 +4,37 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/)
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+
+## [1.12.0] - 2020.06.01
+
+### Added
+- Add url module - @gibkigonzo (#3942)
+- The `response_format` query parameter to the `/api/catalog` endpoint. Currently there is just one additional format supported: `response_format=compact`. When used, the response format got optimized by: a) remapping the results, removing the `_source` from the `hits.hits`; b) compressing the JSON fields names according to the `config.products.fieldsToCompact`; c) removing the JSON fields from the `product.configurable_children` when their values === parent product values; overall response size reduced over -70% - @pkarw
+- The support for `SearchQuery` instead of the ElasticSearch DSL as for the input to `/api/catalog` - using `storefront-query-builder` package - @pkarw - https://github.com/DivanteLtd/vue-storefront/issues/2167
+- Create attribute service that allows to fetch attributes with specific options - used for products aggregates - @gibkigonzo (https://github.com/DivanteLtd/vue-storefront/pull/4001, https://github.com/DivanteLtd/mage2vuestorefront/pull/99)
+- Add ElasticSearch client support for HTTP authentication - @cewald (#397)
+- Endpoint for reset password with reset token. Only for Magento 2 - @Fifciu
+- Varnish Cache with autoinvalidation by Cache tags as addon - @Fifciu
+- Add `resetPasswordUsingResetToken` to `magento1` platform - @cewald (#415)
+- Update to `storefront-query-builder` version `1.0.0` - @cewald (#429)
+- Add `composeError` in './magento2/o2m.js' - @flancer64 (#422)
+- Explicit data extraction from 'Error' objects - @flancer64 (#424)
+- Merge scripts: db7 => db, elastic7=>elastic and add support for es7 in yarn mage2vs import - @gibkigonzo (#436)
+
+### Fixed
+- add es7 support for map url module and fixed default index for es config - @gibkigonzo
+- Add correct paths for production build - @cewald (#407)
+- Fix MSI default stock id value
+- Add outputFormatter to response from cache - @gibkigonzo (#428)
+- disable showing stack for invalid requests - @gibkigonzo (#431)
+- Improve `_outputFormatter` on cache catalog-response to prevent exception - @cewald (#432)
+- use ts for compiling additional scripts - @gibkigonzo (#437)
+- Bugfix for wrong JSON scheme url's and hostname resolution for `magento1` platform - @cewald (#443)
+- check if headers has been already sent in image middleware - @gibkigonzo (#434)
+- Missing packages in Docker Makefile (make, python and g++) - @proxiblue (#448)
+- change path in ecosystem for o2m - @gibkigonzo (#434)
+- udpate typings for query params - @gibkigonzo (#446)
+
 ## [1.11.1] - 2020.03.17
 
 ### Added
@@ -23,7 +54,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - Fixed some smaller issues with graphql so that it is now working again with the fronted - #350
 - Replaced the old `crop` function call which has been removed from Sharp image processor - @grimasod (#381)
-
+- Add product processor to new URL mapper endpoint #401 - @cewald (#401, #403)
+- Add fallback for `sourcePriceInclTax` and `finalPriceInclTax` in `magento1` platform - @cewald (#398)
 
 ## [1.11.0-rc.1] - 2019.10.03
 
@@ -39,6 +71,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Added support for tax calculation where the values from customer_tax_class_ids is used - @resubaka (#307)
 - The `db` context object - passed to every api endpoint now has two usefull methods: `getElasticClient` and `getRedisClient` for accesing the data stores - @pkarw (#328)
 - The `lib/utils` got two new methods `getStoreCode(req: Express.Request)` and `getStoreView(code: string)` for getting the current multistore context from `vue-storefront` frontend requests - @pkarw
+- Check message property instead of errorMessage in apiError function - @cdshotels-liborpansky (#378)
 
 ### Removed
 - The `scripts/seo.js` tool has been removed, the legacy `migrations` scripts have been removed, the unused legacy extensions (`gls-parcelshop-dk`, `postnord-parcelshop-dk`) - @pkarw (#342)

README.md

@@ -111,22 +111,7 @@ Starting from [Elasitc 6 and 7](https://www.elastic.co/guide/en/elasticsearch/re
 
 From now on, we're using the separate indexes per each entity type. The convention is: `${indexName}_${entityType}`. If your' **logical index name** is `vue_storefront_catalog` then it will be mapped to the **physical indexes** of: `vue_storefront_catalog_product`, `vue_storefront_catalog_category` ...
 
-To take the advantage of this new logical->physical index distinction we've provided new Elastic tools: `db7`, `migrate7`, `dump7`, `restore7` tools. They can be used exactly the same way [like the old tools](https://docs.vuestorefront.io/guide/data/database-tool.html) were. The only distinction is that they work on separate indexes.
-
-**Create new index**
-
-Before restoreing or importing data you might want to create a new Elastic index with the proper data types/schema applied. You can run just the `yarn db7 new` command in order to do that.
-
-**Restore the data**
-
-After you ran the docker file and have Elastic7 up and running you might want to:
-
-a) restore the demo data by running `yarn restore7` and `yarn restore7_it; yarn restore7_de` for default multistores. The data is restored from `var/catalog_product.json`, `var/catalog_category.json` and so on...
-
-b) import the data from Magento to proper physical indexes. To do so, currently you can do this only with [mage2vuestorefront](https://github.com/DivanteLtd/mage2vuestorefront/pull/96).
-
-**Note:** After 1.11 stable release (around November, 2019) we'll **replace** the standard tools: `db`, `migrate`, `dump`, `restore` with the Elastic 7 equivalents and it will become the default.
-
+[Tools](https://docs.vuestorefront.io/guide/data/database-tool.html) are adjusted to ES7. You can use `yarn db new`, `yarn restore`, `yarn mage2vs import`. Just make sure that you have set up `config.elasticsearch.apiVersion` to `7.1`.
 
 ## API access
 Catalog API calls are compliant with ElasticSearch (it works like a filtering proxy to ES). More on ES queries: [ElasticSearch queries tutorial](http://okfnlabs.org/blog/2013/07/01/elasticsearch-query-tutorial.html)

config/default.json

@@ -9,7 +9,8 @@
     "availableCacheTags": ["P", "C", "T", "A", "product", "category", "attribute", "taxrule"],
     "invalidateCacheKey": "aeSu7aip",
     "invalidateCacheForwarding": false,
-    "invalidateCacheForwardUrl": "http://localhost:3000/invalidate?key=aeSu7aip&tag="
+    "invalidateCacheForwardUrl": "http://localhost:3000/invalidate?key=aeSu7aip&tag=",
+    "showErrorStack": false
   },
   "orders": {
     "useServerQueue": false
@@ -38,7 +39,89 @@
       "taxrule",
       "review"
     ],
-    "apiVersion": "5.6"
+    "apiVersion": "5.6",
+
+    "searchScoring": {
+      "attributes": {
+        "attribute_code": {
+          "scoreValues": { "attribute_value": { "weight": 1 } }
+        }
+      },
+      "fuzziness": 2,
+      "cutoff_frequency":  0.01,
+      "max_expansions": 3,
+      "minimum_should_match": "75%",
+      "prefix_length": 2,
+      "boost_mode": "multiply",
+      "score_mode": "multiply",
+      "max_boost": 100,
+      "function_min_score": 1
+    },
+    "searchableAttributes": {
+      "name": {
+        "boost": 4
+      },
+      "sku": {
+        "boost": 2
+      },
+      "category.name": {
+        "boost": 1
+      }
+    }
+  },
+  "products": {
+    "fieldsToCompress": ["max_regular_price", "max_price", "max_regular_price", "minimal_regular_price", "final_price", "price", "special_price", "original_final_price", "original_price", "original_special_price", "final_price_incl_tax", "price_incl_tax", "special_price_incl_tax", "final_price_tax", "price_tax", "special_price_tax", "image", "small_image", "thumbnail"],
+    "fieldsToCompact": {
+      "minimal_price": "mp",
+      "has_options": "ho",
+      "url_key": "u",
+      "status": "s",
+      "required_options": "ro",
+      "name": "nm",
+      "tax_class_id": "tci",
+      "description": "desc",
+      "minimal_regular_price": "mrp",
+      "final_price": "fp", 
+      "price": "p", 
+      "special_price": "sp",
+      "original_final_price": "ofp",
+      "original_price": "op",
+      "original_special_price": "osp",
+      "final_price_incl_tax": "fpit", 
+      "original_price_incl_tax": "opit",
+      "price_incl_tax": "pit",
+      "special_price_incl_tax": "spit", 
+      "final_price_tax": "fpt",
+      "price_tax": "pt",
+      "special_price_tax": "spt",
+      "original_price_tax": "opt",
+      "image": "i", 
+      "small_image": "si",
+      "thumbnail": "t"
+    },
+    "filterFieldMapping": {
+      "category.name": "category.name.keyword"
+    },
+    "filterAggregationSize": {
+      "default": 10,
+      "size": 10,
+      "color": 10
+    },    
+    "priceFilterKey": "final_price",
+    "priceFilters": {
+      "ranges": [
+        { "from": 0, "to": 50 },
+        { "from": 50, "to": 100 },
+        { "from": 100, "to": 150 },
+        { "from": 150 }
+      ]
+    }    
+  },
+  "varnish": {
+    "host": "185.246.52.88",
+    "port": 80,
+    "method": "BAN",
+    "enabled": false
   },
   "redis": {
     "host": "localhost",
@@ -78,7 +161,7 @@
       "tax": {
         "defaultCountry": "DE",
         "defaultRegion": "",
-        "deprecatedPriceFieldsSupport": true,
+        "deprecatedPriceFieldsSupport": false,
         "calculateServerSide": true,
         "sourcePriceIncludesTax": false,
         "finalPriceIncludesTax": true,
@@ -115,7 +198,7 @@
         "defaultRegion": "",
         "calculateServerSide": true,
         "sourcePriceIncludesTax": false,
-        "deprecatedPriceFieldsSupport": true,
+        "deprecatedPriceFieldsSupport": false,
         "finalPriceIncludesTax": true,
         "userGroupId": null,
         "useOnlyDefaultUserGroupId": false
@@ -145,7 +228,7 @@
     "usePlatformTotals": true,
     "setConfigurableProductOptions": true,
     "sourcePriceIncludesTax": false,
-    "deprecatedPriceFieldsSupport": true,
+    "deprecatedPriceFieldsSupport": false,
     "finalPriceIncludesTax": false,
     "userGroupId": null,
     "useOnlyDefaultUserGroupId": false
@@ -164,6 +247,7 @@
     "cms-data",
     "mail-service",
     "example-processor",
+    "example-custom-filter",
     "elastic-stock"
   ],
   "extensions": {
@@ -188,6 +272,9 @@
       "resultProcessors": {
         "product": "my-product-processor"
       }
+    },
+    "example-custom-filter": {
+      "catalogFilter": [ "SampleFilter" ]
     }
   },
   "magento2": {
@@ -257,7 +344,8 @@
         "includeFields": [ "children_data", "id", "children_count", "sku", "name", "is_active", "parent_id", "level", "url_key" ]
       },
       "attribute": {
-        "includeFields": [ "attribute_code", "id", "entity_type_id", "options", "default_value", "is_user_defined", "frontend_label", "attribute_id", "default_frontend_label", "is_visible_on_front", "is_visible", "is_comparable" ]
+        "includeFields": [ "attribute_code", "id", "entity_type_id", "options", "default_value", "is_user_defined", "frontend_label", "attribute_id", "default_frontend_label", "is_visible_on_front", "is_visible", "is_comparable" ],
+        "loadByAttributeMetadata": false
       },
       "productList": {
         "sort": "",
@@ -284,5 +372,12 @@
     "description": 1,
     "sku": 1,
     "configurable_children.sku": 1
+  },
+  "urlModule": {
+    "map": {
+      "includeFields": ["url_path", "identifier", "id", "slug", "sku", "type_id"],
+      "searchedFields": ["url_path", "identifier"],
+      "searchedEntities": ["product", "category", "cms_page"]
+    }
   }
 }

config/elastic.schema.product.json

@@ -76,6 +76,25 @@
        "tax_class_id": {"type": "integer"},
        "gender": {"type": "integer"},
        "material": {"type": "integer"},
-       "category_gear": {"type": "integer"}
+       "category_gear": {"type": "integer"},
+       "attributes_metadata": {
+            "properties": {
+                "id": {"type": "integer"},
+                "attribute_id": {"type": "integer"},
+                "default_frontend_label": {"type": "text"},
+                "is_visible_on_front": {"type": "text"},
+                "is_visible" : {"type": "boolean"},
+                "frontend_input": {"type": "text"},
+                "is_user_defined": {"type": "boolean"},
+                "is_comparable": {"type": "text"},
+                "attribute_code": {"type": "text"},
+                "options": {
+                    "properties": {
+                        "value": {"type": "text"},
+                        "label": {"type": "text"}
+                    }
+                }
+            }
+        }
     }
 }

docker-compose.nodejs.yml

@@ -26,3 +26,4 @@ services:
       - /var/www/dist
     ports:
       - '8080:8080'
+      

docker-compose.varnish.yml

@@ -0,0 +1,10 @@
+version: '3.0'
+services:
+  varnish:
+    build:
+      context: .
+      dockerfile: varnish/Dockerfile
+    volumes:
+      - ./docker/varnish/config.vcl:/usr/local/etc/varnish/default.vcl
+    ports:
+      - '1234:80'

docker/varnish/Dockerfile

@@ -0,0 +1,19 @@
+FROM cooptilleuls/varnish:6.0-stretch
+
+# install varnish-modules
+RUN apt-get update -y && \
+	    apt-get install -y build-essential automake libtool curl git python-docutils && \
+	    curl -s https://packagecloud.io/install/repositories/varnishcache/varnish60/script.deb.sh | bash;
+
+RUN apt-get install -y pkg-config libvarnishapi1 libvarnishapi-dev autotools-dev;
+
+RUN git clone https://github.com/varnish/varnish-modules.git /tmp/vm;
+RUN cd /tmp/vm; \
+			git checkout 6.0; \
+	    ./bootstrap && \
+	    ./configure;
+
+RUN cd /tmp/vm && \
+			make && \
+	    make check && \
+	    make install;