Merge pull request #3100 from nexB/add-license-dump-option

Add license dump option

Author: AyanSinhaMahapatra

.gitignore

@@ -95,3 +95,20 @@ tcl
 /00-*.txt
 /z-todo-licenses-*
 
+# Extra ignores from licensedb
+*.pyc
+*.db
+.installed.cfg
+parts
+develop-eggs
+eggs
+downloads
+.settings
+TAGS
+Procfile
+local.cfg
+geckodriver.log
+var
+.metaflow
+selenium
+/dist/

CHANGELOG.rst

@@ -53,15 +53,30 @@ License detection:
   detected license only once. This data can contain the reference license text
   as an option.
 
-- We can now detect licenses using custom license texts and license rules.
-  These can be provided as a one off in a directory or packaged as a plugin
-  for consistent reuse and deployment.
-
 - There is a new "scancode-reindex-licenses" command that replace the
   "scancode --reindex-licenses" command line option which has been
   removed. This new command supports simpler reindexing using custom
   license texts and license rules contributed by plugins or stored in an
-  additional directory. 
+  additional directory. The "--reindex-licenses-for-all-languages" CLI option
+  is also moved to the "scancode-reindex-licenses" command as an option
+  "--all-languages".
+
+- We can now detect licenses using custom license texts and license rules.
+  These can be provided as a one off in a directory or packaged as a plugin
+  for consistent reuse and deployment. There is an option "--additional-directory"
+  with the "scancode-reindex-licenses" command and also a new "--only-builtin"
+  option to only use the builtin licenses to build the cache.
+
+- Scancode LICENSE and RULE files now also contain their data as YAML frontmatter,
+  which previously used to be in their respective YAML files. This reduces number of
+  files in those directories, 'rules' and 'licenses' to half. Git line history is
+  preserved for the files.
+
+- A new command line option "--get-license-data" is added to dump license data in
+  JSON, YAML and HTML formats, and also generates a local index and a static website
+  to view the data. This will essentially be an API/way to get scancode license data
+  as opposed to just reading the files. 
+
 
 Package detection:
 ~~~~~~~~~~~~~~~~~~~~~

requirements.txt

@@ -56,6 +56,7 @@ pygmars==0.7.0
 Pygments==2.12.0
 pymaven-patch==0.3.0
 pyparsing==3.0.9
+python-frontmatter==1.0.0
 pytz==2022.1
 PyYAML==6.0
 rdflib==6.2.0

setup.cfg

@@ -101,6 +101,7 @@ install_requires =
     pygmars >= 0.7.0
     pygments
     pymaven_patch >= 0.2.8
+    python-frontmatter >= 1.0.0
     requests >= 2.7.0
     saneyaml >= 0.5.2
     spdx_tools >= 0.7.0a3

src/licensedcode/data/inactive/rules/free-unknown_url_glc_90.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-P-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 http://opensource.org/licenses/liliq-p-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_90.yml

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-P-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 https://opensource.org/licenses/liliq-p-1.1

src/licensedcode/data/inactive/rules/free-unknown_url_glc_91.RULE

@@ -1 +1,12 @@
+---
+license_expression: free-unknown
+relevance: 100
+notes: |
+    License test derived from a file of the BSD-licensed repository at:
+    https://raw.githubusercontent.com/google/licensecheck/v0.3.1/testdata/ccurls.t1
+    originally expected to be detected as LiLiQ-R-1.1
+    with coverage of 0
+    This is a URL test extracted from ccurls.t1.
+---
+
 http://opensource.org/licenses/liliq-r-1.1