Merge pull request #10270 from jf205/download-db-vs-code

Update documentation about downloading CodeQL databases using the VS Code extension

Author: jf205

docs/codeql/codeql-cli/creating-codeql-databases.rst

@@ -5,11 +5,17 @@ Creating CodeQL databases
 
 Before you analyze your code using CodeQL, you need to create a CodeQL
 database containing all the data required to run queries on your code.
+You can create CodeQL databases yourself using the CodeQL CLI,
+or download them from GitHub.com.
 
 CodeQL analysis relies on extracting relational data from your code, and
 using it to build a :ref:`CodeQL database <codeql-database>`. CodeQL
 databases contain all of the important information about a codebase, which can
-be analyzed by executing CodeQL queries against it.
+be analyzed by executing CodeQL queries against it. GitHub creates and 
+stores CodeQL databases for a large number of open-source projects. For more information,
+see ":ref:`Downloading CodeQL databases from GitHub.com <downloading-codeql-databases-from-github-com>`."
+
+You can also create CodeQL databases yourself using the CodeQL CLI. 
 Before you generate a CodeQL database, you need to:
 
 - Install and set up the CodeQL CLI. For more information, see
@@ -379,24 +385,15 @@ The following example shows how you could use indirect build tracing in an Azure
        # `codeql database analyze`
        # then `codeql github upload-results` ...
 
-Obtaining databases from LGTM.com
----------------------------------
-
-`LGTM.com <https://lgtm.com>`__ analyzes thousands of open-source projects using
-CodeQL. For each project on LGTM.com, you can download an archived CodeQL
-database corresponding to the most recently analyzed revision of the code. These
-databases can also be analyzed using the CodeQL CLI or used with the CodeQL
-extension for Visual Studio Code.
-
-.. include:: ../reusables/download-lgtm-database.rst
-
-.. pull-quote::
-
-   Note
-
-   .. include:: ../reusables/index-files-note.rst
+.. _downloading-codeql-databases-from-github-com:
 
+Downloading databases from GitHub.com
+-------------------------------------
 
+.. include:: ../reusables/download-github-database.rst
+   
+Before running an analysis with the CodeQL CLI, you must unzip the databases.
+   
 Further reading
 ---------------
 

docs/codeql/codeql-cli/getting-started-with-the-codeql-cli.rst

@@ -47,22 +47,11 @@ Conditions <https://securitylab.github.com/tools/codeql/license>`__.
 
 .. pull-quote:: Important
 
-   There are different versions of the CLI available to download, depending
+   There are several versions of the CLI available to download, depending
    on your use case:
 
-   - If you want to use the most up to date CodeQL tools and features, download the
-     version tagged ``latest``.
-
-   - If you want to create CodeQL databases to upload to LGTM Enterprise, download
-     the version that is compatible with the relevant LGTM Enterprise version
-     number. Compatibility information is included in the description for each
-     release on the `CodeQL CLI releases page
-     <https://github.com/github/codeql-cli-binaries/releases>`__ on GitHub. Using the
-     correct version of the CLI ensures that your CodeQL databases are
-     compatible with your version of LGTM Enterprise. For more information,
-     see `Preparing CodeQL databases to upload to LGTM
-     <https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html>`__
-     in the LGTM admin help.
+   * If you want to use the most up to date CodeQL tools and features, download the version tagged ``latest``.
+   * If you want to generate code scanning data to upload to GitHub Enterprise server, then download the version that is compatible with the CodeQL CLI used in your CI system. For more information, see "`Installing CodeQL CLI in your CI system <https://docs.github.com/en/enterprise-server/code-security/code-scanning/using-codeql-code-scanning-with-your-existing-ci-system/installing-codeql-cli-in-your-ci-system#downloading-the-codeql-cli>`__" in the GitHub documentation.
 
 If you use Linux, Windows, or macOS version 10.14 ("Mojave") or earlier, simply
 `download the zip archive
@@ -232,15 +221,7 @@ see ":doc:`About CodeQL packs <about-codeql-packs>`."
    - For the most up to date CodeQL queries, check out the ``main`` branch.
      This branch represents the very latest version of CodeQL's analysis.
 
-   - For the queries used in a particular LGTM Enterprise release, check out the
-     branch tagged with the relevant release number. For example, the branch
-     tagged ``v1.27.0`` corresponds to LGTM Enterprise 1.27. You must use this
-     version if you want to upload data to LGTM Enterprise. For further
-     information, see `Preparing CodeQL databases to upload to LGTM
-     <https://help.semmle.com/lgtm-enterprise/admin/help/prepare-database-upload.html>`__
-     in the LGTM admin help.
-
-1. Extract the zip archive
+4. Extract the zip archive
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 For Linux, Windows, and macOS users (version 10.14 "Mojave", and earlier)

docs/codeql/codeql-for-visual-studio-code/analyzing-your-projects.rst

@@ -14,7 +14,7 @@ To analyze a project, you need to add a :ref:`CodeQL database <codeql-database>`
 
 #. Open the CodeQL Databases view in the sidebar.
 
-#. Hover over the **Databases** title bar and click the appropriate icon to add your database. You can add a database from a local ZIP archive or folder, from a public URL, or from a project slug or URL on LGTM.com.
+#. Hover over the **Databases** title bar and click the appropriate icon to add your database. You can add a database from a local ZIP archive or folder, from a public URL, or from a project's URL on GitHub.com.
 
    .. image:: ../images/codeql-for-visual-studio-code/choose-database.png
       :width: 350
@@ -31,14 +31,14 @@ If you have a CodeQL database saved locally, as an unarchived folder or as a ZIP
 
 - To create a database with the CodeQL CLI, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`."
 
-- .. include:: ../reusables/download-lgtm-database.rst
-
 - To analyze a test database, add a ``.testproj`` folder to the Databases view.
   Test databases (that is, folders with a ``.testproj`` extension) are generated when you run regression tests on custom queries using the :ref:`CodeQL CLI <codeql-cli>`.
   If a query fails a regression test, you may want to analyze the test database in Visual Studio Code to debug the failure.
 
   For more information about running query tests, see ":ref:`Testing custom queries <testing-custom-queries>`" in the CodeQL CLI help.
 
+.. include:: ../reusables/download-github-database.rst
+
 Running a query
 ------------------------
 

docs/codeql/codeql-for-visual-studio-code/customizing-settings.rst

@@ -28,7 +28,7 @@ Editing settings
 Choosing a version of the CodeQL CLI
 --------------------------------------
 
-The CodeQL extension uses the CodeQL CLI to run commands. If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from LGTM.com. Otherwise, the extension automatically manages access to the executable of the CLI for you. For more information about creating databases, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`" in the CLI help.
+The CodeQL extension uses the CodeQL CLI to run commands. If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from GitHub.com. Otherwise, the extension automatically manages access to the executable of the CLI for you. For more information about creating databases, see ":ref:`Creating CodeQL databases <creating-codeql-databases>`" in the CLI help.
 
 To override the default behavior and use a different CLI, you can specify the CodeQL CLI **Executable Path**.
 

docs/codeql/codeql-for-visual-studio-code/setting-up-codeql-in-visual-studio-code.rst

@@ -28,7 +28,7 @@ Configuring access to the CodeQL CLI
 
 The extension uses the CodeQL CLI to compile and run queries.
 
-If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from LGTM.com. For more information, see ":ref:`CodeQL CLI <codeql-cli>`."
+If you already have the CLI installed and added to your ``PATH``, the extension uses that version. This might be the case if you create your own CodeQL databases instead of downloading them from GitHub.com. For more information, see ":ref:`CodeQL CLI <codeql-cli>`."
 
 Otherwise, the extension automatically manages access to the executable of the CLI for you. This ensures that the CLI is compatible with the CodeQL extension. You can also check for updates with the **CodeQL: Check for CLI Updates** command.
 
@@ -57,20 +57,6 @@ There are two ways to do this:
 
    For CLI users there is a third option: If you have followed the instructions in ":ref:`Getting started with the CodeQL CLI <getting-started-with-the-codeql-cli>`" to create a CodeQL directory (for example ``codeql-home``) containing the CodeQL libraries, you can open this directory in VS Code. This also gives the extension access to the CodeQL libraries.
 
-.. container:: toggle
-
-   .. container:: name
-
-      **Click to show information for LGTM Enterprise users**
-
-   Your local version of the CodeQL queries and libraries should match your version of LGTM Enterprise. For example, if you
-   use LGTM Enterprise 1.27, then you should clone the ``1.27.0`` branch of the `starter workspace <https://github.com/github/vscode-codeql-starter/>`__ (or the appropriate ``1.27.x`` branch, corresponding to each maintenance release).
-
-   This ensures that the queries and libraries you write in VS Code also work in the query console on LGTM Enterprise.
-
-   If you prefer to add the CodeQL queries and libraries to an :ref:`existing workspace <existing-workspace>` instead of the starter workspace, then you should
-   clone the appropriate branch of the `CodeQL repository <https://github.com/github/codeql>`__ and add it to your workspace.
-
 .. _starter-workspace:
 
 Using the starter workspace

docs/codeql/images/codeql-for-visual-studio-code/choose-database.png

@@ -0,0 +1,14 @@
+GitHub stores CodeQL databases for over 200,000 repos on GitHub.com, which you can download using the REST API. The list of repos is constantly growing and evolving to make sure that it includes the most interesting codebases for security research.
+
+You can check if a repository has any CodeQL databases available for download using the ``/repos/<owner>/<repo>/code-scanning/codeql/databases`` endpoint.
+For example, to check for CodeQL databases using the `GitHub CLI <https://cli.github.com/manual/gh_api>`__ you would run::
+
+   gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/
+
+This command returns information about any CodeQL databases that are available for a repository, including the language the database represents, and when the database was last updated. If no CodeQL databases are available, the response is empty.
+
+When you have confirmed that a CodeQL database exists for the language you are interested in, you can download it using the following command:: 
+
+   gh api /repos/<owner>/<repo>/code-scanning/codeql/databases/<language> -H 'Accept: application/zip' > path/to/local/database.zip
+
+For more information, see the documentation for the `Get CodeQL database <https://docs.github.com/en/rest/reference/code-scanning#get-codeql-database>`__ endpoint in the GitHub REST API documentation.