Skip to content

Commit a297b67

Browse files
mbaludambaluda
authored
Merge branch 'main' into GeekMasher-patch-1
2 parents 24e4b89 + ad612bb commit a297b67

File tree

325 files changed

+53559
-14783
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

325 files changed

+53559
-14783
lines changed

‎.github/instructions/extractors_cds_tools_ts.instructions.md‎

Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
---
2+
applyTo: 'extractors/cds/tools/**/*.ts'
3+
description: 'Instructions for CodeQL CDS extractor TypeScript source and test files.'
4+
---
5+
6+
# Copilot Instructions for `extractors/cds/tools/**/*.ts` files
7+
8+
## PURPOSE
9+
10+
This file contains instructions for working with TypeScript source code files in the `extractors/cds/tools/` directory of the `codeql-sap-js` repository. This includes the main `cds-extractor.ts` entry-point, modular source files in `src/**/*.ts`, and comprehensive test files in `test/**/*.test.ts`.
11+
12+
## REQUIREMENTS
13+
14+
## COMMON REQUIREMENTS
15+
16+
- ALWAYS use modern TypeScript syntax and features compatible with the configured target (ES2020).
17+
- ALWAYS follow best practices for implementing secure and efficient CodeQL extractor functionality.
18+
- ALWAYS order imports, definitions, static lists, and similar constructs alphabetically.
19+
- ALWAYS follow a test-driven development (TDD) approach by writing comprehensive tests for new features or bug fixes.
20+
- ALWAYS fix lint errors by running `npm run lint:fix` from the `extractors/cds/tools/` directory before committing changes.
21+
- ALWAYS maintain consistency between the CDS extractor's compilation behavior and the `extractors/cds/tools/test/cds-compilation-for-actions.test.sh` script to prevent CI/CD workflow failures.
22+
- **ALWAYS run `npm run build:all` from the `extractors/cds/tools/` directory and ensure it passes completely before committing any changes. This is MANDATORY and includes lint checks, test coverage, and bundle validation.**
23+
24+
### CDS EXTRACTOR SOURCE REQUIREMENTS
25+
26+
The following requirements are specific to the CDS extractor main entry-point `cds-extractor.ts` and source files matching `extractors/cds/tools/src/**/*.ts`.
27+
28+
- ALWAYS keep the main entry-point `cds-extractor.ts` focused on orchestration, delegating specific tasks to well-defined modules in `src/`.
29+
- ALWAYS gracefully handle extraction failures using tool-level diagnostics in order to avoid disrupting the overall CodeQL extraction process. Instead of exiting with a non-zero code, the CDS extractor should generate a diagnostic error (or warning) that points to the relative path (from source root) of the problematic source (e.g. `.cds`) file.
30+
31+
### CDS EXTRACTOR TESTING REQUIREMENTS
32+
33+
The following requirements are specific to the CDS extractor test files matching `extractors/cds/tools/test/**/*.test.ts`.
34+
35+
- ALWAYS write unit tests for new functions and classes in corresponding `test/src/**/*.test.ts` files.
36+
- ALWAYS use Jest testing framework with the configured `ts-jest` preset.
37+
- ALWAYS follow the AAA pattern (Arrange, Act, Assert) for test structure.
38+
- ALWAYS mock external dependencies (filesystem, child processes, network calls) using Jest mocks or `mock-fs`.
39+
- ALWAYS test both success and error scenarios with appropriate edge cases.
40+
- ALWAYS maintain test coverage above the established threshold.
41+
- **ALWAYS run `npm test` or `npm run test:coverage` from the `extractors/cds/tools/` directory and ensure all tests pass before committing changes.**
42+
43+
## PREFERENCES
44+
45+
- PREFER modular design with each major functionality implemented in its own dedicated file or module under `src/`.
46+
- PREFER the existing architectural patterns:
47+
- `src/cds/compiler/` for CDS compiler specific logic
48+
- `src/cds/parser/` for CDS parser specific logic
49+
- `src/logging/` for unified logging and performance tracking
50+
- `src/packageManager/` for dependency management and caching
51+
- `src/codeql.ts` for CodeQL JavaScript extractor integration
52+
- `src/environment.ts` for environment setup and validation
53+
- PREFER comprehensive error handling with diagnostic reporting through the `src/diagnostics.ts` module.
54+
- PREFER performance-conscious implementations that minimize filesystem operations and dependency installations.
55+
- PREFER project-aware processing that understands CDS file relationships and dependencies.
56+
57+
## CONSTRAINTS
58+
59+
- NEVER leave any trailing whitespace on any line.
60+
- NEVER directly modify any compiled files in the `dist/` directory; all changes must be made in the corresponding `src/` files and built using the build process.
61+
- NEVER commit changes without verifying that `npm run build:all` passes completely when run from the `extractors/cds/tools/` directory.
62+
- NEVER modify compilation behavior without updating the corresponding test script `extractors/cds/tools/test/cds-compilation-for-actions.test.sh`.
63+
- NEVER process CDS files in isolation - maintain project-aware context for accurate extraction.
64+
- NEVER bypass the unified logging system - use `src/logging/` utilities for all output and diagnostics.
65+
- NEVER commit extra documentation files that purely explain what has been changed and/or fixed; use git commit messages instead of adding any `.md` files that you have not explicitly been requested to create.

‎.github/pull_request_template.md‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
## What This PR Contributes
2+
3+
<!-- Explain in Markdown bullet points what is covered in this PR:
4+
1. Organize the bullet points in a reasonable level of hierarchy, and
5+
2. Be as EXHAUSTIVE as possible. -->
6+
7+
## Future Works
8+
9+
<!-- Explain in Markdown bullet points what is OUT OF SCOPE of this PR.
10+
Also organize them with bullet points in a reasonable of hierarchy. -->

‎.github/workflows/cds-extractor-dist-bundle.yml‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
name: CDS Extractor Distribution Bundle
2+
3+
on:
4+
push:
5+
branches: [ main ]
6+
paths:
7+
- 'extractors/cds/**'
8+
pull_request:
9+
branches: [ main ]
10+
paths:
11+
- 'extractors/cds/**'
12+
workflow_dispatch:
13+
# This job can be manually triggered to validate the CDS extractor bundle
14+
15+
jobs:
16+
bundle-validation:
17+
name: CDS extractor bundle validation
18+
runs-on: ubuntu-latest
19+
20+
steps:
21+
- name: Checkout repository
22+
uses: actions/checkout@v4
23+
24+
- name: Setup Node.js
25+
uses: actions/setup-node@v4
26+
with:
27+
node-version: '20'
28+
cache: 'npm'
29+
cache-dependency-path: 'extractors/cds/tools/package-lock.json'
30+
31+
- name: Install node dependencies
32+
working-directory: extractors/cds/tools
33+
run: npm ci
34+
35+
- name: Run TS code linter
36+
working-directory: extractors/cds/tools
37+
run: npm run lint
38+
39+
- name: Run TS code unit tests with coverage report
40+
working-directory: extractors/cds/tools
41+
run: npm run test:coverage
42+
43+
- name: Build and validate the CDS extractor bundle
44+
working-directory: extractors/cds/tools
45+
run: npm run build:validate
46+
47+
- name: Validate CDS extractor JS bundle and map files
48+
working-directory: extractors/cds/tools
49+
run: |
50+
_bundle_file="dist/cds-extractor.bundle.js"
51+
_bundle_map_file="${_bundle_file}.map"
52+
if [ -f "$_bundle_file" ]; then
53+
echo "✅ Bundle file exists."
54+
else
55+
echo "❌ Bundle file not found."
56+
exit 2
57+
fi
58+
59+
if [ -f "$_bundle_map_file" ]; then
60+
echo "✅ CDS extractor JS bundle source map file exists."
61+
else
62+
echo "❌ CDS extractor JS bundle source map file not found."
63+
exit 3
64+
fi
65+
66+
# Check if the built bundle and map files differ
67+
# from the versions committed to git.
68+
if git diff --exit-code "$_bundle_file" "$_bundle_map_file"; then
69+
echo "✅ CDS JS bundle and map files match committed versions."
70+
else
71+
echo "❌ CDS JS bundle and/or map file(s) differ from committed version(s)."
72+
echo "The built bundle and/or source map do not match the committed versions."
73+
echo "Please rebuild the bundle and commit the changes:"
74+
echo " cd extractors/cds/tools"
75+
echo " npm install"
76+
echo " npm run build:all"
77+
echo " git add dist/cds-extractor.bundle.*"
78+
echo " git commit -m 'Update CDS extractor dist bundle'"
79+
exit 4
80+
fi
81+
82+
# Check if bundle file starts with the expected shebang for `node`.
83+
if head -n 1 "${_bundle_file}" | grep -q "#!/usr/bin/env node"; then
84+
echo "✅ Bundle has Node.js shebang"
85+
else
86+
echo "❌ Bundle missing Node.js shebang"
87+
exit 5
88+
fi

‎.github/workflows/code_scanning.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,9 @@ on:
1010
- cron: '39 12 * * 2'
1111
workflow_dispatch:
1212

13+
env:
14+
CODEQL_ACTION_DIFF_INFORMED_QUERIES: false
15+
1316
jobs:
1417
analyze-javascript:
1518
name: Analyze

‎.github/workflows/javascript.sarif.expected‎

Lines changed: 1 addition & 1 deletion
Large diffs are not rendered by default.

‎.github/workflows/run-codeql-unit-tests-javascript.yml‎

Lines changed: 20 additions & 16 deletions
Original file line numberDiff line numberDiff line change
@@ -77,24 +77,28 @@ jobs:
7777
run: |
7878
qlt query run install-packs
7979
80-
- name: Ensure presence of cds shell command
81-
run: |
82-
if ! command -v cds &> /dev/null
83-
then
84-
npm install -g @sap/cds-dk
85-
fi
80+
- name: Setup Node.js for CDS compilation
81+
uses: actions/setup-node@v4
82+
with:
83+
node-version: '18'
84+
cache: 'npm'
85+
cache-dependency-path: 'extractors/cds/tools/package-lock.json'
8686

87-
# Compile .cds files to .cds.json files.
87+
- name: Verify Node.js and npm tools
88+
run: |
89+
echo "Node.js version: $(node --version)"
90+
echo "npm version: $(npm --version)"
91+
echo "npx version: $(npx --version)"
92+
# Verify npx can access @sap/cds-dk without installing globally
93+
echo "Testing npx access to @sap/cds-dk..."
94+
npx --yes --package @sap/cds-dk@latest cds --version || echo "CDS will be installed per-project as needed"
95+
96+
# Compile .cds files to .cds.json files using the dedicated test script
8897
- name: Compile CAP CDS files
8998
run: |
90-
for cds_file in $(find . -type f \( -iname '*.cds' \) -print)
91-
do
92-
echo "I am compiling $cds_file"
93-
cds compile $cds_file \
94-
-2 json \
95-
-o "$cds_file.json" \
96-
--locations
97-
done
99+
# Use the dedicated CDS compilation script that includes proper version resolution
100+
# This script follows the same logic as the CDS extractor's resolveCdsVersions function
101+
./extractors/cds/tools/workflow/cds-compilation-for-actions.sh
98102
99103
- name: Run test suites
100104
id: run-test-suites
@@ -105,7 +109,7 @@ jobs:
105109
CODEQL_STDLIB_IDENT: ${{matrix.codeql_standard_library_ident}}
106110
RUNNER_TMP: ${{ runner.temp }}
107111
LGTM_INDEX_XML_MODE: all
108-
LGTM_INDEX_FILETYPES: ".json:JSON"
112+
LGTM_INDEX_FILETYPES: ".json:JSON\n.cds:JSON"
109113

110114
shell: bash
111115
run: >

‎.github/workflows/update-codeql.yml‎

Lines changed: 111 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
name: "Update the CodeQL CLI dependencies"
2+
3+
on:
4+
workflow_dispatch:
5+
# nightly runs to update the CodeQL CLI dependencies
6+
schedule:
7+
- cron: '30 0 * * *'
8+
9+
permissions:
10+
contents: write
11+
pull-requests: write
12+
13+
jobs:
14+
update-codeql:
15+
name: Update CodeQL CLI dependencies
16+
runs-on: ubuntu-latest
17+
18+
steps:
19+
- name: Checkout repository
20+
uses: actions/checkout@v4
21+
22+
- name: Check latest CodeQL CLI version and update qlt.conf.json
23+
id: check-version
24+
env:
25+
GH_TOKEN: ${{ github.token }}
26+
run: |
27+
echo "Checking latest CodeQL CLI version"
28+
current_version=$(jq .CodeQLCLI qlt.conf.json -r)
29+
latest_version=$(gh release list --repo github/codeql-cli-binaries --json 'tagName,isLatest' --jq '.[] | select(.isLatest == true) | .tagName')
30+
echo "Current CodeQL CLI version: $current_version"
31+
echo "Latest CodeQL CLI version: $latest_version"
32+
33+
# Remove 'v' prefix if present for comparison with current version
34+
latest_clean=$(echo "$latest_version" | sed 's/^v//')
35+
36+
if [ "$latest_clean" != "$current_version" ]; then
37+
echo "Updating CodeQL CLI from $current_version to $latest_clean"
38+
echo "update_needed=true" >> $GITHUB_OUTPUT
39+
echo "latest_version=$latest_clean" >> $GITHUB_OUTPUT
40+
echo "latest_version_tag=$latest_version" >> $GITHUB_OUTPUT
41+
42+
# Update qlt.conf.json with all properties
43+
echo "Updating qlt.conf.json with all properties for version $latest_clean"
44+
jq --arg cli_version "$latest_clean" \
45+
--arg std_lib "codeql-cli/$latest_version" \
46+
--arg bundle "codeql-bundle-$latest_version" \
47+
'.CodeQLCLI = $cli_version | .CodeQLStandardLibrary = $std_lib | .CodeQLCLIBundle = $bundle' \
48+
qlt.conf.json > qlt.conf.json.tmp && mv qlt.conf.json.tmp qlt.conf.json
49+
50+
echo "Updated qlt.conf.json contents:"
51+
cat qlt.conf.json
52+
else
53+
echo "CodeQL CLI is already up-to-date at version $current_version."
54+
echo "update_needed=false" >> $GITHUB_OUTPUT
55+
fi
56+
57+
- name: Install QLT
58+
if: steps.check-version.outputs.update_needed == 'true'
59+
id: install-qlt
60+
uses: advanced-security/codeql-development-toolkit/.github/actions/install-qlt@main
61+
with:
62+
qlt-version: 'latest'
63+
add-to-path: true
64+
65+
- name: Install CodeQL
66+
if: steps.check-version.outputs.update_needed == 'true'
67+
id: install-codeql
68+
shell: bash
69+
run: |
70+
echo "Installing CodeQL"
71+
qlt codeql run install
72+
echo "-----------------------------"
73+
echo "CodeQL Home: $QLT_CODEQL_HOME"
74+
echo "CodeQL Binary: $QLT_CODEQL_PATH"
75+
76+
- name: Upgrade CodeQL pack lock files
77+
if: steps.check-version.outputs.update_needed == 'true'
78+
shell: bash
79+
run: |
80+
echo "Upgrading CodeQL pack lock files"
81+
echo "Finding all directories with qlpack.yml files..."
82+
83+
# Find all directories containing qlpack.yml files
84+
find . -name "qlpack.yml" -type f | while read -r qlpack_file; do
85+
pack_dir=$(dirname "$qlpack_file")
86+
echo "Upgrading pack in directory: $pack_dir"
87+
88+
# Change to the directory and run codeql pack upgrade
89+
cd "$pack_dir"
90+
$QLT_CODEQL_PATH pack upgrade
91+
cd - > /dev/null
92+
done
93+
94+
echo "Finished upgrading all CodeQL pack lock files"
95+
96+
- name: Create Pull Request
97+
if: steps.check-version.outputs.update_needed == 'true'
98+
uses: peter-evans/create-pull-request@271a8d0340265f705b14b6d32b9829c1cb33d45e # v7.0.8
99+
with:
100+
title: "Upgrade CodeQL CLI dependency to ${{ steps.check-version.outputs.latest_version_tag }}"
101+
body: |
102+
This PR upgrades the CodeQL CLI version to ${{ steps.check-version.outputs.latest_version_tag }}.
103+
104+
**Changes made:**
105+
- Updated `CodeQLCLI` to `${{ steps.check-version.outputs.latest_version }}`
106+
- Updated `CodeQLStandardLibrary` to `codeql-cli/${{ steps.check-version.outputs.latest_version_tag }}`
107+
- Updated `CodeQLCLIBundle` to `codeql-bundle-${{ steps.check-version.outputs.latest_version_tag }}`
108+
- Upgraded all CodeQL pack lock files using `codeql pack upgrade`
109+
commit-message: "Upgrade CodeQL CLI dependency to ${{ steps.check-version.outputs.latest_version_tag }}"
110+
delete-branch: true
111+
branch: "codeql/upgrade-to-${{ steps.check-version.outputs.latest_version_tag }}"

‎.gitignore‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,7 @@ typings/
6262

6363
# Misc
6464
.DS_Store
65+
.*.swp
6566

6667
dist/
6768
tmp/
@@ -70,3 +71,5 @@ tmp/
7071
**.testproj
7172
dbs
7273
*.cds.json
74+
.cds-extractor-cache
75+

0 commit comments

Comments
 (0)