Dub Integration doesn't work

[Clouud9]

Xmake Version

v3.0.3+master.8bcc3569c

Operating System Version and Architecture

Windows 11 24H2

Describe Bug

XMake doesn't seem to respect the dub.json configuration of a package that's been installed (dmd). Additionally it's unable to fetch packages properly too.

Expected Behavior

XMake should be looking at files inside DMD's compiler directory, instead of inside src. For fetching packages, xmake should just properly execute the dub fetch command.

Project Configuration

The DMD issue prevents the fetching issue from cropping up, so "dmd" would have to be removed from add_packages to

add_rules("mode.debug", "mode.release")
add_requires("dub::dmd", {alias = "dmd"})
add_requires("dub::emsi_containers", {alias = "emsi_containers"})

target("debuggd-plugin")
    set_default(true)
    set_kind("shared")
    add_files("src/*.d|src/main.d")

-- Add these as subdirs? 
target("debuggd-d")
    set_default(false)
    set_kind("static")
    
target("debuggd-cpp")
    set_default(false)
    set_kind("static")

target("debuggd")
    set_kind("binary")
    add_files("src/*.d")
    add_deps("debuggd-d", "debuggd-cpp")
    add_packages("dmd", "emsi_containers")

Additional Information and Error Logs

For the DMD Issue:

PS C:\Users\user\Base\Projects\D\debuggd> xmake build
checking for Microsoft Visual Studio (x64) version ... 2022
[ 67%]: compiling.release src\main.d
[ 67%]: compiling.release src\main.d
error: src\main.d(2): Error: unable to read module `frontend`
import dmd.frontend;
       ^
src\main.d(2):        Expected 'dmd\frontend.d' or 'dmd\frontend\package.d' in one of the following import paths:
import path[0] = C:\Users\user\AppData\Local\dub\packages\dmd\2.111.0\dmd\src
import path[1] = C:\Users\user\AppData\Local\dub\packages\emsi_containers\0.9.0\emsi_containers\src
import path[2] = C:\Users\user\Base\Programs\D\dmd2\windows\bin64\..\..\src\phobos
import path[3] = C:\Users\user\Base\Programs\D\dmd2\windows\bin64\..\..\src\druntime\import
  > in src\main.d

For fetching emsi_containers:

PS C:\Users\user\Base\Projects\D\debuggd> xmake
[ 33%]: compiling.release src\main.d
[ 54%]: compiling.release src\main.d
error: src\main.d(2): Error: unable to read module `slist`
import containers.slist;
       ^
src\main.d(2):        Expected 'containers\slist.d' or 'containers\slist\package.d' in one of the following import paths:
import path[0] = C:\Users\user\Base\Programs\D\dmd2\windows\bin64\..\..\src\phobos
import path[1] = C:\Users\user\Base\Programs\D\dmd2\windows\bin64\..\..\src\druntime\import
  > in src\main.d

I have heard that even when xmake recognizes that it needs to install a dub package, it just errors out without an explanation. This has been brought up in other issues but they've all been closed without being resolved. For this specific example I believe that what's happening instead is that emsi_containers is just not recognized as missing. When I fetched it myself with dub it was still unable to properly import the files, as well.

[waruqi]

XMake doesn't seem to respect the dub.json configuration of a package that's been installed (dmd). Additionally it's unable to fetch packages properly too.

It will parse dub.json.

xmake/xmake/modules/package/manager/dub/find_package.lua

Line 80 in 6b27644

for _, importPath in ipairs(dubjson.importPaths) do

	"description": "The DMD compiler",
	"importPaths": [
		"src/"
	],

But it does not support subPackages in dub.json.

{
	"authors": [
		"Walter Bright"
	],
	"copyright": "Copyright © 1999-2018, The D Language Foundation",
	"dependencies": {
		"dmd:frontend": "*"
	},
	"description": "The DMD compiler",
	"importPaths": [
		"src/"
	],
	"license": "BSL-1.0",
	"name": "dmd",
	"sourcePaths": [
		"src/"
	],
	"subPackages": [
		{
			"dflags": [
				"-preview=dip1000"
			],
			"dflags-windows": [
				"-L/STACK:16777216"
			],
			"importPaths": [
				"compiler/src"
			],
			"name": "compiler",
			"preGenerateCommands-posix": [
				"echo -n /etc > SYSCONFDIR.imp"
			],
			"sourcePaths": [
				"compiler/src/dmd"
			],
			"stringImportPaths": [
				"compiler/src/dmd/res",
				"."
			],
			"targetName": "dmd",
			"targetType": "executable"
		},
		{
			"importPaths": [
				"compiler/src"
			],
			"name": "root",
			"sourcePaths": [
				"compiler/src/dmd/common",
				"compiler/src/dmd/root"
			],
			"targetType": "library"
		},
		{
			"dependencies": {
				"dmd:root": "*"
			},
			"importPaths": [
				"compiler/src"
			],
			"name": "lexer",
			"preGenerateCommands": [
				"\"$DUB_EXE\" \"--compiler=$DC\" --single \"${DUB_PACKAGE_DIR}config.d\" -- \"${DUB_PACKAGE_DIR}generated/dub\" \"${DUB_PACKAGE_DIR}VERSION\" /etc"
			],
			"sourceFiles": [
				"compiler/src/dmd/console.d",
				"compiler/src/dmd/entity.d",
				"compiler/src/dmd/errors.d",
				"compiler/src/dmd/file_manager.d",
				"compiler/src/dmd/globals.d",
				"compiler/src/dmd/id.d",
				"compiler/src/dmd/identifier.d",
				"compiler/src/dmd/lexer.d",
				"compiler/src/dmd/location.d",
				"compiler/src/dmd/sarif.d",
				"compiler/src/dmd/tokens.d",
				"compiler/src/dmd/utils.d",
				"compiler/src/dmd/errorsink.d"
			],
			"sourcePaths": [],
			"stringImportPaths": [
				"compiler/src/dmd/res",
				"generated/dub"
			],
			"targetType": "library",
			"versions": [
				"CallbackAPI",
				"DMDLIB"
			]
		},
		{
			"dependencies": {
				"dmd:lexer": "*"
			},
			"importPaths": [
				"compiler/src"
			],
			"name": "parser",
			"sourceFiles": [
				"compiler/src/dmd/astbase.d",
				"compiler/src/dmd/parse.d",
				"compiler/src/dmd/visitor/transitive.d",
				"compiler/src/dmd/visitor/permissive.d",
				"compiler/src/dmd/visitor/strict.d"
			],
			"sourcePaths": [],
			"targetType": "library",
			"versions": [
				"CallbackAPI"
			]
		},
		{
			"dependencies": {
				"dmd:parser": "*"
			},
			"excludedSourceFiles": [
				"compiler/src/dmd/backend/*",
				"compiler/src/dmd/root/*",
				"compiler/src/dmd/common/*",
				"compiler/src/dmd/{astbase,console,entity,errors,file_manager,globals,id,identifier,lexer,parse,tokens,utf,utils,visitor/permissive,visitor/strict,visitor/transitive,}.d",
				"compiler/src/dmd/{dmsc,e2ir,eh,glue,iasmdmd,iasmgcc,irstate,lib/package,lib/elf,lib/mach,lib/mscoff,lib/scanelf,lib/scanmach,lib/scanmscoff,link,objc_glue,s2ir,tocsym,toctype,tocvdebug,toobj,todt,toir}.d"
			],
			"importPaths": [
				"compiler/src"
			],
			"name": "frontend",
			"sourcePaths": [
				"compiler/src/dmd"
			],
			"stringImportPaths": [
				"compiler/src/dmd/res"
			],
			"targetType": "library",
			"versions": [
				"NoBackend",
				"GC",
				"NoMain",
				"MARS",
				"CallbackAPI"
			]
		}
	],
	"targetType": "none",
	"toolchainRequirements": {
		"dub": ">=1.29.0"
	},
	"version": "2.111.0"
}

[Clouud9]

I'm going to have to add the add this myself then, aren't I?

Do you know where a sub-configuration option could go, and what file handles this?

[waruqi]

I haven't had time lately to analyze how to support it. It may be necessary to resolve submodule paths in extracted subPackages.

you can try it in dub/find_package.lua

https://github.com/xmake-io/xmake/blob/dev/xmake/modules/package/manager/dub/find_package.lua

[Clouud9]

I haven't had time lately to analyze how to support it. It may be necessary to resolve submodule paths in extracted subPackages.

you can try it in dub/find_package.lua

https://github.com/xmake-io/xmake/blob/dev/xmake/modules/package/manager/dub/find_package.lua

IMO the smarter way to do it is to include it in the package configuration itself, or in add_packages(). I could try and add a new function for dub specifically but I'm not sure that's optimal and I have no experience with Lua.

[waruqi]

You should only modify find_package.lua, don't modify anything else.

add_requires/add_packages will use it.

[Clouud9]

You should only modify find_package.lua, don't modify anything else.

add_requires/add_packages will use it.

Ah wait, you're right. I was thinking of sub-configurations instead of sub-packages, my bad.

[Clouud9]

Do you have any information about how to debug the lua scripts as they're running? I have an idea of what to do but I don't know what's going on when the script itself is running.

[waruqi]

see https://xmake.io/guide/best-practices/faq.html#how-to-debug-xmake-source-code

and you only need to debug dub/find_package.lua