Improve handling of importing with DirectoryLayerDirectory

Importing a bunch of different values for a DirectoryLayerDirectory seems to run into a bunch of conflicts and other issues allocating entries.
We need to improve this but I'm pulling this work out of #3578