x86/mm/init: Handle the special case of device private pages in add_pages(), to not increase max_pfn and trigger dma_addressing_limited() bounce buffers

JIRA: https://issues.redhat.com/browse/RHEL-110657

This patch is a backport of the following upstream commit:
commit 7170130
Author: Balbir Singh <balbirs@nvidia.com>
Date:   Tue Apr 1 11:07:52 2025 +1100

    x86/mm/init: Handle the special case of device private pages in add_pages(), to not increase max_pfn and trigger dma_addressing_limited() bounce buffers

    As Bert Karwatzki reported, the following recent commit causes a
    performance regression on AMD iGPU and dGPU systems:

      7ffb791 ("x86/kaslr: Reduce KASLR entropy on most x86 systems")

    It exposed a bug with nokaslr and zone device interaction.

    The root cause of the bug is that, the GPU driver registers a zone
    device private memory region. When KASLR is disabled or the above commit
    is applied, the direct_map_physmem_end is set to much higher than 10 TiB
    typically to the 64TiB address. When zone device private memory is added
    to the system via add_pages(), it bumps up the max_pfn to the same
    value. This causes dma_addressing_limited() to return true, since the
    device cannot address memory all the way up to max_pfn.

    This caused a regression for games played on the iGPU, as it resulted in
    the DMA32 zone being used for GPU allocations.

    Fix this by not bumping up max_pfn on x86 systems, when pgmap is passed
    into add_pages(). The presence of pgmap is used to determine if device
    private memory is being added via add_pages().

    More details:

    devm_request_mem_region() and request_free_mem_region() request for
    device private memory. iomem_resource is passed as the base resource
    with start and end parameters. iomem_resource's end depends on several
    factors, including the platform and virtualization. On x86 for example
    on bare metal, this value is set to boot_cpu_data.x86_phys_bits.
    boot_cpu_data.x86_phys_bits can change depending on support for MKTME.
    By default it is set to the same as log2(direct_map_physmem_end) which
    is 46 to 52 bits depending on the number of levels in the page table.
    The allocation routines used iomem_resource's end and
    direct_map_physmem_end to figure out where to allocate the region.

    [ arch/powerpc is also impacted by this problem, but this patch does not fix
      the issue for PowerPC. ]

    Testing:

     1. Tested on a virtual machine with test_hmm for zone device inseration

     2. A previous version of this patch was tested by Bert, please see:
        https://lore.kernel.org/lkml/d87680bab997fdc9fb4e638983132af235d9a03a.camel@web.de/

    [ mingo: Clarified the comments and the changelog. ]

    Reported-by: Bert Karwatzki <spasswolf@web.de>
    Tested-by: Bert Karwatzki <spasswolf@web.de>
    Fixes: 7ffb791 ("x86/kaslr: Reduce KASLR entropy on most x86 systems")
    Signed-off-by: Balbir Singh <balbirs@nvidia.com>
    Signed-off-by: Ingo Molnar <mingo@kernel.org>
    Cc: Brian Gerst <brgerst@gmail.com>
    Cc: Juergen Gross <jgross@suse.com>
    Cc: H. Peter Anvin <hpa@zytor.com>
    Cc: Linus Torvalds <torvalds@linux-foundation.org>
    Cc: Andrew Morton <akpm@linux-foundation.org>
    Cc: Christoph Hellwig <hch@lst.de>
    Cc: Pierre-Eric Pelloux-Prayer <pierre-eric.pelloux-prayer@amd.com>
    Cc: Alex Deucher <alexander.deucher@amd.com>
    Cc: Christian König <christian.koenig@amd.com>
    Cc: David Airlie <airlied@gmail.com>
    Cc: Simona Vetter <simona@ffwll.ch>
    Link: https://lore.kernel.org/r/20250401000752.249348-1-balbirs@nvidia.com

Signed-off-by: Ricardo Robaina <rrobaina@redhat.com>

Author: rprobaina

arch/x86/mm/init_64.c

@@ -967,9 +967,18 @@ int add_pages(int nid, unsigned long start_pfn, unsigned long nr_pages,
 	ret = __add_pages(nid, start_pfn, nr_pages, params);
 	WARN_ON_ONCE(ret);
 
-	/* update max_pfn, max_low_pfn and high_memory */
-	update_end_of_memory_vars(start_pfn << PAGE_SHIFT,
-				  nr_pages << PAGE_SHIFT);
+	/*
+	 * Special case: add_pages() is called by memremap_pages() for adding device
+	 * private pages. Do not bump up max_pfn in the device private path,
+	 * because max_pfn changes affect dma_addressing_limited().
+	 *
+	 * dma_addressing_limited() returning true when max_pfn is the device's
+	 * addressable memory can force device drivers to use bounce buffers
+	 * and impact their performance negatively:
+	 */
+	if (!params->pgmap)
+		/* update max_pfn, max_low_pfn and high_memory */
+		update_end_of_memory_vars(start_pfn << PAGE_SHIFT, nr_pages << PAGE_SHIFT);
 
 	return ret;
 }