How to properly disable a PCIe device?

Question

I am writing a device driver in linux for PCIe endpoint implemented on Xilinx UltrascaleMPSoC FPGA part. I have implemented the remove function correctly. I connect my device using an adapter to my PC, turn on device, enable its Endpoint and then turn on PC and everything works correctly. However, when I try to unload the driver module using rmmod command, the process hangs.

I went through Linux documentation and for pci_disable_device() documentation, it says

Note we don't actually disable the device until all callers of pci_enable_device() have called pci_disable_device().

Does it mean, my driver has to wait untill every other pcie driver in linux calls pci_disable_device() and only then device gets disabled? I really doubt upon this :(

I tried using modprobe -r and also listed the module usage count using "lsmod". lsmod shows usage count as "0". But still I am not able to unload the module :( I also added print statements.

void remove(struct pci_dev pdev)
{
    pci_unmap_single(pdev, privdata->dma_mem,
    PAGE_SIZE * (1 << memorder),
    PCI_DMA_FROMDEVICE);
    printk(KERN_INFO"unmap_single() complete\n");
    free_pages ((unsigned long) privdata->mem, memorder);
    printk(KERN_INFO"free_pages() complete\n");
    free_irq(pdev->irq, privdata);
    printk(KERN_INFO"free_irq() complete\n");
    pci_disable_msi(pdev);
    printk(KERN_INFO"MSI disable complete\n");
    pci_clear_master(pdev); /* Nobody seems to do this */
    printk(KERN_INFO"clear_master() complete\n");
    pci_iounmap(pdev, privdata->registers);
    printk(KERN_INFO"iounmap() complete\n");
    pci_disable_device(pdev);
    printk(KERN_INFO"disable_device() complete\n");
    pci_release_regions(pdev);
    printk(KERN_INFO"release_regions() complete\n");
}

Expected: The device must get disabled. I can't conclude. When I do a dmesg in another terminal, I get prints till "disable_device() complete" and the terminal hangs.Also,I checked the files: /proc/iomem and /proc/interrupts-> when I unload the module, corresponding entries of my device is removed in both of my files! I only complete the execution of pci_disable_device() and the process hangs.

Note: Only the rmmod process hangs and I cannot use the current terminal, but I can open another terminal, do all things.

Finally, when the do a restart, the system hangs again and everytime, I am doing force restart by holding power button for 10-15 seconds. Check this reboot_hang

Why is the pci_release_regions() hanging my system even though lsmod shows my module usage count as "0".

Thanks in advance :) Also, should I interchange pci_disable_device() and pci_release_regions() methods in above code? -> I tried this as well, but then I get prints till release_regions() complete but disable_device complete is not printed. And process hangs :(

Please help

Have you debugged where/why it is hanging? Have you checked how other drivers do it? I would suggest that you ask this kind of question in the LKML, which is the appropriate forum for this. — Acorn, Oct 08 '19 at 08:20

luizinho · Answer 1 · 2019-10-08T09:06:34.237

Difficult to answer without more information on what you really use on your driver.

But, I will try to give you an answer.

Does it mean, my driver has to wait untill every other pcie driver in linux calls pci_disable_device() and only then device gets disabled?

No, this would make impractical and not very modular.

It seems you're freeing the DMA pages. How are you allocating them ? Are you using coherent allocation? Take a look here to see how to use properly the DMA.

For the remainder, can you try to move release regions before clear_master? Example :

pci_disable_msi(pdev);
ioummap(ADDRESS);
pci_release_regions(pdev);
pci_clear_master(pdev);
pci_disable_device(pdev);
pci_set_drvdata(pdev, NULL);

For more information on how to write PCI Drivers, check the documentation here. It provides a lot of information.

thank you for quick answer. I am using streaming DMA Mappings. I do a pci_map_single() in my probe() method. I tried your above method of putting pci_clear_master() after release region. Now, I get the prints untill clear_master() complete. Now, only the pci_disable_device() will not get printed. Also, I found this link LINUX KERNEL driver hangs/freeze after handling mapped register -> it says hardware module owning the register is either powered down, or not clocked. Is that true? any ideas? — HarishAG, Oct 09 '19 at 09:23

How to properly disable a PCIe device?

1 Answers1