[pci] Disable decoding while setting a BAR value

Setting the base address for a 64-bit BAR requires two separate 32-bit
writes to configuration space, and so will necessarily result in the
BAR temporarily holding an invalid partially written address.

Some hypervisors (observed on an AWS EC2 c7a.medium instance in
eu-west-2) will assume that guests will write BAR values only while
decoding is disabled, and may not rebuild MMIO mappings for the guest
if the BAR registers are written while decoding is enabled.  The
effect of this is that MMIO accesses are not routed through to the
device even though inspection from within the guest shows that every
single PCI configuration register has the correct value.  Writes to
the device will be ignored, and reads will return the all-ones pattern
that typically indicates a nonexistent device.

With the ENA network driver now using low latency transmit queues,
this results in the transmit descriptors being lost (since the MMIO
writes to BAR2 never reach the device), which in turn causes the
device to lock up as soon as the transmit doorbell is rung for the
first time.

Fix by disabling decoding of memory and I/O cycles while setting a BAR
address (as we already do while sizing a BAR), so that the invalid
partial address can never be decoded and so that hypervisors will
rebuild MMIO mappings as expected.

Signed-off-by: Michael Brown <mcb30@ipxe.org>

diff --git a/src/drivers/bus/pci.c b/src/drivers/bus/pci.c
index f3ea6d47808dc09e63fdc9a5ea5881ba8bb9cffe..43ccb751dd282a0ad7b6ea722f4606d61970223e 100644 (file)
--- a/src/drivers/bus/pci.c
+++ b/src/drivers/bus/pci.c
@@ -116,6 +116,13 @@ void pci_bar_set ( struct pci_device *pci, unsigned int reg,
        unsigned int type;
        uint32_t low;
        uint32_t high;
+       uint16_t cmd;
+
+       /* Save the original command register and disable decoding */
+       pci_read_config_word ( pci, PCI_COMMAND, &cmd );
+       pci_write_config_word ( pci, PCI_COMMAND,
+                               ( cmd & ~( PCI_COMMAND_MEM |
+                                          PCI_COMMAND_IO ) ) );
 
        /* Check for a 64-bit BAR */
        pci_read_config_dword ( pci, reg, &low );
@@ -135,6 +142,9 @@ void pci_bar_set ( struct pci_device *pci, unsigned int reg,
                }
                pci_write_config_dword ( pci, reg + 4, high );
        }
+
+       /* Restore the original command register */
+       pci_write_config_word ( pci, PCI_COMMAND, cmd );
 }
 
 /**