Repository navigation
Conversation
nvme-cli 3.2 dropped -H as the short name of "nvme nbft show --hfi", because it collides with a global option. "-H" now selects that global option, which is deprecated, prints a warning and will be removed. The NBFT is still shown only because "nbft show" without a section option shows every section. Once the global option is gone, "nvme nbft show -H" fails, and parse-nvmf-boot-connections.sh skips the NBFT: the boot interfaces get no network configuration. Use the long option names, which are stable across releases. nvme-cli has had --hfi, --subsystem and --output-format since 2.5, the release that introduced "nbft show". Signed-off-by: Daniel Wagner <dwagner@suse.de>
nvme-cli 2.7 renamed 71-nvmf-iopolicy-netapp.rules to 71-nvmf-netapp.rules. inst_rules skips a rule that does not exist, so with any newer nvme-cli the rule, which sets the I/O policy for NetApp ONTAP controllers, is not in the initrd. Install the rule under both names. Signed-off-by: Daniel Wagner <dwagner@suse.de>
nvme-cli 3 configures the connections in /etc/nvme/nvme-fabrics.conf and the drop-ins in /etc/nvme/nvme-fabrics.conf.d/. It reads the legacy discovery.conf and config.json only as long as there is no nvme-fabrics.conf, and converts them into it. The nvmf module neither installs these files nor looks for them: - A host configured with nvme-cli 3 gets no configured connections in the initrd, and check() fails with "No discovery arguments present" unless there is an NBFT or FC. - Once nvme-fabrics.conf is in the initrd, the rd.nvmf.discover connections, which are appended to discovery.conf, are ignored. Install nvme-fabrics.conf and its drop-ins, and take them into account in check(), in nvmf-autoconnect.sh and for rd.nvmf.nostatic and rd.nvmf.discover=fc,auto. With nvme-cli 3, write each rd.nvmf.discover connection as a [Discovery Controller] section of a drop-in. nvme-cli 3 describes its --config option as an "INI configuration file", which tells it apart from nvme-cli 2. nvme-cli 2 still gets discovery.conf. Signed-off-by: Daniel Wagner <dwagner@suse.de>
parse-nvmf-boot-connections.sh writes rd.nvmf.hostnqn and rd.nvmf.hostid to /etc/nvme/hostnqn and /etc/nvme/hostid, and appends rd.nvmf.discover to /etc/nvme/discovery.conf. Only a hostonly initrd has /etc/nvme, from the host's files. In any other initrd, the writes fail with "No such file or directory": the connections are made with a generated host NQN, which a target with an access list refuses, and with nvme-cli 2 the rd.nvmf.discover connections are lost. Create the directory first. Signed-off-by: Daniel Wagner <dwagner@suse.de>
rd.nvmf.nonbft disables connecting to the targets of the NBFT. The NBFT is not parsed then, but once the initqueue times out, nvmf-autoconnect.sh runs a plain "nvme connect-all", and nvme-cli 2.5 and later connect the NBFT's subsystems as well unless --no-nbft is given. With a network up anyway (DHCP), the system then boots from the NBFT's root despite rd.nvmf.nonbft. Pass --no-nbft in the timeout case too when rd.nvmf.nonbft is set and nvme-cli supports it. Signed-off-by: Daniel Wagner <dwagner@suse.de>
ip= takes the MAC address of an interface in the BOOTIF form (52-54-00-12-34-56, ip= splits at ':'), and ifup resolves it. But parse-ip-opts keeps that form in /tmp/net.ifaces, while ifup marks the interface set up and up under its name and its address as sysfs shows it, /tmp/net.52:54:00:12:34:56.did-setup and .up. So the wait hook of the interface, all_ifaces_up() and all_ifaces_setup() never see it, and the bootdev check of the automatic DHCP never matches. Keep the address in the sysfs form, as for BOOTIF=, and don't warn at the command line hook that it doesn't exist as an interface. With systemd and a network root, the wait hook also ends when the root device appears, which hid it; iscsiroot's all_ifaces_setup() checks and an initramfs without systemd see it. Signed-off-by: Daniel Wagner <dwagner@suse.de>
systemd-network-generator reads ip=, vlan=, ifname= etc. from the kernel command line only, and it runs before the cmdline hooks. What the hooks write to /etc/cmdline.d never reaches networkd: the ip= and vlan= that nvmf writes for an NBFT HFI, or those of the iBFT. With no .network generated, networkd-config.sh adds the default one, DHCP on every interface. An NBFT boot with a static HFI only worked where a DHCP server happened to hand out the same address (QEMU's user network does: 10.0.2.15 via 10.0.2.2). Run the generator again with dracut's command line (SYSTEMD_PROC_CMDLINE, its override of /proc/cmdline), which has the kernel's and /etc/cmdline.d's. Found with an NBFT boot of a guest on a tap device without DHCP: the HFI's address was never set. Signed-off-by: Daniel Wagner <dwagner@suse.de>
nvme-cli hands the initrd's connections over to nvme-discoverd in the real root when they are connected with the owner pre-os: it records them for nvme-discoverd, which keeps them connected and never disconnects them, and connects them with ctrl_loss_tmo=-1. Pass --owner=pre-os to every connect-all if the --owner help of nvme mentions it, and route the FC discovery events through nvmf-autoconnect.sh so they get the owner too. Signed-off-by: Daniel Wagner <dwagner@suse.de>
github-actions
Bot
added
docs
man
modules
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Nothing in the real root disconnects the pre-OS controllers, as the root may be on them. At power-off the connections just end, and the target waits for the keep-alive timeout. If the initrd connected controllers with the owner pre-os (or nbft, as nvme-cli up to 3.2 records it), ask for the shutdown initramfs and disconnect them there after the root is unmounted. Signed-off-by: Daniel Wagner <dwagner@suse.de>
dracut renames the interface of each NBFT HFI to nbftN so the ip= it writes can name it. An nvme that follows the pre-OS model finds the interface by its MAC address and doesn't need the name. With systemd-networkd, which matches alternative names too, add nbftN as an alternative name instead of renaming the interface. The other network modules keep the rename. Signed-off-by: Daniel Wagner <dwagner@suse.de>
…gacy network-legacy matches interface names only, but its ip= takes a MAC address in the BOOTIF form. With an nvme that follows the pre-OS model, name the HFI's interface by its MAC address instead of renaming it. An HFI with a VLAN is still renamed, as vlan= needs a name for the parent. Signed-off-by: Daniel Wagner <dwagner@suse.de>
nm-initrd-generator can read the NBFT itself (rd.nvmf.nm) and matches the HFI's interface by its MAC address, VLANs included. It loads libnvme with dlopen(), so dracut doesn't install it, and nvme-cli 3 only brings libnvme3.so.1, which NetworkManager up to 1.58 doesn't load. With network-manager, install the libnvme its generator loads if the system has it. Then, with an nvme that follows the pre-OS model, default to rd.nvmf.nm and write no ip= or ifname= for the NBFT. Otherwise the HFI is renamed as before. Signed-off-by: Daniel Wagner <dwagner@suse.de>
2046188 to
29b5566
Compare
|
CC @mwilck |
| # then leaves the NBFT's network to NetworkManager. | ||
| __nvmf_nm_reads_nbft() { | ||
| local _gen _lib _dir | ||
| for _gen in /usr/libexec/nm-initrd-generator /usr/lib/nm-initrd-generator; do |
There was a problem hiding this comment.
What about strings "$_gen" | grep libnvme.*\.so?
@tbzatek: why does NM use dlopen at all? Can't it just link to libnvme?
There was a problem hiding this comment.
NM devs explicitly asked for that, apparently there are use cases where very kilobyte matters. nm-initrd-generator covers more functionality than NBFT and in many cases nvme is not needed/desired.
There was a problem hiding this comment.
It's tough, I'll need to talk to our dracut maintainers to see how much stuff we can pull unconditionally in. Last time it was the jq dependency that was subject to objections. And with the last nvme-cli we had to disable libarchive (for now) as it pulled whole new dependency tree in the initramfs.
|
Cc @tbzatek, @MichaelRabek, @johnmeneghini
To the best of my knowledge, this already works since 00df743 (only with NetworkManager). systemd-networkd wasn't in the list of network management tools that support NBFT boot until now. I am pleased if it works, but I'd like to know what has been tested so far.
I am not sure if we need this. Like in #2759, it would be reasonable to say that sticking with Also, I believe that this feature is orthogonal to In general I would say that this PR is a bit overloaded 😄
I'd call this fix, not a feature. |
What was tested with systemd-networkd, in the nvme-lab boot tests: QEMU guests boot from an NVMe/TCP root through this module, with an NBFT generated for the guest ( NBFT boot with networkd, interface renamed to
pre-OS NBFT boot with networkd,
Not covered: real firmware, DH-CHAP/TLS in the NBFT. |
PR #2820 fixes the legacy setup which is needed with nvme-cli < 3.3, for backwards compatibility.
Yes, though the
I know github includes the patches from #2820 in the view. Only the last 5 patches are actually part of this PR.
Right, let's sort this out first. |
The dracut side of the NVMe-oF pre-OS model, linux-nvme/nvme-cli#4409: the initrd connects as the owner
pre-os, nvme records these connections, and nvme-discoverd keeps them in the real root without owning or disconnecting them.This is stacked on #2820: the first 7 commits are #2820, the last 5 are new:
feat(nvmf): connect as the pre-OS environment:--owner=pre-oson everyconnect-allifnvme connect-all --helpmentions it (checked withgrep, which busybox supports, unlike thesedform in chore(nvmf): drop supporting nvmi-cli < 2.5 #2759); FC discovery events go throughnvmf-autoconnect.shtoo.feat(nvmf): disconnect the pre-OS controllers in the shutdown initramfs: nothing in the real root disconnects them, so the shutdown initramfs does, after the root is unmounted.feat(nvmf): keep the name of an NBFT interface with the pre-OS model: with systemd-networkd,nbftNbecomes an alternative name instead of a rename.feat(nvmf): name an NBFT interface by its MAC address with network-legacyfeat(nvmf): let NetworkManager read the NBFT with the pre-OS model: install the libnvme nm-initrd-generator loads, and default tord.nvmf.nm.With an nvme that doesn't follow the model (nvme-cli up to 3.2, and 2.x), nothing changes.
Testing
nvme-lab, QEMU guests booting from NVMe/TCP and NVMe/FC through this module:
Draft until linux-nvme/nvme-cli#4409 and #2820 are merged.