What's New in Proxmox 8.0

Let's take a look at the new features and highlights for V8 Beta1

I bounce back and forth between Harvester and Proxmox. Right now, I have one Proxmox server, which is running the Rancher instance that spins up my Kubernetes cluster on Harvester (stay tuned for that post).

But I recently noticed that the first Beta for Proxmox 8.0 was released on June 9. Let’s take a look at some of the highlighted features and improvements.

---

Here are a few of the key features that will be introduced in the new version of Proxmox:

1. Debian Bookworm: Proxmox VE 8.0 is based on the world-renowned Debian Bookworm, a testament to the high level of reliability and quality to expect from this release.

2. Seamless Upgrade from Proxmox VE 7.4: Upgrading from Proxmox VE 7.4 to the new 8.0 Beta1 is a seamless process. Refer to the detailed guide on upgrading from Proxmox VE 7 to 8 for more information.

3. Ceph Quincy Enterprise Repository: Through any Proxmox VE subscription, users can now access the most stable Ceph repository, taking full advantage of cutting-edge storage technology.

4. Access Realm Sync Jobs: With the new feature to synchronize users and groups from an LDAP/AD server automatically, IT administrators can manage user access more efficiently and accurately.

5. Improved Host Network Bridge and VNet Access: Proxmox VE 8.0 Beta1 integrates host network bridge and VNet access into its ACL system. The new SDN.Use privilege and the /sdn/zones/<zone>/<bridge-or-vnet>/<vlan-tag> ACL object path allow users to grant precise usage permissions for specific networks, streamlining the management of network resources.

---

Here is the full changelog for v8.0 Beta1:

Enhancements in the web interface (GUI)

• The Ceph repository selection now takes into account the subscription status of the complete cluster and recommends the optimal version for the cluster.

• Improved Dark color theme:

The Dark color theme, introduced in Proxmox VE 7.4, received a lot of feedback from our community, which resulted in further improvements.

• Set strict SameSite attribute on the Authorization cookie

• The Markdown parser, used in notes, has been improved:

  • it allows setting the target for links, to make any link open in a new tab or window.

  • it allows providing URLs with a scheme different from HTTP/HTTPS;

    You can now directly link to resources like rdp://<rest-of-url>, providing convenience links in the guest notes.

  • tag-names and protocols are matched case-insensitive.

• The mobile UI code was refactored to not suffer from incompatible changes made for the web-based GUI.

• The generated CSR used by the built-in ACME client now sets the correct CSR version (0 instead of 2).

• Uploading files now only computes the MD5 sum of the uploaded file, if it can be used for comparison with the user-provided one.

• Firewall settings: Improve the alignment of permissions checked by the web UI with the permissions actually required by the API.

• Explicitly disallow internal-only tmpfilename parameter for file uploads.

• Fix multipart HTTP uploads without Content-Type header.

• Show Ceph pool number in the web UI, as it is often mentioned in Ceph warnings and errors.

• Improved translations, among others:

  • Ukrainian (NEW)

  • Japanese

  • Simplified Chinese

  • Traditional Chinese

  • The size units (Bytes, KB, MiB,...) are now passed through the translation framework as well, allowing localized variants (e.g., for French).

  • The language selection is now localized and displayed in the currently selected language

Virtual Machines (KVM/QEMU)

• New QEMU Version 8.0:

  • The virtiofsd codebase was replaced by a new and improved implementation based on Rust, which is packaged separately.

  • QEMU Guest Agent now has initial support for NetBSD and OpenBSD.

  • Many more changes, see the upstream changelog for details.

• Avoid invalid smm machine flag for aarch64 VM when using serial display and SeaBIOS.

• Warn if a network interface is not connected to a bridge on VM startup. This can happen if the user manually edited the VM config.

• Fix an issue with the improved code for volume size information gathering for passed through disks during backup preparation.

• Workaround breaking driver changes in newer Nvidia grid drivers, which prevented mediated devices (mdev) to be reclaimed upon guest exit.

• Prefer an explicit configured SMBIOS UUID for Nvidia vGPU passthrough.

  If a uuid command line parameter is present, it will be preferred over the default auto-generated UUID, containing the VMID and mdev index.

This fixes an issue, with software inside the guest, which relies on a specific and unique UUID setting.

• Improved gathering of current setting for live memory unplugging.

• Avoid sending a temporary size of zero to QEMU when resizing block devices. Previously, this was done when resizing RBD volumes, but it is not necessary anymore.

• When resizing a disk, spawn a worker task to avoid HTTP request timeout (issue 2315).

• Allow resizing qcow2 disk images with snapshots (issue 517).

• cloud-init improvements:

  • Introduce ciupgrade option that controls whether machines should upgrade packages on boot (issue 3428).

  • Better align privilege checks in the web UI with the actual privileges required in the backend.

  • Fix an issue where the hostname was not properly set on Fedora/CentOS machines, by passing the hostname via the fqdn option.

  • Fix an issue where displaying pending changes via qm and pvesh caused an error.

  • Allow setting network options with VM.Config.Cloudinit privileges, instead of requiring the more powerful VM.Config.Network privilege.

• Drop unused QMP commands for getting the link and creating/deleting internal snapshots.

• Replace usages of deprecated -no-hpet QEMU option with the hpet=off machine flag.

Containers (LXC)

• Improve handling of /etc/machine-id on clone operations - the file is now only truncated, if the source did not explicitly set it to 'uninitialized' or remove it. Thus, the admin can decide if they want first-boot semantics or not (see machine-id (5)).

• Set memory.high cgroup limit to 99.6% of configured memory. This setting gives the container a chance to free memory before directly running into an Out-of-Memory (OOM) condition. It is applied on lxc.conf generation and on hot-plugging memory to a running container.

• Warn users on conflicting, manual, lxc.idmap entries.

  Custom mappings can become quite complicated and cause overlaps fast.

  By issuing a warning upon container start, the user should find the wrong entry directly.

• When resizing a disk, perform plausibility checks already before spawning the worker task. This allows invalid requests to fail earlier.

• General code improvements, adhering to best practices for Perl code.

General improvements for virtual guests

• When cloning guests, the validation of the provided name of the clone is now happening in the frontend, improving UX.

• Add config files in /etc/pve/mapping and privileges Mapping.* in preparation for cluster-wide mapping of PCI/USB devices.

HA Manager

• Stability improvements of manual maintenance mode:

  • Fix an issue where a request for enabling maintenance mode on a node is lost, in case the rebooted node is the current active Cluster Resource Manager (CRM).

  • Fix an issue where a shutdown policy other than migrate could cause a node in maintenance mode to leave maintenance mode too early or fence itself.

  • Fix an issue where ha-rebalance-on-start could cause a newly added and already-running service to be shut down and migrated to another node.

    Now, ha-rebalance-on-start ignores services that are already running.

• When enabling or disabling maintenance mode via the CLI, the ha-manager command now checks whether the provided node exists.

  This avoids misconfigurations, e.g., due to a typo in the node name.

Improved management for Proxmox VE clusters

• The rsync invocation used when joining nodes via ssh, which is deprecated, has been adapted to changes in rsync CLI argument parsing in Bookworm.

Backup/Restore

• Improve performance of backups that use zstd on fast disks, by invoking zstd without the --rsyncable flag (issue 4605).

• Suppress harmless but confusing "storing login ticket failed" errors when backing up to Proxmox Backup Server.

• When restoring from backups via the web UI, the VM/CT name is now validated client-side before sending an API request. This helps catching invalid names early.

• The web UI now sorts backups by date, whereas it previously sorted backups first by VMID and then by date. The VMID is added as an extra column for users who would like to restore the previous sorting order (issue 4678).

• Fix an issue where the backup job editor window occasionally did not show the selected guests (issue 4627).

• The fs-freeze-on-backup option of the QEMU guest agent, which controls whether the filesystem should be frozen for backups, can now be set in the web UI.

• Improve permission model for backup jobs: Editing backup jobs now generally requires the Datastore.Allocate privilege on the target storage, and editing backup jobs with dumpdir requires root privileges.

• Clarify description of the ionice setting.

Storage

• The file-based storage-types have two new config options create-base-path and create-subdirs. They replace the mkdir option and separate two different concepts:

create-base-path decides if the path to the storage should be created if it does not exist,

create-subdirs decides if the content-specific sub-directories (guest images, ISO, container template, backups) should be created.

Conflating both settings in the single mkdir option caused a few unwanted effects in certain situations (issue 3214).

• The CIFS storage type can now be configured with custom mount options, as it was already possible for the NFS storage type.

• The subdir option of the CIFS storage type can now be configured in the web interface. The option can be used to mount a subdirectory of a SMB/CIFS share and was previously only accessible via the API/CLI.

• Improve API documentation for the upload method.

• The API now allows to also query replication jobs that are disabled.

• Allow @ in directory storage path, as it is often used to signify Btrfs subvolumes.

• When resizing RBD volumes, always round up sizes to the nearest integer. This avoids errors due to passing a floating-point size to the RBD tooling.

Ceph

• Add support for new Ceph enterprise repositories. When installing Ceph via pveceph install or the web UI, you can now choose between the test, no-subscription and enterprise (default) repositories. The -test-repository option of the pveceph install command was removed.

• Add pveceph osddetails command to show information about OSDs on the command line, with a level of detail that is comparable to the web UI/API.

• Drop support for hyper-converged Ceph Octopus and Pacific, as they are not supported in Proxmox VE 8.

  Proxmox VE 8 will support managing Quincy and newer Ceph server releases, setups still using Pacific can upgrade to Ceph Quincy before upgrading Proxmox VE from 7 to 8.

  The Ceph 17.2 Quincy client will still support accessing older Ceph server setups as a client.

• Remove overly restrictive validation of public_network during monitor creation. Configuring a public network like 0::/0 or 0::/1 caused a superfluous "value does not look like a valid CIDR network" error.

• The Ceph installation wizard in the web UI does not create monitors and managers called localhost anymore and uses the actual node name instead.

Access Control

• Add possibility to define realm sync jobs in the web UI. These allow to synchronize users and groups from an LDAP server automatically at regular intervals.

• Add TFA/TOTP lockout to protect against an attacker who has obtained the user password and attempts to guess the second factor:

  • If TFA failed too many times in a row, lock this user account out of TFA for an hour. If TOTP failed too many times in a row, disable TOTP for the user account. Using a recovery key will unlock a user account.

  • Add pveum tfa unlock command and /access/users/{userid}/unlock-tfa API endpoint for manually unlocking users.

  • Add TFA lockout status to responses of /access/tfa and /access/users endpoints.

• Fix validity check for LDAP base DNs that was overly strict starting from Proxmox VE 7.4. For example, the check rejected base DNs containing both dashes and spaces (issue 4609).

• When authenticating via PAM, pass the PAM_RHOST item. With this, it is possible to manually configure PAM such that certain users (for example root@pam) can only log in from certain hosts.

• Add pveum tfa list command for listing second factors on the command line.

• The access/ticket API endpoint does not support the deprecated login API (using new-format=0) anymore.

• Remove the Permission.Modify privilege from the PVESysAdmin and PVEAdmin roles and restrict it to the Administrator role. This reduces the chances of accidentally granting privilege modification privileges.

• Login with TFA: In order to improve UX, fix wording of messages related to recovery keys.

• Forbid creating roles with names starting with PVE to reserve these role names for use in future upgrades.

• SDN.Use is required on a bridge/vnet (or its zone) in order to configure it in a guest vNIC.

  • use /sdn/zones/localnetwork or /sdn/zones/localnetwork/<bridge> to allow usage of all or specific local bridges.

  • use /sdn/zones/<zone> or /sdn/zones/<zone>/<bridge> to allow usage of all or specific vnets in a given SDN zone.

• Users with VM.Allocate/Datastore.Allocate/Pool.Allocate privileges, but without the Permissions.Modify privilege, can now only assign a subset of their own privileges to specific VM/storage/pool paths, instead of arbitrary roles.

Firewall & Software Defined Networking

• Allow to distinguish IP sets and aliases with the same name defined on the datacenter level and on the guest level by providing an explicit prefix (issue 4556). Previously, the innermost IP set/alias took precedence, which is still the default behavior if no prefix is provided.

• Fix an issue where an allowed special ICMP-type could accidentally be added as destination port for a layer 4 protocol, breaking firewall rule loading.

• Fix setting the correct vlan-protocol for QinQ zones if the bridge is vlan-aware.(issue 4683

• Fix an issue where routing between zones was enabled by default in exit nodes. This has been fixed by adding null-routes for each other zone prefix to each zone (issue 4389).

• Correctly order vrf and router bgp vrf entries by vrf name in the frr configuration. (issue 4662)

• For setups where a node is primary exit node for one vrf and secondary exit for a different vrf, the configuration now also adds the second vrf's default route. (issue 4657)

• Allow specifying a custom vxlan-tunnel port per interface.

• Update the frr configuration generation to the version of frr shipped in Debian Bookworm.

• Fix an issue where reloading the network configuration on a remote node created an error, which hid the actual issue with the network configuration.

• Add support for IPv6 SLAAC and router advertisement configuration in /etc/network/interfaces to ifupdown2.

• Fix live reloading when changing VLAN and VXLAN specific attributes.

• Add support for creating an OVS bridge which tags traffic with a specific VLAN tag to ifupdown2.

This is to match the possibility in ifupdown.

Improvements for the management of Proxmox VE Nodes

• pve7to8 compatibility check script added.

As with previous major upgrades, Proxmox VE 7 ships a script checking for issues with the current node/cluster. It should point to any issues which might prevent a successful major upgrade.

• Outdated pve6to7 compatibility check script was removed.

• Fix an issue where the web UI would display no APT repositories during a major upgrade.

• The new version of grub2 provided by Debian Bookworm (2.06-13) fixes an issue where a host using LVM would fail to boot with a message disk `lvmid/...` not found, even though the LVM setup is healthy.

Installation ISO

• The version of BusyBox shipped with the ISO was updated to version 1.36.1.

• The Proxmox-provided Ceph Quincy repo will be set-up by default, providing updates for a modern Ceph client even if Proxmox Ceph hyper-converged setup is not in use.

• Detection of unreasonable system time.

If the system time is older than the time the installer was created, the system notifies the user with a warning.

• ethtool is now shipped with the ISO and installed on all systems.

• systemd-boot is provided by its own package instead of systemd in Debian Bookworm and is installed with the new ISO.

Notable bug fixes and general improvments

• Most git repositories now have a dsc Makefile target to create a Debian Source Package and additionally a sbuild target to create the source package and build it using sbuild.

---

There are some bugs as well:

• Storage activation now checks that every content type uses a different directory, in order to prevent unexpected interactions between different content types. This breaks setups in which the content-dirs option was set up to map different content types to the same directory, and setups in which some content directories were manually set up as symlinks to a common directory.

• QEMU 8.0 removed some previously deprecated features. Proxmox VE 8 won't use the -chardev tty and -chardev parport aliases anymore, and no other features were used by the Proxmox VE stack. Thus, only installations using args inside their guest configs need to check the compatibility. See the Qemu changelog on the topic for details.

  • The removed features in QEMU 8.0 also include the Proxmox VE-specific, but unused/deprecated QMP commands get_link_status, snapshot-drive and delete-drive-snapshot.

• The lxc.id_map configuration key has been deprecated for a long time by lxc and was replaced by lxc.idmap. With this release, its presence is considered an error. The key can only be present if it was manually added to a guest configuration.

• The lxcfs is now built with fuse 3. This upgrade is done on a major release, since all running containers need to be restarted afterwards.

• Permission System

  • There is a new SDN.Use privilege (and corresponding PVESDNUser role) that is required to configure virtual NICs in guests. See SDN section above for details!

  • The Permission.Modify privilege has been removed from the PVESysAdmin and PVEAdmin roles, in order to reduce the chances of accidentally granting the privilege to modify privileges. If a particular setup requires a role with this privilege, it is necessary to define a new custom role and use that instead of PVESysAdmin/PVEAdmin.

  • Users with VM.Allocate/Datastore.Allocate/Pool.Allocate privileges, but without the Permissions.Modify privilege, can now only assign a subset of their own privileges to specific VM/storage/pool paths. Previously they could assign any role to specific VM/storage/pool paths. As the privileges usable on specific VM/storage/pool paths were quite limited, this did not allow privilege escalation, but restricting the capabilities now allows adding more powerful privileges in future versions without breaking changes.

• Editing backup jobs now generally requires the Datastore.Allocate privilege on the target storage, and editing backup jobs with dumpdir requires root privileges.

• User accounts will now be locked after too many attempts to authenticate with a second factor. This is intended to protect against an attacker who has obtained the user password and attempts to guess the second factor. Unlocking requires either a successful login with a recovery key or a manual unlock by an administrator.

• The API can handle array-type data differently, while staying backward compatible.

  Instead of being able to pass the individual elements separated by null bytes, you can pass the data directly as array.

---

Overall, I am really looking forward to this release. I may just have to upgrade my server tonight! Watch me break everything and have to clean install everything 😔.

Cheers,

Joe