CPU topology on s390x

Since QEMU 8.2, CPU topology on s390x provides up to 3 levels of topology containers: drawers, books and sockets. They define a tree-shaped hierarchy.

The socket container has one or more CPU entries. Each of these CPU entries consists of a bitmap and three CPU attributes:

  • CPU type

  • entitlement

  • dedication

Each bit set in the bitmap correspond to a core-id of a vCPU with matching attributes.

This documentation provides general information on S390 CPU topology, how to enable it and explains the new CPU attributes. For information on how to modify the S390 CPU topology and how to monitor polarization changes, see docs/devel/s390-cpu-topology.rst.

Prerequisites

To use the CPU topology, you need to run with KVM on a s390x host that uses the Linux kernel v6.0 or newer (which provide the so-called KVM_CAP_S390_CPU_TOPOLOGY capability that allows QEMU to signal the CPU topology facility via the so-called STFLE bit 11 to the VM).

Enabling CPU topology

Currently, CPU topology is only enabled in the host model by default.

Enabling CPU topology in a CPU model is done by setting the CPU flag ctop to on as in:

-cpu gen16b,ctop=on

Having the topology disabled by default allows migration between old and new QEMU without adding new flags.

Default topology usage

The CPU topology can be specified on the QEMU command line with the -smp or the -device QEMU command arguments.

Note also that since 7.2 threads are no longer supported in the topology and the -smp command line argument accepts only threads=1.

If none of the containers attributes (drawers, books, sockets) are specified for the -smp flag, the number of these containers is 1.

Thus the following two options will result in the same topology:

-smp cpus=5,drawer=1,books=1,sockets=8,cores=4,maxcpus=32

and

-smp cpus=5,sockets=8,cores=4,maxcpus=32

When a CPU is defined by the -smp command argument, its position inside the topology is calculated by adding the CPUs to the topology based on the core-id starting with core-0 at position 0 of socket-0, book-0, drawer-0 and filling all CPUs of socket-0 before filling socket-1 of book-0 and so on up to the last socket of the last book of the last drawer.

When a CPU is defined by the -device command argument, the tree topology attributes must all be defined or all not defined.

-device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1

or

-device gen16b-s390x-cpu,core-id=1,dedicated=true

If none of the tree attributes (drawer, book, sockets), are specified for the -device argument, like for all CPUs defined with the -smp command argument the topology tree attributes will be set by simply adding the CPUs to the topology based on the core-id.

QEMU will not try to resolve collisions and will report an error if the CPU topology defined explicitly or implicitly on a -device argument collides with the definition of a CPU implicitly defined on the -smp argument.

When the topology modifier attributes are not defined for the -device command argument they takes following default values:

  • dedicated: false

  • entitlement: medium

Hot plug

New CPUs can be plugged using the device_add hmp command as in:

(qemu) device_add gen16b-s390x-cpu,core-id=9

The placement of the CPU is derived from the core-id as described above.

The topology can of course also be fully defined:

(qemu) device_add gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1

Examples

In the following machine we define 8 sockets with 4 cores each.

$ qemu-system-s390x -m 2G \
  -cpu gen16b,ctop=on \
  -smp cpus=5,sockets=8,cores=4,maxcpus=32 \
  -device host-s390x-cpu,core-id=14 \

A new CPUs can be plugged using the device_add hmp command as before:

(qemu) device_add gen16b-s390x-cpu,core-id=9

The core-id defines the placement of the core in the topology by starting with core 0 in socket 0 up to maxcpus.

In the example above:

  • There are 5 CPUs provided to the guest with the -smp command line They will take the core-ids 0,1,2,3,4 As we have 4 cores in a socket, we have 4 CPUs provided to the guest in socket 0, with core-ids 0,1,2,3. The last CPU, with core-id 4, will be on socket 1.

  • the core with ID 14 provided by the -device command line will be placed in socket 3, with core-id 14

  • the core with ID 9 provided by the device_add qmp command will be placed in socket 2, with core-id 9

Polarization, entitlement and dedication

Polarization

The polarization affects how the CPUs of a shared host are utilized/distributed among guests. The guest determines the polarization by using the PTF instruction.

Polarization defines two models of CPU provisioning: horizontal and vertical.

The horizontal polarization is the default model on boot and after subsystem reset. When horizontal polarization is in effect all vCPUs should have about equal resource provisioning.

In the vertical polarization model vCPUs are unequal, but overall more resources might be available. The guest can make use of the vCPU entitlement information provided by the host to optimize kernel thread scheduling.

A subsystem reset puts all vCPU of the configuration into the horizontal polarization.

Entitlement

The vertical polarization specifies that the guest’s vCPU can get different real CPU provisioning:

  • a vCPU with vertical high entitlement specifies that this vCPU gets 100% of the real CPU provisioning.

  • a vCPU with vertical medium entitlement specifies that this vCPU shares the real CPU with other vCPUs.

  • a vCPU with vertical low entitlement specifies that this vCPU only gets real CPU provisioning when no other vCPUs needs it.

In the case a vCPU with vertical high entitlement does not use the real CPU, the unused “slack” can be dispatched to other vCPU with medium or low entitlement.

A vCPU can be “dedicated” in which case the vCPU is fully dedicated to a single real CPU.

The dedicated bit is an indication of affinity of a vCPU for a real CPU while the entitlement indicates the sharing or exclusivity of use.

Defining the topology on the command line

The topology can entirely be defined using -device cpu statements, with the exception of CPU 0 which must be defined with the -smp argument.

For example, here we set the position of the cores 1,2,3 to drawer 1, book 1, socket 2 and cores 0,9 and 14 to drawer 0, book 0, socket 0 without defining entitlement or dedication. Core 4 will be set on its default position on socket 1 (since we have 4 core per socket) and we define it as dedicated and with vertical high entitlement.

$ qemu-system-s390x -m 2G \
  -cpu gen16b,ctop=on \
  -smp cpus=1,sockets=8,cores=4,maxcpus=32 \
  \
  -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=1 \
  -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=2 \
  -device gen16b-s390x-cpu,drawer-id=1,book-id=1,socket-id=2,core-id=3 \
  \
  -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=9 \
  -device gen16b-s390x-cpu,drawer-id=0,book-id=0,socket-id=0,core-id=14 \
  \
  -device gen16b-s390x-cpu,core-id=4,dedicated=on,entitlement=high

The entitlement defined for the CPU 4 will only be used after the guest successfully enables vertical polarization by using the PTF instruction.