Starting CPUs on ARM64

The systems I am working with have 80 or more cores in them. I’ve recently had to investigate processes around core start up. Here are my notes.

The Linux kernel start up process can be investigated from the start kernel function. We see a reference to smp_setup_processor_id fairly early on in the process. This essentially tells the CPU to read the Multiprocessor Affintiy Register. Ift gets ANDed with a bitmask of

#define MPIDR_HWID_BITMASK UL(0xff00ffffff)

I am assuming that the 16 masked out bits should not be used in identifying the core, but most of this register is used.

The first thing that is done is to explicitly set this first CPU in the __cpu_logical_map. This allows us to use sequential integers starting at 0 to refer to the cores instead of the HW based identifiers pulled from the register above.

IRQs are disabled to begin.

The first CPU is initialized. This used the CPU Id set up in the map abover, and sets fields on the CPU object: online, active, present and possible. These all are bit fields in the CPU mask set structure….

The next call is to setup the NR of CPUs actually on the system, nr_cpu_ids is a global variable, and thus needs to be initialized before anything can make use of it.

The per CPU areas refer to memory management. I think that ARM64 uses the generic function; other architectures have their own specific functions, but I don’t see an ARM64. I do see a NUMA one, though. I know we enable NUMA in our builds, so I suspect that it is built as a driver and used if explicitly enabled in the Kernel command line.

There are couple other Boot CPU initialization calls. Later on we hit the setup for the scheduler. This is till before the initialization of the other CPUs. This happens in the call to smp_preprate_cpus, and this is the big one. There are a couple elevel of Ops (function pointers in structures) calls till we get down to where the real work happens: This resolves down to power state control interface (psci) CPU boot function which in turn calls the psci_ops->cpu_on function, implemented in firmware. There is a reference to where this is defined is in Documentation, like this.
cpu_on = <0x95c10002>;

The address seems to be defined by the device tree files, such as:

arch/arm/boot/dts/artpec6.dtsi:81: cpu_on = <0x84000003>;

Thus, the CPU device has an address that turns on the CPU. There are comparable addresses for other operations. What is interesting is that this address is a function pointer. The function takes two parameters. The first is the HW identifier of the CPU, looked up from the map. The other is a value looked up in the call:

phys_addr_t pa_secondary_entry = __pa_symbol(function_nocfi(secondary_entry));

This is a function implemented in assembly code here.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.