How do you know what is inside your computer? There are a couple tools. If the hardware is on the PCI bus, from the command line you can run lspci, which will in turn enumerate the discovered devices on that bus. But what if the hardware is not on the PCI bus? And how does the Kernel discover it in the first place? For the hardware that I have to work with, the answer is that it is enumerated by the Unified Extensible Firmware Interface (UEFI) coded embedded in the device and exposed via the Advanced Configuration and Power Interface (ACPI). This world is full of four letter acronyms. Here are my notes on some of them.
What I am about to write about is based on what I’ve learned from looking at Ampere System(s)-on-a-Chip (SoC). While details change from generation to generation, this is the general pattern I’ve learned. It is not specific to ARM servers, but it is common in ARM servers, or so I’ve been told by people who have spent more time in this world than I have.
Adam, we ain’t in Keystone anymore.
We don’t tend to call them Central Processing Units (CPUs) anymore. It seems kinda silly to call them Central when there are dozens or hundreds of them on a single SoC. We’ll call them cores. When a core needs to talk to another component on the SoC it uses some form of interconnect. The one that I have used is called Platform Communication Channel (PCC).
There are lots of channels, on a board for communication between many components. There are two items of note in a PCC; the mailbox and the doorbell. I love that these two terms are NOT ACRONYMS. They are, of course, analogies, and good ones, if not perfect. A mailbox is a shared memory location (offset and length in physical memory). A Doorbell is a register. The idea is that you put your message in the mailbox, and then push the doorbell.
The analogy breaks down when you realize that the doorbell is not just a bit, but rather a full number that can carry additional information about the package that was left in the mailbox. And, because this is a multi user operating system, there is a locking and synchronization process involved.
The Channel is a generic communication mechanism. On an AltraMax system it is used by (at least) the xgene_hwmon Linux kernel module and the cppc_cpufrew kernel module. hwmon is for temperature and power monitoring. The PPC in CPPC is not the same PPC as above. It stands for Collaborative Processor Performance Control.
ACPI information is exposed in what the spec calls tables. For the most part, these are normalized tuples, but there is some nesting. Two tables of interest for communication between parts of an Ampere System on a Chip (SoC) are the Differentiated System Description Table (DSTD) and the Platform Communication Channel Table (PCCT.) I wanted to take a closer look at what is in these tables. To do so, I used acpidump with the -b switch to extract the tables and the iasl command to convert these two tables to their human readable source code forms.
The PCCT table has a short header, and then a repeated series of entries like this one:
[032h 0050 4] Platform Interrupt : 00000038
[036h 0054 1] Flags (Decoded Below) : 00
Polarity : 0
Mode : 0
[037h 0055 1] Reserved : 00
[038h 0056 8] Base Address : 00000000F0C10000
[040h 0064 8] Address Length : 0000000000004000
[048h 0072 12] Doorbell Register : [Generic Address Structure]
[048h 0072 1] Space ID : 00 [SystemMemory]
[049h 0073 1] Bit Width : 20
[04Ah 0074 1] Bit Offset : 00
[04Bh 0075 1] Encoded Access Width : 03 [DWord Access:32]
[04Ch 0076 8] Address : 0000100001540010
[054h 0084 8] Preserve Mask : 0000000000000000
[05Ch 0092 8] Write Mask : 0000000053000040
[064h 0100 4] Command Latency : 000003E8
[068h 0104 4] Maximum Access Rate : 00000000
[06Ch 0108 2] Minimum Turnaround Time : 0000
[06Eh 0110 12] Platform ACK Register : [Generic Address Structure]
[06Eh 0110 1] Space ID : 00 [SystemMemory]
[06Fh 0111 1] Bit Width : 20
[070h 0112 1] Bit Offset : 00
[071h 0113 1] Encoded Access Width : 03 [DWord Access:32]
[072h 0114 8] Address : 0000100001540020
[07Ah 0122 8] ACK Preserve Mask : 0000000000000000
[082h 0130 8] ACK Write Mask : 0000000000010001
[08Ah 0138 1] Subtable Type : 02 [HW-Reduced Comm Subspace Type2]
[08Bh 0139 1] Length : 5A
Lets compare this with entries in the DSDT.
Device (C000)
{
Name (_HID, "ACPI0007" /* Processor Device */) // _HID: Hardware ID
Name (_UID, Zero) // _UID: Unique ID
Method (_LPI, 0, NotSerialized) // _LPI: Low Power Idle States
{
Return (PLPI) /* \_SB_.PLPI */
}
Name (PCPC, Package (0x17)
{
0x17,
0x03,
ResourceTemplate ()
{
Register (PCC,
0x20, // Bit Width
0x00, // Bit Offset
0x0000000000000000, // Address
0x02, // Access Size
)
},
What I have learned so far is that the HID (Hardware Identifier? I think) for this device is ACPI0007, which is a Processor, or Core using the earlier terminology. It has inside it a PCPC package. I have not been able to expand this acronym but I have been told that they are registers within CPPC, so possibly the letters mean the same thing but are in a different order. Maybe Processor Collaborative Performance Counter?
The two values we are interested in here are the Access Size and Address values. The Access Size is actually a zero relative index. This points to the third entry in the PCCT. This particular register has an offset of 0. The next has 0x0000000000000004, and then 0x0000000000000008, and then 0x000000000000000C, and so on.
My understanding is that in PCC there are registers and there are shared memory regions. Because a shared memory region might be off the SoC in SDRAM, it can have a lot higher latency than a register. Thus, to keep latency down, these use the on chip registers.
More on this as I learn more. And I might be WAAAAY off on this, so please do not use it to design your hardware.