amrishhallberg.comNovember 6, 2026
Amrish Hallberg

Bare-Metal Multicore on the Raspberry Pi Zero 2 W

Registers x0Instructions ldrFunctions kernel_mainCores core 0

Okay, so now that we have the single core firmware for the Raspberry Pi Zero 2w working, lets get all four cores running. Why is four cores better than one? We have many tasks that we may want to complete with parallism, rather than just concurrency, in an embedded context. Picture a system where you have flight controls, radar imaging, detonation controls, etc. You may not want all of those in your critical loop. If radar processing takes an unexpected amount of time, we don't want our critical loop to stall. There are multiple ways to handle that, such as watchdogs, but multiple parallel cores opens up a lot of cool things we can do, and imposes little overhead, granted we don't race ourselves into oblivion

First things first, all cores share the same memory addresses, because there is no OS underneath to abstract or restrict the space away from us. Remember, unless we write a filesystem, there isn't one. There is nowhere to put data outside of RAM. We have the RPI-given 512-MB of memory. All four cores can see all 512-MB of memory. We need to give each of them their own stack to start so they don't step on each others toes.

linker.lde2d34ec
1
2 ENTRY(_start) /* we declare where we want to start*/
3
4 SECTIONS {
5
6
7 /*beginning memory address*/
8 . = 0x80000;
9
10 /*our text section*/
11 .text :
12 {
13 KEEP(*(.text.boot))
14
15 /*rest of exe code */
16 *(.text*)
17 }
18+
19 . = ALIGN(16);
20 .rodata :
21 {
22 *(.rodata*)
23 }
24+
25+
26 . = ALIGN(16);
27 .data :
28 {
29 *(.data*)
30 }
31+
32 . = ALIGN(16);
33 .bss (NOLOAD) :
34 {
35 __bss_start = .;
36 *(.bss*)
37 *(COMMON)
38 __bss_end = .;
39 }
40+
41 . = ALIGN(16);
42+
43 .stack (NOLOAD):
44 {
45 . = ALIGN(16);
46 /* save the address*/
47 __stack_bottom = .;
48+
49 . += 4096; /*push it 4kb*/
50+
51 __stack_top = .;
52 /*snag that thang again. now we have the top and bottem
53 *to reference*/
54+
55+ /*now we want 4 stacks, lets give them equal size*/
56+
57+ __core1_stack_bottom = .; /*save lowest address */
58+ . += 4096; /* increment by 4k */
59+ __core1_stack_top = .; /*grab our core1 bottom top address*/
60+
61+ __core2_stack_bottom = .;
62+ . += 4096;
63+ __core2_stack_top = .;
64+
65+ __core3_stack_bottom = .;
66+ . += 4096;
67+ __core3_stack_top = .;
68+
69+
70 }
71+
72+
73 }

Okay, now we have 16-KB of memory reserved for our four core stack. Stacks grow down, so we grab their lowest address first, increment by 4K, and grab their highest possible address.

Four contiguous 4 KiB stacks, with higher addresses above and downward stack growth.

Now we can see that they each have their own 4KB to play with, and will not step on each others toes (unless we fuckup. Don't fuck up).

We need to initialize their stacks now, because we currently only handle core 0. Core 0 also has some extra responsibilities that we don't want to race, so we need to jmp the other cores to their own initialization procedure. Here I called it init_stack, and the general flow is:

  1. we need to find the stack top. This is labeled in our linker, and is the highest address of the core 0 stack.
  2. Because we calculate the other stack by just incrementing 4096 bytes each time, we can recalculate here once we know that starting address. For any given core, number N, we can get the top stack address with
N stack top = (core 0 stack top address) + ( N * 4096)

which we place in the x0 register. Remember, each core has its own registers, so each core has its own x0. We are placing the stack location of each core's stack in their own x0, and then moving that into each of their own, unshared stack pointer. Then each core has a stack pointer register that knows where it starts (and from which we can compute an end). 3. Right now we are then hanging the core again

boot.S33e7a67
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 _start:
6 //we should do this before initalizing the stack
7+ mrs x1, mpidr_el1 //this tells each core who it is
8 and x1, x1, #0xFF //we want the lowest 8 bits
9+
10+ //instead of ending their careers, we want them to each intialize
11+ cbnz x1, init_stack //everyone not core 0 jumps to init_stack
12+
13 ldr x0, =__stack_top
14 mov sp, x0
15+ //core 0 is special, it is one that does bss clearing and initalized the peripherals
16 ldr x0, =__bss_start
17 ldr x1, =__bss_end
18+
19 clear_bss:
20 cmp x0, x1
21 b.hs bss_done //branch to bss_done if x0 >= x1
22+
23 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
24 b clear_bss
25+
26 bss_done:
27 bl kernel_main
28+
29 hang:
30 wfe
31 b hang
32+
33+//remember we auto incremented by 4KB, so we can just recalculate the addresses
34+//instead of hard labeling everything
35+//if we change stack size, this also needs to be adjusted
36+//n core stack top = __stack_top + (n * 4096)
37+//each core will enter, compute its value, and initalize
38+init_stack:
39+ ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
40+ add x0, x0, x1, lsl #12 //shifting left
41+ mov sp, x0 //move the stack top into each of their own stack pointer registers
42+ b hang //placeholder

Now, that start entry is really only for core 0. We will actually give each of the other cores their own entry, which we can just call something like secondary_entry. Remember, each core needs the mrs instruction to inform the core who they are. The mpidr_el register will tell us our core number, like a "whatsmyip" for a core. Then remember we apply the AND to get the lowest 8 bits in the x1 register. Because only core 0 uses _start now we can remove the jmp to init_stack.

boot.S4151b36
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5+//exclusive core 0 entry now
6 _start:
7+ //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 bl kernel_main
27
28 hang:
29 wfe
30 b hang
31
32+
33+//entry for all other cores
34 //remember we auto incremented by 4KB, so we can just recalculate the addresses
35 //instead of hard labeling everything
36 //if we change stack size, this also needs to be adjusted
37 //n core stack top = __stack_top + (n * 4096)
38 //each core will enter, compute its value, and initalize
39+
40+.global secondary_entry
41+secondry_entry:
42+ mrs x1, mpidr_el1 //this tells each core who it is
43+ and x1, x1, #0xFF //we want the lowest 8 bits
44+
45 init_stack:
46 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
47 add x0, x0, x1, lsl #12 //shifting left
48 mov sp, x0 //move the stack top into each of their own stack pointer registers
49 b hang //placeholder

Misspelled secondary

User-selected dog reaction image at the misspelled secondary label.
boot.S0141ad0
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 bl kernel_main
27
28 hang:
29 wfe
30 b hang
31
32
33 //entry for all other cores
34 //remember we auto incremented by 4KB, so we can just recalculate the addresses
35 //instead of hard labeling everything
36 //if we change stack size, this also needs to be adjusted
37 //n core stack top = __stack_top + (n * 4096)
38 //each core will enter, compute its value, and initalize
39
40 .global secondary_entry
41+secondary_entry:
42 mrs x1, mpidr_el1 //this tells each core who it is
43 and x1, x1, #0xFF //we want the lowest 8 bits
44
45 init_stack:
46 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
47 add x0, x0, x1, lsl #12 //shifting left
48 mov sp, x0 //move the stack top into each of their own stack pointer registers
49 b hang //placeholder

Now, we need core 0 to write the entry addresses into the correct slot, and we need it to send them a wake event.

Development screenshot: Pasted image 20261106085046.png

We can see in their official examples that they use 0xe0, 0xe8, and 0xf0 for the cores. Lets start by just unleashing one extra core

User-selected angry cat reaction image at the worker release.

Now, because core 0 is the chosen one and has a super special entry, we can use it to prime the other cores. We branch from bss_done to the kernel_main in our C file, so immediately prior to branching there is a good place to put the multi-core initialization.

So first we need to put the location of the entry function into the x0 register. Then we put the 0xe0 location that we stole from the RPi foundations homework, into x1. Next we store what is in x0, into the address pointed to by x1. dsb is a special ARM instruction that stands for Data Synchronization Barrier, and forces the cpu to wait for all loads, stores, cache ops, and translation lookaside buffer operations to be done. This ensures we don't do anything before the values we expect are actually instantiated. If we don't, a worker could wake up and read its address before it has a value other than 0, and go back to sleep. sev is the Send Event instruction.

boot.Sd5e0d72
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26+ ldr x0, =secondary_entry //entry code location
27+ mov x1, #0xe0 //core 1 slot locatoin
28+ str x0, [x1] //put x0 value in 0xe0
29+
30+ dsb sy //barrier
31+ sev //
32+
33 bl kernel_main
34
35 hang:
36+ wfe //wait for event
37 b hang
38
39
40 //entry for all other cores
41 //remember we auto incremented by 4KB, so we can just recalculate the addresses
42 //instead of hard labeling everything
43 //if we change stack size, this also needs to be adjusted
44 //n core stack top = __stack_top + (n * 4096)
45 //each core will enter, compute its value, and initalize
46
47 .global secondary_entry
48 secondary_entry:
49 mrs x1, mpidr_el1 //this tells each core who it is
50 and x1, x1, #0xFF //we want the lowest 8 bits
51
52 init_stack:
53 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
54 add x0, x0, x1, lsl #12 //shifting left
55 mov sp, x0 //move the stack top into each of their own stack pointer registers
56 b hang //placeholder

Alright cool, now its trivial to get each of the other two up and running as well

boot.S6706747
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30+ mov x1, #0xe8
31+ str x0, [x1]
32+
33+ mov x1, #0xf0
34+ str x0, [x1]
35+
36 dsb sy //barrier
37+ sev //send event
38
39 bl kernel_main
40
41 hang:
42 wfe //wait for event
43 b hang
44
45
46 //entry for all other cores
47 //remember we auto incremented by 4KB, so we can just recalculate the addresses
48 //instead of hard labeling everything
49 //if we change stack size, this also needs to be adjusted
50 //n core stack top = __stack_top + (n * 4096)
51 //each core will enter, compute its value, and initalize
52
53 .global secondary_entry
54 secondary_entry:
55 mrs x1, mpidr_el1 //this tells each core who it is
56 and x1, x1, #0xFF //we want the lowest 8 bits
57
58 init_stack:
59 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
60 add x0, x0, x1, lsl #12 //shifting left
61 mov sp, x0 //move the stack top into each of their own stack pointer registers
62 b hang //placeholder

If this has gotten a little confusing, that is perfectly okay

Earlier build walkthrough: core 0 releases workers, each worker derives its stack from its core number.

Hopefully this graphic can help you trace the logic of each core. Now, each core is theoretically alive, but right now we are just hanging them and would need electrical traces to determine if they were actually doing something. In order to get some sort of signal, we can take advantage of the fact that they all share the same RAM, and give them unique derived locations to write to, as a "flag", indicating that they are indeed working as intended. This is the first thing we have really done where we need to be worried about any kind of interference from other cores. How do we make sure that they don't accidentally overwrite each others data? We can use their unique core numbers, so that even running the same calculation they can never derive the same memory address.

We can follow an algorithm something roughly like

flag address = flag base + (core n * 8)

now, we need somewhere to put this new 32 bytes worth of flag memory. .section .bss changes our mode from executable instructions to uninitialized storage. "aw" means allocateable and writeable. @nobits means we dont have inital data to go into the range. .balign 8 aligns us to the 8 byte boundary We are just calling this memory region core_alive because that is what we are checking .skip 32 reserves our 32 bytes (8 bytes for 4 cores) if we run through our flag address algorithm

Address Slot
core_alive + 0 core 0’s slot
core_alive + 8 core 1’s slot
core_alive + 16 core 2’s slot
core_alive + 24 core 3’s slot
boot.Sfb00b4d
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38
39 bl kernel_main
40
41 hang:
42 wfe //wait for event
43 b hang
44
45
46 //entry for all other cores
47 //remember we auto incremented by 4KB, so we can just recalculate the addresses
48 //instead of hard labeling everything
49 //if we change stack size, this also needs to be adjusted
50 //n core stack top = __stack_top + (n * 4096)
51 //each core will enter, compute its value, and initalize
52
53 .global secondary_entry
54 secondary_entry:
55 mrs x1, mpidr_el1 //this tells each core who it is
56 and x1, x1, #0xFF //we want the lowest 8 bits
57
58 init_stack:
59 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
60 add x0, x0, x1, lsl #12 //shifting left
61 mov sp, x0 //move the stack top into each of their own stack pointer registers
62 b hang //placeholder
63+
64+
65+.section .bss, "aw", @nobits
66+.balign 8
67+core_alive:
68+ .skip 32

Now we need to have each worker locate its own slot in the core_alive section, and to store a "1" there, so we know they completed their procedure.We'll put that in init_stack. First we want to put the memory address of core_alive into one of their registers, so they know where to write. Then we want to calculate their writing spot. lsl 3 means shift left by 3 bits, which is the same thing as multiplying a number by 2^3 or 8. for example if our bits start at 00000010 and we shift left 3 bits, we get 00010000 , or in other words we started with 2 and ended with 16 (a multiplicative of 8).

boot.Sbbfabd2
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38
39 bl kernel_main
40
41 hang:
42 wfe //wait for event
43 b hang
44
45
46 //entry for all other cores
47 //remember we auto incremented by 4KB, so we can just recalculate the addresses
48 //instead of hard labeling everything
49 //if we change stack size, this also needs to be adjusted
50 //n core stack top = __stack_top + (n * 4096)
51 //each core will enter, compute its value, and initalize
52
53 .global secondary_entry
54 secondary_entry:
55 mrs x1, mpidr_el1 //this tells each core who it is
56 and x1, x1, #0xFF //we want the lowest 8 bits
57
58 init_stack:
59 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
60 add x0, x0, x1, lsl #12 //shifting left
61 mov sp, x0 //move the stack top into each of their own stack pointer registers
62+ ldr x2, =core_alive
63+ add x2, x2, x1, lsl #3
64+
65 b hang //placeholder
66
67
68 .section .bss, "aw", @nobits
69 .balign 8
70 core_alive:
71 .skip 32

So now x2 holds our intended write address for each core, we need to write 1. We can just use x3 for this, and we can store the value in x3 into the memory indicated by the address in x2.

boot.Sc6226ff
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38
39 bl kernel_main
40
41 hang:
42 wfe //wait for event
43 b hang
44
45
46 //entry for all other cores
47 //remember we auto incremented by 4KB, so we can just recalculate the addresses
48 //instead of hard labeling everything
49 //if we change stack size, this also needs to be adjusted
50 //n core stack top = __stack_top + (n * 4096)
51 //each core will enter, compute its value, and initalize
52
53 .global secondary_entry
54 secondary_entry:
55 mrs x1, mpidr_el1 //this tells each core who it is
56 and x1, x1, #0xFF //we want the lowest 8 bits
57
58 init_stack:
59 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
60 add x0, x0, x1, lsl #12 //shifting left
61 mov sp, x0 //move the stack top into each of their own stack pointer registers
62 ldr x2, =core_alive
63 add x2, x2, x1, lsl #3
64+ mov x3, #1
65+ stlr x3, [x2]
66 b hang //placeholder
67
68
69 .section .bss, "aw", @nobits
70 .balign 8
71 core_alive:
72 .skip 32

now lets go back to core 0. So each core 1-3 SHOULD have, by now, awoken, initialized its stack, determined where its core_alive reservation is, written 1, and hung. Core 0 should check these to ensure all the cores are alive. Remember core 0 doesnt need to write there, it just needs to read it. So we get the address of core_alive, add 8 so we are looking at core 1's slot, and then we read the data in the x2 memory address into our x3 register. If x3 = 0, the core is not alive yet. If x3 = 1, the core is alive.

terminal
1 ldr x2, =core_alive
2 add x2, x2, #8
3 ldar x3, [x2]
Black-and-white animated Frankenstein scene: It is alive! It is alive!

now, we need that to be reusable, so instead of placing it in a specific place in our path, lets make it a new procedure. Remember, in proper languages (which are exclusively assembly and HDL variants), where things go in a file matters, so the procedure needs to be inside of the executable .text section, before the .bss section. So we are going to pass in the argument for which core we are looking for in x0.

  1. We read the core_alive address into x2
  2. we increment x2 by 8, x0 number of times.
  3. we then read the value in that 8 byte memory region into x0, which is our return
  4. we return so now our function will either return 0 (core is not alive) or 1 (core is alive)

We also need to add the read_core_alive function to our main.c, so it is expecting that definition somewhere (which comes from our boot.S).

boot.S1307efe
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38 bl kernel_main
39
40 hang:
41 wfe //wait for event
42 b hang
43
44
45 //entry for all other cores
46 //remember we auto incremented by 4KB, so we can just recalculate the addresses
47 //instead of hard labeling everything
48 //if we change stack size, this also needs to be adjusted
49 //n core stack top = __stack_top + (n * 4096)
50 //each core will enter, compute its value, and initalize
51
52 .global secondary_entry
53 secondary_entry:
54 mrs x1, mpidr_el1 //this tells each core who it is
55 and x1, x1, #0xFF //we want the lowest 8 bits
56
57 init_stack:
58 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
59 add x0, x0, x1, lsl #12 //shifting left
60 mov sp, x0 //move the stack top into each of their own stack pointer registers
61 ldr x2, =core_alive
62 add x2, x2, x1, lsl #3
63 mov x3, #1
64 stlr x3, [x2]
65 b hang //placeholder
66
67+.global read_core_alive
68+read_core_alive:
69+ //we are gonna say we want the caller (core 0) to pass in
70+ //the core number in x0
71+ //so you wont see it set in this function, we expect it to already be there
72+ ldr x2, =core_alive
73+ add x2, x2, x0, lsl #3
74+ //then we wanna pass the return value back in x0
75+ ldar x0, [x2]
76+ ret
77+
78
79 .section .bss, "aw", @nobits
80 .balign 8
81 core_alive:
82 .skip 32
main.c1307efe
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5+
6+unsigned long read_core_alive(unsigned long core);
7
8 void kernel_main(void)
9 {
10 uart_init();
11 uart_puts("terminal ready\r\n");
12
13 while (1) {
14 uart_puts("Pi> ");
15 uart_readline(input, sizeof(input));
16 uart_puts("Received: ");
17 uart_puts(input);
18 uart_puts("\r\n");
19 }
20 }

We need to give ourselves some capabilities most OS's would provide in order to not be bound by program execution in terms of things like time tracking. We need to know two main things:

  1. What is the timer tick value?
  2. How many ticks are one second? This lets our counter run independently of our polling loop, which helps us keep real-time deadlines.

For the first helper, getting the current tick value, we want use the mrs instruction to query the Counter-timer Physical Count Register. ISB is the instruction synchronization barrier, and ensures the read comes after earlier instructions, so we have an accurate timer change assessment.

Now in the other helper, we want to read the frequncy of the system in Hz, so we know how to do our frequncy calculations. This is called the Counter-timer Frequency Register.

main.c3e08b0e
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7+
8+
9+static unsigned long timer_ticks(void){
10+ unsigned long ticks;
11+
12+
13+ __asm__ volatile(
14+ "isb\n"
15+ "mrs %0, cntpct_el0" //read counter timer register
16+ : "=r"(ticks)
17+ :
18+ : "memory"
19+ );
20+ return ticks;
21+
22+}
23+
24+static unsigned long timer_frequency(void)
25+{
26+
27+ unsigned long frequency;
28+
29+
30+ __asm__ volatile(
31+
32+ "mrs %0, cntfrq_el0"
33+ : "=r"(frequency)
34+ );
35+ return frequency;
36+}
37+
38+
39
40 void kernel_main(void)
41 {
42 uart_init();
43 uart_puts("terminal ready\r\n");
44
45 while (1) {
46 uart_puts("Pi> ");
47 uart_readline(input, sizeof(input));
48 uart_puts("Received: ");
49 uart_puts(input);
50 uart_puts("\r\n");
51 }
52 }

Okay, now to our kernel_main, we need to add the checking. We need to figure out how long one second is, what time we are starting at, and then we need to start polling the flags. You may notice there is a second condition beyond all the flags being set, and that is that the elapsement of timer ticks has been less than one seconds worth. This is a simple example (bounded polling) of how we can meet hard, real-time deadlines with our embedded systems. If it takes longer than one second, something is very, very wrong. Imagine if this was your car, and the car systems are busy doing a bunch of calculations. There is only so long we can allow that before we need to get it back to paying attention to your braking, steering, air bag deployment, etc. otherwise your car won't respond and you will have a pink panther kind of day on the road.

main.c77a796f
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39
40 void kernel_main(void)
41 {
42 uart_init();
43+ //now we need to add in the core reads
44+ unsigned long timeout = timer_frequency(); //one seconds worth
45+ unsigned long start = timer_ticks(); //get our starting time
46+
47+ unsigned long core1_alive = 0;
48+ unsigned long core2_alive = 0;
49+ unsigned long core3_alive = 0;
50+
51+ while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
52+ core1_alive = read_core_alive(1);
53+ core2_alive = read_core_alive(2);
54+ core3_alive = read_core_alive(3);
55+ }
56+
57+
58 uart_puts("terminal ready\r\n");
59
60 while (1) {
61 uart_puts("Pi> ");
62 uart_readline(input, sizeof(input));
63 uart_puts("Received: ");
64 uart_puts(input);
65 uart_puts("\r\n");
66 }
67 }

Now that we are reading them, we want to give our console some sort of validation that the cores are awake. So we will reuse the uart_puts to do it. For each core we will just use a simple ternary operator to put one string if true, and one string if false.

main.cb374ae9
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39
40 void kernel_main(void)
41 {
42 uart_init();
43 //now we need to add in the core reads
44 unsigned long timeout = timer_frequency(); //one seconds worth
45 unsigned long start = timer_ticks(); //get our starting time
46
47 unsigned long core1_alive = 0;
48 unsigned long core2_alive = 0;
49 unsigned long core3_alive = 0;
50
51 while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
52 core1_alive = read_core_alive(1);
53 core2_alive = read_core_alive(2);
54 core3_alive = read_core_alive(3);
55 }
56
57+ uart_puts(core1_alive ? "Core 1: alive\r\n" : "Core 1: startup timed out\r\n");
58+ uart_puts(core1_alive ? "Core 2: alive\r\n" : "Core 2: startup timed out\r\n");
59+ uart_puts(core1_alive ? "Core 3: alive\r\n" : "Core 3: startup timed out\r\n");
60+
61
62 uart_puts("terminal ready\r\n");
63
64 while (1) {
65 uart_puts("Pi> ");
66 uart_readline(input, sizeof(input));
67 uart_puts("Received: ");
68 uart_puts(input);
69 uart_puts("\r\n");
70 }
71 }

Now, we have successful output, but I want to know how long startup took. This will help us have an idea of how long it takes to go from applying power to doing that things that we want. So lets add a uart_put_ulong in uart.c and uart.h so we can put unsigned long numbers to the console. 20 characters fit the biggest 64-bit unsigned decimal value. the addition of '0' to the value converts it into the proper character, and we can take that character and put it to the serial console now with uart_putc.

main.cc5f2643
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39
40 void kernel_main(void)
41 {
42 uart_init();
43 //now we need to add in the core reads
44 unsigned long timeout = timer_frequency(); //one seconds worth
45 unsigned long start = timer_ticks(); //get our starting time
46
47 unsigned long core1_alive = 0;
48 unsigned long core2_alive = 0;
49 unsigned long core3_alive = 0;
50
51 while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
52 core1_alive = read_core_alive(1);
53 core2_alive = read_core_alive(2);
54 core3_alive = read_core_alive(3);
55 }
56
57+ unsigned long elapsed_ticks = timer_ticks() - start; //how long that took in ticks
58+ unsigned long elapsed_ms = (elapsed_ticks * 1000UL) / timeout; //convert to real milliseconds
59+
60 uart_puts(core1_alive ? "Core 1: alive\r\n" : "Core 1: startup timed out\r\n");
61+ uart_puts(core2_alive ? "Core 2: alive\r\n" : "Core 2: startup timed out\r\n");
62+ uart_puts(core3_alive ? "Core 3: alive\r\n" : "Core 3: startup timed out\r\n");
63+ uart_puts("Worker startup wait time: ");
64+ uart_put_ulong(elapsed_ms);
65+ uart_puts(" ms\r\n");
66
67 uart_puts("terminal ready\r\n");
68
69 while (1) {
70 uart_puts("Pi> ");
71 uart_readline(input, sizeof(input));
72 uart_puts("Received: ");
73 uart_puts(input);
74 uart_puts("\r\n");
75 }
76 }
uart.cc5f2643
1
2 #include <stdint.h>
3 #include "uart.h"
4
5 #define AUX_BASE 0x3F215000UL
6 #define AUX_MU_LCR (AUX_BASE + 0x4CUL) //8-bit char's
7 #define AUX_MU_BAUD (AUX_BASE + 0x68UL) // 115200 baud rate
8 #define AUX_ENABLES (AUX_BASE + 0X04UL) //enable mini uart
9 #define AUX_MU_CNTL (AUX_BASE +0X60UL) //lever for tx and rx
10 #define AUX_MU_IER (AUX_BASE + 0x44UL) //interrupt enable
11 #define AUX_MU_IIR (AUX_BASE + 0x48UL) //FIFO control
12 #define AUX_MU_MCR (AUX_BASE + 0x50UL) //modem control
13 #define AUX_MU_IO (AUX_BASE + 0x40UL) // data register, sents to tx FIFO
14 #define AUX_MU_LSR (AUX_BASE + 0x54UL) //line status register
15
16
17
18 #define GPIO_BASE 0x3F200000UL
19 #define GPFSEL1 (GPIO_BASE + 0x04UL)
20 #define GPPUD (GPIO_BASE + 0x94UL)
21 #define GPPUDCLK0 (GPIO_BASE + 0x98UL)
22
23
24
25 static uint32_t mmio_read32(uintptr_t address){
26 return *(volatile uint32_t *)address;
27 }
28
29 static void mmio_write32(uintptr_t address, uint32_t value){
30
31 *(volatile uint32_t *)address = value;
32
33 }
34
35
36 void gpio_uart_pins(void){
37 uint32_t selector = mmio_read32(GPFSEL1);
38+
39 selector &= ~((7U << 12) | (7U << 15));
40+
41 selector |= (2u << 12) | (2u << 15);
42+
43 mmio_write32(GPFSEL1, selector);
44 }
45
46 void mini_uart_enable(void){
47 uint32_t value = mmio_read32(AUX_ENABLES);
48 value |= (1u << 0);
49 mmio_write32(AUX_ENABLES, value);
50+
51 mmio_write32(AUX_MU_CNTL,0U);
52
53+
54 }
55
56 void mini_uart_configure(void)
57 {
58 mmio_write32(AUX_MU_LCR, 3u); //write the 8-bit mode
59 mmio_write32(AUX_MU_BAUD, 270u); //set the baud to 115200 @ 250mhz
60 //we're gonna poll instead of using interrupts
61 mmio_write32(AUX_MU_IER, 0u); //no uart interrupts
62 mmio_write32(AUX_MU_MCR, 0u); //no modem
63 mmio_write32(AUX_MU_IIR, 6u); //clear all the rx tx FIFO's
64
65 }
66
67
68 static void delay_cycles(void)
69 {
70 for (volatile unsigned i = 0; i < 200; i++) {
71 __asm__ volatile("nop");
72 }
73 }
74
75
76 void gpio_uart_disable_pulls(void)
77 {
78 mmio_write32(GPPUD, 0u); //disable pull-up and pull-down resistors
79 delay_cycles(); //add our 200 nops
80
81 mmio_write32(GPPUDCLK0, (1u << 14) | (1u << 15)); //configure GPIO 14 and 15
82 delay_cycles();
83+
84 mmio_write32(GPPUD, 0u); //Clear control reg
85 mmio_write32(GPPUDCLK0, 0u); //clear clock reg
86 }
87
88 void uart_putc(char c)
89 {
90 while ((mmio_read32(AUX_MU_LSR) & (1u << 5)) == 0u) {
91 } //just continuously polling until its ready
92 mmio_write32(AUX_MU_IO, (uint8_t)c); //send the character, cast to 8 bit uint
93 }
94
95+void uart_put_ulong(unsigned long value)
96+{
97+ char digits[20];
98+ size_t count = 0;
99+
100+ do {
101+ digits[count++] = (char)('0' + (value % 10));
102+ value /= 10;
103+ } while (value != 0);
104+
105+ while (count > 0) {
106+ uart_putc(digits[--count]);
107+ }
108+}
109
110 char uart_getc(void)
111 {
112 while ((mmio_read32(AUX_MU_LSR) & (1u << 0)) == 0u) { //polling
113 }
114 return (char)(mmio_read32(AUX_MU_IO) & 0xFFu);
115 }
116
117 //this is effectively our printf alternative
118 void uart_puts(const char *str)
119 {
120 while (*str != '\0') {
121 uart_putc(*str);
122 str++;
123 }
124 }
125
126
127 //now we need an initalization function
128
129 void uart_init(void){
130+
131 mini_uart_enable();
132 mini_uart_configure();
133 gpio_uart_disable_pulls();
134 gpio_uart_pins();
135 mmio_write32(AUX_MU_CNTL, 3u); //enable tx and rx
136 }
137
138+//basically need an interactive terminal
139 size_t uart_readline(char *buffer, size_t capacity)
140 {
141 if (buffer == NULL || capacity == 0) {
142 return 0;
143 }
144
145 size_t length = 0;
146
147 while (1) {
148 char c = uart_getc();
149 if (c == '\r' || c == '\n') {
150 buffer[length] = '\0';
151 uart_puts("\r\n");
152 return length;
153 }
154 if (c == '\b' || c == 127) {
155 if (length > 0) {
156 length--;
157 uart_puts("\b \b");
158 }
159 continue;
160 }
161 if ((unsigned char)c < 32) {
162 continue;
163 }
164 if (length < capacity - 1) {
165 buffer[length++] = c;
166 uart_putc(c);
167 }
168 else {
169 uart_putc('\a'); //buffer is too full
170 }
171 }
172 }
uart.hc5f2643
1 #ifndef PI_ZERO2_UART_H
2 #define PI_ZERO2_UART_H
3
4 #include <stddef.h>
5
6 void uart_init(void);
7 void uart_putc(char c);
8 char uart_getc(void);
9 void uart_puts(const char *str);
10+void uart_put_ulong(unsigned long value);
11 size_t uart_readline(char *buffer, size_t capacity);
12
13 #endif /* PI_ZERO2_UART_H */

Okay, we are past a lot of the scaffolding, and we need to actually give the cores something to do other than hang. so lets replace the b hang with something that enters them into a function. Currently we have the registers identify in x1, and the ARM64 ABI says we need to pass our first argument in x0, so we need to move that first. Then we branch with link to a function I'm calling secondary_main. So now we need to define secondary_main.

boot.S90b3358
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38 bl kernel_main
39
40 hang:
41 wfe //wait for event
42 b hang
43
44
45 //entry for all other cores
46 //remember we auto incremented by 4KB, so we can just recalculate the addresses
47 //instead of hard labeling everything
48 //if we change stack size, this also needs to be adjusted
49 //n core stack top = __stack_top + (n * 4096)
50 //each core will enter, compute its value, and initalize
51
52 .global secondary_entry
53 secondary_entry:
54 mrs x1, mpidr_el1 //this tells each core who it is
55 and x1, x1, #0xFF //we want the lowest 8 bits
56
57 init_stack:
58 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
59 add x0, x0, x1, lsl #12 //shifting left
60 mov sp, x0 //move the stack top into each of their own stack pointer registers
61 ldr x2, =core_alive
62 add x2, x2, x1, lsl #3
63 mov x3, #1
64 stlr x3, [x2]
65+
66+ mov x0, x1
67+ bl secondary_main
68+ b hang //safe thing to do if secondary_main acts unexpectedly
69+
70
71 .global read_core_alive
72 read_core_alive:
73 //we are gonna say we want the caller (core 0) to pass in
74 //the core number in x0
75 //so you wont see it set in this function, we expect it to already be there
76 ldr x2, =core_alive
77 add x2, x2, x0, lsl #3
78 //then we wanna pass the return value back in x0
79 ldar x0, [x2]
80 ret
81
82
83 .section .bss, "aw", @nobits
84 .balign 8
85 core_alive:
86 .skip 32

I think the best way if we have given functions in mind, is to use a case statement, and assign each core a function based on its core number.

main.c19c3ca0
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39
40 void kernel_main(void)
41 {
42 uart_init();
43 //now we need to add in the core reads
44 unsigned long timeout = timer_frequency(); //one seconds worth
45 unsigned long start = timer_ticks(); //get our starting time
46
47 unsigned long core1_alive = 0;
48 unsigned long core2_alive = 0;
49 unsigned long core3_alive = 0;
50
51 while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
52 core1_alive = read_core_alive(1);
53 core2_alive = read_core_alive(2);
54 core3_alive = read_core_alive(3);
55 }
56
57 unsigned long elapsed_ticks = timer_ticks() - start; //how long that took in ticks
58 unsigned long elapsed_ms = (elapsed_ticks * 1000UL) / timeout; //convert to real milliseconds
59
60 uart_puts(core1_alive ? "Core 1: alive\r\n" : "Core 1: startup timed out\r\n");
61 uart_puts(core2_alive ? "Core 2: alive\r\n" : "Core 2: startup timed out\r\n");
62 uart_puts(core3_alive ? "Core 3: alive\r\n" : "Core 3: startup timed out\r\n");
63 uart_puts("Worker startup wait time: ");
64 uart_put_ulong(elapsed_ms);
65 uart_puts(" ms\r\n");
66
67 uart_puts("terminal ready\r\n");
68
69 while (1) {
70 uart_puts("Pi> ");
71 uart_readline(input, sizeof(input));
72 uart_puts("Received: ");
73 uart_puts(input);
74 uart_puts("\r\n");
75 }
76 }
77+
78+
79+void secondary_main(unsigned long core)
80+{
81+
82+switch (core){
83+ case 1:
84+ break;
85+
86+ case 2:
87+ break;
88+
89+ case 3:
90+ break;
91+ for (;;){
92+ __asm__ volatile("wfe");
93+ }
94+}
95+
96+}

Okay, so this should technically work but we have not actually defined anything that they should be doing. We can make a small token function to just have them doing work.

main.cdfaa25c
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39+//simple continuous function where each core will count by a different increment
40+static void counter_worker(unsigned long step)
41+{
42+ volatile unsigned long counter = 0;
43+ for (;;){
44+ counter += step;
45+ }
46+}
47
48 void kernel_main(void)
49 {
50 uart_init();
51 //now we need to add in the core reads
52 unsigned long timeout = timer_frequency(); //one seconds worth
53 unsigned long start = timer_ticks(); //get our starting time
54
55 unsigned long core1_alive = 0;
56 unsigned long core2_alive = 0;
57 unsigned long core3_alive = 0;
58
59 while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
60 core1_alive = read_core_alive(1);
61 core2_alive = read_core_alive(2);
62 core3_alive = read_core_alive(3);
63 }
64
65 unsigned long elapsed_ticks = timer_ticks() - start; //how long that took in ticks
66 unsigned long elapsed_ms = (elapsed_ticks * 1000UL) / timeout; //convert to real milliseconds
67
68 uart_puts(core1_alive ? "Core 1: alive\r\n" : "Core 1: startup timed out\r\n");
69 uart_puts(core2_alive ? "Core 2: alive\r\n" : "Core 2: startup timed out\r\n");
70 uart_puts(core3_alive ? "Core 3: alive\r\n" : "Core 3: startup timed out\r\n");
71 uart_puts("Worker startup wait time: ");
72 uart_put_ulong(elapsed_ms);
73 uart_puts(" ms\r\n");
74
75 uart_puts("terminal ready\r\n");
76
77 while (1) {
78 uart_puts("Pi> ");
79 uart_readline(input, sizeof(input));
80 uart_puts("Received: ");
81 uart_puts(input);
82 uart_puts("\r\n");
83 }
84 }
85
86
87 void secondary_main(unsigned long core)
88 {
89
90 switch (core){
91 case 1:
92+ counter_worker(1);
93 break;
94
95 case 2:
96+ counter_worker(2);
97 break;
98
99 case 3:
100+ counter_worker(3);
101 break;
102+}
103 for (;;){
104 __asm__ volatile("wfe");
105 }
106
107 }

Now, each of these core is currently just writing into their own stack, and that is private. That means our core 0 can't actually access them and tell us any of the values. So instead, I would like to reserve another set of memory address, offset by a size increment times the core number. Once again, this stops the cores from racing, as we can ensure core 1 never writes into core 2's space, core 2 never writes into core 1 or 3's space, and so on. Lets go back into boot.S and add another .bss section. Before we add another toy size section, we should remember this will be a race-free section of essentially "shared" memory, where workers can place and move data around for each other, separate from their private stack. We should also remember that the stack addresses follow the bss section, and will move according to the amount we allocate. I think 64KB per core is a decent place to start, because it gives us a fair amount of memory to use. Now when I say "shared" here, I expect one core to be able to write to one of the four sections, but it is able to read the other 3. We absolutely do not want all 4 cores writing to all four sections of this reserved memory. There is no kernel protection against such writes here (well, we ARE the kernel protection I suppose). So now each core can determine the start of its allowed address by finding

allowed_region = core_shared + (core N * 65536)

and they can write up to 64KB-1 bytes to that section, before they would bleed into their neighbor core's area.

boot.Se15dcf9
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38 bl kernel_main
39
40 hang:
41 wfe //wait for event
42 b hang
43
44
45 //entry for all other cores
46 //remember we auto incremented by 4KB, so we can just recalculate the addresses
47 //instead of hard labeling everything
48 //if we change stack size, this also needs to be adjusted
49 //n core stack top = __stack_top + (n * 4096)
50 //each core will enter, compute its value, and initalize
51
52 .global secondary_entry
53 secondary_entry:
54 mrs x1, mpidr_el1 //this tells each core who it is
55 and x1, x1, #0xFF //we want the lowest 8 bits
56
57 init_stack:
58 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
59 add x0, x0, x1, lsl #12 //shifting left
60 mov sp, x0 //move the stack top into each of their own stack pointer registers
61 ldr x2, =core_alive
62 add x2, x2, x1, lsl #3
63 mov x3, #1
64 stlr x3, [x2]
65
66 mov x0, x1
67 bl secondary_main
68 b hang //safe thing to do if secondary_main acts unexpectedly
69
70
71 .global read_core_alive
72 read_core_alive:
73 //we are gonna say we want the caller (core 0) to pass in
74 //the core number in x0
75 //so you wont see it set in this function, we expect it to already be there
76 ldr x2, =core_alive
77 add x2, x2, x0, lsl #3
78 //then we wanna pass the return value back in x0
79 ldar x0, [x2]
80 ret
81
82
83 .section .bss, "aw", @nobits
84 .balign 8
85 core_alive:
86 .skip 32
87+
88+.balign 64
89+core_shared:
90+ .skip 262144

With that extra memory region now, we should modify the counter function and add a proper assembly helper that ensures we only write to the correct region, and that we only write the first 8 bytes.

  1. First, we will declare write_core_counter as an actual function that can be located
  2. Then we load the core_shared address into x2
  3. Then we apply the
allowed_region = core_shared + (core N * 65536)

formula, shifting 16 bits left to multiply by 64KB (2^10 is 1KB, 2^6 is 64, 2^16=64KB) 4. Then we store the value in x1 into the memory address pointed to by x2 5. return

now, it also needs to be declared in main.c so that we know its defined somewhere at link time. Because each worker was just giving their own core number, only taking the step worked, but now we need to pass in the core number so each time the function is ran, it knows what region it is allowed to write into. We can adjust the counter_worker inputs, and also ensure we use the assembly writer to actually write to the assigned memory.

boot.Se0f719a
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38 bl kernel_main
39
40 hang:
41 wfe //wait for event
42 b hang
43
44
45 //entry for all other cores
46 //remember we auto incremented by 4KB, so we can just recalculate the addresses
47 //instead of hard labeling everything
48 //if we change stack size, this also needs to be adjusted
49 //n core stack top = __stack_top + (n * 4096)
50 //each core will enter, compute its value, and initalize
51
52 .global secondary_entry
53 secondary_entry:
54 mrs x1, mpidr_el1 //this tells each core who it is
55 and x1, x1, #0xFF //we want the lowest 8 bits
56
57 init_stack:
58 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
59 add x0, x0, x1, lsl #12 //shifting left
60 mov sp, x0 //move the stack top into each of their own stack pointer registers
61 ldr x2, =core_alive
62 add x2, x2, x1, lsl #3
63 mov x3, #1
64 stlr x3, [x2]
65
66 mov x0, x1
67 bl secondary_main
68 b hang //safe thing to do if secondary_main acts unexpectedly
69
70
71 .global read_core_alive
72 read_core_alive:
73 //we are gonna say we want the caller (core 0) to pass in
74 //the core number in x0
75 //so you wont see it set in this function, we expect it to already be there
76 ldr x2, =core_alive
77 add x2, x2, x0, lsl #3
78 //then we wanna pass the return value back in x0
79 ldar x0, [x2]
80 ret
81
82
83+.global write_core_counter
84+write_core_counter:
85+ ldr x2, =core_shared
86+ add x2, x2, x0, lsl #16
87+ stlr x1, [x2]
88+ ret
89+
90 .section .bss, "aw", @nobits
91 .balign 8
92 core_alive:
93 .skip 32
94
95 .balign 64
96 core_shared:
97 .skip 262144
main.ce0f719a
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39+void write_core_counter(unsigned long core, unsigned long value);
40 //simple continuous function where each core will count by a different increment
41+static void counter_worker(unsigned long core, unsigned long step)
42 {
43 volatile unsigned long counter = 0;
44 for (;;){
45 counter += step;
46+ write_core_counter(core, counter);
47 }
48 }
49
50 void kernel_main(void)
51 {
52 uart_init();
53 //now we need to add in the core reads
54 unsigned long timeout = timer_frequency(); //one seconds worth
55 unsigned long start = timer_ticks(); //get our starting time
56
57 unsigned long core1_alive = 0;
58 unsigned long core2_alive = 0;
59 unsigned long core3_alive = 0;
60
61 while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
62 core1_alive = read_core_alive(1);
63 core2_alive = read_core_alive(2);
64 core3_alive = read_core_alive(3);
65 }
66
67 unsigned long elapsed_ticks = timer_ticks() - start; //how long that took in ticks
68 unsigned long elapsed_ms = (elapsed_ticks * 1000UL) / timeout; //convert to real milliseconds
69
70 uart_puts(core1_alive ? "Core 1: alive\r\n" : "Core 1: startup timed out\r\n");
71 uart_puts(core2_alive ? "Core 2: alive\r\n" : "Core 2: startup timed out\r\n");
72 uart_puts(core3_alive ? "Core 3: alive\r\n" : "Core 3: startup timed out\r\n");
73 uart_puts("Worker startup wait time: ");
74 uart_put_ulong(elapsed_ms);
75 uart_puts(" ms\r\n");
76
77 uart_puts("terminal ready\r\n");
78
79 while (1) {
80 uart_puts("Pi> ");
81 uart_readline(input, sizeof(input));
82 uart_puts("Received: ");
83 uart_puts(input);
84 uart_puts("\r\n");
85 }
86 }
87
88
89 void secondary_main(unsigned long core)
90 {
91
92 switch (core){
93 case 1:
94+ counter_worker(1, 1);
95 break;
96
97 case 2:
98+ counter_worker(2, 2);
99 break;
100
101 case 3:
102+ counter_worker(3, 3);
103 break;
104 }
105 for (;;){
106 __asm__ volatile("wfe");
107 }
108
109 }

Now, we also need to allow core 0 some way to read and interpret those writes and values. Remember, in this ABI, x0 will have argument 1. We want the first argument to be the core number. When we do our add and lsl, we will do it by the appropriate core number. So lets

  1. Declare our read_core_counter function
  2. Load the core_shared address into its x2 register
  3. shift left by the appropriate 16 bits
  4. load the value the x2 address points to into x0
  5. and use that value as the return value We need to declare this function as well in main.c
boot.S7e127b6
1 .section .text.boot, "ax" //we need an executable code section
2
3 .global _start //this is our entry point
4
5 //exclusive core 0 entry now
6 _start:
7 //this is actuall vestigial now but its acceptable to me preoptimization
8 //we should do this before initalizing the stack
9 mrs x1, mpidr_el1 //this tells each core who it is
10 and x1, x1, #0xFF //we want the lowest 8 bits
11
12 ldr x0, =__stack_top
13 mov sp, x0
14 //core 0 is special, it is one that does bss clearing and initalized the peripherals
15 ldr x0, =__bss_start
16 ldr x1, =__bss_end
17
18 clear_bss:
19 cmp x0, x1
20 b.hs bss_done //branch to bss_done if x0 >= x1
21
22 strb wzr, [x0], #1 //store 0 at x0 mem address, inc x0 by 1
23 b clear_bss
24
25 bss_done:
26 ldr x0, =secondary_entry //entry code location
27 mov x1, #0xe0 //core 1 slot locatoin
28 str x0, [x1] //put x0 value in 0xe0
29
30 mov x1, #0xe8
31 str x0, [x1]
32
33 mov x1, #0xf0
34 str x0, [x1]
35
36 dsb sy //barrier
37 sev //send event
38 bl kernel_main
39
40 hang:
41 wfe //wait for event
42 b hang
43
44
45 //entry for all other cores
46 //remember we auto incremented by 4KB, so we can just recalculate the addresses
47 //instead of hard labeling everything
48 //if we change stack size, this also needs to be adjusted
49 //n core stack top = __stack_top + (n * 4096)
50 //each core will enter, compute its value, and initalize
51
52 .global secondary_entry
53 secondary_entry:
54 mrs x1, mpidr_el1 //this tells each core who it is
55 and x1, x1, #0xFF //we want the lowest 8 bits
56
57 init_stack:
58 ldr x0, =__stack_top //each core has seperate registers, cant rely on shared understanding
59 add x0, x0, x1, lsl #12 //shifting left
60 mov sp, x0 //move the stack top into each of their own stack pointer registers
61 ldr x2, =core_alive
62 add x2, x2, x1, lsl #3
63 mov x3, #1
64 stlr x3, [x2]
65
66 mov x0, x1
67 bl secondary_main
68 b hang //safe thing to do if secondary_main acts unexpectedly
69
70
71 .global read_core_alive
72 read_core_alive:
73 //we are gonna say we want the caller (core 0) to pass in
74 //the core number in x0
75 //so you wont see it set in this function, we expect it to already be there
76 ldr x2, =core_alive
77 add x2, x2, x0, lsl #3
78 //then we wanna pass the return value back in x0
79 ldar x0, [x2]
80 ret
81
82
83 .global write_core_counter
84 write_core_counter:
85 ldr x2, =core_shared
86 add x2, x2, x0, lsl #16
87 stlr x1, [x2]
88 ret
89
90+.global read_core_counter
91+read_core_counter:
92+ ldr x2, =core_shared
93+ add x2, x2, x0, lsl #16
94+ ldar x0, [x2]
95+ ret
96+
97 .section .bss, "aw", @nobits
98 .balign 8
99 core_alive:
100 .skip 32
101
102 .balign 64
103 core_shared:
104 .skip 262144

Now, we dont want to clutter our command line real bad with these values coming at us all the time. It would be better if our shell only printed these when we input something like "core-counters". For that, we need some way to tell if our input value into the shell matches some value we predefined in our exe. Lets add a very simple strings_equal function, and lets add the printing our on console so we can see it.

main.c7e127b6
1 #include "uart.h"
2
3 /* Use .bss rather than our small boot stack for input. */
4 static char input[256];
5
6 unsigned long read_core_alive(unsigned long core);
7
8
9 static unsigned long timer_ticks(void){
10 unsigned long ticks;
11
12
13 __asm__ volatile(
14 "isb\n"
15 "mrs %0, cntpct_el0" //read counter timer register
16 : "=r"(ticks)
17 :
18 : "memory"
19 );
20 return ticks;
21
22 }
23
24 static unsigned long timer_frequency(void)
25 {
26
27 unsigned long frequency;
28
29
30 __asm__ volatile(
31
32 "mrs %0, cntfrq_el0"
33 : "=r"(frequency)
34 );
35 return frequency;
36 }
37
38
39+static int strings_equal(const char *a, const char *b)
40+{
41+ while (*a && *a == *b) {
42+ a++;
43+ b++;
44+ }
45+
46+ return *a == *b;
47+}
48+
49 void write_core_counter(unsigned long core, unsigned long value);
50+unsigned long read_core_counter(unsigned long core);
51 //simple continuous function where each core will count by a different increment
52 static void counter_worker(unsigned long core, unsigned long step)
53 {
54 volatile unsigned long counter = 0;
55 for (;;){
56 counter += step;
57 write_core_counter(core, counter);
58 }
59 }
60
61 void kernel_main(void)
62 {
63 uart_init();
64 //now we need to add in the core reads
65 unsigned long timeout = timer_frequency(); //one seconds worth
66 unsigned long start = timer_ticks(); //get our starting time
67
68 unsigned long core1_alive = 0;
69 unsigned long core2_alive = 0;
70 unsigned long core3_alive = 0;
71
72 while(!(core1_alive && core2_alive && core3_alive) && ((timer_ticks()-start) < timeout)){
73 core1_alive = read_core_alive(1);
74 core2_alive = read_core_alive(2);
75 core3_alive = read_core_alive(3);
76 }
77
78 unsigned long elapsed_ticks = timer_ticks() - start; //how long that took in ticks
79 unsigned long elapsed_ms = (elapsed_ticks * 1000UL) / timeout; //convert to real milliseconds
80
81 uart_puts(core1_alive ? "Core 1: alive\r\n" : "Core 1: startup timed out\r\n");
82 uart_puts(core2_alive ? "Core 2: alive\r\n" : "Core 2: startup timed out\r\n");
83 uart_puts(core3_alive ? "Core 3: alive\r\n" : "Core 3: startup timed out\r\n");
84 uart_puts("Worker startup wait time: ");
85 uart_put_ulong(elapsed_ms);
86 uart_puts(" ms\r\n");
87
88 uart_puts("terminal ready\r\n");
89
90 while (1) {
91 uart_puts("Pi> ");
92 uart_readline(input, sizeof(input));
93+ if (strings_equal(input, "core-counters"))
94+ {
95+ uart_puts("Core 1 counter: ");
96+ uart_put_ulong(read_core_counter(1));
97+ uart_puts("\r\n");
98+
99+ uart_puts("Core 2 counter: ");
100+ uart_put_ulong(read_core_counter(2));
101+ uart_puts("\r\n");
102+
103+ uart_puts("Core 3 counter: ");
104+ uart_put_ulong(read_core_counter(3));
105+ uart_puts("\r\n");
106+
107+ }
108 uart_puts("Received: ");
109 uart_puts(input);
110 uart_puts("\r\n");
111 }
112 }
113
114
115 void secondary_main(unsigned long core)
116 {
117
118 switch (core){
119 case 1:
120 counter_worker(1, 1);
121 break;
122
123 case 2:
124 counter_worker(2, 2);
125 break;
126
127 case 3:
128 counter_worker(3, 3);
129 break;
130 }
131 for (;;){
132 __asm__ volatile("wfe");
133 }
134
135 }

Alright, now, we haven't validated any of this yet because I wrote it between CDL and JFK, and pulling microcomputers and wires out on planes is frowned upon. Lets compile, examine, and test this

terminal
1 clang --target=aarch64-none-elf -c platform/multicore/boot.S -o build/multicore/boot.o
2
3
4 clang --target=aarch64-none-elf -std=c11 -ffreestanding -fno-builtin -fno-stack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-pic -O0 -Wall -Wextra -Iplatform/multicore -c platform/multicore/main.c -o build/multicore/main.o
5
6
7 clang --target=aarch64-none-elf -std=c11 -ffreestanding -fno-builtin -fno-stack-protector -fno-unwind-tables -fno-asynchronous-unwind-tables -fno-pic -O0 -Wall -Wextra -Iplatform/multicore -c platform/multicore/uart.c -o build/multicore/uart.o
8
9 ld.lld -m aarch64elf -T platform/multicore/linker.ld -o build/multicore/kernel8.elf build/multicore/boot.o build/multicore/main.o build/multicore/uart.o
10
11 llvm-objcopy -O binary build/multicore/kernel8.elf build/multicore/kernel8.img
12
13
Development screenshot: Pasted image 20261106121644.png

okay lets inspect it

terminal
1 llvm-objdump -d build/multicore/kernel8.elf

Our timer functions look correct:

Development screenshot: Pasted image 20261106121827.png

counter worker and its dispatcher look good:

Development screenshot: Pasted image 20261106121911.png
Development screenshot: Pasted image 20261106121937.png

and our shared memory functions look correct for reading and writing:

Development screenshot: Pasted image 20261106122043.png

Beautiful. Let's actually run it and see what breaks. Remember to copy the newly compiled kernel8.img to the raspberry pi's SD card.

Development screenshot: Pasted image 20261106122636.png

setup or COM session

PuTTY shows all three workers alive and a startup wait of 0 ms.

alright, i could have used something with finer resolution than a millisecond. That's not very helpful. however, we have what we were looking for.

PuTTY core-counters command returns a distinct counter from each worker.

Okay, so we now effectively have the boot sequence, core initialization, and how we assign different cores different memory regions and functions. Now we can go into more useful and specialized functions.