Computer architecture studies how a computer is organized and how software instructions are executed by hardware.

A useful mental model is:

C / C++ / ML Model
       |
       v
    Compiler
       |
       v
      ISA
       |
       v
+----------------------+
|        CPU           |
|  Registers           |
|  ALU                 |
|  Control Unit        |
|  Pipeline            |
|  Cache               |
+----------+-----------+
           |
           v
      Memory System
   +-------+-------+
   | Cache |  RAM  |
   +-------+-------+
           |
           v
        Storage

For AI systems, there may also be:

                 +------ CPU ------+
                 |                 |
Program --> ISA -+                 +--> RAM
                 |                 |
                 +------ NPU ------+
                        |
                   AI workloads

2. CPU

The CPU (Central Processing Unit) executes instructions.

Important components:

2.1 ALU

Arithmetic Logic Unit

Performs operations such as:

addition
subtraction
comparisons
AND / OR / XOR
shifts

Example:

A = 10
B = 20

ADD A, B

Result = 30

2.2 Control Unit

Controls the execution of instructions.

It determines:

which instruction is being executed
which registers are involved
which operation the ALU should perform
when memory should be accessed

2.3 Registers

Registers are very small, very fast storage locations inside the CPU.

Examples on x86-64:

RAX
RBX
RCX
RDX

RISC-V:

x0 ... x31

Registers are much faster than RAM.

3. Program Counter (PC)

The Program Counter contains the address associated with the next instruction to fetch / execute.

Example:

Address      Instruction
0x1000       LOAD R1, [0x2000]
0x1004       LOAD R2, [0x2004]
0x1008       ADD  R3, R1, R2
0x100C       STORE [0x3000], R3

Initially:

PC = 0x1000

Then execution proceeds to later instruction addresses.

A branch or jump can change the PC.

4. Instruction Cycle

A simplified instruction cycle is:

FETCH
  |
  v
DECODE
  |
  v
EXECUTE
  |
  v
MEMORY ACCESS
  |
  v
WRITE BACK

Fetch

Get the instruction from memory/cache.

Decode

Determine:

operation
source registers
destination register
immediate values
addressing information

Execute

Perform the operation.

Memory Access

Read/write memory when required.

Write Back

Store the result into a register.

5. Registers vs Cache vs RAM

A simplified hierarchy:

Fastest
  |
  v
Registers
L1 Cache
L2 Cache
L3 Cache
RAM
SSD / Storage
  |
  v
Slowest

As we go downward:

capacity generally increases
latency generally increases
cost per bit generally decreases

The exact implementation differs between CPUs.

6. Memory Hierarchy

Why not make all memory as fast as registers?

Because fast memory is expensive and difficult to scale to large capacities.

Therefore systems use multiple levels:

The goal is to keep frequently used data close to the CPU.

7. Cache

A cache is a small, fast memory that stores copies of recently or frequently accessed data/instructions.

CPU --> L1 --> L2 --> L3 --> RAM

7.1 Cache Hit

Requested data is already in cache.

CPU --> Cache --> DATA

Fast.

7.2 Cache Miss

Requested data is not in cache.

CPU --> Cache --MISS--> RAM
                     |
                     v
                 bring data
                     |
                     v
                   Cache
                     |
                     v
                    CPU

A cache miss is more expensive than a cache hit.

8. Locality of Reference

Caches work well because programs tend to exhibit locality.

8.1 Temporal Locality

If something was accessed recently, it is likely to be accessed again soon.

Example:

int x = 10;

for (int i = 0; i < 1000; i++) {
    sum += x;
}

x is repeatedly used.

8.2 Spatial Locality

If one memory location is accessed, nearby locations are likely to be accessed.

Example:

for (int i = 0; i < 1000; i++) {
    sum += arr[i];
}

The accesses are:

arr[0]
arr[1]
arr[2]
arr[3]
...

This is cache-friendly.

9. Cache Lines

Caches normally move data in blocks called cache lines rather than one byte at a time.

Suppose:

arr[0] arr[1] arr[2] arr[3]

are located close together.

When arr[0] is loaded, the cache may fetch a whole block containing nearby elements.

Therefore:

for (int i = 0; i < N; i++)
    sum += arr[i];

often has good spatial locality.

A random access pattern may have much worse locality:

sum += arr[random_index[i]];

10. L1, L2, L3 Cache

A common conceptual hierarchy:

L1

smallest
fastest
usually private to a core
often split into instruction cache and data cache

L2

larger than L1
slower than L1
commonly associated with a core

L3

larger
slower than L2
often shared among several cores

Conceptually:

Core 0 --> L1 --> L2 ----+
                          |
Core 1 --> L1 --> L2 ----+--> L3
                          |
Core 2 --> L1 --> L2 ----+

Actual CPU layouts vary.

11. Instruction Cache vs Data Cache

A CPU needs both:

instructions
data

Therefore an L1 cache is often split conceptually into:

L1
+----------------------+
| Instruction Cache    |
| Data Cache           |
+----------------------+

Instruction cache stores instructions.

Data cache stores data.

12. Cache Mapping

There are three classic cache organizations.

12.1 Direct Mapped

Each memory block has exactly one possible cache location.

Advantages

simple
fast

Disadvantage

conflict misses can occur easily

12.2 Fully Associative

A block can be placed anywhere in the cache.

Advantage

fewer placement restrictions

Disadvantage

more hardware complexity

12.3 Set Associative

A compromise.

For a 4-way set-associative cache:

a block maps to one set
it can occupy any of 4 ways in that set

Set 0: [way 0][way 1][way 2][way 3]
Set 1: [way 0][way 1][way 2][way 3]
...

13. Cache Replacement

When a set is full, something must be evicted.

Common replacement policies:

LRU — Least Recently Used
FIFO — First In First Out
Random

LRU Example

Suppose a 2-entry cache sees:

A B A C

After A B A:

A = recently used
B = less recently used

When C arrives, B is a candidate for eviction under LRU.

14. Cache Write Policies

14.1 Write Through

A write updates both cache and lower-level memory.

CPU
 |
 +--> Cache
 |
 +--> RAM

Advantage

RAM is kept up to date.

Disadvantage

More memory traffic.

14.2 Write Back

A write updates the cache first.

RAM is updated later, often when the cache line is evicted.

Advantage

Fewer immediate writes to lower memory.

Disadvantage

More bookkeeping is required.

15. Cache-Friendly Programming

This is an important connection between programming and architecture.

Good locality:

for (int i = 0; i < N; i++)
    sum += arr[i];

Poorer locality may occur with:

for (int i = 0; i < N; i++)
    sum += arr[random_index[i]];

Why?

The first pattern accesses neighboring memory.

The second can jump around memory.

16. ISA

ISA = Instruction Set Architecture

It defines the interface between software and the processor.

It describes things such as:

instructions
registers
data types
instruction formats
addressing modes
architectural state
memory behavior

Examples:

x86-64
ARM
RISC-V

Think of the ISA as the contract software relies on.

17. ISA vs Microarchitecture

This is a very common interview/OA distinction.

ISA

What the processor provides.

Microarchitecture

How the processor implements that ISA.

Example:

          ISA
           |
   ----------------
   |              |
Implementation A  Implementation B

Two processors can implement the same ISA using very different internal hardware.

18. RISC vs CISC

RISC

Reduced Instruction Set Computer

Typical characteristics:

simpler instructions
regular instruction formats
load/store style
many general-purpose registers
simpler decoding

Examples:

RISC-V
ARM

CISC

Complex Instruction Set Computer

Traditional example:

x86

Typical characteristics:

richer instruction set
more complex instructions
x86 has variable-length instruction encoding

Important trap

Do NOT conclude:

RISC = fast
CISC = slow

Modern processors are much more sophisticated than this simple comparison.

19. Load/Store Architecture

In a load/store design, arithmetic usually operates on registers.

Example:

LOAD R1, [A]
LOAD R2, [B]
ADD  R3, R1, R2
STORE [C], R3

The CPU does not usually perform arbitrary arithmetic directly on memory operands.

This is especially important when thinking about RISC-V.

20. RISC-V

RISC-V is an open instruction set architecture.

Basic concepts:

x0 ... x31

Standard RISC-V has 32 integer registers in the common base ISA.

x0

x0 is the zero register:

x0 = 0

Writes to it do not create a normal persistent nonzero value.

21. SoC

SoC = System on Chip

A SoC integrates many components into a single chip.

Possible components:

+--------------------------------------+
|                  SoC                 |
|                                      |
| CPU cores                            |
| GPU / NPU                            |
| Memory controller                    |
| DMA                                  |
| I/O                                  |
| Security blocks                      |
| Other accelerators                   |
+--------------------------------------+

This is common in phones, embedded systems, edge AI devices, etc.

22. CPU vs GPU vs NPU

CPU

General-purpose processor.

Good for:

operating systems
control logic
branching
varied workloads

GPU

Highly parallel architecture.

Good for:

graphics
matrix-heavy workloads
large data-parallel workloads

NPU / AI Accelerator

Specialized hardware for neural-network computations.

Typical operations include:

matrix multiplication
convolution
multiply-accumulate operations

23. SIMD

SIMD = Single Instruction, Multiple Data

Instead of executing:

one operation at a time, SIMD can conceptually do:

A = [1 2 3 4]
B = [5 6 7 8]

A + B
   |
   v
[6 8 10 12]

One instruction operates on multiple data elements.

This is data-level parallelism.

24. Vector Processing

Vector processing extends the same idea:

vector = [a0 a1 a2 a3 a4 a5 a6 a7]

An instruction can operate across several elements.

Useful for:

ML
image processing
signal processing
scientific computing

The exact implementation differs across architectures.

25. Matrix Acceleration

Neural networks perform large matrix operations such as:

Specialized matrix hardware can perform many multiply-accumulate operations in parallel.

This can provide better:

throughput
energy efficiency
hardware utilization

than using a general-purpose CPU for the same workload.

26. Pipelines

A pipeline overlaps different stages of multiple instructions.

Without pipelining:

I1: FETCH -> DECODE -> EXECUTE -> MEM -> WRITE
I2:                                FETCH -> ...

With pipelining:

Cycle   1   2   3   4   5
I1      F   D   E   M   W
I2          F   D   E   M
I3              F   D   E
I4                  F   D

Multiple instructions can be in different stages simultaneously.

27. Throughput vs Latency

A very important distinction.

Latency

How long one operation takes.

Throughput

How many operations can be completed per unit time.

Pipelining primarily improves throughput.

It does not necessarily reduce the latency of an individual instruction.

Example

If an instruction has 5 pipeline stages:

F D E M W

One instruction may still take about 5 cycles to pass through the whole pipeline.

But after the pipeline fills, ideally a new instruction can finish every cycle.

28. Pipeline Hazards

There are three major types.

28.1 Structural Hazard

Two operations need the same hardware resource at the same time.

Example.

Instruction 1: Execute/Memory write stage
Instruction 2: Fetch stage

28.2 Data Hazard

An instruction depends on another instruction.

Example:

I1: R1 = R2 + R3
I2: R4 = R1 + R5

I2 needs the value produced by I1.

28.3 Control Hazard

Caused by branches and jumps.

Example:

if (x > 10)
    foo();
else
    bar();

The CPU needs to know which path will be executed.

29. Data Hazard Types

The classic names are:

RAW — Read After Write

I1: WRITE R1
I2: READ R1

I2 needs the result of I1.

This is the most intuitive dependency.

WAR — Write After Read

I1: READ R1
I2: WRITE R1

WAW — Write After Write

I1: WRITE R1
I2: WRITE R1

The exact hazards that can occur depend on the pipeline and execution model.

30. Handling Hazards

Stall

Pause pipeline progress.

I1
I2
STALL
STALL
I3

Forwarding / Bypassing

Send a result directly to a later stage without waiting for normal register write-back.

Conceptually:

I1 result
    |
    v
forwarding path
    |
    v
I2

Compiler Scheduling

The compiler may reorder independent instructions to reduce stalls.

This is one of the major places where compiler design and architecture interact.

31. Branch Prediction

Branches create uncertainty:

if (condition)
    A;
else
    B;

The processor predicts which path will be taken.

If prediction is correct:

continue execution

If prediction is wrong:

discard speculative work
fetch/execute correct path

A wrong prediction is a branch misprediction and costs performance.

32. Speculative Execution

A CPU may execute instructions before it knows for certain that they are on the correct path.

Conceptually:

branch
  |
  v
predict
  |
  v
execute predicted path
  |
  v
verify

Correct prediction:

keep the work

Wrong prediction:

discard speculative work

33. Superscalar CPUs

A superscalar processor can issue multiple instructions per cycle if enough hardware and independent instructions are available.

Example:

ADD R1, R2, R3
MUL R4, R5, R6

If independent, the processor may execute them in parallel using different execution resources.

34. Out-of-Order Execution

Processors can sometimes execute independent instructions before earlier long-running instructions finish.

Example:

I1: long memory operation
I2: ADD R1,R2,R3
I3: MUL R4,R5,R6

Potentially:

I1 ------------------------>
     I2 -->
     I3 -->

This increases instruction-level parallelism.

The CPU still has to preserve the program's required architectural behavior.

35. ILP

Instruction-Level Parallelism

The ability to execute multiple independent instructions concurrently.

Sources include:

pipelining
superscalar execution
out-of-order execution

36. Multicore Processors

A multicore CPU contains multiple processing cores.

+-------------------------+
| CPU package             |
|                         |
| Core 0                  |
| Core 1                  |
| Core 2                  |
| Core 3                  |
+-------------------------+

Different threads can execute on different cores.

37. Parallelism Levels

Instruction-Level Parallelism (ILP)

Multiple instructions.

Data-Level Parallelism (DLP)

Same operation on multiple data elements.

SIMD is a common example.

Thread-Level Parallelism (TLP)

Multiple threads.

Task-Level Parallelism

Different tasks execute concurrently.

38. Flynn's Taxonomy

Four classic categories:

SISD

Single Instruction, Single Data

one instruction
     |
     v
one data stream

SIMD

Single Instruction, Multiple Data

one instruction
     |
     +--> data 1
     +--> data 2
     +--> data 3
     +--> data 4

MISD

Multiple Instruction, Single Data

Rare.

MIMD

Multiple Instruction, Multiple Data

Common in multicore and multiprocessor systems.

39. Cache Coherence

Multicore CPUs may have multiple caches.

Example:

Core 1 --> cache --> X = 10
Core 2 --> cache --> X = 10

Suppose Core 1 changes:

X = 20

Core 2's cached copy must not incorrectly remain permanently inconsistent.

This is the cache coherence problem.

Protocols such as MESI are used in many systems.

40. MESI

Classic states:

M = Modified
E = Exclusive
S = Shared
I = Invalid

You do not necessarily need to memorize every state transition for a basic OA, but understand that coherence protocols manage which core owns or shares cache lines.

Suppose:

Thread 1 modifies A
Thread 2 modifies B

But:

A and B are on the same cache line

Even though the threads use different variables, cache coherence may cause unnecessary cache-line invalidations.

This performance problem is called false sharing.

42. Virtual Memory

Programs typically use virtual addresses.

Conceptually:

Program
   |
   v
Virtual Address
   |
   v
MMU
   |
   v
Physical Address
   |
   v
RAM

Benefits include:

process isolation
virtual address spaces
paging
flexible memory management

43. Paging

Virtual memory is divided into pages.

Physical memory is divided into frames.

A page table maps:

Virtual Page Number --> Physical Frame Number

Example:

Virtual:
Page 5 | Offset 100

Page table:
5 --> Frame 20

Physical:
Frame 20 | Offset 100

The offset does not change during this translation.

44. TLB

TLB = Translation Lookaside Buffer

It caches recent virtual-to-physical address translations.

Virtual Address
       |
       v
      TLB
    /     \
  hit     miss
  |        |
  v        v
physical  Page Table
address      |
             v
         translation

TLB Hit

Translation is found quickly.

TLB Miss

The system performs a page-table lookup.

45. Page Fault

A page fault occurs when a needed virtual page cannot currently be accessed through the expected resident physical-memory mapping.

The OS may need to handle the situation by bringing data into RAM from storage or otherwise resolving the mapping.

Important

A cache miss and a page fault are not the same thing.

Cache miss --> cache hierarchy issue
Page fault  --> virtual-memory / OS issue

46. Endianness

Endianness determines byte order for multi-byte values.

Suppose:

0x12345678

Big Endian

Most significant byte first:

12 34 56 78

Little Endian

Least significant byte first:

78 56 34 12

x86-64 is little-endian.

47. Alignment

Data often has preferred alignment boundaries.

For example, a 4-byte integer may ideally be aligned to an address divisible by 4.

Misaligned accesses may:

require extra work
be slower
or be restricted on some architectures

48. Structure Padding

Consider:

struct A {
    char c;
    int x;
};

You might initially expect:

1 + 4 = 5 bytes

But alignment can insert padding:

c
padding
padding
padding
x x x x

Therefore:

sizeof(A)

can be larger than 5.

This is a common C/C++ architecture question.

49. Memory Bandwidth vs Latency

Latency

How long it takes before the requested data becomes available.

Bandwidth

How much data can be transferred per unit time.

Analogy:

Latency  = how long before the first delivery arrives
Bandwidth = how many deliveries can arrive per hour

A system can have high bandwidth and still have noticeable latency.

50. Compute-Bound vs Memory-Bound

Compute-Bound

Performance is primarily limited by computational units.

CPU / GPU / NPU
       |
       v
   bottleneck

Memory-Bound

Performance is primarily limited by moving data.

Cache / RAM / interconnect
           |
           v
       bottleneck

AI workloads can become memory-bound because huge tensors must be moved between memory levels.

51. DMA

DMA = Direct Memory Access

Without DMA:

Device --> CPU --> RAM

The CPU is involved in copying data.

With DMA:

Device --> DMA --> RAM

The CPU typically configures the transfer:

source
destination
size

Then the DMA engine performs the bulk transfer.

The device/DMA may interrupt the CPU when the transfer completes.

52. Why DMA Matters for AI Accelerators

Suppose an NPU needs a large tensor.

Instead of having the CPU copy the entire tensor manually:

RAM
 |
 v
CPU
 |
 v
NPU

DMA can help:

RAM
 |
 v
DMA
 |
 v
NPU local memory

This reduces unnecessary CPU involvement and can improve system throughput.

53. Memory-Mapped I/O

In memory-mapped I/O, device registers appear in the address space.

The CPU can access a device register using memory operations.

Conceptually:

CPU
 |
 | read/write address
 v
Device Register

This is common in embedded systems.

54. Interrupts

A device can notify the CPU when something needs attention.

Example:

Network device receives packet
            |
            v
        Interrupt
            |
            v
           CPU
            |
            v
    Interrupt Handler

This avoids constant polling.

55. Polling vs Interrupts

Polling

CPU repeatedly checks:

"Ready?"
"Ready?"
"Ready?"

Interrupt

Device notifies the CPU when needed.

Polling

Can waste CPU cycles.

Interrupts

Avoid constant checking, but have handling overhead.

56. Addressing Modes

An instruction needs a way to identify its operands.

Common conceptual modes:

Immediate

Value is inside the instruction.

ADD R1, R2, #5

Register

Operand is in a register.

ADD R1, R2, R3

Indirect

A register contains an address.

Base + Offset

Useful for arrays and structures.

address = base_register + offset

57. Calling Convention and ABI

When compiled functions interact, they must agree on conventions.

A calling convention can specify:

where arguments are passed
where return values are placed
which registers the caller/callee must preserve
how the stack is used

ABI = Application Binary Interface

The ABI describes binary-level conventions that allow separately compiled components to work together.

This matters to compilers and runtimes.

58. Stack and Heap

A process's virtual address space can be viewed conceptually as:

High addresses
+------------------+
| Stack            |
+------------------+
|                  |
| Shared libraries |
|                  |
+------------------+
| Heap             |
+------------------+
| Data             |
+------------------+
| Code / Text      |
+------------------+
Low addresses

Exact layouts vary by OS and architecture.

Stack

Commonly used for:

function-call frames
local variables
return state

Heap

Commonly used for dynamically allocated objects.

Both ultimately use the virtual-memory system.

59. Function Call — Architecture View

Suppose:

foo(a, b);

Conceptually:

prepare arguments
      |
      v
call / jump to foo
      |
      v
execute foo
      |
      v
return result
      |
      v
continue caller

Registers, stack, PC, and the calling convention all participate in this process.

60. CPU Performance Equation

A fundamental equation:

[ CPU\ Time = Instruction\ Count \times CPI \times Clock\ Cycle\ Time ]

Since:

[ Clock\ Cycle\ Time = \frac{1}{Clock\ Rate} ]

we get:

[ CPU\ Time = \frac{Instruction\ Count \times CPI}{Clock\ Rate} ]

Where:

Instruction Count (IC) = number of executed instructions
CPI = Cycles Per Instruction
Clock Rate = cycles per second

61. Performance Example

Suppose:

Instruction Count = 1 billion
CPI = 2
Clock Rate = 2 GHz

Then:

[ CPU\ Time = \frac{10^9 \times 2}{2 \times 10^9} ]

[ CPU\ Time = 1\ second ]

62. Higher Clock Speed Does Not Automatically Mean Faster

CPU A:

3 GHz
CPI = 1

CPU B:

4 GHz
CPI = 2

For the same instruction count:

[ T_A = \frac{IC}{3G} ]

[ T_B = \frac{2IC}{4G} = \frac{IC}{2G} ]

So CPU A is faster despite having a lower clock frequency.

Exam takeaway

Do not use GHz alone to compare processors.

63. Amdahl's Law

Amdahl's Law describes the maximum overall speedup obtained by optimizing only part of a program.

[ Speedup = \frac{1} {(1-f)+\frac{f}{s}} ]

Where:

(f) = fraction of execution time improved
(s) = speedup of the improved fraction

64. Amdahl Example

Suppose 20% of the program can be made infinitely fast.

Then:

[ f=0.2 ]

and:

[ s \rightarrow \infty ]

Therefore:

[ Speedup = \frac{1}{0.8} = 1.25 ]

Lesson

Optimizing only a small part of a program cannot give unlimited total speedup.

65. Prefetching

Prefetching attempts to load data before the CPU actually needs it.

Example:

CPU currently accesses arr[10]

Hardware/compiler predicts:

arr[11]
arr[12]
arr[13]

will be needed

So data can be fetched into cache early.

This can reduce the impact of memory latency.

66. Cache Coherence vs Memory Consistency

Do not confuse them.

Cache Coherence

Concerned with consistency of cached copies of the same memory location.

Memory Consistency

Concerned with the ordering/visibility rules for memory operations.

They are related, but not identical concepts.

67. Atomic Operations

Consider:

counter++;

At a conceptual machine level this can involve:

LOAD counter
ADD 1
STORE counter

Two threads can interfere.

An atomic operation makes a supported operation indivisible with respect to the synchronization model.

This is important for concurrent programming.

68. Mutex vs Atomic

Mutex

Used to protect a critical section.

lock
  |
critical section
  |
unlock

Atomic

Used for particular indivisible operations.

Atomic operations can sometimes avoid the overhead of locking for simple shared-state operations.

69. AI and Architecture

Neural-network inference can be thought of as:

Model
  |
  +--> computation
  |
  +--> memory movement
  |
  +--> parallelism

Hardware must optimize all three.

Useful hardware:

CPU
GPU
NPU
SIMD/vector units
matrix engines
cache
local SRAM
DMA

70. Quantization

Quantization represents values using lower precision.

Example:

FP32
  |
  v
INT8

Potential benefits:

less memory
lower memory bandwidth
lower storage requirements
potentially faster execution
potentially lower power

Potential disadvantage:

precision/accuracy can decrease

71. Post-Training Quantization vs QAT

Post-Training Quantization

Quantize a trained model after training.

Quantization-Aware Training (QAT)

Training is performed while accounting for quantization effects.

For OA purposes, know the basic difference.

72. Operator Fusion

Suppose:

Operation A
     |
     v
Operation B

Naively:

A
 |
 v
write intermediate tensor to memory
 |
 v
read intermediate tensor
 |
 v
B

Fusion can combine operations so the intermediate result stays closer to the computation.

Benefits can include:

less memory traffic
lower latency
less overhead
potentially lower energy usage

73. Tiling

Suppose we have a huge matrix.

Instead of processing the whole matrix at once:

+-----------------------+
|                       |
|      HUGE MATRIX      |
|                       |
+-----------------------+

break it into tiles:

+-----+-----+-----+
| T   | T   | T   |
+-----+-----+-----+
| T   | T   | T   |
+-----+-----+-----+
| T   | T   | T   |
+-----+-----+-----+

Why?

Because smaller pieces may fit better into:

cache
local accelerator memory
registers

This improves locality and can reduce expensive memory traffic.

74. Layout Transformation

A tensor can be stored in different layouts.

For example:

NCHW
NHWC

Different hardware may prefer different layouts.

The compiler/runtime can transform the layout to improve:

memory access pattern
vectorization
accelerator utilization
cache behavior

75. Memory Planning

AI runtimes manage the memory used by tensors and intermediate results.

The goal can be to:

reuse buffers
reduce peak memory usage
avoid unnecessary copies
keep frequently used data in appropriate memory

76. Compiler + Architecture

This is particularly important for a compiler/runtime role.

Think:

High-Level Program / AI Model
              |
              v
              IR
              |
              v
       Compiler Optimizations
              |
      +-------+-------+
      |       |       |
      v       v       v
    Cache   SIMD    Fusion
      |       |       |
      +-------+-------+
              |
              v
          Lowering
              |
              v
       Target Hardware

The compiler must understand what the hardware can do efficiently.

77. Example: Simple Loop

Consider:

for (int i = 0; i < N; i++)
    C[i] = A[i] + B[i];

Architecture observations

Arrays are contiguous.
Access pattern is sequential.
Good spatial locality.
Data can potentially be vectorized.
Multiple iterations are independent.
SIMD can process multiple elements together.

Conceptually:

A: [a0 a1 a2 a3]
B: [b0 b1 b2 b3]
       |
       v
   SIMD ADD
       |
       v
C: [c0 c1 c2 c3]

78. Example: Why Matrix Multiplication is Optimized

Naive matrix multiplication:

[ C_{ij} = \sum_k A_{ik}B_{kj} ]

This performs many arithmetic operations and memory accesses.

Architecture-aware optimization can use:

tiling
vectorization
cache blocking
parallel execution
accelerator-specific instructions

The goal is to keep useful data close to the compute units.

79. Common OA Traps

Trap 1

Higher GHz = always faster

False.

CPI and instruction count matter too.

Trap 2

Pipelining makes one instruction execute instantly

False.

Pipelining primarily improves throughput.

Trap 3

Cache is slower than RAM

False.

Cache is faster and smaller.

Trap 4

TLB stores actual program data

False.

TLB stores address translations.

Trap 5

Cache miss = page fault

False.

Different mechanisms.

Trap 6

SIMD means multiple instructions

False.

SIMD = one instruction, multiple data.

Trap 7

RISC-V is a CPU

Not exactly.

RISC-V is an ISA.

Trap 8

DMA means CPU does not participate at all

Not necessarily.

The CPU often configures the transfer; DMA performs the bulk movement.

Trap 9

RISC always means faster than CISC

Too simplistic.

Trap 10

Array traversal is always slow because RAM is slow

Not necessarily.

Sequential access benefits from locality and caching.

80. Fast Comparison Tables

Memory

Level	Speed	Capacity	Typical role
Registers	Highest	Tiny	Active CPU values
L1	Very high	Small	Hot instructions/data
L2	High	Larger	Cache
L3	Lower	Larger	Shared cache
RAM	Much lower	Large	Main memory
SSD	Much lower	Very large	Persistent storage

Parallelism

Type	Idea	Example
ILP	Multiple instructions	Superscalar CPU
DLP	Same operation on many data	SIMD
TLP	Multiple threads	Multicore
Task-level	Different tasks	Parallel services

Memory Concepts

Concept	Meaning
Cache hit	Data is already in cache
Cache miss	Data not in cache
TLB hit	Translation is cached
TLB miss	Translation needs lookup
Page fault	Required page needs OS-level handling
Temporal locality	Reuse recently accessed data
Spatial locality	Access nearby data

81. Exam-Style Questions

Q1

Which is generally fastest?

A. RAM
B. L3 cache
C. L1 cache
D. SSD

Answer: C

Registers would be even faster than L1.

Q2

A program repeatedly accesses the same variable. Which locality?

A. Spatial
B. Temporal
C. Structural
D. Sequential

Answer: B

Q3

A program traverses an array sequentially. Which locality?

A. Temporal only
B. Spatial
C. No locality
D. Control locality

Answer: B

Q4

What does TLB cache?

A. CPU instructions
B. Disk blocks
C. Virtual-to-physical translations
D. Branch targets only

Answer: C

Q5

Which is a control hazard?

A. Two instructions need the same register value
B. Branch direction is not yet known
C. Two instructions need the same hardware unit
D. Cache line is too large

Answer: B

Q6

SIMD stands for:

A. Single Instruction, Multiple Data
B. Single Instruction, Multiple Devices
C. Shared Instruction, Multiple Data
D. Sequential Instruction, Multiple Data

Answer: A

Q7

Which technique allows multiple instructions to occupy different stages simultaneously?

A. Paging
B. Pipelining
C. DMA
D. TLB

Answer: B

Q8

What is DMA mainly used for?

A. Instruction decoding
B. Direct data transfer between device and memory
C. Branch prediction
D. Cache replacement

Answer: B

Q9

RISC-V is:

A. A compiler
B. A CPU model
C. An ISA
D. An operating system

Answer: C

Q10

Which optimization reduces intermediate memory traffic?

A. Operator fusion
B. Randomization
C. Paging
D. Branching

Answer: A

82. Last-Minute OA Checklist

Before the OA, make sure you can explain all of these without looking them up:

CPU

ALU
control unit
registers
PC
instruction cycle

Memory

registers
L1/L2/L3
RAM
storage
memory hierarchy
cache hit/miss
locality
cache line

Cache

direct mapped
set associative
fully associative
LRU
write-through
write-back

ISA

ISA
microarchitecture
RISC vs CISC
load/store
RISC-V

Pipeline

fetch/decode/execute/memory/write-back
throughput
latency
structural hazard
RAW/WAR/WAW
control hazard
forwarding
stalls
branch prediction
speculative execution

Parallelism

superscalar
out-of-order
multicore
ILP
SIMD
vector processing
Flynn's taxonomy

Memory Management

virtual memory
paging
page table
TLB
page fault
alignment
endianness
structure padding

I/O

interrupts
polling
DMA
memory-mapped I/O

AI / Accelerator

CPU vs GPU vs NPU
SIMD
matrix acceleration
quantization
operator fusion
tiling
layout transformation
memory planning
bandwidth vs latency
compute-bound vs memory-bound

Performance

Instruction Count
CPI
clock rate
CPU time equation
Amdahl's Law

83. One-Page Mental Model

                         SOFTWARE
                            |
                            v
                         COMPILER
                            |
                            v
                           ISA
                            |
              +-------------+-------------+
              |                           |
              v                           v
             CPU                    AI ACCELERATOR
              |                           |
      +-------+-------+             SIMD / Matrix
      |       |       |                   |
      v       v       v                   |
 Registers   ALU   Pipeline                |
      |               |                   |
      +-------+-------+-------------------+
              |
              v
            Cache
        L1 -> L2 -> L3
              |
              v
             RAM
              |
              v
           Storage

Performance comes from:
- fewer instructions
- lower CPI
- higher clock rate
- fewer cache misses
- better locality
- better parallelism
- better SIMD/vector usage
- reduced memory movement
- efficient accelerator utilization

Compiler/AI optimizations:
- vectorization
- operator fusion
- tiling
- quantization
- layout transformation
- memory planning
- accelerator-aware scheduling

84. Final Priority for This OA

If time is limited, study in this order:

Cache + memory hierarchy
Pipeline + hazards + branch prediction
ISA + RISC-V + RISC vs CISC
Virtual memory + paging + TLB
SIMD + vector processing + multicore
DMA + bandwidth vs latency
CPU vs GPU vs NPU
Quantization + fusion + tiling
CPI / performance calculations
Endianness + alignment + structure padding

The key idea is:

Computer architecture is largely about moving computation and data efficiently.

For this AI compiler/runtime role, the compiler is the layer that tries to turn a high-level AI workload into hardware-efficient execution by exploiting exactly these properties of the architecture.