Parallel Computing — Overview

Why the same computation can be 100× faster

Emmanuel Pilliat — ENSAI 3A

Course outline

• 0 — Julia for Python users: syntax bridge
• 1 — Interpreted vs compiled: JIT, type specialization
• 2 — Multiple dispatch vs OOP: dispatch, type instability
• 3 — Threads & concurrency: @async, @spawn, races
• 4 — Memory & batching: cache, locality
• 5 — GPU: feed thousands of threads

Running example: Monte-Carlo π — loop → threads → GPU.

Julia = test bench, Python = comparison.

0: 1-based arrays, end, the dot, structs instead of classes; no performance yet. 1: a Julia loop in milliseconds vs Python half a second — the JIT, measuring properly, type specialization. 2: how Julia picks the method to run, and what it costs when the compiler can’t know the type (abstract container ~10×, untyped global > 100×: type instability). 3: @async overlaps waiting on one core; Threads.@spawn splits computing over many — and the parallel π that returns a wrong answer. 4: the processor waits for memory. 5: the bus, and locality again on the card. Running example: same dart count, count_in_disk — loop (0–1), threads (3), GPU (5). Batching also in 4.

How we work

• Slides frame; scripts and notebooks demo: predict, then run.
• Each module: a Julia script + Pluto notebook; modules 1–5: a Python notebook (Colab).
• Short activities in class (5–10 min, Python or Julia).
• We compare the numbers across languages.

Note

Python track = common; Julia track = deep dive.

Same lesson in script (VS Code) and Pluto. 6 decks but 5 Julia notebooks: modules 4 and 5 share one script, one notebook and one Python notebook. Activities: one or two blanks to fill, one striking number, handed out in class. Python track: nobody falls behind; Julia track: @btime, @code_native, @spawn, the GPU.

Start here

• ▶ 0 — Julia for Python users — the onboarding deck.
• 📓 All scripts, notebooks and Python notebooks: course page.

Next: 0 — Julia for Python users ➡

Synthesis

Four ideas

1. Compile and specialize (1): no cost paid per operation.
2. Concrete types (2): generic and fast.
3. Share as little as possible (3): or the result is wrong.
4. Feed the hardware (4–5): cache, batching, busy GPU.

Not a magic language: understand what the machine does. Type known at compile time = fast.

⬅ 5 — GPU · Course page

2: multiple dispatch is generic and fast — and the same * runs on the GPU. Performance comes from understanding the cost of each operation, who shares what, and where the data travels. Through the whole course: type found at run time = slow; type known at compile time = fast.