Study notes I'm keeping as I learn to program. The ideas here come from Imran Ahmad's 50 Algorithms Every Programmer Should Know — I'm sharing them in my own words. The plain-language explanations, the examples, and any mistakes are mine.
The concept in plain language
A neural network is just layers of tiny math units stacked on top of each other. Each unit — a “neuron” — takes some numbers in, multiplies each by a weight, adds them up, and passes the total through an activation function: a nonlinear gate (common ones are ReLU, sigmoid, and tanh) that decides how much signal to let through. One neuron is dumb. A few million of them, arranged in layers, can approximate almost any pattern you can throw at them.
How does it learn? Two words: backpropagation and gradient descent. You show the network an example, measure how wrong its answer is, then work backward through the layers to figure out which weights to blame — and nudge each one a little in the direction that reduces the error. Repeat that a few million times and the network “learns.” Transfer learning is the shortcut: instead of training from scratch, you take a network someone already trained on a huge dataset and fine-tune it on your smaller problem. It’s borrowing an educated brain instead of raising a child.
Language is harder, because computers read numbers, not words. So we tokenize (chop text into pieces), sometimes stem (trim words to their root, so “running” becomes “run”), and score which words actually matter with TF-IDF (a word is important if it’s frequent here but rare everywhere else). The real leap is word embeddings: placing every word as a point in a high-dimensional coordinate space, learned so that meaning becomes geometry. That’s what makes the famous king − man + woman ≈ queen trick work — you can literally do arithmetic on meaning.
Words come in order, so we need models with memory. Recurrent neural networks (RNNs) read one token at a time, carrying a running summary forward. But plain RNNs forget: signals fade as they travel back through many steps — the vanishing-gradient problem. LSTMs and GRUs fix this by adding gates that choose what to keep, update, or discard. Chris Olah’s Understanding LSTM Networks is still the clearest picture of how those gates work.
Then, in 2017, a smarter idea: attention. Instead of shuffling information step by step, let the model look at every word at once and decide, for each word, which other words matter. Jay Alammar’s The Illustrated Transformer walks through it panel by panel. Stack that mechanism into a big network, feed it much of the internet, and you get a large language model — the family that ChatGPT belongs to.
Where you’ve seen it in the real world
The modern deep-learning era has a clean starting gun: 2012, when AlexNet — built by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton — won the ImageNet image-recognition contest with a top-5 error rate of 15.3%, well ahead of the runner-up’s 26.2% (NeurIPS, 2012). A roughly eleven-point gap overnight told everyone that deep neural networks weren’t a curiosity — they were the future of computer vision.
A year later, Tomáš Mikolov and colleagues at Google published word2vec, the embedding method behind the king − man + woman ≈ queen result (arXiv, 2013). Suddenly software had a usable, numerical sense of what words mean.
You’ve used the next milestone without noticing. In 2016 Google Translate switched to a neural system, GNMT, which — per Google’s own write-up — cut translation errors by more than 55–85% across several major language pairs versus the old phrase-based approach (Google Research, 2016). Then came the paper that reset the field — Vaswani et al.’s Attention Is All You Need (2017), which threw out recurrence entirely and built translation purely on attention. That architecture, the Transformer, is the “T” in GPT.
The public met the result on November 30, 2022, when OpenAI released ChatGPT. Its growth was startling: per a UBS analyst note (citing Similarweb data), reported by Reuters, ChatGPT was estimated to have reached about 100 million monthly users roughly two months after launch — with the analysts adding they couldn’t recall a faster ramp in a consumer app (Reuters, 2023). Worth flagging plainly: that 100-million figure is an outside estimate, not an OpenAI-published headcount.
The healthcare angle
This is where it stops being abstract for a physician. In 2016, a Google team led by Varun Gulshan trained a deep network to read retinal photographs for diabetic retinopathy and published it in JAMA, reporting an area under the ROC curve of about 0.99 for referable disease on two validation sets (Gulshan et al., 2016) — algorithm sensitivity and specificity in the range of a trained grader.
Then came a regulatory first. In April 2018 the FDA authorized IDx-DR as the first autonomous AI diagnostic system that returns a screening result without a clinician interpreting the image. Its pivotal trial in primary-care offices reported observed sensitivity of 87.4% and specificity of 89.5% for more-than-mild retinopathy (Abràmoff et al., npj Digital Medicine, 2018). That’s a camera in a family-medicine clinic, not an eye specialist, making the call.
The pattern repeated in cardiology. Awni Hannun and colleagues trained a deep network on more than 90,000 single-lead ECGs and reported arrhythmia detection at a level comparable to expert cardiologists, in Nature Medicine (Hannun et al., 2019).
And the language models came for our exams. In 2023, Kung and colleagues reported that ChatGPT scored at or near the roughly 60% passing threshold across all three USMLE steps, with no specialized medical training (Kung et al., PLOS Digital Health, 2023). “Almost passing” is the honest framing — but it was a general-purpose chatbot, not a medical one.
Why it matters when you work with AI
Here’s the through-line worth carrying into your own coding. Every system above — retinal screener, ECG reader, ChatGPT — is the same machinery: weighted layers, an activation function, backprop, and (for the recent ones) attention over embeddings. Once you see that, the field stops looking like a wall of buzzwords.
A few practical takeaways:
- These are pattern-matchers with error bars, not oracles. A 0.99 AUC or a “cardiologist-level” result is a statistic on a specific dataset — it can drift on patients, cameras, or populations the model never saw. Ask what it was validated on.
- Embeddings and attention are the ideas to actually internalize. They explain both the magic (meaning as geometry, focus on what’s relevant) and the failure modes (a model confidently “hallucinating” is just attention landing on the wrong pattern).
- Transfer learning is why you rarely start from zero. Almost everything you’ll build fine-tunes or prompts a pretrained model rather than training one — which is a gift, and also means you inherit its blind spots.
- Verify before you trust, especially in medicine. The FDA authorized an autonomous tool only after a prospective trial, not a demo. That’s the right instinct to bring to any AI you’d put near a patient — or into production.
Sources
- Krizhevsky, Sutskever & Hinton, ImageNet Classification with Deep Convolutional Neural Networks (2012) — NeurIPS
- Mikolov, Chen, Corrado & Dean, Efficient Estimation of Word Representations in Vector Space (2013) — arXiv
- Mikolov et al., Distributed Representations of Words and Phrases and their Compositionality (2013) — NIPS/arXiv
- Wu et al., Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016) — arXiv
- Google Research, A Neural Network for Machine Translation, at Production Scale (2016) — Google Research Blog
- Vaswani et al., Attention Is All You Need (2017) — arXiv
- Hu, ChatGPT sets record for fastest-growing user base — analyst note (2023) — Reuters
- Gulshan et al., Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs (2016) — JAMA
- Abràmoff et al., Pivotal trial of an autonomous AI-based diagnostic system for detection of diabetic retinopathy in primary care offices (2018) — npj Digital Medicine
- Hannun et al., Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network (2019) — Nature Medicine
- Kung et al., Performance of ChatGPT on USMLE (2023) — PLOS Digital Health
- Alammar, The Illustrated Transformer (2018) — jalammar.github.io
- Olah, Understanding LSTM Networks (2015) — colah's blog
— Jeremy Tabernero, MD · More in this series · Get in touch