lesson
50 min

Layers and Activations

Stack neurons into layers, add activation functions, and build your first neural network in Keras.

Objective

By the end of this lesson, you can describe the layers of a neural network, explain what an activation function does, and build a small network in Keras.

Layers

A neural network connects neurons in layers:

  • Input layer: one value per feature (column).
  • Hidden layers: neurons that combine the inputs in new ways. "Hidden" just means they are in the middle.
  • Output layer: the prediction. One neuron for a number or a yes/no score, or one per category.

Each hidden layer can learn more complex patterns built from the layer before it. A network with many hidden layers is called deep. That is where "deep learning" comes from.

Activation functions

Between layers, each neuron passes its total through an activation function. Without one, stacking layers would add up to just one big straight-line model. Activations let the network learn curves.

Two common ones:

  • ReLU: if the total is negative, output 0. Otherwise, output the total. Simple and fast. Used in most hidden layers.
  • Sigmoid: squeezes any total into the range 0 to 1. Used for yes/no outputs, because the result reads like a probability.
Run it to see what ReLU and sigmoid do to three totals. Add -10 and 10 to the list. What does sigmoid do with very large or very small totals?
Python 3
Ctrl+Enter to run | Tab to indent | Python runs in your browser via Pyodide

Your first network in Keras

Keras is a beginner-friendly library for neural networks that runs on TensorFlow. It is preinstalled in Google Colab. This network solves XOR, which a single neuron could not.

import numpy as np
from tensorflow import keras

X = np.array([[0, 0], [0, 1], [1, 0], [1, 1]])
y = np.array([0, 1, 1, 0])          # XOR

model = keras.Sequential([
    keras.Input(shape=(2,)),
    keras.layers.Dense(8, activation="relu"),     # hidden layer
    keras.layers.Dense(1, activation="sigmoid"),  # output: a score from 0 to 1
])
model.compile(optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"])
model.fit(X, y, epochs=500, verbose=0)
print(model.predict(X).round(2))

Results vary a little between runs, because the starting weights are random. If it has not learned XOR, run it again or raise the epochs.

Judgment check

Neural networks are powerful and hard to explain. For a table of a few dozen columns, a regression or a small tree often does as well, and you can explain it to the person affected. Use a neural network when the data is images, sound, or text, or when simpler models clearly fall short on your test set.

Quiz: Layers check

  1. 01Why do neural networks need activation functions?

    Choose one.

  2. 02You have a table with 12 columns. A small decision tree does as well as a neural network on the test set. Which should you use?

    Choose one.

Ask. Answer. Build together.