What a neural network is doing

A neural network is a function with a very large number of adjustable numbers in it, and training is the process of adjusting them so the function produces the outputs you want.

The structure. Inputs go in, pass through layers of simple units, and an output comes out. Each unit takes the numbers reaching it, multiplies each by a weight, adds them up, and applies a simple non linear function. That is the whole mechanism repeated many times.

Why the non linear part matters. Without it, stacking layers gives you nothing a single layer could not do. The non linearity is what lets a network represent complicated relationships.

How it learns. You show it an example, compare its output with the correct answer, and compute how wrong it was. Then you adjust every weight slightly in the direction that would have reduced the error. Repeat this millions of times over many examples and the weights settle into a configuration that works.

That is gradient descent, and it is the whole of training. Everything else is details about doing it efficiently.

What makes it different from earlier approaches. You do not tell it what features to look at. Show it enough photographs of cats and it works out for itself what distinguishes one, discovering edges, then shapes, then parts, then objects across its layers.

And what it is not. It is not reasoning, and it has no model of the world. It is a very flexible function fitted to data, which explains both its strength and its failures.