Linear Systems

1Equations become one constraint map

A linear system is not just many equations written one under another. It is one transformation A applied to an unknown vector x, with the requirement that the result must equal the target vector b.

That framing matters because it turns algebra into structure. You can ask whether the target lies in the output space of A, whether the columns of A are redundant, and whether the constraints identify one answer or a whole family.

Figure. Each linear equation is a constraint line. A simultaneous solution must sit on every constraint at once, so the intersection is the candidate answer.

Write Ax=b

1. Unknown vectorPut the unknowns in a fixed order, such as x=(x,y).
2. RowsEach equation becomes one row of coefficients in A.
3. TargetThe right-hand sides become the vector b.

2Elimination rewrites constraints without changing the answer

Gaussian elimination is controlled rewriting. You combine equations to remove an unknown while keeping exactly the same solution set.

The safe mental model is not canceling symbols wherever they appear. It is replacing one row by a new row that follows from the old rows, so any solution of the original system still satisfies the rewritten system.

Figure. Subtract the second constraint from the first: (2a + b) − (a + b) = a and 7 − 5 = 2, so a = 2. Back-substitute into a + b = 5 to get b = 3. The pair is unchanged.

Eliminate one variable

1. AlignChoose equations where one unknown can be removed by subtraction or scaling.
2. RewriteReplace a row with a difference that still follows from the original rows.
3. Back-substituteUse the simpler row to recover the remaining unknown.

3A system can be unique, impossible, or underdetermined

Solving is not always about finding one point. Independent constraints can identify a unique solution, dependent constraints can leave many solutions, and contradictory constraints can leave no solution.

The matrix view explains the cases. The columns of A describe what targets are reachable; if b is outside that reachable set, the system is inconsistent.

Figure. The three geometric cases are crossing lines, parallel lines, and the same line. They correspond to one solution, no solution, and infinitely many solutions.

4Normal equations are linear systems for fit parameters

Least squares turns a fitting problem into a linear system in the model parameters. For a line \hat y=a+bt, the unknown vector is \theta=(a,b).

The normal-equation form X^TX\theta=X^Ty appears because the best fit makes the residual orthogonal to the columns of the design matrix. In a tiny example it is still just a 2 by 2 system to solve.

The normal-equation ledger turns a fitting problem into the parameter solution a=1, b=2.

From fit to system

1. Design matrixUse one column for the intercept and one for t.
2. Normal equationsCompute or receive X^TX\theta=X^Ty.
3. Solve parametersSolve the resulting system for a and b.

5A solution is checked by its residual

After solving, the most direct check is to plug the answer back into Ax=b. The difference Ax-b is the residual vector.

For an exact linear system, a correct solution has zero residual. For a least-squares fit, the residual is usually not zero, but its structure still tells you how the fit failed.

Figure. Plug x = 8/3, y = 5/3 back into 2x + y = 7 and x − y = 1. Both sides match, so the residual Ax − b is (0, 0).

Check a candidate

1. MultiplyCompute A\hat x using the candidate solution.
2. SubtractForm the residual A\hat x-b.
3. InterpretZero means exact satisfaction; nonzero means remaining error or inconsistency.