E ExamMaster

Engineering Mathematics · Optimization

Objectives and Parameters

In Engineering Mathematics because optimization begins by naming what can change and what success means numerically.

Before an algorithm can optimize anything, the problem must be written as a score over adjustable parameters. This topic names the knobs, the loss, and the comparison a learner will later descend.

  • Engineering Mathematics
  • Medium level
  • 4 concepts

1Parameters are the adjustable knobs

A parameter is a quantity the model is allowed to change while learning. In a one-feature linear model \hat y=wx, the parameter is w; the input x and target y are observations, not knobs.

Keeping that separation clean matters because optimization asks which parameter value gives the best score on the same fixed data.

Picture a fixed two-row data table beside one movable slider labelled w; the slider moves, the table does not.

Name the pieces

  1. ModelChoose the prediction rule, such as \hat y=wx.
  2. ParameterMark the adjustable part, here the slope w.
  3. DataHold the training pairs fixed while comparing values of w.

Which symbol is allowed to move?

For the two training pairs (1,2) and (2,3), use \hat y=wx. If w=1, list the predictions and errors.

  • Predictions at w=11,2
  • Errors \hat y-y-1,-1
  • Adjustable quantityw, not the two data pairs

Pro tip. An optimizer changes w because w is the model's knob. Changing the data would describe a different training problem.

In \hat y=wx for pairs (1,3),(2,5), what is the parameter?
  1. w
  2. x
  3. y
  4. The dataset size

Training adjusts w; (x,y) are data.

2A loss scores each parameter setting

A loss function turns a parameter value into one number. For squared error, each prediction error is squared, the squares are averaged, and the result is smaller for a better fit.

For the fixed data (1,2) and (2,3) with \hat y=wx, the two-point mean squared error is L(w)=\frac{1}{2}[(w-2)^2+(2w-3)^2].

Figure. An objective assigns a score to each parameter setting. Optimization can compare settings only after this scoring rule exists.

Build the loss

  1. PredictFor each row, compute \hat y=wx.
  2. ErrorSubtract the target: wx_i-y_i.
  3. Average squaresSquare the two errors and divide by 2.

Write and evaluate a two-point MSE

Use (1,2) and (2,3) with \hat y=wx. Write L(w) and evaluate it at w=1 and w=1.5.

  • Loss as a function of wL(w)=\frac{1}{2}[(w-2)^2+(2w-3)^2]
  • L(1)\frac{1}{2}[(-1)^2+(-1)^2]=1
  • L(1.5)\frac{1}{2}[(-0.5)^2+0^2]=0.125
  • Lower lossw=1.5 beats w=1 on this data

Pro tip. The same two rows are used twice; only w changes. That is what makes the comparison meaningful.

Loss L(w)=\frac{1}{2}[(w-1)^2+(2w-4)^2]. What is L(2)?
  1. 0.5
  2. 0
  3. 2
  4. 1

(2-1)^2+(4-4)^2=1, times 1/2 gives 0.5.

3The optimum is the argument, not the score

\min L(w) is the smallest loss value, while \arg\min_w L(w) is the parameter value that attains it. Optimizers return parameters; the loss value is the evidence that one setting is better.

In machine learning language, training means searching parameter space for an argument that makes the chosen objective small.

A comparison table would have candidate parameters in one column and losses in another; the selected row is the one with the smallest loss.

Read argmin

  1. ObjectiveName the function being minimized.
  2. CandidatesEvaluate the same function at competing parameter values.
  3. Return knobReport the parameter that gives the smallest value.

Score versus winning parameter

For the same loss, compare L(1)=1, L(1.5)=0.125, and L(2)=0.5.

  • Candidate losses1,0.125,0.5
  • Smallest loss among the three0.125
  • Best candidate parameterw=1.5

Pro tip. The score is 0.125, but the candidate solution is w=1.5. Mixing those up is a common early optimization error.

If L(1)=4, L(2)=1, L(3)=2, then \arg\min_w L(w) is?
  1. 2
  2. 1
  3. 4
  4. 3

Argmin returns the parameter that achieves the smallest loss, here w=2.

Notes

  • Parameters are the adjustable quantities the optimizer is allowed to change.
  • The objective function assigns a score or cost to each parameter setting.
  • A loss is an objective written so that smaller is better.

Formulas

  • \theta^* = \arg\min_\theta J(\theta) names a parameter choice that minimizes the objective.
  • For two squared-error observations, L(w)=\frac{1}{2}\sum_{i=1}^{2}(wx_i-y_i)^2.

Exam traps & shortcuts

  • Write down the objective before discussing the algorithm; otherwise there is no way to tell what better means.
  • Do not move the data when comparing parameter settings; only the parameters change.

Reference tables

Use these words before writing an update rule.

Optimization vocabulary
WordMeaningIn \hat y=wx
DataFixed observations used to score the model(x_i,y_i) pairs
ParameterAdjustable model quantityw
ObjectiveFunction the optimizer tries to improveL(w)
LossObjective where smaller is betterMSE

Recap

Optimization starts by naming the fixed data, the movable parameters, and the score being minimized.

Knobs
Parameters are the quantities the optimizer may change; data are held fixed during a comparison.
Loss
A loss function maps one parameter setting to one numeric score.
Two-point MSE
For (1,2),(2,3) and \hat y=wx, L(w)=\frac{1}{2}[(w-2)^2+(2w-3)^2].
Argmin
\arg\min names the winning parameter, not the minimum loss value itself.

Practise Objectives and Parameters

Reading is free and needs no account. Practice, mocks and progress live in the app.

  • A 5-question practice set that ends the chapter
  • 4 quick checks with worked explanations
  • Timed mocks scored with the real marking scheme
  • Readiness tracked per topic, kept on your device
Continue with Google — freeNo card, no trial. Works offline once installed.