Optimization: Nonlinear Programming & Trust-Region Methods
Optimization: Nonlinear Programming & Trust-Region Methods
💡 Key Takeaways
- Formulate nonlinear optimization problems: objective, decision variables, and constraints that define the feasible set.
- Understand the trust-region subproblem: build a local quadratic model and minimize it within a capped search region to decide the step.
- Compare trust-region methods with line-search approaches, and discuss convergence, stability, and handling nonconvexity.
- Learn the practical steps of a trust-region algorithm: solve the subproblem, assess predicted vs. actual reduction, update the radius, and decide on accepting the step.
❓ Frequently Asked Questions
What is nonlinear programming (NLP) in optimization?
NLP refers to optimizing an objective that is nonlinear in the decision variables, possibly with nonlinear constraints. It includes unconstrained problems as well as constrained ones solved with methods like gradient-based, Newton-type, barrier, or penalty approaches.
What is a trust-region method?
A trust-region method builds a local model (usually quadratic) of the objective around the current point and minimizes this model inside a neighborhood (the trust region) where the model is trusted. The region radius adapts based on how well the model predicts actual improvement.
What does the trust-region subproblem look like?
Typically: minimize m_k(p) = g_k^T p + 1/2 p^T B_k p subject to ||p|| ≤ Delta_k, where g_k is the gradient and B_k is an (approximate) Hessian. The solution p is the proposed step.
How is trust-region different from line-search methods?
Line-search methods pick a direction and move along a line with a step size chosen for sufficient decrease. Trust-region methods constrain the step to lie within a radius and decide the step based on the agreement between predicted and actual improvement, often offering greater robustness for nonlinear or ill-conditioned problems.