The Conjugate Gradient Method for Solving Linear Systems

I know there's a number of ways to find and reference the minimum of a function, but how can I reference the argmin? E.g.if it were of the form f(x), how can I reference the minimizing value of x ?
EDIT: I was able to figure out what was going wrong! It came down to me not using "matlabFunction(f)", so nothing would work in the first place. Thanks everyone for making me look to other issues rather than continuing to think I had misinterpreted the purpose of "fminbnd()"
(For later search purposes): Argmin Arg min Minimum argument Function minimizer Algorithm Iterative process Conjugate gradient method Truncated newton method Convex analysis

How important it is to be able to write your own functions vs using something someone else already made? (specially in Math heavy fields such as Optimization, Machine Learning and CFD)

Hello AskProgramming . This question is probably trivial but I wonder because I come from a non-programming background and this semester I'm taking a course on Optimization. It's been math and programming heavy and I struggled a lot in the first assignments where we had to write our own codes to minimize the objective functions even if I understood the math behind the methods (at that time it was single and multivariable nonconstrained optimization so things like Golden-section, Conjugate gradient, Gradient descent, Quasi-Newton methods and so on) due to the fact that for me in the past programming has been mainly a hobby and my ability to translate those equations into code is subpar for not saying straight up bad.
Fast forward a few weeks and for the subsequent assignments the instructor allows us to use existing code and functions to solve the problems and I have been doing much better. For this particular case we're dealing with linear programming, sequential quadratic programming and penalty function methods to solve linear and nonlinear constrained optimization problems. I have used functions from MATLAB and from users who posted them online and I have been able to obtain the results I wanted after some time debugging and changing the code for my particular needs.
Now my question is for someone who is interested in this field (or in other highly mathematical heavy fields such as CFD), how important it is to be able to write code from scratch vs modifying and using existing code, functions and libraries from someone else to do the job?

I'm havng trouble understanding why SGD, RMSProp, and LBFGS have trouble converging on a solution to this problem (data included)

Here's a dropbox link to a simple data set:
https://dl.dropboxusercontent.com106825941/IPData.tar.gz
It's about 84,000 examples polled from an cart-and-pendulum simulation. The columns are:

position theta velocity angular-velocity force-applied-to-cart value

...Where "value" is a simple objective function whose inputs are all taken from the first four columns (x, theta, v, w). All inputs are scaled such that their mean is 0 and they range more or less within [-3, 3]. The output is scaled such that the mean is 0.5, and all values fall within the interval [0.2, 0.8].
A 5-25-1 feedforward network tasked with learning the value function and trained in Matlab can converge on an almost perfect solution with Levenberg-Marquardt or Scaled Conjugate Gradient descent very quickly. However, using very similar network architecture in my own code (one difference being that my output neuron is a sigmoid, while Matlab's output neuron is linear) SGD and RMSprop fail to converge to a good answer. I've tried minibatches with SGD, using the entire dataset per epoch, and lots of different learning rates and learning rate decay values. I've spent a similar amount of time tweaking hyperparameters with RMSProp.
RISO's LBFGS implementation also fails with this dataset, although I haven't put as much time into playing with it.
I see three possibilities:
1.) A bug in my code. This is of course the thing I've been most suspicious of, but I'm begining to doubt this is the cause. My code passes this test, and successfully extracts gabor shapes from natural images when used for autoencoding. It also passes simpler tests, like learning XOR^* .
2.) Something about this dataset is particularly difficult for stochastic methods. This seems unlikely; if you put together a scatter plot of value-vs-theta-vs-omega, you can see that it's a rather simple structure.
3.) SGD and RMSProp are incredibly sensitive to hyperparameter values, or perhaps weight initialization, and I've just been setting them wrong. Right now I'm initializing weights with a uniform random variable -1 I'm hoping someone can give me some insight into why my SGD and RMSProp are failing here. This should be an easy problem, but I can't find anything to point to that's demonstrably wrong.
*: In regards to XOR, my code also seems to be very senstve to hyperparamters when solving this. A 2-3-1 network needs over 10000 iterations to converge, and won't do so if the batch size is anything other than 1. Starting froma configuration that converges, and reducing the learning rate by a decade and also increasing the number of training iterations by a decade does not result in a network that also converges. This seems wrong.

Matlab homework help

Numerical Analysis and Programming

I've recently become quite interested in numerical methods for PDEs; I've done some work with development of some methods, but everything has so far been done in MATLAB. I really enjoy working in Matlab, but whenever I look at groups working on scientific computing and numerical methods, it seems that they're often looking for students with backgrounds in C++, which I do not have any background in. I do have some basic capabilities in python (although I haven't tried out Scipy or numpy yet); not sure if that is relevant or not.
I'm hoping for a few things: One would be a textbook recommendation for learning C++, ideally one geared towards numerical methods. I have no particular interest in C++ or general software development, so my main goal is really to learn just what I need so that I could be useful in a group that works on scientific computation. I have no clue how much that means I need to learn.
Another would be any recommendations on a numerical analysis/methods for PDE books. I'm already familiar with the basic ideas of the Finite Element Method (I used a book Numerical Treatment of PDEs by Christian Grossman I believe, and I'm getting Brenner and Scott's FEM book). I also have Saad's Iterative Methods book. For my theoretical background, I have plenty of graduate level real analysis and functional analysis, but relatively scarce amounts of numerical analysis. I know the basic theory of Galerkin methods on elliptic PDEs (variational formulation, Lax Milgram, Sobolev spaces/interpolation estimates with polynomials), and a bit about weak formulations for parabolic problems, but that's about it. I'm also familiar with some of the iterative techniques for solving linear systems, such as Gauss-Seidel, Jacobi, SOR, and the conjugate gradient and preconditioned conjugate gradient method.
Third, if there are any numerical methods/scientific computing graduate students or researchers, if you have any advice or suggestions on what sort of skills to develop, things to learn, or general advice, that would be greatly appreciated. I'm currently a graduate student, and most of my work has been theoretical, but I'm really interested in learning more about modeling, simulations, and developing numerical methods for scientific problems.
Thanks!

Default Activation Function in Matlab Neural Network Toolbox

I'm trying to confirm the default activation function in Matlab's Neural Network Toolbox. Is it the tan-sigmoid function? Apparently my google-fu is weak this morning and I'm getting more than one result for the "default". (also, is there a way to plot the activation function? I know how to use "plot"... but plotting the default function is eluding me right now). Thank you.
EDIT: I believe I found it. It turns out that it's not listed in the documentation as a "default" as such... although the "trainscg" is described. The line that states it is found when you export an "Advanced Script" from the example files (as opposed to the "Simple Script")... line 33:

net.trainFcn = 'trainscg'; % Scaled conjugate gradient

This is not the "log-sigmoid" transfer function that one would normally assume it would be... but seems to work just fine.

Implementation details of Olshausen and Fields sparse coding paper in Nature

I want to find a working implementation of the 1996 Nature paper Emergence of simple-cell receptive field properties by learning a sparse code for natural images or the authors' other paper with more details published on Vision Research Sparse coding with an overcomplete basis set: a strategy employed by V1? to help me learn their method.
They have a matlab version calling c implementation of fast conjugate gradient here. However I haven't successfully made in run on my computer. Does anyone have code available (preferably in python) reproduced their results?
Another question is what is the reasoning behind the "rate parameter for the gain adjustment" $\alpha$ in Eq.(18) of the second paper?

conjugate gradient matlab video

The Conjugate Gradient Method is an iterative technique for solving large sparse systems of linear equations. As a linear algebra and matrix manipulation technique, it is a useful tool in approximating solutions to linearized partial di erential equations. The conjugate gradient method aims to solve a system of linear equations, Ax=b, where A is symmetric, without calculation of the inverse of A. It only requires a very small amount of membory, hence is particularly suitable for large scale systems. It is faster than other approach such as Gaussian elimination if A is well-conditioned. For example, The conjugate gradient method is an algorithm for the numerical solution of particular systems of linear equations, namely those whose matrix is symmetric and positive-definite. The conjugate gradient method is often implemented as an iterative algorithm , applicable to sparse systems that are too large to be handled by a direct implementation or other direct methods such as the Cholesky decomposition. So the conjugate gradient method ﬁnds the exact solution in at most n iterations. The convergence analysis shows that kx− xkkA typically becomes small quite rapidly and we can stop the iteration with k much smaller that n. It is this rapid convergence which makes the method interesting and in practice an iterative method. The preconditioned conjugate gradients method (PCG) was developed to exploit the structure of symmetric positive definite matrices. Several other algorithms can operate on symmetric positive definite matrices, but PCG is the quickest and most reliable at solving those types of systems . You are now following this Submission. You will see updates in your activity feed; You may receive emails, depending on your notification preferences Conjugate Gradient to Solve Symmetric Linear Systems The conjugate gradient method is an iterative method that is taylored to solve large symmetric linear systems $Ax=b$. We first give an example using a full explicit matrix $A$, but one should keep in mind that this method is efficient especially when the matrix $A$ is sparse or more generally when it is fast to apply $A$ to a vector. The scaled conjugate gradient algorithm (SCG), developed by Moller [Moll93], was designed to avoid the time-consuming line search. This algorithm is too complex to explain in a few lines, but the basic idea is to combine the model-trust region approach (used in the Levenberg-Marquardt algorithm described later), with the conjugate gradient approach. matlab nmr regularization tomography conjugate-gradient inverse-problems gmres fista image-deblurring krylov-subspace-methods Updated Oct 30, 2020 MATLAB

conjugate gradient matlab top

Mod-01 Lec-33 Conjugate Gradient Method, Matrix ...

Advanced Numerical Analysis by Prof. Sachin C. Patwardhan,Department of Chemical Engineering,IIT Bombay.For more details on NPTEL visit http://nptel.ac.in This is a brief introduction to the optimization algorithm called conjugate gradient. About Press Copyright Contact us Creators Advertise Developers Terms Privacy Policy & Safety How YouTube works Test new features Press Copyright Contact us Creators ... This video demonstrates the convergence of the Conjugate Gradient Method with an Incomplete LU Decomposition (ILU) preconditioner on the Laplace equation on ... This MATLAB session implements a fully numerical steepest ascent method by using the finite-difference method to evaluate the gradient. A simple visualizati... This video will explain the working of the Conjugate Gradient (Fletcher Reeves) Method for solving the Unconstrained Optimization problems.Steepest Descent M... Video lecture on the Conjugate Gradient Method In this tutorial I explain the method of Conjugate Gradients for solving a particular system of linear equations Ax=b, with a positive semi-definite and symm...