Classification

In this problem we'll set up a simple classifier using sklearn.

For this exercise, we'll use the polish bankruptcy data, available here: https://archive.ics.uci.edu/ml/datasets/Polish+companies+bankruptcy+data

First, download the data and save it to your local machine. Then we will read in the file as a csv:

In [1]:
f = open("/home/jmcauley/datasets/mooc/bankruptcy/5year.arff", 'r')

The data has a slightly odd format (try opening the above file in a text editor to see), so we have to skip the first few lines before we get to the actual csv data.

In [2]:
while not '@data' in f.readline():
    pass

Other than that we read it like a regular csv file, along with some simple data processing as we go

In [3]:
dataset = []
for l in f:
    if '?' in l: # Missing entry
        continue
    l = l.split(',')
    values = [1] + [float(x) for x in l]
    values[-1] = values[-1] > 0 # Convert to bool
    dataset.append(values)
In [4]:
len(dataset)
Out[4]:
3031

Looking at just the last entry (i.e., the label), how many positive values are there?

In [5]:
sum([x[-1] for x in dataset])
Out[5]:
102

Generate feature and label matrices from the data (in this case, we're just using the features directly from the csv)

In [6]:
X = [values[:-1] for values in dataset]
In [7]:
y = [values[-1] for values in dataset]

We'll use the "linear_model" library from sklearn, which has a Logistic Regression class

In [8]:
from sklearn import linear_model
In [9]:
model = linear_model.LogisticRegression()
In [10]:
model.fit(X, y)
Out[10]:
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
          verbose=0, warm_start=False)

Once the model has finished training, we can use the model to make predictions. Here we just use our training data to make predictions, but we could also use new (i.e., previously unseen) datapoints

Simple diagnostics - model accuracy

In [11]:
predictions = model.predict(X)

The predictions made my the model are just a vector of "True" and "False" values for each datapoint

In [12]:
predictions
Out[12]:
array([False, False, False, ..., False, False, False])

Next we convert this to an array indicating which of the predictions were correct

In [13]:
correctPredictions = predictions == y
In [14]:
correctPredictions
Out[14]:
array([ True,  True,  True, ..., False, False, False])

To compute the accuracy of the model, we are just counting the fraction of "True" values in the above vector

In [15]:
sum(correctPredictions) / len(correctPredictions)
Out[15]:
0.9663477400197954

Classification with separate training/test splits

In the previous example, we calculated accuracy, but used the same data for evaluation as we used for training. A more robust evaluation would consist of evaluating the model on new data.

Again, we start by reading in the csv data as before.

In [16]:
f = open("/home/jmcauley/datasets/mooc/bankruptcy/5year.arff", 'r')
In [17]:
while not '@data' in f.readline():
    pass
In [18]:
dataset = []
for l in f:
    if '?' in l:
        continue
    l = l.split(',')
    values = [1] + [float(x) for x in l]
    values[-1] = values[-1] > 0 # Convert to bool
    dataset.append(values)

We start by splitting the data into training and testing portions. It is important to randomly shuffle the data first, so that these two portions are similar samples of the data.

In [19]:
import random
In [20]:
random.shuffle(dataset)
In [21]:
X = [values[:-1] for values in dataset]
In [22]:
y = [values[-1] for values in dataset]

Here we split the data so that the first half (of the shuffled data) is used for training and the rest is used for testing

In [23]:
N = len(X)
X_train = X[:N//2]
X_test = X[N//2:]
y_train = y[:N//2]
y_test = y[N//2:]
In [24]:
len(X), len(X_train), len(X_test)
Out[24]:
(3031, 1515, 1516)

Again we'll use the same Logistic Regression model for training, though this time we'll only train on the training data

In [25]:
from sklearn import linear_model
In [26]:
model = linear_model.LogisticRegression()
In [27]:
model.fit(X_train, y_train)
Out[27]:
LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
          intercept_scaling=1, max_iter=100, multi_class='ovr', n_jobs=1,
          penalty='l2', random_state=None, solver='liblinear', tol=0.0001,
          verbose=0, warm_start=False)

Below we make predictions in the same way as before, but now do so seperately for the training and test sets.

In [28]:
predictionsTrain = model.predict(X_train)
predictionsTest = model.predict(X_test)
In [29]:
correctPredictionsTrain = predictionsTrain == y_train
correctPredictionsTest = predictionsTest == y_test

Training accuracy:

In [30]:
sum(correctPredictionsTrain) / len(correctPredictionsTrain)
Out[30]:
0.9643564356435643

Test accuracy:

In [31]:
sum(correctPredictionsTest) / len(correctPredictionsTest)
Out[31]:
0.9617414248021108

Gradient descent (regression)

Here we'll show how to implement regression using gradient descent. Of course this is not necessary when using a library, though the purpose of this exercise is to demonstrate how gradient-descent-based learning operates in general.

This exercise is based on the UCI PM2.5 (Air Quality) dataset, available here: https://archive.ics.uci.edu/ml/datasets/Beijing+PM2.5+Data

As usual, this data should first be saved to your local disk.

We first read the data as usual, and extract the pm2.5 measurements as outputs to be predicted:

In [32]:
path = "datasets/PRSA_data_2010.1.1-2014.12.31.csv"
f = open(path, 'r')
In [33]:
dataset = []
header = f.readline().strip().split(',')
for line in f:
    line = line.split(',')
    dataset.append(line)
In [34]:
header.index('pm2.5')
Out[34]:
5
In [35]:
dataset = [d for d in dataset if d[5] != 'NA']

For demonstration we'll build a simple feature vector consisting of only a single feature (temperature), plus an offset. Thus our goal is to predict air quality as a function of temperature.

In [36]:
def feature(datum):
    feat = [1, float(datum[7])] # Temperature
    return feat
In [37]:
X = [feature(d) for d in dataset]
y = [float(d[5]) for d in dataset]
In [38]:
X[0]
Out[38]:
[1, -4.0]

Below we initialize our parameter vector (theta).

In [39]:
K = len(X[0])
K
Out[39]:
2
In [40]:
theta = [0.0]*K

It can help to initialize the model to a solution that's "close to" the optimal value. A good initial guess for the offset term is to initialize it to the average value of the label (i.e., the average air quality measurement)

In [41]:
theta[0] = sum(y) / len(y)

Below are a few utility functions to compute the inner product and norm of a vector. These are also implemented in various libraries (e.g. numpy), and using the library functions would likely result in a more efficient implementation, but the functions are written here just for demonstration purposes.

In [42]:
def inner(x,y):
    return sum([a*b for (a,b) in zip(x,y)])
In [43]:
def norm(x):
    return sum([a*a for a in x]) # equivalently, inner(x,x)

Derivative function:

Most important is our function to compute the derivative. The expression being computed here is the derivative of the objective (the Mean Squared Error) with respect to the parameters (theta), at our current estimate of theta.

In [44]:
def derivative(X, y, theta):
    dtheta = [0.0]*len(theta) # Initialize the derivative vector to be a vector of zeros
    K = len(theta)
    N = len(X)
    MSE = 0 # Compute the MSE as we go, just to print it for debugging
    for i in range(N):
        error = inner(X[i],theta) - y[i]
        for k in range(K):
            dtheta[k] += 2*X[i][k]*error/N # See the lectures to understand how this expression was derived
        MSE += error*error/N
    return dtheta, MSE
In [45]:
learningRate = 0.003

Now we iteratively call our derivative function to improve our estimate of theta, by following the direction of the derivative. The details of this function are quite difficult to get right, e.g. if the learning rate or the convergence criteria are not set well the function may not produce a good solution.

In [46]:
while (True):
    dtheta,MSE = derivative(X, y, theta)
    m = norm(dtheta)
    print("norm(dtheta) = " + str(m) + " MSE = " + str(MSE))
    for k in range(K):
        theta[k] -= learningRate * dtheta[k]
    if m < 0.01: break

norm(dtheta) = 41178.18180276439 MSE = 8473.070863045637
norm(dtheta) = 27391.037310856456 MSE = 8461.470120068407
norm(dtheta) = 18227.59616521803 MSE = 8453.69132148803
norm(dtheta) = 12137.19327414051 MSE = 8448.453092769803
norm(dtheta) = 8089.214952690785 MSE = 8444.903795606504
norm(dtheta) = 5398.687652667358 MSE = 8442.47740594691
norm(dtheta) = 3610.359546776664 MSE = 8440.797726662375
norm(dtheta) = 2421.658035196417 MSE = 8439.614724709467
norm(dtheta) = 1631.4852346051546 MSE = 8438.762219100408
norm(dtheta) = 1106.18608452748 MSE = 8438.12975918841
norm(dtheta) = 756.9298365044901 MSE = 8437.643934223193
norm(dtheta) = 524.6771841240179 MSE = 8437.255951177423
norm(dtheta) = 370.1890137650022 MSE = 8436.933378638194
norm(dtheta) = 267.3859519156493 MSE = 8436.65465957273
norm(dtheta) = 198.93494191748107 MSE = 8436.40546436858
norm(dtheta) = 153.3159642979812 MSE = 8436.176266962746
norm(dtheta) = 122.87250029772292 MSE = 8435.960733873406
norm(dtheta) = 102.5158175537563 MSE = 8435.75465350741
norm(dtheta) = 88.86380355398373 MSE = 8435.55522454821
norm(dtheta) = 79.66861121441336 MSE = 8435.360583000387
norm(dtheta) = 73.43625675281844 MSE = 8435.16948784892
norm(dtheta) = 69.17379421549077 MSE = 8434.981112142255
norm(dtheta) = 66.22128087510248 MSE = 8434.794904138284
norm(dtheta) = 64.14009691955206 MSE = 8434.610495019098
norm(dtheta) = 62.63871971662855 MSE = 8434.427637555958
norm(dtheta) = 61.52339139602519 MSE = 8434.246165342996
norm(dtheta) = 60.665331198765905 MSE = 8434.065965704867
norm(dtheta) = 59.97894392367798 MSE = 8433.88696168824
norm(dtheta) = 59.407336675322206 MSE = 8433.709100093938
norm(dtheta) = 58.91269290537571 MSE = 8433.532343522953
norm(dtheta) = 58.46987474181126 MSE = 8433.356665088584
norm(dtheta) = 58.062170924866656 MSE = 8433.182044904057
norm(dtheta) = 57.678470770872075 MSE = 8433.008467745542
norm(dtheta) = 57.31138591259816 MSE = 8432.835921500664
norm(dtheta) = 56.956001957263624 MSE = 8432.664396136894
norm(dtheta) = 56.609048804775064 MSE = 8432.493883015399
norm(dtheta) = 56.268349218818344 MSE = 8432.32437443488
norm(dtheta) = 55.932452332260944 MSE = 8432.155863329805
norm(dtheta) = 55.60039006483615 MSE = 8431.988343067256
norm(dtheta) = 55.27151523169225 MSE = 8431.821807313729
norm(dtheta) = 54.945393945834 MSE = 8431.656249946522
norm(dtheta) = 54.62173410577471 MSE = 8431.491664993653
norm(dtheta) = 54.30033786637083 MSE = 8431.328046594657
norm(dtheta) = 53.98107004954366 MSE = 8431.165388974281
norm(dtheta) = 53.66383714906062 MSE = 8431.0036864248
norm(dtheta) = 53.348573376422465 MSE = 8430.842933295029
norm(dtheta) = 53.035231386457276 MSE = 8430.683123981258
norm(dtheta) = 52.723776113192045 MSE = 8430.524252922716
norm(dtheta) = 52.41418067285116 MSE = 8430.366314598441
norm(dtheta) = 52.1064236407871 MSE = 8430.20930352342
norm(dtheta) = 51.80048724150597 MSE = 8430.053214248157
norm(dtheta) = 51.496356145612964 MSE = 8429.89804135734
norm(dtheta) = 51.19401667008471 MSE = 8429.743779468017
norm(dtheta) = 50.8934562466372 MSE = 8429.59042323061
norm(dtheta) = 50.59466306827325 MSE = 8429.437967326843
norm(dtheta) = 50.29762585424491 MSE = 8429.286406470104
norm(dtheta) = 50.00233369371205 MSE = 8429.135735405336
norm(dtheta) = 49.708775941736214 MSE = 8428.985948908385
norm(dtheta) = 49.41694215001274 MSE = 8428.837041785991
norm(dtheta) = 49.12682202074671 MSE = 8428.689008875079
norm(dtheta) = 48.83840537585026 MSE = 8428.541845043446
norm(dtheta) = 48.55168213637392 MSE = 8428.39554518844
norm(dtheta) = 48.266642308712775 MSE = 8428.250104238334
norm(dtheta) = 47.98327597533746 MSE = 8428.105517150003
norm(dtheta) = 47.701573288497045 MSE = 8427.961778910514
norm(dtheta) = 47.42152446594685 MSE = 8427.818884536893
norm(dtheta) = 47.14311978797191 MSE = 8427.676829074711
norm(dtheta) = 46.86634959532466 MSE = 8427.535607598908
norm(dtheta) = 46.59120428771475 MSE = 8427.395215213031
norm(dtheta) = 46.3176743227119 MSE = 8427.255647049551
norm(dtheta) = 46.04575021491526 MSE = 8427.116898269882
norm(dtheta) = 45.77542253525628 MSE = 8426.978964063663
norm(dtheta) = 45.506681910481284 MSE = 8426.841839648168
norm(dtheta) = 45.239519022643194 MSE = 8426.705520269606
norm(dtheta) = 44.97392460870626 MSE = 8426.570001201555
norm(dtheta) = 44.709889460135464 MSE = 8426.435277745728
norm(dtheta) = 44.447404422549376 MSE = 8426.301345230748
norm(dtheta) = 44.18646039536908 MSE = 8426.16819901387
norm(dtheta) = 43.92704833147488 MSE = 8426.035834477872
norm(dtheta) = 43.66915923689382 MSE = 8425.904247034483
norm(dtheta) = 43.412784170470445 MSE = 8425.773432121185
norm(dtheta) = 43.15791424354429 MSE = 8425.643385202384
norm(dtheta) = 42.90454061965805 MSE = 8425.514101769604
norm(dtheta) = 42.652654514233575 MSE = 8425.385577340398
norm(dtheta) = 42.402247194262856 MSE = 8425.257807458742
norm(dtheta) = 42.153309978015955 MSE = 8425.130787694989
norm(dtheta) = 41.905834234733966 MSE = 8425.004513645092
norm(dtheta) = 41.65981138432469 MSE = 8424.878980931146
norm(dtheta) = 41.41523289707373 MSE = 8424.75418520095
norm(dtheta) = 41.17209029334125 MSE = 8424.630122127597
norm(dtheta) = 40.930375143264264 MSE = 8424.50678741006
norm(dtheta) = 40.690079066482845 MSE = 8424.384176771944
norm(dtheta) = 40.45119373183271 MSE = 8424.26228596231
norm(dtheta) = 40.21371085705438 MSE = 8424.141110755641
norm(dtheta) = 39.977622208516024 MSE = 8424.020646950064
norm(dtheta) = 39.74291960092808 MSE = 8423.900890369347
norm(dtheta) = 39.509594897052686 MSE = 8423.781836861775
norm(dtheta) = 39.27764000742123 MSE = 8423.663482299384
norm(dtheta) = 39.047046890065026 MSE = 8423.545822578835
norm(dtheta) = 38.817807550217815 MSE = 8423.428853620675
norm(dtheta) = 38.589914040062695 MSE = 8423.312571369768
norm(dtheta) = 38.36335845842787 MSE = 8423.19697179458
norm(dtheta) = 38.13813295054101 MSE = 8423.082050886864
norm(dtheta) = 37.91422970773989 MSE = 8422.967804662638
norm(dtheta) = 37.691640967203114 MSE = 8422.854229160846
norm(dtheta) = 37.47035901168313 MSE = 8422.741320443873
norm(dtheta) = 37.25037616924744 MSE = 8422.6290745967
norm(dtheta) = 37.03168481299565 MSE = 8422.51748772811
norm(dtheta) = 36.814277360804326 MSE = 8422.406555969317
norm(dtheta) = 36.59814627507069 MSE = 8422.296275474131
norm(dtheta) = 36.38328406243685 MSE = 8422.186642418983
norm(dtheta) = 36.16968327353829 MSE = 8422.077653003624
norm(dtheta) = 35.95733650274599 MSE = 8421.969303448015
norm(dtheta) = 35.74623638791095 MSE = 8421.861589996814
norm(dtheta) = 35.53637561009955 MSE = 8421.754508914913
norm(dtheta) = 35.32774689335541 MSE = 8421.648056489941
norm(dtheta) = 35.12034300442977 MSE = 8421.542229031185
norm(dtheta) = 34.91415675254284 MSE = 8421.43702286952
norm(dtheta) = 34.70918098913189 MSE = 8421.332434357239
norm(dtheta) = 34.50540860760036 MSE = 8421.22845986863
norm(dtheta) = 34.30283254307713 MSE = 8421.125095798518
norm(dtheta) = 34.10144577215993 MSE = 8421.022338563458
norm(dtheta) = 33.90124131268834 MSE = 8420.920184600274
norm(dtheta) = 33.70221222348834 MSE = 8420.818630367852
norm(dtheta) = 33.504351604139515 MSE = 8420.717672345196
norm(dtheta) = 33.30765259472809 MSE = 8420.617307031825
norm(dtheta) = 33.112108375618085 MSE = 8420.517530948266
norm(dtheta) = 32.91771216721237 MSE = 8420.418340634737
norm(dtheta) = 32.72445722970786 MSE = 8420.319732652626
norm(dtheta) = 32.5323368628795 MSE = 8420.221703583458
norm(dtheta) = 32.34134440582878 MSE = 8420.12425002808
norm(dtheta) = 32.15147323676982 MSE = 8420.027368607953
norm(dtheta) = 31.96271677278863 MSE = 8419.93105596389
norm(dtheta) = 31.775068469622177 MSE = 8419.835308756947
norm(dtheta) = 31.58852182141841 MSE = 8419.740123667572
norm(dtheta) = 31.403070360532265 MSE = 8419.645497395428
norm(dtheta) = 31.21870765728403 MSE = 8419.551426659831
norm(dtheta) = 31.035427319738435 MSE = 8419.457908199509
norm(dtheta) = 30.853222993488487 MSE = 8419.364938771932
norm(dtheta) = 30.672088361436305 MSE = 8419.272515153889
norm(dtheta) = 30.492017143567512 MSE = 8419.180634140985
norm(dtheta) = 30.313003096737592 MSE = 8419.08929254782
norm(dtheta) = 30.135040014453637 MSE = 8418.998487207371
norm(dtheta) = 29.958121726662267 MSE = 8418.908214971261
norm(dtheta) = 29.782242099530617 MSE = 8418.818472710042
norm(dtheta) = 29.607395035235676 MSE = 8418.729257312052
norm(dtheta) = 29.43357447176383 MSE = 8418.640565684305
norm(dtheta) = 29.260774382675972 MSE = 8418.552394751658
norm(dtheta) = 29.08898877692344 MSE = 8418.464741457456
norm(dtheta) = 28.918211698627772 MSE = 8418.377602762357
norm(dtheta) = 28.748437226874206 MSE = 8418.290975645652
norm(dtheta) = 28.579659475510372 MSE = 8418.2048571035
norm(dtheta) = 28.411872592941098 MSE = 8418.119244150555
norm(dtheta) = 28.245070761926446 MSE = 8418.034133818242
norm(dtheta) = 28.079248199375787 MSE = 8417.949523156112
norm(dtheta) = 27.914399156150917 MSE = 8417.865409230066
norm(dtheta) = 27.7505179168669 MSE = 8417.78178912462
norm(dtheta) = 27.587598799691193 MSE = 8417.698659939879
norm(dtheta) = 27.42563615614992 MSE = 8417.616018794495
norm(dtheta) = 27.264624370930918 MSE = 8417.533862822434
norm(dtheta) = 27.10455786168938 MSE = 8417.452189176052
norm(dtheta) = 26.945431078849094 MSE = 8417.370995023111
norm(dtheta) = 26.787238505418138 MSE = 8417.290277549177
norm(dtheta) = 26.629974656796964 MSE = 8417.210033955147
norm(dtheta) = 26.473634080578105 MSE = 8417.130261458939
norm(dtheta) = 26.318211356369662 MSE = 8417.05095729546
norm(dtheta) = 26.163701095600587 MSE = 8416.972118714788
norm(dtheta) = 26.01009794133549 MSE = 8416.893742983406
norm(dtheta) = 25.857396568086727 MSE = 8416.815827384075
norm(dtheta) = 25.705591681635806 MSE = 8416.738369215696
norm(dtheta) = 25.554678018841063 MSE = 8416.661365792479
norm(dtheta) = 25.40465034746656 MSE = 8416.58481444482
norm(dtheta) = 25.25550346598612 MSE = 8416.508712518602
norm(dtheta) = 25.10723220341728 MSE = 8416.433057375338
norm(dtheta) = 24.959831419131387 MSE = 8416.357846392088
norm(dtheta) = 24.813296002680985 MSE = 8416.28307696121
norm(dtheta) = 24.667620873622052 MSE = 8416.208746490274
norm(dtheta) = 24.522800981337753 MSE = 8416.134852402454
norm(dtheta) = 24.378831304861723 MSE = 8416.061392135709
norm(dtheta) = 24.2357068527036 MSE = 8415.988363143235
norm(dtheta) = 24.093422662679945 MSE = 8415.91576289297
norm(dtheta) = 23.951973801735438 MSE = 8415.843588867716
norm(dtheta) = 23.81135536578125 MSE = 8415.771838565428
norm(dtheta) = 23.671562479516293 MSE = 8415.700509498272
norm(dtheta) = 23.532590296263216 MSE = 8415.629599193391
norm(dtheta) = 23.394433997796476 MSE = 8415.55910519225
norm(dtheta) = 23.25708879418115 MSE = 8415.489025050765
norm(dtheta) = 23.120549923598567 MSE = 8415.419356339276
norm(dtheta) = 22.984812652191454 MSE = 8415.350096642293
norm(dtheta) = 22.849872273890856 MSE = 8415.281243558795
norm(dtheta) = 22.715724110258286 MSE = 8415.212794701209
norm(dtheta) = 22.582363510317332 MSE = 8415.144747696519
norm(dtheta) = 22.449785850403156 MSE = 8415.077100185443
norm(dtheta) = 22.317986533990126 MSE = 8415.009849823055
norm(dtheta) = 22.18696099154325 MSE = 8414.942994277368
norm(dtheta) = 22.05670468034994 MSE = 8414.876531230215
norm(dtheta) = 21.92721308437249 MSE = 8414.810458377837
norm(dtheta) = 21.79848171407615 MSE = 8414.744773429364
norm(dtheta) = 21.67050610629594 MSE = 8414.679474107083
norm(dtheta) = 21.54328182406271 MSE = 8414.614558147576
norm(dtheta) = 21.416804456456198 MSE = 8414.550023299678
norm(dtheta) = 21.291069618453108 MSE = 8414.485867326332
norm(dtheta) = 21.166072950774698 MSE = 8414.42208800292
norm(dtheta) = 21.04181011973317 MSE = 8414.358683118333
norm(dtheta) = 20.918276817085236 MSE = 8414.295650474356
norm(dtheta) = 20.795468759878414 MSE = 8414.232987885529
norm(dtheta) = 20.673381690304033 MSE = 8414.170693179245
norm(dtheta) = 20.552011375556262 MSE = 8414.108764196044
norm(dtheta) = 20.43135360767183 MSE = 8414.047198788388
norm(dtheta) = 20.311404203396275 MSE = 8413.985994822107
norm(dtheta) = 20.19215900403339 MSE = 8413.925150175086
norm(dtheta) = 20.07361387530204 MSE = 8413.864662737613
norm(dtheta) = 19.95576470719376 MSE = 8413.804530413086
norm(dtheta) = 19.838607413827784 MSE = 8413.744751116186
norm(dtheta) = 19.722137933311465 MSE = 8413.685322774807
norm(dtheta) = 19.60635222759813 MSE = 8413.626243327923
norm(dtheta) = 19.491246282348694 MSE = 8413.56751072787
norm(dtheta) = 19.376816106794227 MSE = 8413.509122938069
norm(dtheta) = 19.263057733590333 MSE = 8413.451077934109
norm(dtheta) = 19.149967218687117 MSE = 8413.393373703691
norm(dtheta) = 19.03754064118838 MSE = 8413.33600824606
norm(dtheta) = 18.925774103218153 MSE = 8413.278979572326
norm(dtheta) = 18.814663729783195 MSE = 8413.222285705573
norm(dtheta) = 18.704205668639947 MSE = 8413.165924679864
norm(dtheta) = 18.594396090162856 MSE = 8413.109894541243
norm(dtheta) = 18.485231187204054 MSE = 8413.054193347209
norm(dtheta) = 18.376707174973447 MSE = 8412.998819166596
norm(dtheta) = 18.268820290898166 MSE = 8412.943770079206
norm(dtheta) = 18.16156679449375 MSE = 8412.889044176709
norm(dtheta) = 18.05494296723574 MSE = 8412.834639562047
norm(dtheta) = 17.948945112432284 MSE = 8412.780554348516
norm(dtheta) = 17.843569555092312 MSE = 8412.726786661307
norm(dtheta) = 17.738812641802582 MSE = 8412.673334635778
norm(dtheta) = 17.634670740595624 MSE = 8412.620196419339
norm(dtheta) = 17.531140240828574 MSE = 8412.56737016932
norm(dtheta) = 17.42821755305394 MSE = 8412.514854054189
norm(dtheta) = 17.325899108901645 MSE = 8412.462646253336
norm(dtheta) = 17.224181360946055 MSE = 8412.410744956629
norm(dtheta) = 17.12306078259144 MSE = 8412.35914836467
norm(dtheta) = 17.022533867945153 MSE = 8412.307854688625
norm(dtheta) = 16.922597131695845 MSE = 8412.256862149829
norm(dtheta) = 16.823247108994412 MSE = 8412.206168980534
norm(dtheta) = 16.724480355334343 MSE = 8412.155773423374
norm(dtheta) = 16.626293446432022 MSE = 8412.105673730845
norm(dtheta) = 16.528682978104086 MSE = 8412.055868166257
norm(dtheta) = 16.43164556615692 MSE = 8412.006355002257
norm(dtheta) = 16.33517784626026 MSE = 8411.957132522917
norm(dtheta) = 16.23927647383967 MSE = 8411.908199021347
norm(dtheta) = 16.143938123953717 MSE = 8411.859552800912
norm(dtheta) = 16.049159491180887 MSE = 8411.811192175206
norm(dtheta) = 15.954937289506697 MSE = 8411.763115467316
norm(dtheta) = 15.861268252210223 MSE = 8411.715321010734
norm(dtheta) = 15.768149131741072 MSE = 8411.667807148193
norm(dtheta) = 15.675576699626033 MSE = 8411.620572232243
norm(dtheta) = 15.583547746337592 MSE = 8411.573614625597
norm(dtheta) = 15.4920590811918 MSE = 8411.526932699755
norm(dtheta) = 15.40110753224068 MSE = 8411.480524836574
norm(dtheta) = 15.310689946155058 MSE = 8411.434389426906
norm(dtheta) = 15.220803188118378 MSE = 8411.388524871423
norm(dtheta) = 15.131444141720232 MSE = 8411.342929579549
norm(dtheta) = 15.042609708844456 MSE = 8411.297601971379
norm(dtheta) = 14.954296809564383 MSE = 8411.252540474296
norm(dtheta) = 14.866502382033095 MSE = 8411.207743526635
norm(dtheta) = 14.779223382382694 MSE = 8411.163209575201
norm(dtheta) = 14.692456784612387 MSE = 8411.118937075797
norm(dtheta) = 14.606199580489585 MSE = 8411.074924493509
norm(dtheta) = 14.520448779438428 MSE = 8411.031170302487
norm(dtheta) = 14.435201408444536 MSE = 8410.987672985686
norm(dtheta) = 14.350454511944497 MSE = 8410.944431035372
norm(dtheta) = 14.266205151729217 MSE = 8410.901442951674
norm(dtheta) = 14.182450406836075 MSE = 8410.8587072449
norm(dtheta) = 14.099187373455038 MSE = 8410.816222432828
norm(dtheta) = 14.01641316482071 MSE = 8410.773987042825
norm(dtheta) = 13.934124911115491 MSE = 8410.731999610412
norm(dtheta) = 13.852319759372312 MSE = 8410.690258679806
norm(dtheta) = 13.770994873373162 MSE = 8410.648762803969
norm(dtheta) = 13.690147433547887 MSE = 8410.607510544187
norm(dtheta) = 13.6097746368838 MSE = 8410.566500470324
norm(dtheta) = 13.529873696820118 MSE = 8410.525731160325
norm(dtheta) = 13.450441843159329 MSE = 8410.48520120072
norm(dtheta) = 13.371476321965696 MSE = 8410.444909186597
norm(dtheta) = 13.292974395469992 MSE = 8410.404853720687
norm(dtheta) = 13.214933341979119 MSE = 8410.365033414344
norm(dtheta) = 13.13735045577665 MSE = 8410.32544688722
norm(dtheta) = 13.060223047030943 MSE = 8410.286092766562
norm(dtheta) = 12.983548441702316 MSE = 8410.246969688114
norm(dtheta) = 12.907323981450663 MSE = 8410.208076295172
norm(dtheta) = 12.831547023540713 MSE = 8410.169411239654
norm(dtheta) = 12.756214940755495 MSE = 8410.130973180716
norm(dtheta) = 12.68132512129732 MSE = 8410.092760785972
norm(dtheta) = 12.606874968706714 MSE = 8410.054772730413
norm(dtheta) = 12.532861901766177 MSE = 8410.017007697246
norm(dtheta) = 12.459283354408386 MSE = 8409.979464376578
norm(dtheta) = 12.3861367756382 MSE = 8409.942141467092
norm(dtheta) = 12.31341962943124 MSE = 8409.90503767501
norm(dtheta) = 12.24112939465166 MSE = 8409.868151713612
norm(dtheta) = 12.169263564969668 MSE = 8409.831482304053
norm(dtheta) = 12.097819648765046 MSE = 8409.795028175007
norm(dtheta) = 12.026795169047475 MSE = 8409.758788063142
norm(dtheta) = 11.956187663370413 MSE = 8409.722760711056
norm(dtheta) = 11.88599468373809 MSE = 8409.686944870302
norm(dtheta) = 11.816213796533527 MSE = 8409.65133929906
norm(dtheta) = 11.74684258242323 MSE = 8409.615942762488
norm(dtheta) = 11.677878636277482 MSE = 8409.580754033965
norm(dtheta) = 11.609319567088024 MSE = 8409.545771893043
norm(dtheta) = 11.541162997883463 MSE = 8409.510995127026
norm(dtheta) = 11.473406565645892 MSE = 8409.476422530164
norm(dtheta) = 11.406047921231119 MSE = 8409.44205290391
norm(dtheta) = 11.339084729288835 MSE = 8409.407885056327
norm(dtheta) = 11.272514668173748 MSE = 8409.373917803128
norm(dtheta) = 11.206335429877134 MSE = 8409.340149966656
norm(dtheta) = 11.140544719934892 MSE = 8409.306580376091
norm(dtheta) = 11.07514025735662 MSE = 8409.273207867453
norm(dtheta) = 11.010119774541263 MSE = 8409.240031283727
norm(dtheta) = 10.945481017202818 MSE = 8409.207049474977
norm(dtheta) = 10.881221744287263 MSE = 8409.174261297276
norm(dtheta) = 10.817339727897524 MSE = 8409.141665614094
norm(dtheta) = 10.753832753219339 MSE = 8409.109261295249
norm(dtheta) = 10.690698618438377 MSE = 8409.077047217432
norm(dtheta) = 10.627935134667803 MSE = 8409.04502226328
norm(dtheta) = 10.56554012587081 MSE = 8409.013185323078
norm(dtheta) = 10.503511428787036 MSE = 8408.981535292774
norm(dtheta) = 10.441846892855105 MSE = 8408.950071074963
norm(dtheta) = 10.38054438013972 MSE = 8408.91879157879
norm(dtheta) = 10.319601765256069 MSE = 8408.887695719915
norm(dtheta) = 10.259016935299448 MSE = 8408.85678242017
norm(dtheta) = 10.198787789767763 MSE = 8408.82605060763
norm(dtheta) = 10.138912240491797 MSE = 8408.795499217074
norm(dtheta) = 10.079388211561279 MSE = 8408.76512718913
norm(dtheta) = 10.020213639251379 MSE = 8408.734933470649
norm(dtheta) = 9.961386471955958 MSE = 8408.704917015139
norm(dtheta) = 9.90290467011354 MSE = 8408.675076781592
norm(dtheta) = 9.84476620613396 MSE = 8408.6454117356
norm(dtheta) = 9.786969064332414 MSE = 8408.615920848666
norm(dtheta) = 9.729511240858137 MSE = 8408.58660309827
norm(dtheta) = 9.672390743623751 MSE = 8408.557457468007
norm(dtheta) = 9.61560559223895 MSE = 8408.528482947328
norm(dtheta) = 9.559153817937052 MSE = 8408.499678531589
norm(dtheta) = 9.503033463512262 MSE = 8408.47104322257
norm(dtheta) = 9.447242583249425 MSE = 8408.442576026875
norm(dtheta) = 9.3917792428517 MSE = 8408.414275957908
norm(dtheta) = 9.336641519385903 MSE = 8408.386142034362
norm(dtheta) = 9.281827501201247 MSE = 8408.358173280709
norm(dtheta) = 9.227335287873986 MSE = 8408.330368727586
norm(dtheta) = 9.173162990135584 MSE = 8408.302727410619
norm(dtheta) = 9.119308729809973 MSE = 8408.275248371938
norm(dtheta) = 9.065770639747804 MSE = 8408.247930658266
norm(dtheta) = 9.012546863760564 MSE = 8408.220773323063
norm(dtheta) = 8.959635556557492 MSE = 8408.193775424508
norm(dtheta) = 8.907034883681394 MSE = 8408.166936026306
norm(dtheta) = 8.854743021444783 MSE = 8408.140254198152
norm(dtheta) = 8.802758156867045 MSE = 8408.113729015282
norm(dtheta) = 8.751078487611283 MSE = 8408.08735955767
norm(dtheta) = 8.699702221921422 MSE = 8408.061144911055
norm(dtheta) = 8.648627578559669 MSE = 8408.035084166691
norm(dtheta) = 8.597852786748465 MSE = 8408.009176421154
norm(dtheta) = 8.547376086104963 MSE = 8407.98342077611
norm(dtheta) = 8.49719572657756 MSE = 8407.957816338414
norm(dtheta) = 8.447309968395938 MSE = 8407.932362220816
norm(dtheta) = 8.397717081996937 MSE = 8407.907057540226
norm(dtheta) = 8.348415347975719 MSE = 8407.881901419663
norm(dtheta) = 8.29940305702046 MSE = 8407.856892986874
norm(dtheta) = 8.250678509853744 MSE = 8407.832031374934
norm(dtheta) = 8.202240017176166 MSE = 8407.807315721577
norm(dtheta) = 8.154085899603873 MSE = 8407.782745170565
norm(dtheta) = 8.106214487613524 MSE = 8407.758318869253
norm(dtheta) = 8.058624121483058 MSE = 8407.734035971127
norm(dtheta) = 8.011313151236171 MSE = 8407.709895634169
norm(dtheta) = 7.964279936579522 MSE = 8407.685897021478
norm(dtheta) = 7.917522846852565 MSE = 8407.662039301025
norm(dtheta) = 7.871040260967071 MSE = 8407.63832164591
norm(dtheta) = 7.824830567353106 MSE = 8407.614743233373
norm(dtheta) = 7.7788921639013875 MSE = 8407.591303246172
norm(dtheta) = 7.733223457906038 MSE = 8407.568000871619
norm(dtheta) = 7.687822866016787 MSE = 8407.54483530185
norm(dtheta) = 7.642688814173441 MSE = 8407.52180573358
norm(dtheta) = 7.5978197375610055 MSE = 8407.498911368637
norm(dtheta) = 7.553214080548943 MSE = 8407.476151412868
norm(dtheta) = 7.508870296640569 MSE = 8407.453525077273
norm(dtheta) = 7.464786848418734 MSE = 8407.431031577882
norm(dtheta) = 7.420962207492114 MSE = 8407.408670133967
norm(dtheta) = 7.377394854440988 MSE = 8407.38643997106
norm(dtheta) = 7.334083278767304 MSE = 8407.364340317961
norm(dtheta) = 7.291025978840811 MSE = 8407.342370408705
norm(dtheta) = 7.248221461846111 MSE = 8407.32052948134
norm(dtheta) = 7.205668243733508 MSE = 8407.298816778877
norm(dtheta) = 7.163364849164114 MSE = 8407.277231548538
norm(dtheta) = 7.121309811463044 MSE = 8407.255773041585
norm(dtheta) = 7.079501672563091 MSE = 8407.23444051443
norm(dtheta) = 7.037938982958779 MSE = 8407.213233227192
norm(dtheta) = 6.99662030165467 MSE = 8407.19215044477
norm(dtheta) = 6.955544196113743 MSE = 8407.171191436482
norm(dtheta) = 6.914709242211573 MSE = 8407.150355475194
norm(dtheta) = 6.874114024181758 MSE = 8407.129641838681
norm(dtheta) = 6.833757134572087 MSE = 8407.109049808889
norm(dtheta) = 6.793637174192047 MSE = 8407.0885786718
norm(dtheta) = 6.7537527520650205 MSE = 8407.068227717597
norm(dtheta) = 6.7141024853829405 MSE = 8407.047996240704
norm(dtheta) = 6.674684999452592 MSE = 8407.027883539982
norm(dtheta) = 6.635498927653557 MSE = 8407.007888917798
norm(dtheta) = 6.596542911389402 MSE = 8406.988011680889
norm(dtheta) = 6.557815600038147 MSE = 8406.968251140555
norm(dtheta) = 6.519315650906874 MSE = 8406.948606611095
norm(dtheta) = 6.4810417291871945 MSE = 8406.929077411889
norm(dtheta) = 6.442992507905444 MSE = 8406.909662865517
norm(dtheta) = 6.405166667879567 MSE = 8406.89036229939
norm(dtheta) = 6.367562897672344 MSE = 8406.871175043558
norm(dtheta) = 6.3301798935442175 MSE = 8406.852100433376
norm(dtheta) = 6.293016359410807 MSE = 8406.83313780747
norm(dtheta) = 6.256071006796991 MSE = 8406.81428650814
norm(dtheta) = 6.21934255479288 MSE = 8406.79554588199
norm(dtheta) = 6.182829730006318 MSE = 8406.776915279328
norm(dtheta) = 6.146531266522132 MSE = 8406.75839405399
norm(dtheta) = 6.110445905858291 MSE = 8406.7399815641
norm(dtheta) = 6.0745723969202166 MSE = 8406.721677171186
norm(dtheta) = 6.0389094959579 MSE = 8406.70348024045
norm(dtheta) = 6.003455966523536 MSE = 8406.685390141321
norm(dtheta) = 5.968210579428697 MSE = 8406.667406246197
norm(dtheta) = 5.933172112700426 MSE = 8406.649527932126
norm(dtheta) = 5.898339351540743 MSE = 8406.631754578724
norm(dtheta) = 5.863711088283802 MSE = 8406.61408557023
norm(dtheta) = 5.829286122351973 MSE = 8406.596520293715
norm(dtheta) = 5.79506326021802 MSE = 8406.579058140222
norm(dtheta) = 5.761041315360571 MSE = 8406.561698504349
norm(dtheta) = 5.72721910822477 MSE = 8406.544440784453
norm(dtheta) = 5.693595466180308 MSE = 8406.527284382182
norm(dtheta) = 5.660169223481796 MSE = 8406.510228702256
norm(dtheta) = 5.626939221227056 MSE = 8406.493273153812
norm(dtheta) = 5.593904307317195 MSE = 8406.476417148631
norm(dtheta) = 5.561063336419379 MSE = 8406.459660102775
norm(dtheta) = 5.528415169922351 MSE = 8406.443001434887
norm(dtheta) = 5.495958675900879 MSE = 8406.426440567599
norm(dtheta) = 5.463692729075016 MSE = 8406.409976926589
norm(dtheta) = 5.4316162107691826 MSE = 8406.393609941335
norm(dtheta) = 5.399728008879352 MSE = 8406.377339043927
norm(dtheta) = 5.368027017825994 MSE = 8406.361163670796
norm(dtheta) = 5.336512138523955 MSE = 8406.345083260763
norm(dtheta) = 5.305182278339076 MSE = 8406.329097256274
norm(dtheta) = 5.27403635105107 MSE = 8406.313205103144
norm(dtheta) = 5.243073276818286 MSE = 8406.297406250624
norm(dtheta) = 5.212291982136689 MSE = 8406.281700150777
norm(dtheta) = 5.181691399808355 MSE = 8406.266086258922
norm(dtheta) = 5.1512704688957545 MSE = 8406.250564034177
norm(dtheta) = 5.1210281346934465 MSE = 8406.235132937752
norm(dtheta) = 5.090963348687057 MSE = 8406.219792435239
norm(dtheta) = 5.061075068518089 MSE = 8406.20454199443
norm(dtheta) = 5.031362257946162 MSE = 8406.189381086462
norm(dtheta) = 5.0018238868174025 MSE = 8406.174309186119
norm(dtheta) = 4.972458931023121 MSE = 8406.159325770614
norm(dtheta) = 4.943266372467748 MSE = 8406.144430320594
norm(dtheta) = 4.9142451990332585 MSE = 8406.129622319286
norm(dtheta) = 4.885394404542797 MSE = 8406.114901253615
norm(dtheta) = 4.856712988727933 MSE = 8406.100266613097
norm(dtheta) = 4.828199957192588 MSE = 8406.085717890544
norm(dtheta) = 4.799854321376891 MSE = 8406.071254581224
norm(dtheta) = 4.771675098525423 MSE = 8406.05687618379
norm(dtheta) = 4.743661311654046 MSE = 8406.042582199645
norm(dtheta) = 4.715811989512046 MSE = 8406.028372133575
norm(dtheta) = 4.688126166551498 MSE = 8406.014245492453
norm(dtheta) = 4.660602882894586 MSE = 8406.000201786734
norm(dtheta) = 4.6332411842964545 MSE = 8405.986240529563
norm(dtheta) = 4.606040122115351 MSE = 8405.9723612369
norm(dtheta) = 4.578998753280559 MSE = 8405.958563427543
norm(dtheta) = 4.552116140254186 MSE = 8405.944846622837
norm(dtheta) = 4.525391351005803 MSE = 8405.931210347437
norm(dtheta) = 4.4988234589766005 MSE = 8405.91765412862
norm(dtheta) = 4.472411543045239 MSE = 8405.904177496494
norm(dtheta) = 4.446154687500669 MSE = 8405.89077998335
norm(dtheta) = 4.420051982005939 MSE = 8405.877461125165
norm(dtheta) = 4.3941025215694465 MSE = 8405.864220459942
norm(dtheta) = 4.368305406512175 MSE = 8405.8510575286
norm(dtheta) = 4.342659742438019 MSE = 8405.83797187496
norm(dtheta) = 4.317164640200418 MSE = 8405.824963045277
norm(dtheta) = 4.291819215874232 MSE = 8405.812030588375
norm(dtheta) = 4.266622590721817 MSE = 8405.799174056034
norm(dtheta) = 4.241573891166978 MSE = 8405.786393002467
norm(dtheta) = 4.216672248759206 MSE = 8405.773686984563
norm(dtheta) = 4.191916800149748 MSE = 8405.76105556167
norm(dtheta) = 4.167306687055962 MSE = 8405.748498296043
norm(dtheta) = 4.142841056234084 MSE = 8405.736014752103
norm(dtheta) = 4.118519059451897 MSE = 8405.723604497387
norm(dtheta) = 4.0943398534552 MSE = 8405.711267101213
norm(dtheta) = 4.070302599939965 MSE = 8405.699002136384
norm(dtheta) = 4.046406465525083 MSE = 8405.686809176943
norm(dtheta) = 4.022650621721926 MSE = 8405.67468780071
norm(dtheta) = 3.9990342449047174 MSE = 8405.662637587075
norm(dtheta) = 3.9755565162843656 MSE = 8405.65065811856
norm(dtheta) = 3.9522166218772115 MSE = 8405.638748979774
norm(dtheta) = 3.929013752480015 MSE = 8405.626909757651
norm(dtheta) = 3.9059471036397957 MSE = 8405.615140041753
norm(dtheta) = 3.8830158756261755 MSE = 8405.603439424276
norm(dtheta) = 3.860219273403352 MSE = 8405.59180749929
norm(dtheta) = 3.837556506602933 MSE = 8405.580243863717
norm(dtheta) = 3.815026789498603 MSE = 8405.568748116191
norm(dtheta) = 3.792629340975085 MSE = 8405.557319858644
norm(dtheta) = 3.7703633845036 MSE = 8405.545958694724
norm(dtheta) = 3.748228148113654 MSE = 8405.53466423034
norm(dtheta) = 3.726222864367211 MSE = 8405.52343607408
norm(dtheta) = 3.704346770332819 MSE = 8405.51227383679
norm(dtheta) = 3.6825991075556495 MSE = 8405.501177131366
norm(dtheta) = 3.6609791220363643 MSE = 8405.490145572925
norm(dtheta) = 3.6394860642007267 MSE = 8405.479178778978
norm(dtheta) = 3.6181191888748634 MSE = 8405.46827636945
norm(dtheta) = 3.5968777552610196 MSE = 8405.457437966557
norm(dtheta) = 3.5757610269092117 MSE = 8405.446663194092
norm(dtheta) = 3.5547682716934945 MSE = 8405.435951678764
norm(dtheta) = 3.5338987617868503 MSE = 8405.425303048909
norm(dtheta) = 3.513151773633923 MSE = 8405.414716935984
norm(dtheta) = 3.4925265879278946 MSE = 8405.404192972343
norm(dtheta) = 3.4720224895857634 MSE = 8405.393730793297
norm(dtheta) = 3.451638767721272 MSE = 8405.383330036047
norm(dtheta) = 3.4313747156218595 MSE = 8405.372990340238
norm(dtheta) = 3.411229630725325 MSE = 8405.362711347136
norm(dtheta) = 3.391202814592178 MSE = 8405.352492700444
norm(dtheta) = 3.3712935728846354 MSE = 8405.34233404589
norm(dtheta) = 3.35150121534084 MSE = 8405.332235031414
norm(dtheta) = 3.331825055751766 MSE = 8405.322195306468
norm(dtheta) = 3.312264411936251 MSE = 8405.312214523225
norm(dtheta) = 3.292818605718733 MSE = 8405.302292335908
norm(dtheta) = 3.2734869629050123 MSE = 8405.292428400251
norm(dtheta) = 3.2542688132589372 MSE = 8405.282622374174
norm(dtheta) = 3.2351634904793647 MSE = 8405.272873917798
norm(dtheta) = 3.2161703321762873 MSE = 8405.263182693228
norm(dtheta) = 3.197288679849065 MSE = 8405.253548364262
norm(dtheta) = 3.1785178788630546 MSE = 8405.243970597117
norm(dtheta) = 3.1598572784268306 MSE = 8405.234449059333
norm(dtheta) = 3.141306231568942 MSE = 8405.224983421516
norm(dtheta) = 3.1228640951176905 MSE = 8405.215573354639
norm(dtheta) = 3.104530229675594 MSE = 8405.20621853317
norm(dtheta) = 3.086303999600027 MSE = 8405.196918632111
norm(dtheta) = 3.068184772980324 MSE = 8405.187673329558
norm(dtheta) = 3.0501719216150067 MSE = 8405.178482304567
norm(dtheta) = 3.0322648209915766 MSE = 8405.169345239143
norm(dtheta) = 3.0144628502619186 MSE = 8405.160261815758
norm(dtheta) = 2.9967653922254662 MSE = 8405.151231719707
norm(dtheta) = 2.9791718333037323 MSE = 8405.142254638244
norm(dtheta) = 2.961681563520135 MSE = 8405.133330259761
norm(dtheta) = 2.94429397648014 MSE = 8405.124458274919
norm(dtheta) = 2.9270084693488982 MSE = 8405.115638376199
norm(dtheta) = 2.909824442830276 MSE = 8405.106870257681
norm(dtheta) = 2.8927413011467897 MSE = 8405.098153615685
norm(dtheta) = 2.875758452018722 MSE = 8405.089488147596
norm(dtheta) = 2.8588753066443773 MSE = 8405.080873553281
norm(dtheta) = 2.842091279677085 MSE = 8405.072309533965
norm(dtheta) = 2.8254057892077187 MSE = 8405.063795792723
norm(dtheta) = 2.808818256743901 MSE = 8405.0553320342
norm(dtheta) = 2.792328107188348 MSE = 8405.046917965206
norm(dtheta) = 2.7759347688205276 MSE = 8405.038553294075
norm(dtheta) = 2.7596376732769237 MSE = 8405.030237730625
norm(dtheta) = 2.7434362555302796 MSE = 8405.021970986505
norm(dtheta) = 2.727329953870888 MSE = 8405.01375277543
norm(dtheta) = 2.7113182098859325 MSE = 8405.005582811908
norm(dtheta) = 2.695400468442428 MSE = 8404.997460813076
norm(dtheta) = 2.6795761776644613 MSE = 8404.989386497582
norm(dtheta) = 2.6638447889177317 MSE = 8404.981359584759
norm(dtheta) = 2.6482057567886876 MSE = 8404.973379796904
norm(dtheta) = 2.63265853906515 MSE = 8404.965446857277
norm(dtheta) = 2.6172025967187347 MSE = 8404.95756049074
norm(dtheta) = 2.6018373938850736 MSE = 8404.949720423701
norm(dtheta) = 2.5865623978466252 MSE = 8404.941926384663
norm(dtheta) = 2.5713770790125214 MSE = 8404.934178103254
norm(dtheta) = 2.5562809109013545 MSE = 8404.926475310916
norm(dtheta) = 2.541273370123415 MSE = 8404.918817740283
norm(dtheta) = 2.526353936359953 MSE = 8404.911205126293
norm(dtheta) = 2.511522092348313 MSE = 8404.903637204663
norm(dtheta) = 2.4967773238623816 MSE = 8404.896113713301
norm(dtheta) = 2.4821191196944126 MSE = 8404.888634391136
norm(dtheta) = 2.4675469716389573 MSE = 8404.881198978872
norm(dtheta) = 2.453060374472927 MSE = 8404.873807218997
norm(dtheta) = 2.438658825939929 MSE = 8404.866458854958
norm(dtheta) = 2.424341826733074 MSE = 8404.85915363185
norm(dtheta) = 2.410108880475002 MSE = 8404.851891296996
norm(dtheta) = 2.3959594937041624 MSE = 8404.844671597923
norm(dtheta) = 2.3818931758547874 MSE = 8404.837494284926
norm(dtheta) = 2.367909439241745 MSE = 8404.830359108459
norm(dtheta) = 2.3540077990435737 MSE = 8404.823265821668
norm(dtheta) = 2.340187773284331 MSE = 8404.816214178567
norm(dtheta) = 2.32644888281768 MSE = 8404.809203934708
norm(dtheta) = 2.3127906513110537 MSE = 8404.80223484669
norm(dtheta) = 2.299212605227309 MSE = 8404.795306673273
norm(dtheta) = 2.2857142738103073 MSE = 8404.788419174281
norm(dtheta) = 2.2722951890665435 MSE = 8404.781572110498
norm(dtheta) = 2.258954885751355 MSE = 8404.774765244849
norm(dtheta) = 2.2456929013508904 MSE = 8404.767998341264
norm(dtheta) = 2.2325087760656928 MSE = 8404.761271165264
norm(dtheta) = 2.219402052797579 MSE = 8404.754583483424
norm(dtheta) = 2.2063722771310244 MSE = 8404.747935063999
norm(dtheta) = 2.193418997316872 MSE = 8404.741325676474
norm(dtheta) = 2.1805417642613723 MSE = 8404.734755091495
norm(dtheta) = 2.1677401315038516 MSE = 8404.728223081773
norm(dtheta) = 2.155013655207055 MSE = 8404.721729420266
norm(dtheta) = 2.142361894138815 MSE = 8404.715273881968
norm(dtheta) = 2.129784409656879 MSE = 8404.708856243175
norm(dtheta) = 2.1172807656949204 MSE = 8404.702476281567
norm(dtheta) = 2.104850528746057 MSE = 8404.696133775613
norm(dtheta) = 2.0924932678488886 MSE = 8404.689828505676
norm(dtheta) = 2.080208554572029 MSE = 8404.6835602529
norm(dtheta) = 2.0679959629993445 MSE = 8404.677328800448
norm(dtheta) = 2.0558550697149256 MSE = 8404.671133931432
norm(dtheta) = 2.043785453788945 MSE = 8404.664975431695
norm(dtheta) = 2.031786696763132 MSE = 8404.658853087685
norm(dtheta) = 2.019858382635129 MSE = 8404.652766686935
norm(dtheta) = 2.008000097845684 MSE = 8404.646716018886
norm(dtheta) = 1.9962114312622796 MSE = 8404.640700872886
norm(dtheta) = 1.9844919741682403 MSE = 8404.63472104112
norm(dtheta) = 1.9728413202437718 MSE = 8404.628776316067
norm(dtheta) = 1.9612590655561095 MSE = 8404.622866491429
norm(dtheta) = 1.9497448085439901 MSE = 8404.61699136273
norm(dtheta) = 1.9382981500031742 MSE = 8404.611150726114
norm(dtheta) = 1.9269186930734106 MSE = 8404.60534437871
norm(dtheta) = 1.9156060432234148 MSE = 8404.59957211965
norm(dtheta) = 1.9043598082394035 MSE = 8404.59383374863
norm(dtheta) = 1.8931795982099344 MSE = 8404.58812906676
norm(dtheta) = 1.882065025511983 MSE = 8404.582457876031
norm(dtheta) = 1.8710157047984544 MSE = 8404.57681998022
norm(dtheta) = 1.860031252985515 MSE = 8404.571215183438
norm(dtheta) = 1.8491112892370631 MSE = 8404.565643291906
norm(dtheta) = 1.838255434953247 MSE = 8404.56010411182
norm(dtheta) = 1.8274633137574998 MSE = 8404.554597451473
norm(dtheta) = 1.8167345514820097 MSE = 8404.549123120009
norm(dtheta) = 1.806068776156715 MSE = 8404.54368092744
norm(dtheta) = 1.7954656179937396 MSE = 8404.538270685145
norm(dtheta) = 1.784924709378124 MSE = 8404.532892205541
norm(dtheta) = 1.7744456848514243 MSE = 8404.527545302546
norm(dtheta) = 1.764028181101764 MSE = 8404.522229790009
norm(dtheta) = 1.753671836950078 MSE = 8404.51694548413
norm(dtheta) = 1.743376293337205 MSE = 8404.51169220157
norm(dtheta) = 1.7331411933124756 MSE = 8404.506469760336
norm(dtheta) = 1.722966182020697 MSE = 8404.501277979183
norm(dtheta) = 1.712850906690311 MSE = 8404.496116678198
norm(dtheta) = 1.7027950166194794 MSE = 8404.490985678609
norm(dtheta) = 1.692798163167306 MSE = 8404.485884802203
norm(dtheta) = 1.6828599997384335 MSE = 8404.48081387235
norm(dtheta) = 1.672980181772143 MSE = 8404.475772713145
norm(dtheta) = 1.6631583667315042 MSE = 8404.470761149703
norm(dtheta) = 1.6533942140894489 MSE = 8404.465779008515
norm(dtheta) = 1.643687385319369 MSE = 8404.460826116943
norm(dtheta) = 1.6340375438811252 MSE = 8404.455902302776
norm(dtheta) = 1.624444355210423 MSE = 8404.451007395568
norm(dtheta) = 1.6149074867079005 MSE = 8404.446141225526
norm(dtheta) = 1.6054266077260333 MSE = 8404.441303624486
norm(dtheta) = 1.5960013895587908 MSE = 8404.436494423699
norm(dtheta) = 1.5866315054300457 MSE = 8404.4317134574
norm(dtheta) = 1.5773166304817734 MSE = 8404.426960559382
norm(dtheta) = 1.5680564417634508 MSE = 8404.422235564783
norm(dtheta) = 1.558850618219937 MSE = 8404.417538309857
norm(dtheta) = 1.5496988406824719 MSE = 8404.41286863209
norm(dtheta) = 1.5406007918543252 MSE = 8404.408226369127
norm(dtheta) = 1.53155615630261 MSE = 8404.403611360212
norm(dtheta) = 1.5225646204460073 MSE = 8404.399023445372
norm(dtheta) = 1.5136258725442369 MSE = 8404.394462465365
norm(dtheta) = 1.5047396026868125 MSE = 8404.389928262324
norm(dtheta) = 1.4959055027835146 MSE = 8404.38542067872
norm(dtheta) = 1.4871232665520708 MSE = 8404.380939558814
norm(dtheta) = 1.4783925895085712 MSE = 8404.376484746517
norm(dtheta) = 1.4697131689571379 MSE = 8404.372056087737
norm(dtheta) = 1.461084703978487 MSE = 8404.367653429284
norm(dtheta) = 1.4525068954199696 MSE = 8404.363276617909
norm(dtheta) = 1.4439794458853086 MSE = 8404.358925502454
norm(dtheta) = 1.4355020597245007 MSE = 8404.354599931394
norm(dtheta) = 1.427074443022993 MSE = 8404.350299755262
norm(dtheta) = 1.4186963035916085 MSE = 8404.346024824652
norm(dtheta) = 1.4103673509567163 MSE = 8404.34177499166
norm(dtheta) = 1.402087296350276 MSE = 8404.337550108829
norm(dtheta) = 1.3938558526990033 MSE = 8404.33335002966
norm(dtheta) = 1.3856727346153832 MSE = 8404.329174608481
norm(dtheta) = 1.3775376583876697 MSE = 8404.325023700609
norm(dtheta) = 1.3694503419686461 MSE = 8404.3208971619
norm(dtheta) = 1.3614105049681813 MSE = 8404.316794849663
norm(dtheta) = 1.3534178686417966 MSE = 8404.312716621547
norm(dtheta) = 1.3454721558809857 MSE = 8404.308662335843
norm(dtheta) = 1.3375730912050003 MSE = 8404.304631852485
norm(dtheta) = 1.329720400749589 MSE = 8404.300625031348
norm(dtheta) = 1.3219138122589789 MSE = 8404.296641733685
norm(dtheta) = 1.3141530550752762 MSE = 8404.292681821431
norm(dtheta) = 1.306437860129655 MSE = 8404.288745157206
norm(dtheta) = 1.2987679599334532 MSE = 8404.284831604657
norm(dtheta) = 1.2911430885678918 MSE = 8404.280941027613
norm(dtheta) = 1.2835629816752034 MSE = 8404.277073291787
norm(dtheta) = 1.2760273764499253 MSE = 8404.273228262864
norm(dtheta) = 1.2685360116298963 MSE = 8404.269405807545
norm(dtheta) = 1.2610886274859603 MSE = 8404.265605793475
norm(dtheta) = 1.2536849658142217 MSE = 8404.261828088389
norm(dtheta) = 1.246324769927014 MSE = 8404.258072561883
norm(dtheta) = 1.2390077846424952 MSE = 8404.254339083262
norm(dtheta) = 1.2317337562782598 MSE = 8404.250627523477
norm(dtheta) = 1.2245024326406684 MSE = 8404.246937753722
norm(dtheta) = 1.2173135630166971 MSE = 8404.243269646055
norm(dtheta) = 1.2101668981649936 MSE = 8404.23962307319
norm(dtheta) = 1.203062190307724 MSE = 8404.23599790904
norm(dtheta) = 1.1959991931219502 MSE = 8404.232394027385
norm(dtheta) = 1.188977661730101 MSE = 8404.228811303761
norm(dtheta) = 1.181997352693578 MSE = 8404.225249613608
norm(dtheta) = 1.1750580240014072 MSE = 8404.221708833755
norm(dtheta) = 1.1681594350642224 MSE = 8404.218188841292
norm(dtheta) = 1.161301346705365 MSE = 8404.214689514
norm(dtheta) = 1.1544835211517666 MSE = 8404.21121073093
norm(dtheta) = 1.1477057220264923 MSE = 8404.207752371074
norm(dtheta) = 1.1409677143406622 MSE = 8404.204314315
norm(dtheta) = 1.1342692644844652 MSE = 8404.200896443015
norm(dtheta) = 1.1276101402202336 MSE = 8404.197498636908
norm(dtheta) = 1.120990110672819 MSE = 8404.194120778817
norm(dtheta) = 1.1144089463234907 MSE = 8404.190762751514
norm(dtheta) = 1.1078664190003495 MSE = 8404.187424439004
norm(dtheta) = 1.1013623018707392 MSE = 8404.184105724753
norm(dtheta) = 1.0948963694348708 MSE = 8404.180806494629
norm(dtheta) = 1.0884683975159593 MSE = 8404.177526633654
norm(dtheta) = 1.0820781632532321 MSE = 8404.17426602806
norm(dtheta) = 1.0757254450951255 MSE = 8404.171024565114
norm(dtheta) = 1.069410022789677 MSE = 8404.16780213225
norm(dtheta) = 1.0631316773789932 MSE = 8404.16459861809
norm(dtheta) = 1.0568901911897688 MSE = 8404.161413910942
norm(dtheta) = 1.0506853478271154 MSE = 8404.158247900838
norm(dtheta) = 1.0445169321671055 MSE = 8404.155100478025
norm(dtheta) = 1.0383847303476716 MSE = 8404.151971533178
norm(dtheta) = 1.0322885297627458 MSE = 8404.148860957737
norm(dtheta) = 1.0262281190544562 MSE = 8404.145768644194
norm(dtheta) = 1.0202032881062222 MSE = 8404.142694485092
norm(dtheta) = 1.014213828034166 MSE = 8404.139638373854
norm(dtheta) = 1.0082595311817515 MSE = 8404.136600204754
norm(dtheta) = 1.002340191110346 MSE = 8404.133579872194
norm(dtheta) = 0.9964556025945245 MSE = 8404.130577271551
norm(dtheta) = 0.9906055616126364 MSE = 8404.127592298673
norm(dtheta) = 0.9847898653417982 MSE = 8404.12462485037
norm(dtheta) = 0.9790083121487723 MSE = 8404.121674823096
norm(dtheta) = 0.9732607015854422 MSE = 8404.118742115365
norm(dtheta) = 0.967546834379226 MSE = 8404.115826624911
norm(dtheta) = 0.9618665124277189 MSE = 8404.112928250886
norm(dtheta) = 0.956219538792281 MSE = 8404.11004689302
norm(dtheta) = 0.9506057176897074 MSE = 8404.107182450967
norm(dtheta) = 0.9450248544867528 MSE = 8404.10433482547
norm(dtheta) = 0.9394767556924114 MSE = 8404.101503918053
norm(dtheta) = 0.9339612289512208 MSE = 8404.098689630486
norm(dtheta) = 0.9284780830384881 MSE = 8404.09589186523
norm(dtheta) = 0.923027127850572 MSE = 8404.09311052544
norm(dtheta) = 0.9176081744008605 MSE = 8404.090345514154
norm(dtheta) = 0.9122210348117635 MSE = 8404.08759673585
norm(dtheta) = 0.9068655223091202 MSE = 8404.08486409544
norm(dtheta) = 0.901541451214828 MSE = 8404.082147497707
norm(dtheta) = 0.8962486369413254 MSE = 8404.079446848684
norm(dtheta) = 0.8909868959843686 MSE = 8404.076762055181
norm(dtheta) = 0.8857560459171469 MSE = 8404.074093023204
norm(dtheta) = 0.8805559053839525 MSE = 8404.071439660742
norm(dtheta) = 0.875386294093915 MSE = 8404.068801876176
norm(dtheta) = 0.870247032814134 MSE = 8404.066179577409
norm(dtheta) = 0.8651379433646837 MSE = 8404.0635726738
norm(dtheta) = 0.8600588486107774 MSE = 8404.060981074928
norm(dtheta) = 0.8550095724583471 MSE = 8404.058404690872
norm(dtheta) = 0.8499899398470547 MSE = 8404.055843432488
norm(dtheta) = 0.8449997767433427 MSE = 8404.05329721058
norm(dtheta) = 0.8400389101372663 MSE = 8404.050765937385
norm(dtheta) = 0.8351071680326743 MSE = 8404.048249524943
norm(dtheta) = 0.8302043794444622 MSE = 8404.045747885988
norm(dtheta) = 0.8253303743903643 MSE = 8404.043260933713
norm(dtheta) = 0.8204849838872011 MSE = 8404.040788581979
norm(dtheta) = 0.815668039943066 MSE = 8404.038330745028
norm(dtheta) = 0.8108793755524205 MSE = 8404.035887337754
norm(dtheta) = 0.8061188246897455 MSE = 8404.033458275293
norm(dtheta) = 0.8013862223055426 MSE = 8404.03104347359
norm(dtheta) = 0.7966814043181318 MSE = 8404.028642848754
norm(dtheta) = 0.7920042076095747 MSE = 8404.026256317553
norm(dtheta) = 0.787354470019591 MSE = 8404.02388379718
norm(dtheta) = 0.7827320303395878 MSE = 8404.021525205895
norm(dtheta) = 0.7781367283076559 MSE = 8404.0191804612
norm(dtheta) = 0.7735684046029535 MSE = 8404.01684948249
norm(dtheta) = 0.769026900839648 MSE = 8404.014532188225
norm(dtheta) = 0.7645120595620158 MSE = 8404.012228498737
norm(dtheta) = 0.7600237242383284 MSE = 8404.009938333633
norm(dtheta) = 0.7555617392564863 MSE = 8404.007661613898
norm(dtheta) = 0.7511259499173194 MSE = 8404.00539826062
norm(dtheta) = 0.7467162024303535 MSE = 8404.003148194932
norm(dtheta) = 0.7423323439076689 MSE = 8404.000911338871
norm(dtheta) = 0.7379742223590205 MSE = 8403.99868761541
norm(dtheta) = 0.7336416866864538 MSE = 8403.996476946977
norm(dtheta) = 0.7293345866794154 MSE = 8403.994279256887
norm(dtheta) = 0.7250527730083857 MSE = 8403.992094469171
norm(dtheta) = 0.7207960972219011 MSE = 8403.989922508148
norm(dtheta) = 0.7165644117386241 MSE = 8403.987763298259
norm(dtheta) = 0.712357569844248 MSE = 8403.985616764787
norm(dtheta) = 0.7081754256861811 MSE = 8403.983482833095
norm(dtheta) = 0.7040178342672709 MSE = 8403.981361429858
norm(dtheta) = 0.6998846514425167 MSE = 8403.979252480727
norm(dtheta) = 0.6957757339125056 MSE = 8403.977155913146
norm(dtheta) = 0.6916909392191044 MSE = 8403.97507165384
norm(dtheta) = 0.6876301257411899 MSE = 8403.972999631147
norm(dtheta) = 0.6835931526885692 MSE = 8403.970939772951
norm(dtheta) = 0.6795798800975031 MSE = 8403.96889200796
norm(dtheta) = 0.6755901688263255 MSE = 8403.96685626493
norm(dtheta) = 0.6716238805498801 MSE = 8403.96483247352
norm(dtheta) = 0.6676808777555808 MSE = 8403.962820563636
norm(dtheta) = 0.6637610237375299 MSE = 8403.960820465025
norm(dtheta) = 0.6598641825927843 MSE = 8403.958832108903
norm(dtheta) = 0.6559902192164803 MSE = 8403.956855426175
norm(dtheta) = 0.6521389992966565 MSE = 8403.95489034813
norm(dtheta) = 0.6483103893094552 MSE = 8403.952936806858
norm(dtheta) = 0.6445042565159388 MSE = 8403.950994734552
norm(dtheta) = 0.6407204689555099 MSE = 8403.949064063807
norm(dtheta) = 0.6369588954429256 MSE = 8403.947144727737
norm(dtheta) = 0.6332194055625805 MSE = 8403.94523665966
norm(dtheta) = 0.6295018696651237 MSE = 8403.94333979376
norm(dtheta) = 0.6258061588616792 MSE = 8403.941454064028
norm(dtheta) = 0.6221321450204609 MSE = 8403.939579404907
norm(dtheta) = 0.618479700762242 MSE = 8403.93771575209
norm(dtheta) = 0.6148486994545388 MSE = 8403.935863040304
norm(dtheta) = 0.6112390152097709 MSE = 8403.934021205363
norm(dtheta) = 0.607650522878174 MSE = 8403.932190183681
norm(dtheta) = 0.6040830980454751 MSE = 8403.930369911523
norm(dtheta) = 0.6005366170273236 MSE = 8403.928560326081
norm(dtheta) = 0.597010956866092 MSE = 8403.926761364299
norm(dtheta) = 0.5935059953250617 MSE = 8403.924972964078
norm(dtheta) = 0.5900216108863373 MSE = 8403.923195063238
norm(dtheta) = 0.5865576827447138 MSE = 8403.921427600106
norm(dtheta) = 0.5831140908040131 MSE = 8403.919670513513
norm(dtheta) = 0.5796907156738447 MSE = 8403.917923742567
norm(dtheta) = 0.5762874386642725 MSE = 8403.916187226692
norm(dtheta) = 0.5729041417820507 MSE = 8403.914460905691
norm(dtheta) = 0.5695407077270845 MSE = 8403.912744719493
norm(dtheta) = 0.5661970198874788 MSE = 8403.911038608741
norm(dtheta) = 0.5628729623364945 MSE = 8403.909342514356
norm(dtheta) = 0.5595684198274378 MSE = 8403.907656377572
norm(dtheta) = 0.5562832777905102 MSE = 8403.905980139903
norm(dtheta) = 0.5530174223282581 MSE = 8403.90431374285
norm(dtheta) = 0.5497707402123733 MSE = 8403.902657129422
norm(dtheta) = 0.5465431188787933 MSE = 8403.901010241554
norm(dtheta) = 0.5433344464246895 MSE = 8403.899373022141
norm(dtheta) = 0.5401446116040908 MSE = 8403.897745414648
norm(dtheta) = 0.5369735038238889 MSE = 8403.896127362757
norm(dtheta) = 0.5338210131406265 MSE = 8403.894518810008
norm(dtheta) = 0.5306870302560557 MSE = 8403.892919700944
norm(dtheta) = 0.5275714465134478 MSE = 8403.891329979806
norm(dtheta) = 0.5244741538947182 MSE = 8403.88974959187
norm(dtheta) = 0.5213950450149327 MSE = 8403.888178482315
norm(dtheta) = 0.5183340131204438 MSE = 8403.88661659633
norm(dtheta) = 0.5152909520839002 MSE = 8403.885063879903
norm(dtheta) = 0.5122657564008923 MSE = 8403.883520279265
norm(dtheta) = 0.5092583211867157 MSE = 8403.881985740783
norm(dtheta) = 0.5062685421725608 MSE = 8403.880460211542
norm(dtheta) = 0.5032963157011985 MSE = 8403.878943638492
norm(dtheta) = 0.5003415387245973 MSE = 8403.877435968718
norm(dtheta) = 0.49740410879910646 MSE = 8403.875937150353
norm(dtheta) = 0.4944839240830003 MSE = 8403.874447131515
norm(dtheta) = 0.49158088333217437 MSE = 8403.872965860293
norm(dtheta) = 0.48869488589710625 MSE = 8403.871493285304
norm(dtheta) = 0.4858258317187077 MSE = 8403.87002935559
norm(dtheta) = 0.48297362132619404 MSE = 8403.86857402047
norm(dtheta) = 0.4801381558316261 MSE = 8403.867127229198
norm(dtheta) = 0.47731933692879813 MSE = 8403.865688931939
norm(dtheta) = 0.4745170668876984 MSE = 8403.864259078808
norm(dtheta) = 0.4717312485525456 MSE = 8403.86283762012
norm(dtheta) = 0.468961785337882 MSE = 8403.861424506415
norm(dtheta) = 0.46620858122530595 MSE = 8403.860019689058
norm(dtheta) = 0.46347154075996444 MSE = 8403.858623119022
norm(dtheta) = 0.46075056904750356 MSE = 8403.85723474818
norm(dtheta) = 0.4580455717508191 MSE = 8403.855854528296
norm(dtheta) = 0.45535645508643324 MSE = 8403.85448241128
norm(dtheta) = 0.45268312582160175 MSE = 8403.853118349896
norm(dtheta) = 0.45002549127070257 MSE = 8403.851762296734
norm(dtheta) = 0.4473834592925797 MSE = 8403.850414204653
norm(dtheta) = 0.44475693828684426 MSE = 8403.849074026944
norm(dtheta) = 0.4421458371910368 MSE = 8403.847741717442
norm(dtheta) = 0.43955006547702113 MSE = 8403.846417229544
norm(dtheta) = 0.43696953314835313 MSE = 8403.845100517658
norm(dtheta) = 0.43440415073693767 MSE = 8403.843791535988
norm(dtheta) = 0.43185382929978844 MSE = 8403.842490238952
norm(dtheta) = 0.42931848041621706 MSE = 8403.841196581892
norm(dtheta) = 0.42679801618480545 MSE = 8403.83991051959
norm(dtheta) = 0.4242923492195669 MSE = 8403.838632007595
norm(dtheta) = 0.42180139264836736 MSE = 8403.837361001391
norm(dtheta) = 0.41932506010866105 MSE = 8403.836097457279
norm(dtheta) = 0.4168632657447456 MSE = 8403.834841331181
norm(dtheta) = 0.41441592420525974 MSE = 8403.833592579665
norm(dtheta) = 0.4119829506400461 MSE = 8403.832351159215
norm(dtheta) = 0.4095642606966541 MSE = 8403.831117027123
norm(dtheta) = 0.407159770518251 MSE = 8403.829890140314
norm(dtheta) = 0.4047693967399519 MSE = 8403.82867045645
norm(dtheta) = 0.40239305648660734 MSE = 8403.827457933048
norm(dtheta) = 0.40003066736944676 MSE = 8403.826252528333
norm(dtheta) = 0.39768214748356673 MSE = 8403.825054200212
norm(dtheta) = 0.39534741540480595 MSE = 8403.82386290748
norm(dtheta) = 0.39302639018684743 MSE = 8403.822678608512
norm(dtheta) = 0.3907189913588795 MSE = 8403.821501262491
norm(dtheta) = 0.38842513892256253 MSE = 8403.820330828581
norm(dtheta) = 0.38614475334871284 MSE = 8403.819167265856
norm(dtheta) = 0.38387775557573717 MSE = 8403.818010534254
norm(dtheta) = 0.3816240670056765 MSE = 8403.816860593815
norm(dtheta) = 0.37938360950230116 MSE = 8403.815717404452
norm(dtheta) = 0.3771563053879468 MSE = 8403.814580926473
norm(dtheta) = 0.374942077441137 MSE = 8403.813451120712
norm(dtheta) = 0.372740848893607 MSE = 8403.812327947742
norm(dtheta) = 0.37055254342771465 MSE = 8403.811211368928
norm(dtheta) = 0.368377085174274 MSE = 8403.81010134537
norm(dtheta) = 0.36621439870887607 MSE = 8403.808997838469
norm(dtheta) = 0.3640644090506912 MSE = 8403.807900810027
norm(dtheta) = 0.3619270416582249 MSE = 8403.806810222346
norm(dtheta) = 0.35980222242823284 MSE = 8403.805726037059
norm(dtheta) = 0.357689877692517 MSE = 8403.804648216937
norm(dtheta) = 0.3555899342151783 MSE = 8403.803576724751
norm(dtheta) = 0.3535023191899349 MSE = 8403.802511522881
norm(dtheta) = 0.35142696023866954 MSE = 8403.801452574788
norm(dtheta) = 0.3493637854074968 MSE = 8403.800399843547
norm(dtheta) = 0.3473127231657021 MSE = 8403.799353292707
norm(dtheta) = 0.34527370240186933 MSE = 8403.798312886116
norm(dtheta) = 0.3432466524221599 MSE = 8403.79727858753
norm(dtheta) = 0.34123150294805976 MSE = 8403.796250361107
norm(dtheta) = 0.3392281841134936 MSE = 8403.795228171404
norm(dtheta) = 0.33723662646239433 MSE = 8403.794211982638
norm(dtheta) = 0.3352567609469125 MSE = 8403.793201759854
norm(dtheta) = 0.3332885189241452 MSE = 8403.79219746798
norm(dtheta) = 0.3313318321541033 MSE = 8403.791199072015
norm(dtheta) = 0.3293866327977781 MSE = 8403.790206537456
norm(dtheta) = 0.3274528534144963 MSE = 8403.789219830149
norm(dtheta) = 0.32553042695906365 MSE = 8403.788238915578
norm(dtheta) = 0.3236192867802887 MSE = 8403.787263759621
norm(dtheta) = 0.3217193666180954 MSE = 8403.786294328904
norm(dtheta) = 0.31983060060125956 MSE = 8403.785330589364
norm(dtheta) = 0.31795292324576163 MSE = 8403.784372508035
norm(dtheta) = 0.31608626945163104 MSE = 8403.783420051168
norm(dtheta) = 0.3142305745011956 MSE = 8403.782473186151
norm(dtheta) = 0.31238577405680595 MSE = 8403.781531880257
norm(dtheta) = 0.3105518041582952 MSE = 8403.780596100283
norm(dtheta) = 0.30872860122120555 MSE = 8403.779665814418
norm(dtheta) = 0.30691610203433506 MSE = 8403.778740989852
norm(dtheta) = 0.30511424375775203 MSE = 8403.777821595046
norm(dtheta) = 0.3033229639200233 MSE = 8403.776907597816
norm(dtheta) = 0.30154220041684954 MSE = 8403.775998966245
norm(dtheta) = 0.29977189150841826 MSE = 8403.775095669522
norm(dtheta) = 0.2980119758172522 MSE = 8403.774197675693
norm(dtheta) = 0.29626239232655616 MSE = 8403.77330495393
norm(dtheta) = 0.2945230803772618 MSE = 8403.772417473
norm(dtheta) = 0.2927939796669302 MSE = 8403.771535202583
norm(dtheta) = 0.2910750302467257 MSE = 8403.770658111658
norm(dtheta) = 0.2893661725201624 MSE = 8403.76978617008
norm(dtheta) = 0.287667347240311 MSE = 8403.76891934774
norm(dtheta) = 0.2859784955081924 MSE = 8403.768057613997
norm(dtheta) = 0.284299558770399 MSE = 8403.767200939741
norm(dtheta) = 0.2826304788178661 MSE = 8403.76634929462
norm(dtheta) = 0.2809711977826993 MSE = 8403.765502649541
norm(dtheta) = 0.27932165813700643 MSE = 8403.764660974819
norm(dtheta) = 0.27768180269024145 MSE = 8403.763824241754
norm(dtheta) = 0.27605157458836 MSE = 8403.762992420789
norm(dtheta) = 0.2744309173104407 MSE = 8403.762165483182
norm(dtheta) = 0.2728197746678083 MSE = 8403.76134340073
norm(dtheta) = 0.27121809080120146 MSE = 8403.760526144453
norm(dtheta) = 0.2696258101797318 MSE = 8403.759713686082
norm(dtheta) = 0.26804287759831497 MSE = 8403.758905997602
norm(dtheta) = 0.26646923817603263 MSE = 8403.758103051
norm(dtheta) = 0.2649048373541322 MSE = 8403.757304818228
norm(dtheta) = 0.2633496208939653 MSE = 8403.756511271846
norm(dtheta) = 0.26180353487608665 MSE = 8403.755722384218
norm(dtheta) = 0.26026652569642456 MSE = 8403.754938128264
norm(dtheta) = 0.25873854006671476 MSE = 8403.75415847607
norm(dtheta) = 0.25721952501074585 MSE = 8403.753383401616
norm(dtheta) = 0.2557094278637044 MSE = 8403.75261287718
norm(dtheta) = 0.2542081962698827 MSE = 8403.751846876465
norm(dtheta) = 0.2527157781810767 MSE = 8403.751085372796
norm(dtheta) = 0.25123212185429633 MSE = 8403.75032833985
norm(dtheta) = 0.24975717585066295 MSE = 8403.749575751288
norm(dtheta) = 0.24829088903331895 MSE = 8403.74882758115
norm(dtheta) = 0.24683321056528934 MSE = 8403.748083803384
norm(dtheta) = 0.24538408990830535 MSE = 8403.747344392194
norm(dtheta) = 0.24394347682069953 MSE = 8403.746609321874
norm(dtheta) = 0.2425113213560161 MSE = 8403.745878567173
norm(dtheta) = 0.24108757386062102 MSE = 8403.745152102689
norm(dtheta) = 0.23967218497262036 MSE = 8403.744429903023
norm(dtheta) = 0.2382651056199165 MSE = 8403.743711943343
norm(dtheta) = 0.23686628701852078 MSE = 8403.742998198639
norm(dtheta) = 0.23547568067068667 MSE = 8403.742288644253
norm(dtheta) = 0.23409323836364088 MSE = 8403.741583255622
norm(dtheta) = 0.23271891216746796 MSE = 8403.740882008202
norm(dtheta) = 0.23135265443370082 MSE = 8403.740184877583
norm(dtheta) = 0.22999441779365926 MSE = 8403.739491839828
norm(dtheta) = 0.2286441551565546 MSE = 8403.738802870712
norm(dtheta) = 0.227301819708347 MSE = 8403.738117946643
norm(dtheta) = 0.22596736490965494 MSE = 8403.737437043432
norm(dtheta) = 0.2246407444944781 MSE = 8403.736760137652
norm(dtheta) = 0.22332191246816596 MSE = 8403.736087206158
norm(dtheta) = 0.22201082310633855 MSE = 8403.735418225246
norm(dtheta) = 0.22070743095303483 MSE = 8403.734753171757
norm(dtheta) = 0.219411690818992 MSE = 8403.734092022622
norm(dtheta) = 0.21812355778042586 MSE = 8403.733434755111
norm(dtheta) = 0.21684298717721717 MSE = 8403.732781346138
norm(dtheta) = 0.2155699346113276 MSE = 8403.732131773377
norm(dtheta) = 0.21430435594576044 MSE = 8403.731486014247
norm(dtheta) = 0.2130462073020901 MSE = 8403.730844046115
norm(dtheta) = 0.21179544505989437 MSE = 8403.730205847023
norm(dtheta) = 0.21055202585479213 MSE = 8403.72957139446
norm(dtheta) = 0.2093159065767465 MSE = 8403.72894066704
norm(dtheta) = 0.20808704436913686 MSE = 8403.728313642192
norm(dtheta) = 0.2068653966267166 MSE = 8403.727690298598
norm(dtheta) = 0.20565092099442805 MSE = 8403.727070614865
norm(dtheta) = 0.20444357536596733 MSE = 8403.726454568692
norm(dtheta) = 0.20324331788208141 MSE = 8403.725842139675
norm(dtheta) = 0.2020501069294422 MSE = 8403.725233305837
norm(dtheta) = 0.2008639011387677 MSE = 8403.724628046715
norm(dtheta) = 0.19968465938394894 MSE = 8403.724026340478
norm(dtheta) = 0.19851234078003988 MSE = 8403.723428167092
norm(dtheta) = 0.19734690468252641 MSE = 8403.722833505515
norm(dtheta) = 0.1961883106848936 MSE = 8403.72224233482
norm(dtheta) = 0.19503651861840604 MSE = 8403.721654635061
norm(dtheta) = 0.1938914885499482 MSE = 8403.721070385549
norm(dtheta) = 0.1927531807808531 MSE = 8403.720489566103
norm(dtheta) = 0.19162155584546112 MSE = 8403.71991215646
norm(dtheta) = 0.19049657450972216 MSE = 8403.719338136838
norm(dtheta) = 0.18937819777043904 MSE = 8403.71876748712
norm(dtheta) = 0.18826638685266417 MSE = 8403.718200187594
norm(dtheta) = 0.18716110320956228 MSE = 8403.717636218662
norm(dtheta) = 0.18606230852064432 MSE = 8403.717075560742
norm(dtheta) = 0.18496996468991067 MSE = 8403.716518194171
norm(dtheta) = 0.18388403384555976 MSE = 8403.715964099914
norm(dtheta) = 0.1828044783380688 MSE = 8403.71541325876
norm(dtheta) = 0.18173126073837692 MSE = 8403.714865651255
norm(dtheta) = 0.18066434383792054 MSE = 8403.714321258944
norm(dtheta) = 0.1796036906461232 MSE = 8403.71378006254
norm(dtheta) = 0.17854926438962662 MSE = 8403.71324204346
norm(dtheta) = 0.17750102851116464 MSE = 8403.71270718292
norm(dtheta) = 0.176458946667898 MSE = 8403.712175462622
norm(dtheta) = 0.17542298273043386 MSE = 8403.711646863874
norm(dtheta) = 0.17439310078138046 MSE = 8403.711121368457
norm(dtheta) = 0.17336926511432613 MSE = 8403.710598958332
norm(dtheta) = 0.17235144023260687 MSE = 8403.710079614879
norm(dtheta) = 0.17133959084760264 MSE = 8403.709563320685
norm(dtheta) = 0.1703336818782864 MSE = 8403.709050057261
norm(dtheta) = 0.16933367844914263 MSE = 8403.708539807436
norm(dtheta) = 0.16833954588969605 MSE = 8403.70803255317
norm(dtheta) = 0.16735124973309554 MSE = 8403.707528276818
norm(dtheta) = 0.16636875571449303 MSE = 8403.707026960912
norm(dtheta) = 0.16539202977060088 MSE = 8403.706528588347
norm(dtheta) = 0.16442103803778493 MSE = 8403.706033141407
norm(dtheta) = 0.16345574685145495 MSE = 8403.705540603489
norm(dtheta) = 0.1624961227444275 MSE = 8403.705050956954
norm(dtheta) = 0.16154213244630913 MSE = 8403.704564185282
norm(dtheta) = 0.16059374288166317 MSE = 8403.704080271087
norm(dtheta) = 0.15965092116956844 MSE = 8403.70359919802
norm(dtheta) = 0.15871363462189558 MSE = 8403.70312094934
norm(dtheta) = 0.1577818507427347 MSE = 8403.702645508227
norm(dtheta) = 0.15685553722641493 MSE = 8403.702172858546
norm(dtheta) = 0.15593466195757863 MSE = 8403.701702983491
norm(dtheta) = 0.15501919300885797 MSE = 8403.701235867096
norm(dtheta) = 0.15410909864064798 MSE = 8403.700771493084
norm(dtheta) = 0.15320434729968838 MSE = 8403.700309845442
norm(dtheta) = 0.15230490761766297 MSE = 8403.699850907913
norm(dtheta) = 0.15141074841078248 MSE = 8403.699394664727
norm(dtheta) = 0.15052183867815905 MSE = 8403.698941100276
norm(dtheta) = 0.149638147600836 MSE = 8403.698490198502
norm(dtheta) = 0.14875964454102963 MSE = 8403.69804194389
norm(dtheta) = 0.14788629904049935 MSE = 8403.69759632087
norm(dtheta) = 0.14701808082006498 MSE = 8403.697153314091
norm(dtheta) = 0.14615495977824025 MSE = 8403.696712908231
norm(dtheta) = 0.14529690599018275 MSE = 8403.696275087726
norm(dtheta) = 0.14444388970692573 MSE = 8403.695839837796
norm(dtheta) = 0.1435958813540289 MSE = 8403.695407142954
norm(dtheta) = 0.14275285153047806 MSE = 8403.694976988629
norm(dtheta) = 0.14191477100837532 MSE = 8403.694549359496
norm(dtheta) = 0.14108161073088027 MSE = 8403.694124241127
norm(dtheta) = 0.1402533418121971 MSE = 8403.693701618311
norm(dtheta) = 0.13942993553568847 MSE = 8403.693281476686
norm(dtheta) = 0.13861136335361735 MSE = 8403.69286380174
norm(dtheta) = 0.13779759688568227 MSE = 8403.692448578868
norm(dtheta) = 0.13698860791832002 MSE = 8403.69203579373
norm(dtheta) = 0.13618436840350095 MSE = 8403.691625432006
norm(dtheta) = 0.13538485045794668 MSE = 8403.691217479305
norm(dtheta) = 0.13459002636197773 MSE = 8403.690811921893
norm(dtheta) = 0.13379986855877904 MSE = 8403.690408745319
norm(dtheta) = 0.13301434965310097 MSE = 8403.69000793569
norm(dtheta) = 0.13223344241092627 MSE = 8403.689609479232
norm(dtheta) = 0.1314571197576115 MSE = 8403.689213361935
norm(dtheta) = 0.13068535477790666 MSE = 8403.688819570263
norm(dtheta) = 0.1299181207143802 MSE = 8403.688428090541
norm(dtheta) = 0.12915539096660017 MSE = 8403.688038908998
norm(dtheta) = 0.12839713909040187 MSE = 8403.68765201233
norm(dtheta) = 0.12764333879694884 MSE = 8403.68726738709
norm(dtheta) = 0.1268939639517419 MSE = 8403.686885020084
norm(dtheta) = 0.1261489885734099 MSE = 8403.686504897527
norm(dtheta) = 0.12540838683357608 MSE = 8403.686127006962
norm(dtheta) = 0.12467213305516361 MSE = 8403.685751334719
norm(dtheta) = 0.12394020171202953 MSE = 8403.685377868042
norm(dtheta) = 0.12321256742753944 MSE = 8403.685006594049
norm(dtheta) = 0.1224892049746174 MSE = 8403.684637499711
norm(dtheta) = 0.12177008927388662 MSE = 8403.684270572057
norm(dtheta) = 0.12105519539326903 MSE = 8403.683905798947
norm(dtheta) = 0.12034449854708357 MSE = 8403.683543167183
norm(dtheta) = 0.11963797409525602 MSE = 8403.683182664305
norm(dtheta) = 0.11893559754221929 MSE = 8403.68282427793
norm(dtheta) = 0.1182373445362338 MSE = 8403.682467995388
norm(dtheta) = 0.1175431908686708 MSE = 8403.682113805004
norm(dtheta) = 0.11685311247289411 MSE = 8403.681761693702
norm(dtheta) = 0.11616708542354752 MSE = 8403.681411649559
norm(dtheta) = 0.11548508593589198 MSE = 8403.681063660653
norm(dtheta) = 0.11480709036462226 MSE = 8403.680717714567
norm(dtheta) = 0.11413307520338374 MSE = 8403.6803737995
norm(dtheta) = 0.11346301708382078 MSE = 8403.680031903757
norm(dtheta) = 0.11279689277471618 MSE = 8403.67969201478
norm(dtheta) = 0.11213467918120386 MSE = 8403.679354121576
norm(dtheta) = 0.11147635334417814 MSE = 8403.679018212004
norm(dtheta) = 0.11082189243898276 MSE = 8403.67868427452
norm(dtheta) = 0.11017127377549432 MSE = 8403.67835229762
norm(dtheta) = 0.10952447479627031 MSE = 8403.678022269605
norm(dtheta) = 0.10888147307653018 MSE = 8403.677694179149
norm(dtheta) = 0.1082422463232364 MSE = 8403.67736801479
norm(dtheta) = 0.10760677237410134 MSE = 8403.67704376525
norm(dtheta) = 0.10697502919688993 MSE = 8403.67672141954
norm(dtheta) = 0.10634699488891795 MSE = 8403.67640096624
norm(dtheta) = 0.10572264767589597 MSE = 8403.676082394053
norm(dtheta) = 0.10510196591149815 MSE = 8403.675765692462
norm(dtheta) = 0.1044849280762625 MSE = 8403.675450850033
norm(dtheta) = 0.10387151277742591 MSE = 8403.675137856015
norm(dtheta) = 0.10326169874760471 MSE = 8403.674826699385
norm(dtheta) = 0.1026554648442001 MSE = 8403.674517369678
norm(dtheta) = 0.1020527900489141 MSE = 8403.674209855963
norm(dtheta) = 0.10145365346665529 MSE = 8403.67390414747
norm(dtheta) = 0.10085803432520726 MSE = 8403.673600233851
norm(dtheta) = 0.1002659119741123 MSE = 8403.673298104744
norm(dtheta) = 0.09967726588432274 MSE = 8403.672997749021
norm(dtheta) = 0.09909207564717554 MSE = 8403.67269915682
norm(dtheta) = 0.09851032097396926 MSE = 8403.672402317477
norm(dtheta) = 0.09793198169483112 MSE = 8403.672107220962
norm(dtheta) = 0.09735703775872535 MSE = 8403.67181385683
norm(dtheta) = 0.09678546923196486 MSE = 8403.671522215154
norm(dtheta) = 0.09621725629808754 MSE = 8403.67123228538
norm(dtheta) = 0.09565237925685799 MSE = 8403.670944057783
norm(dtheta) = 0.0950908185237992 MSE = 8403.6706575226
norm(dtheta) = 0.09453255462930209 MSE = 8403.670372669514
norm(dtheta) = 0.09397756821822276 MSE = 8403.670089488793
norm(dtheta) = 0.09342584004892887 MSE = 8403.669807970475
norm(dtheta) = 0.09287735099266498 MSE = 8403.669528104756
norm(dtheta) = 0.09233208203321476 MSE = 8403.669249882247
norm(dtheta) = 0.0917900142658315 MSE = 8403.668973293075
norm(dtheta) = 0.0912511288967157 MSE = 8403.668698327898
norm(dtheta) = 0.09071540724260645 MSE = 8403.668424976884
norm(dtheta) = 0.09018283072969022 MSE = 8403.668153230634
norm(dtheta) = 0.08965338089349813 MSE = 8403.667883079926
norm(dtheta) = 0.08912703937760916 MSE = 8403.667614515054
norm(dtheta) = 0.08860378793356817 MSE = 8403.667347527025
norm(dtheta) = 0.08808360842014507 MSE = 8403.667082106336
norm(dtheta) = 0.08756648280232737 MSE = 8403.666818243926
norm(dtheta) = 0.08705239315133019 MSE = 8403.666555930575
norm(dtheta) = 0.08654132164325554 MSE = 8403.66629515737
norm(dtheta) = 0.08603325055923616 MSE = 8403.666035914906
norm(dtheta) = 0.08552816228407373 MSE = 8403.665778194658
norm(dtheta) = 0.08502603930620711 MSE = 8403.665521987119
norm(dtheta) = 0.0845268642168745 MSE = 8403.665267284003
norm(dtheta) = 0.08403061970938246 MSE = 8403.66501407609
norm(dtheta) = 0.08353728857879314 MSE = 8403.664762354852
norm(dtheta) = 0.08304685372110995 MSE = 8403.664512111493
norm(dtheta) = 0.08255929813260969 MSE = 8403.664263337081
norm(dtheta) = 0.08207460490969705 MSE = 8403.664016023245
norm(dtheta) = 0.08159275724781329 MSE = 8403.663770161365
norm(dtheta) = 0.08111373844113157 MSE = 8403.663525742755
norm(dtheta) = 0.08063753188173947 MSE = 8403.663282759226
norm(dtheta) = 0.0801641210594795 MSE = 8403.663041202106
norm(dtheta) = 0.07969348956095829 MSE = 8403.662801063236
norm(dtheta) = 0.07922562106915607 MSE = 8403.662562334397
norm(dtheta) = 0.07876049936292413 MSE = 8403.662325006746
norm(dtheta) = 0.07829810831632 MSE = 8403.662089072623
norm(dtheta) = 0.07783843189795361 MSE = 8403.661854523758
norm(dtheta) = 0.07738145417075167 MSE = 8403.661621351534
norm(dtheta) = 0.07692715929101565 MSE = 8403.661389548382
norm(dtheta) = 0.07647553150818431 MSE = 8403.661159106057
norm(dtheta) = 0.07602655516419968 MSE = 8403.660930016727
norm(dtheta) = 0.07558021469270962 MSE = 8403.660702272258
norm(dtheta) = 0.07513649461914247 MSE = 8403.660475865063
norm(dtheta) = 0.0746953795593012 MSE = 8403.660250786746
norm(dtheta) = 0.07425685421963771 MSE = 8403.660027030091
norm(dtheta) = 0.07382090339630726 MSE = 8403.659804586852
norm(dtheta) = 0.07338751197472917 MSE = 8403.659583449715
norm(dtheta) = 0.0729566649289499 MSE = 8403.659363610797
norm(dtheta) = 0.07252834732138036 MSE = 8403.659145062373
norm(dtheta) = 0.07210254430214527 MSE = 8403.658927797207
norm(dtheta) = 0.07167924110831693 MSE = 8403.658711807504
norm(dtheta) = 0.07125842306394786 MSE = 8403.658497085924
norm(dtheta) = 0.07084007557896993 MSE = 8403.658283624867
norm(dtheta) = 0.07042418414916053 MSE = 8403.658071417149
norm(dtheta) = 0.07001073435538352 MSE = 8403.657860455161
norm(dtheta) = 0.06959971186320878 MSE = 8403.657650731568
norm(dtheta) = 0.0691911024222371 MSE = 8403.657442239406
norm(dtheta) = 0.06878489186582253 MSE = 8403.657234971055
norm(dtheta) = 0.06838106611050813 MSE = 8403.657028919686
norm(dtheta) = 0.0679796111555264 MSE = 8403.656824078202
norm(dtheta) = 0.06758051308212351 MSE = 8403.656620439086
norm(dtheta) = 0.06718375805360886 MSE = 8403.656417995544
norm(dtheta) = 0.06678933231404509 MSE = 8403.656216740397
norm(dtheta) = 0.06639722218877996 MSE = 8403.65601666714
norm(dtheta) = 0.06600741408302888 MSE = 8403.655817768138
norm(dtheta) = 0.06561989448211725 MSE = 8403.655620036947
norm(dtheta) = 0.06523464995043517 MSE = 8403.65542346672
norm(dtheta) = 0.06485166713149546 MSE = 8403.65522805041
norm(dtheta) = 0.06447093274706152 MSE = 8403.655033781384
norm(dtheta) = 0.06409243359691581 MSE = 8403.654840652795
norm(dtheta) = 0.06371615655846559 MSE = 8403.65464865806
norm(dtheta) = 0.06334208858581927 MSE = 8403.654457790639
norm(dtheta) = 0.0629702167100286 MSE = 8403.654268043703
norm(dtheta) = 0.06260052803805544 MSE = 8403.654079410646
norm(dtheta) = 0.06223300975274614 MSE = 8403.653891885328
norm(dtheta) = 0.061867649111934216 MSE = 8403.653705460547
norm(dtheta) = 0.06150443344853397 MSE = 8403.653520130349
norm(dtheta) = 0.061143350169662145 MSE = 8403.653335888315
norm(dtheta) = 0.060784386756427174 MSE = 8403.653152727842
norm(dtheta) = 0.060427530763387236 MSE = 8403.652970642725
norm(dtheta) = 0.060072769818246795 MSE = 8403.65278962656
norm(dtheta) = 0.05972009162122615 MSE = 8403.652609673258
norm(dtheta) = 0.05936948394485001 MSE = 8403.65243077622
norm(dtheta) = 0.05902093463339096 MSE = 8403.652252929644
norm(dtheta) = 0.0586744316025947 MSE = 8403.65207612721
norm(dtheta) = 0.05832996283889669 MSE = 8403.65190036269
norm(dtheta) = 0.05798751639962914 MSE = 8403.65172562975
norm(dtheta) = 0.05764708041181567 MSE = 8403.651551923087
norm(dtheta) = 0.057308643072443564 MSE = 8403.651379235967
norm(dtheta) = 0.05697219264787903 MSE = 8403.65120756284
norm(dtheta) = 0.05663771747309002 MSE = 8403.651036897458
norm(dtheta) = 0.05630520595183298 MSE = 8403.650867233966
norm(dtheta) = 0.0559746465556366 MSE = 8403.650698566677
norm(dtheta) = 0.05564602782398246 MSE = 8403.650530889456
norm(dtheta) = 0.05531933836343344 MSE = 8403.650364196894
norm(dtheta) = 0.05499456684761406 MSE = 8403.650198482777
norm(dtheta) = 0.05467170201651617 MSE = 8403.650033741556
norm(dtheta) = 0.05435073267630654 MSE = 8403.64986996754
norm(dtheta) = 0.05403164769877382 MSE = 8403.649707155073
norm(dtheta) = 0.05371443602116191 MSE = 8403.64954529824
norm(dtheta) = 0.05339908664562935 MSE = 8403.649384391765
norm(dtheta) = 0.05308558863884255 MSE = 8403.649224430063
norm(dtheta) = 0.05277393113170105 MSE = 8403.649065407346
norm(dtheta) = 0.05246410331892111 MSE = 8403.648907318311
norm(dtheta) = 0.05215609445861628 MSE = 8403.648750157423
norm(dtheta) = 0.05184989387199945 MSE = 8403.648593919064
norm(dtheta) = 0.051545490943042 MSE = 8403.648438598073
norm(dtheta) = 0.05124287511782125 MSE = 8403.64828418879
norm(dtheta) = 0.05094203590461046 MSE = 8403.648130686088
norm(dtheta) = 0.05064296287320722 MSE = 8403.647978084666
norm(dtheta) = 0.05034564565460408 MSE = 8403.647826379198
norm(dtheta) = 0.05005007394065292 MSE = 8403.647675564236
norm(dtheta) = 0.04975623748379609 MSE = 8403.647525634677
norm(dtheta) = 0.049464126096562616 MSE = 8403.647376585535
norm(dtheta) = 0.04917372965135999 MSE = 8403.64722841117
norm(dtheta) = 0.048885038080016464 MSE = 8403.64708110681
norm(dtheta) = 0.04859804137349569 MSE = 8403.64693466726
norm(dtheta) = 0.04831272958146062 MSE = 8403.646789087528
norm(dtheta) = 0.048029092812004445 MSE = 8403.646644362294
norm(dtheta) = 0.047747121231413496 MSE = 8403.646500486693
norm(dtheta) = 0.04746680506350854 MSE = 8403.64635745596
norm(dtheta) = 0.04718813458972112 MSE = 8403.646215264807
norm(dtheta) = 0.04691110014838113 MSE = 8403.64607390843
norm(dtheta) = 0.046635692134573546 MSE = 8403.645933382046
norm(dtheta) = 0.04636190099977378 MSE = 8403.645793680733
norm(dtheta) = 0.04608971725154542 MSE = 8403.645654799264
norm(dtheta) = 0.045819131453183455 MSE = 8403.645516733399
norm(dtheta) = 0.04555013422339957 MSE = 8403.645379478108
norm(dtheta) = 0.04528271623578272 MSE = 8403.645243028572
norm(dtheta) = 0.04501686821904494 MSE = 8403.645107380145
norm(dtheta) = 0.04475258095601469 MSE = 8403.644972528144
norm(dtheta) = 0.04448984528370541 MSE = 8403.644838467724
norm(dtheta) = 0.044228652093070114 MSE = 8403.64470519411
norm(dtheta) = 0.043968992328383465 MSE = 8403.644572703271
norm(dtheta) = 0.04371085698710896 MSE = 8403.644440990176
norm(dtheta) = 0.04345423711955852 MSE = 8403.644310050446
norm(dtheta) = 0.04319912382868193 MSE = 8403.644179879366
norm(dtheta) = 0.0429455082695991 MSE = 8403.644050472454
norm(dtheta) = 0.04269338164931888 MSE = 8403.643921825265
norm(dtheta) = 0.04244273522644469 MSE = 8403.643793933388
norm(dtheta) = 0.04219356031108364 MSE = 8403.64366679235
norm(dtheta) = 0.041945848264189387 MSE = 8403.643540397803
norm(dtheta) = 0.04169959049740777 MSE = 8403.643414745118
norm(dtheta) = 0.04145477847298359 MSE = 8403.643289830326
norm(dtheta) = 0.04121140370312368 MSE = 8403.643165648742
norm(dtheta) = 0.04096945774989176 MSE = 8403.64304219626
norm(dtheta) = 0.04072893222502841 MSE = 8403.642919468557
norm(dtheta) = 0.040489818789355966 MSE = 8403.642797461356
norm(dtheta) = 0.04025210915264846 MSE = 8403.642676170479
norm(dtheta) = 0.040015795073489985 MSE = 8403.642555591567
norm(dtheta) = 0.03978086835879528 MSE = 8403.642435720665
norm(dtheta) = 0.039547320863426404 MSE = 8403.642316553358
norm(dtheta) = 0.03931514449031956 MSE = 8403.642198085889
norm(dtheta) = 0.03908433118976404 MSE = 8403.642080313823
norm(dtheta) = 0.03885487295947397 MSE = 8403.64196323318
norm(dtheta) = 0.03862676184384011 MSE = 8403.641846839848
norm(dtheta) = 0.03839998993430065 MSE = 8403.641731130036
norm(dtheta) = 0.03817454936849365 MSE = 8403.641616099318
norm(dtheta) = 0.037950432330397806 MSE = 8403.641501744032
norm(dtheta) = 0.037727631049708096 MSE = 8403.641388059943
norm(dtheta) = 0.03750613780182702 MSE = 8403.641275043337
norm(dtheta) = 0.037285944907511685 MSE = 8403.641162690465
norm(dtheta) = 0.03706704473249556 MSE = 8403.641050997137
norm(dtheta) = 0.03684942968754479 MSE = 8403.640939959258
norm(dtheta) = 0.03663309222780381 MSE = 8403.640829573706
norm(dtheta) = 0.036418024852679635 MSE = 8403.640719835737
norm(dtheta) = 0.036204220105885365 MSE = 8403.640610742266
norm(dtheta) = 0.03599167057452782 MSE = 8403.640502289203
norm(dtheta) = 0.03578036888952574 MSE = 8403.640394473152
norm(dtheta) = 0.03557030772493765 MSE = 8403.640287289723
norm(dtheta) = 0.03536147979783333 MSE = 8403.640180735681
norm(dtheta) = 0.03515387786809913 MSE = 8403.640074807108
norm(dtheta) = 0.034947494738078906 MSE = 8403.639969500562
norm(dtheta) = 0.034742323252360464 MSE = 8403.63986481211
norm(dtheta) = 0.03453835629761028 MSE = 8403.639760738406
norm(dtheta) = 0.034335586802116416 MSE = 8403.63965727558
norm(dtheta) = 0.03413400773590455 MSE = 8403.639554420242
norm(dtheta) = 0.033933612110034414 MSE = 8403.63945216882
norm(dtheta) = 0.03373439297677045 MSE = 8403.63935051765
norm(dtheta) = 0.033536343428993184 MSE = 8403.639249463227
norm(dtheta) = 0.03333945660040694 MSE = 8403.639149001894
norm(dtheta) = 0.03314372566478962 MSE = 8403.63904913054
norm(dtheta) = 0.03294914383604023 MSE = 8403.638949845566
norm(dtheta) = 0.032755704367924285 MSE = 8403.63885114355
norm(dtheta) = 0.03256340055382512 MSE = 8403.638753020843
norm(dtheta) = 0.03237222572656159 MSE = 8403.638655474018
norm(dtheta) = 0.03218217325791679 MSE = 8403.638558500143
norm(dtheta) = 0.031993236558721626 MSE = 8403.63846209558
norm(dtheta) = 0.03180540907847393 MSE = 8403.638366256966
norm(dtheta) = 0.03161868430513645 MSE = 8403.63827098101
norm(dtheta) = 0.031433055764803375 MSE = 8403.638176264432
norm(dtheta) = 0.031248517021770755 MSE = 8403.638082103816
norm(dtheta) = 0.03106506167793935 MSE = 8403.63798849623
norm(dtheta) = 0.030882683372841488 MSE = 8403.637895437962
norm(dtheta) = 0.030701375783305018 MSE = 8403.63780292591
norm(dtheta) = 0.030521132623413733 MSE = 8403.637710957288
norm(dtheta) = 0.030341947643965122 MSE = 8403.637619528352
norm(dtheta) = 0.03016381463260831 MSE = 8403.637528636345
norm(dtheta) = 0.02998672741335037 MSE = 8403.637438278003
norm(dtheta) = 0.02981067984654331 MSE = 8403.637348450084
norm(dtheta) = 0.029635665828475585 MSE = 8403.637259149466
norm(dtheta) = 0.029461679291462345 MSE = 8403.63717037321
norm(dtheta) = 0.029288714203210595 MSE = 8403.637082118088
norm(dtheta) = 0.02911676456697217 MSE = 8403.636994380977
norm(dtheta) = 0.028945824421191754 MSE = 8403.636907159073
norm(dtheta) = 0.028775887839275593 MSE = 8403.636820449286
norm(dtheta) = 0.02860694892952219 MSE = 8403.636734248475
norm(dtheta) = 0.028439001834667098 MSE = 8403.636648553662
norm(dtheta) = 0.02827204073193309 MSE = 8403.636563362144
norm(dtheta) = 0.028106059832758448 MSE = 8403.6364786709
norm(dtheta) = 0.027941053382455096 MSE = 8403.636394476473
norm(dtheta) = 0.02777701566023232 MSE = 8403.636310776548
norm(dtheta) = 0.027613940978772726 MSE = 8403.636227568013
norm(dtheta) = 0.027451823684263663 MSE = 8403.636144847993
norm(dtheta) = 0.027290658156022433 MSE = 8403.636062613527
norm(dtheta) = 0.02713043880632341 MSE = 8403.635980861993
norm(dtheta) = 0.02697116008036158 MSE = 8403.635899590216
norm(dtheta) = 0.026812816455820516 MSE = 8403.63581879561
norm(dtheta) = 0.026655402442921347 MSE = 8403.635738475468
norm(dtheta) = 0.026498912583994855 MSE = 8403.635658626794
norm(dtheta) = 0.026343341453469407 MSE = 8403.635579246855
norm(dtheta) = 0.026188683657677955 MSE = 8403.635500333052
norm(dtheta) = 0.026034933834598543 MSE = 8403.635421882445
norm(dtheta) = 0.025882086653525916 MSE = 8403.635343892533
norm(dtheta) = 0.0257301368152934 MSE = 8403.635266360274
norm(dtheta) = 0.02557907905170359 MSE = 8403.6351892833
norm(dtheta) = 0.025428908125538 MSE = 8403.635112658967
norm(dtheta) = 0.025279618830259466 MSE = 8403.635036484271
norm(dtheta) = 0.025131205990061058 MSE = 8403.634960757032
norm(dtheta) = 0.024983664459269892 MSE = 8403.634885474177
norm(dtheta) = 0.024836989122616754 MSE = 8403.634810633326
norm(dtheta) = 0.02469117489482152 MSE = 8403.634736231874
norm(dtheta) = 0.02454621672040998 MSE = 8403.634662267212
norm(dtheta) = 0.02440210957367119 MSE = 8403.634588736939
norm(dtheta) = 0.0242588484583143 MSE = 8403.634515638096
norm(dtheta) = 0.024116428407381277 MSE = 8403.63444296854
norm(dtheta) = 0.02397484448319716 MSE = 8403.63437072562
norm(dtheta) = 0.02383409177688324 MSE = 8403.63429890684
norm(dtheta) = 0.02369416540856279 MSE = 8403.634227509698
norm(dtheta) = 0.023555060526941413 MSE = 8403.634156531607
norm(dtheta) = 0.023416772309092452 MSE = 8403.634085970445
norm(dtheta) = 0.023279295960603366 MSE = 8403.634015823181
norm(dtheta) = 0.023142626715117437 MSE = 8403.633946088077
norm(dtheta) = 0.02300675983417573 MSE = 8403.633876762246
norm(dtheta) = 0.022871690607292473 MSE = 8403.633807843311
norm(dtheta) = 0.02273741435148229 MSE = 8403.63373932912
norm(dtheta) = 0.022603926411403664 MSE = 8403.633671217132
norm(dtheta) = 0.022471222158927048 MSE = 8403.633603504963
norm(dtheta) = 0.022339296993177753 MSE = 8403.63353619032
norm(dtheta) = 0.022208146340224943 MSE = 8403.63346927092
norm(dtheta) = 0.022077765653038227 MSE = 8403.63340274442
norm(dtheta) = 0.02194815041126102 MSE = 8403.633336608578
norm(dtheta) = 0.021819296121065652 MSE = 8403.633270860963
norm(dtheta) = 0.021691198315035895 MSE = 8403.633205499329
norm(dtheta) = 0.021563852551969846 MSE = 8403.633140521428
norm(dtheta) = 0.02143725441669331 MSE = 8403.63307592477
norm(dtheta) = 0.02131139952002923 MSE = 8403.633011707561
norm(dtheta) = 0.021186283498584445 MSE = 8403.63294786746
norm(dtheta) = 0.021061902014429614 MSE = 8403.632884402003
norm(dtheta) = 0.02093825075529206 MSE = 8403.632821309217
norm(dtheta) = 0.020815325434127535 MSE = 8403.632758586611
norm(dtheta) = 0.020693121788993078 MSE = 8403.632696232524
norm(dtheta) = 0.02057163558309337 MSE = 8403.632634244352
norm(dtheta) = 0.020450862604443013 MSE = 8403.632572620183
norm(dtheta) = 0.02033079866578412 MSE = 8403.632511357828
norm(dtheta) = 0.020211439604426805 MSE = 8403.632450454974
norm(dtheta) = 0.020092781282208302 MSE = 8403.632389909859
norm(dtheta) = 0.01997481958512442 MSE = 8403.632329720076
norm(dtheta) = 0.01985755042345594 MSE = 8403.63226988391
norm(dtheta) = 0.019740969731355335 MSE = 8403.632210398593
norm(dtheta) = 0.019625073467003304 MSE = 8403.632151262871
norm(dtheta) = 0.01950985761213408 MSE = 8403.632092474143
norm(dtheta) = 0.019395318172301383 MSE = 8403.632034030543
norm(dtheta) = 0.01928145117625362 MSE = 8403.631975930186
norm(dtheta) = 0.019168252676205557 MSE = 8403.631918170862
norm(dtheta) = 0.019055718747547277 MSE = 8403.631860750551
norm(dtheta) = 0.01894384548864255 MSE = 8403.631803667615
norm(dtheta) = 0.018832629020802037 MSE = 8403.63174691963
norm(dtheta) = 0.01872206548816359 MSE = 8403.631690504604
norm(dtheta) = 0.018612151057389958 MSE = 8403.63163442093
norm(dtheta) = 0.018502881917670286 MSE = 8403.6315786667
norm(dtheta) = 0.01839425428065121 MSE = 8403.631523239497
norm(dtheta) = 0.01828626438015348 MSE = 8403.63146813802
norm(dtheta) = 0.018178908472136792 MSE = 8403.631413359735
norm(dtheta) = 0.018072182834497783 MSE = 8403.631358903138
norm(dtheta) = 0.01796608376701732 MSE = 8403.631304766293
norm(dtheta) = 0.017860607591208656 MSE = 8403.63125094728
norm(dtheta) = 0.017755750650177104 MSE = 8403.631197444236
norm(dtheta) = 0.017651509308466196 MSE = 8403.631144255321
norm(dtheta) = 0.01754787995203689 MSE = 8403.631091378578
norm(dtheta) = 0.01744485898790332 MSE = 8403.631038812227
norm(dtheta) = 0.017342442844384184 MSE = 8403.63098655465
norm(dtheta) = 0.017240627970653014 MSE = 8403.63093460386
norm(dtheta) = 0.017139410836720128 MSE = 8403.630882958048
norm(dtheta) = 0.01703878793336643 MSE = 8403.630831615339
norm(dtheta) = 0.01693875577195959 MSE = 8403.630780574123
norm(dtheta) = 0.016839310884270826 MSE = 8403.63072983241
norm(dtheta) = 0.01674044982263173 MSE = 8403.63067938875
norm(dtheta) = 0.016642169159374817 MSE = 8403.630629241356
norm(dtheta) = 0.016544465487115727 MSE = 8403.630579388047
norm(dtheta) = 0.016447335418472053 MSE = 8403.630529827873
norm(dtheta) = 0.01635077558579218 MSE = 8403.630480558257
norm(dtheta) = 0.016254782641394253 MSE = 8403.63043157816
norm(dtheta) = 0.016159353257134954 MSE = 8403.630382885507
norm(dtheta) = 0.01606448412441758 MSE = 8403.630334478448
norm(dtheta) = 0.01597017195413277 MSE = 8403.630286355945
norm(dtheta) = 0.015876413476404 MSE = 8403.630238515923
norm(dtheta) = 0.015783205440597994 MSE = 8403.630190956548
norm(dtheta) = 0.01569054461516728 MSE = 8403.630143676675
norm(dtheta) = 0.015598427787493865 MSE = 8403.630096674176
norm(dtheta) = 0.01550685176386617 MSE = 8403.630049947553
norm(dtheta) = 0.015415813369318632 MSE = 8403.630003495487
norm(dtheta) = 0.015325309447450586 MSE = 8403.62995731603
norm(dtheta) = 0.01523533686054666 MSE = 8403.629911407697
norm(dtheta) = 0.015145892489133967 MSE = 8403.629865768731
norm(dtheta) = 0.015056973232163654 MSE = 8403.629820397853
norm(dtheta) = 0.014968576006823888 MSE = 8403.629775293357
norm(dtheta) = 0.014880697748253385 MSE = 8403.629730453615
norm(dtheta) = 0.014793335409766488 MSE = 8403.629685877168
norm(dtheta) = 0.014706485962387578 MSE = 8403.629641562438
norm(dtheta) = 0.01462014639506737 MSE = 8403.629597507765
norm(dtheta) = 0.014534313714363147 MSE = 8403.629553711924
norm(dtheta) = 0.014448984944429085 MSE = 8403.62951017308
norm(dtheta) = 0.014364157126858013 MSE = 8403.629466889766
norm(dtheta) = 0.014279827320656276 MSE = 8403.629423860677
norm(dtheta) = 0.014195992602055183 MSE = 8403.629381084214
norm(dtheta) = 0.014112650064491876 MSE = 8403.629338558854
norm(dtheta) = 0.014029796818408128 MSE = 8403.62929628298
norm(dtheta) = 0.013947429991290497 MSE = 8403.629254255604
norm(dtheta) = 0.013865546727410407 MSE = 8403.629212474749
norm(dtheta) = 0.013784144187894245 MSE = 8403.629170939128
norm(dtheta) = 0.013703219550416262 MSE = 8403.629129647603
norm(dtheta) = 0.013622770009308708 MSE = 8403.629088598234
norm(dtheta) = 0.013542792775387091 MSE = 8403.62904779006
norm(dtheta) = 0.013463285075753053 MSE = 8403.62900722146
norm(dtheta) = 0.013384244153860258 MSE = 8403.628966891012
norm(dtheta) = 0.013305667269354345 MSE = 8403.62892679741
norm(dtheta) = 0.013227551697881064 MSE = 8403.628886938977
norm(dtheta) = 0.013149894731231617 MSE = 8403.628847314647
norm(dtheta) = 0.01307269367695354 MSE = 8403.62880792285
norm(dtheta) = 0.012995945858414765 MSE = 8403.628768762397
norm(dtheta) = 0.012919648614803367 MSE = 8403.628729832066
norm(dtheta) = 0.012843799300812404 MSE = 8403.628691129981
norm(dtheta) = 0.012768395286772184 MSE = 8403.628652655118
norm(dtheta) = 0.012693433958313672 MSE = 8403.628614406243
norm(dtheta) = 0.01261891271655535 MSE = 8403.628576381978
norm(dtheta) = 0.012544828977800756 MSE = 8403.628538580817
norm(dtheta) = 0.012471180173553212 MSE = 8403.628501001785
norm(dtheta) = 0.012397963750332553 MSE = 8403.62846364311
norm(dtheta) = 0.012325177169744045 MSE = 8403.628426503956
norm(dtheta) = 0.012252817908221816 MSE = 8403.628389582687
norm(dtheta) = 0.012180883457034275 MSE = 8403.628352878395
norm(dtheta) = 0.01210937132218114 MSE = 8403.628316389379
norm(dtheta) = 0.012038279024337122 MSE = 8403.628280114759
norm(dtheta) = 0.011967604098671501 MSE = 8403.628244052788
norm(dtheta) = 0.011897344094866959 MSE = 8403.628208202868
norm(dtheta) = 0.011827496576981369 MSE = 8403.62817256326
norm(dtheta) = 0.01175805912335232 MSE = 8403.628137132953
norm(dtheta) = 0.011689029326586602 MSE = 8403.628101910692
norm(dtheta) = 0.011620404793373283 MSE = 8403.628066895211
norm(dtheta) = 0.011552183144470674 MSE = 8403.628032085151
norm(dtheta) = 0.011484362014606602 MSE = 8403.627997479512
norm(dtheta) = 0.011416939052379027 MSE = 8403.62796307702
norm(dtheta) = 0.011349911920260029 MSE = 8403.627928876691
norm(dtheta) = 0.011283278294328607 MSE = 8403.627894876954
norm(dtheta) = 0.011217035864394256 MSE = 8403.627861076906
norm(dtheta) = 0.011151182333828943 MSE = 8403.62782747516
norm(dtheta) = 0.011085715419421865 MSE = 8403.627794070962
norm(dtheta) = 0.011020632851413479 MSE = 8403.627760862651
norm(dtheta) = 0.01095593237337664 MSE = 8403.627727849314
norm(dtheta) = 0.010891611742123273 MSE = 8403.627695029725
norm(dtheta) = 0.010827668727610302 MSE = 8403.627662403022
norm(dtheta) = 0.010764101112955294 MSE = 8403.627629967714
norm(dtheta) = 0.01070090669415397 MSE = 8403.627597722905
norm(dtheta) = 0.01063808328031018 MSE = 8403.627565667332
norm(dtheta) = 0.010575628693268708 MSE = 8403.627533799903
norm(dtheta) = 0.010513540767733419 MSE = 8403.627502119632
norm(dtheta) = 0.010451817351068865 MSE = 8403.627470625426
norm(dtheta) = 0.010390456303283141 MSE = 8403.627439316158
norm(dtheta) = 0.010329455497002886 MSE = 8403.627408190638
norm(dtheta) = 0.010268812817264251 MSE = 8403.627377247822
norm(dtheta) = 0.010208526161571276 MSE = 8403.627346486566
norm(dtheta) = 0.010148593439787018 MSE = 8403.62731590605
norm(dtheta) = 0.010089012573987862 MSE = 8403.62728550503
norm(dtheta) = 0.010029781498496738 MSE = 8403.6272552825
norm(dtheta) = 0.009970898159743034 MSE = 8403.627225237291

Once gradient descent has converged, we can examine the learned value of theta:

In [47]:
theta
Out[47]:
[107.00031826701057, -0.6803048266097109]

Gradient descent in Tensorflow

Although the above gradient descent code produced an acceptable solution, it was (a) difficult to implement; (b) difficult to tune the optimization process (e.g. learning rates and stopping criteria); and (c) inefficient (i.e., slow). Using libraries like tensorflow can help to alleviate these issues.

First you will need to set up a working install of tensorflow. See installation instructions here: https://www.tensorflow.org/install

In [48]:
import tensorflow as tf

Otherwise we read in the data as before

In [49]:
path = "datasets/PRSA_data_2010.1.1-2014.12.31.csv"
f = open(path, 'r')
In [50]:
dataset = []
header = f.readline().strip().split(',')
for line in f:
    line = line.split(',')
    dataset.append(line)
In [51]:
header.index('pm2.5')
Out[51]:
5
In [52]:
dataset = [d for d in dataset if d[5] != 'NA']
In [53]:
def feature(datum):
    feat = [1, float(datum[7]), float(datum[8]), float(datum[10])] # Temperature, pressure, and wind speed
    return feat
In [54]:
X = [feature(d) for d in dataset]
y = [float(d[5]) for d in dataset]
In [55]:
y = tf.constant(y, shape=[len(y),1])
In [56]:
K = len(X[0])

The main advantage of tensorflow is that we don't have to compute the gradient - tensorflow will compute it for us. Instead, we have to implement our objective (i.e., the MSE) in terms of tensorflow operations:

In [57]:
def MSE(X, y, theta):
  return tf.reduce_mean((tf.matmul(X,theta) - y)**2)

Next we tell tensorflow that theta is our vector of variables to be optimized (we also specify its shape and initial values)

In [58]:
theta = tf.Variable(tf.constant([0.0]*K, shape=[K,1]))

Here we select an optimizer, which is essentially a specific gradient descent implementation. The parameter passed to the optimizer is a learning rate.

In [59]:
optimizer = tf.train.AdamOptimizer(0.01)

Then we tell tensorflow that our MSE function is the objective to be optimized.

In [60]:
objective = MSE(X,y,theta)

Then we tell tensorflow that this objective should be minimized (i.e., we are trying to minimize an error, rather than maximizing an accuracy), and initialize the session.

In [61]:
train = optimizer.minimize(objective)
In [62]:
init = tf.global_variables_initializer()
In [63]:
sess = tf.Session()
sess.run(init)

Finally we run 1000 iterations of gradient descent. Note how fast this is compared to our own implementation!

In [64]:
for iteration in range(1000):
  cvalues = sess.run([train, objective])
  print("objective = " + str(cvalues[1]))

objective = 18197.11
objective = 16256.274
objective = 14544.915
objective = 13065.931
objective = 11818.9375
objective = 10800.008
objective = 10000.627
objective = 9406.955
objective = 8999.841
objective = 8755.391
objective = 8645.959
objective = 8641.206
objective = 8709.448
objective = 8819.531
objective = 8943.106
objective = 9056.544
objective = 9142.379
objective = 9189.911
objective = 9194.929
objective = 9158.805
objective = 9087.197
objective = 8988.592
objective = 8872.917
objective = 8750.339
objective = 8630.25
objective = 8520.496
objective = 8426.892
objective = 8352.981
objective = 8299.985
objective = 8267.004
objective = 8251.36
objective = 8249.125
objective = 8255.695
objective = 8266.347
objective = 8276.795
objective = 8283.601
objective = 8284.418
objective = 8278.101
objective = 8264.656
objective = 8245.025
objective = 8220.871
objective = 8194.206
objective = 8167.128
objective = 8141.5273
objective = 8118.9033
objective = 8100.2065
objective = 8085.7983
objective = 8075.5405
objective = 8068.804
objective = 8064.702
objective = 8062.159
objective = 8060.178
objective = 8057.8687
objective = 8054.585
objective = 8049.991
objective = 8044.02
objective = 8036.89
objective = 8028.9556
objective = 8020.693
objective = 8012.5757
objective = 8005.0317
objective = 7998.3564
objective = 7992.705
objective = 7988.108
objective = 7984.434
objective = 7981.4814
objective = 7978.9946
objective = 7976.7285
objective = 7974.457
objective = 7972.029
objective = 7969.367
objective = 7966.4536
objective = 7963.3574
objective = 7960.16
objective = 7956.988
objective = 7953.955
objective = 7951.1484
objective = 7948.6055
objective = 7946.35
objective = 7944.3745
objective = 7942.6025
objective = 7940.986
objective = 7939.452
objective = 7937.957
objective = 7936.4424
objective = 7934.8994
objective = 7933.3184
objective = 7931.711
objective = 7930.116
objective = 7928.5527
objective = 7927.0493
objective = 7925.6167
objective = 7924.2705
objective = 7923.0073
objective = 7921.831
objective = 7920.6895
objective = 7919.5986
objective = 7918.53
objective = 7917.4653
objective = 7916.4033
objective = 7915.338
objective = 7914.2803
objective = 7913.23
objective = 7912.189
objective = 7911.176
objective = 7910.1875
objective = 7909.2217
objective = 7908.2915
objective = 7907.379
objective = 7906.496
objective = 7905.62
objective = 7904.752
objective = 7903.8965
objective = 7903.036
objective = 7902.1914
objective = 7901.345
objective = 7900.516
objective = 7899.687
objective = 7898.877
objective = 7898.0684
objective = 7897.277
objective = 7896.499
objective = 7895.7334
objective = 7894.973
objective = 7894.217
objective = 7893.472
objective = 7892.7363
objective = 7892.0024
objective = 7891.2725
objective = 7890.5522
objective = 7889.838
objective = 7889.1323
objective = 7888.4326
objective = 7887.7427
objective = 7887.056
objective = 7886.382
objective = 7885.7144
objective = 7885.053
objective = 7884.3984
objective = 7883.7427
objective = 7883.1035
objective = 7882.4565
objective = 7881.833
objective = 7881.2036
objective = 7880.591
objective = 7879.9775
objective = 7879.368
objective = 7878.776
objective = 7878.1836
objective = 7877.5933
objective = 7877.0264
objective = 7876.445
objective = 7875.885
objective = 7875.3228
objective = 7874.7705
objective = 7874.2134
objective = 7873.6724
objective = 7873.138
objective = 7872.603
objective = 7872.082
objective = 7871.5654
objective = 7871.049
objective = 7870.5454
objective = 7870.0444
objective = 7869.551
objective = 7869.062
objective = 7868.579
objective = 7868.1025
objective = 7867.626
objective = 7867.158
objective = 7866.703
objective = 7866.239
objective = 7865.7935
objective = 7865.3457
objective = 7864.913
objective = 7864.467
objective = 7864.0547
objective = 7863.6226
objective = 7863.206
objective = 7862.7915
objective = 7862.3833
objective = 7861.978
objective = 7861.579
objective = 7861.195
objective = 7860.805
objective = 7860.4243
objective = 7860.045
objective = 7859.6714
objective = 7859.298
objective = 7858.945
objective = 7858.579
objective = 7858.231
objective = 7857.8843
objective = 7857.537
objective = 7857.2036
objective = 7856.8643
objective = 7856.53
objective = 7856.2036
objective = 7855.892
objective = 7855.5713
objective = 7855.2593
objective = 7854.951
objective = 7854.6426
objective = 7854.3438
objective = 7854.0522
objective = 7853.753
objective = 7853.468
objective = 7853.1865
objective = 7852.8975
objective = 7852.623
objective = 7852.3594
objective = 7852.09
objective = 7851.8247
objective = 7851.5625
objective = 7851.308
objective = 7851.054
objective = 7850.8076
objective = 7850.5557
objective = 7850.3174
objective = 7850.0806
objective = 7849.8423
objective = 7849.6167
objective = 7849.383
objective = 7849.1606
objective = 7848.9395
objective = 7848.724
objective = 7848.5073
objective = 7848.29
objective = 7848.0854
objective = 7847.881
objective = 7847.684
objective = 7847.4785
objective = 7847.287
objective = 7847.094
objective = 7846.8994
objective = 7846.714
objective = 7846.5293
objective = 7846.352
objective = 7846.1704
objective = 7845.998
objective = 7845.822
objective = 7845.655
objective = 7845.4863
objective = 7845.3193
objective = 7845.158
objective = 7845.008
objective = 7844.8423
objective = 7844.6875
objective = 7844.5347
objective = 7844.3804
objective = 7844.243
objective = 7844.095
objective = 7843.958
objective = 7843.8154
objective = 7843.678
objective = 7843.539
objective = 7843.411
objective = 7843.278
objective = 7843.1504
objective = 7843.02
objective = 7842.903
objective = 7842.7734
objective = 7842.6567
objective = 7842.5396
objective = 7842.4224
objective = 7842.3096
objective = 7842.196
objective = 7842.0874
objective = 7841.982
objective = 7841.873
objective = 7841.7666
objective = 7841.6597
objective = 7841.5664
objective = 7841.4624
objective = 7841.3687
objective = 7841.2705
objective = 7841.1733
objective = 7841.0884
objective = 7840.9893
objective = 7840.903
objective = 7840.8184
objective = 7840.731
objective = 7840.6436
objective = 7840.564
objective = 7840.483
objective = 7840.4014
objective = 7840.3203
objective = 7840.25
objective = 7840.172
objective = 7840.095
objective = 7840.023
objective = 7839.9487
objective = 7839.879
objective = 7839.8115
objective = 7839.7456
objective = 7839.6753
objective = 7839.615
objective = 7839.5464
objective = 7839.4834
objective = 7839.423
objective = 7839.3647
objective = 7839.3027
objective = 7839.2495
objective = 7839.1846
objective = 7839.1323
objective = 7839.0835
objective = 7839.022
objective = 7838.9688
objective = 7838.917
objective = 7838.862
objective = 7838.8135
objective = 7838.7656
objective = 7838.716
objective = 7838.6743
objective = 7838.625
objective = 7838.582
objective = 7838.536
objective = 7838.4917
objective = 7838.451
objective = 7838.4077
objective = 7838.3716
objective = 7838.326
objective = 7838.299
objective = 7838.252
objective = 7838.214
objective = 7838.179
objective = 7838.1406
objective = 7838.1016
objective = 7838.072
objective = 7838.0366
objective = 7838.0044
objective = 7837.968
objective = 7837.9355
objective = 7837.9033
objective = 7837.871
objective = 7837.842
objective = 7837.816
objective = 7837.7837
objective = 7837.754
objective = 7837.7334
objective = 7837.6978
objective = 7837.676
objective = 7837.649
objective = 7837.6274
objective = 7837.5967
objective = 7837.575
objective = 7837.551
objective = 7837.525
objective = 7837.4995
objective = 7837.481
objective = 7837.4556
objective = 7837.432
objective = 7837.4126
objective = 7837.3896
objective = 7837.37
objective = 7837.3574
objective = 7837.338
objective = 7837.3164
objective = 7837.2964
objective = 7837.2812
objective = 7837.2627
objective = 7837.244
objective = 7837.2236
objective = 7837.213
objective = 7837.19
objective = 7837.1807
objective = 7837.154
objective = 7837.1484
objective = 7837.1333
objective = 7837.112
objective = 7837.104
objective = 7837.0854
objective = 7837.0747
objective = 7837.0596
objective = 7837.0527
objective = 7837.041
objective = 7837.023
objective = 7837.012
objective = 7836.995
objective = 7836.9907
objective = 7836.9785
objective = 7836.967
objective = 7836.957
objective = 7836.949
objective = 7836.937
objective = 7836.93
objective = 7836.9194
objective = 7836.9062
objective = 7836.895
objective = 7836.8896
objective = 7836.877
objective = 7836.866
objective = 7836.863
objective = 7836.8486
objective = 7836.8403
objective = 7836.836
objective = 7836.825
objective = 7836.8145
objective = 7836.813
objective = 7836.8027
objective = 7836.797
objective = 7836.7944
objective = 7836.788
objective = 7836.782
objective = 7836.773
objective = 7836.7686
objective = 7836.7593
objective = 7836.75
objective = 7836.75
objective = 7836.7407
objective = 7836.735
objective = 7836.733
objective = 7836.7188
objective = 7836.715
objective = 7836.7134
objective = 7836.7085
objective = 7836.704
objective = 7836.6978
objective = 7836.6934
objective = 7836.688
objective = 7836.68
objective = 7836.681
objective = 7836.6777
objective = 7836.6743
objective = 7836.667
objective = 7836.661
objective = 7836.661
objective = 7836.6587
objective = 7836.655
objective = 7836.6465
objective = 7836.648
objective = 7836.644
objective = 7836.637
objective = 7836.638
objective = 7836.6396
objective = 7836.6304
objective = 7836.6274
objective = 7836.63
objective = 7836.6245
objective = 7836.6226
objective = 7836.6167
objective = 7836.621
objective = 7836.614
objective = 7836.6113
objective = 7836.6074
objective = 7836.606
objective = 7836.606
objective = 7836.6045
objective = 7836.603
objective = 7836.601
objective = 7836.598
objective = 7836.5957
objective = 7836.594
objective = 7836.5938
objective = 7836.589
objective = 7836.585
objective = 7836.5806
objective = 7836.586
objective = 7836.582
objective = 7836.5845
objective = 7836.578
objective = 7836.579
objective = 7836.5767
objective = 7836.571
objective = 7836.5723
objective = 7836.5737
objective = 7836.566
objective = 7836.5674
objective = 7836.5674
objective = 7836.5645
objective = 7836.563
objective = 7836.5605
objective = 7836.5615
objective = 7836.562
objective = 7836.5586
objective = 7836.56
objective = 7836.56
objective = 7836.5586
objective = 7836.5586
objective = 7836.555
objective = 7836.5522
objective = 7836.5493
objective = 7836.5493
objective = 7836.553
objective = 7836.5537
objective = 7836.5522
objective = 7836.551
objective = 7836.547
objective = 7836.548
objective = 7836.544
objective = 7836.546
objective = 7836.548
objective = 7836.5415
objective = 7836.54
objective = 7836.5337
objective = 7836.5337
objective = 7836.5386
objective = 7836.5415
objective = 7836.5376
objective = 7836.5386
objective = 7836.5347
objective = 7836.5386
objective = 7836.537
objective = 7836.537
objective = 7836.5337
objective = 7836.531
objective = 7836.5347
objective = 7836.537
objective = 7836.535
objective = 7836.531
objective = 7836.5283
objective = 7836.5293
objective = 7836.532
objective = 7836.5293
objective = 7836.528
objective = 7836.528
objective = 7836.5264
objective = 7836.5264
objective = 7836.5264
objective = 7836.528
objective = 7836.528
objective = 7836.5293
objective = 7836.527
objective = 7836.5293
objective = 7836.5293
objective = 7836.5244
objective = 7836.527
objective = 7836.528
objective = 7836.527
objective = 7836.5264
objective = 7836.5264
objective = 7836.528
objective = 7836.528
objective = 7836.5244
objective = 7836.528
objective = 7836.5264
objective = 7836.5254
objective = 7836.5244
objective = 7836.5244
objective = 7836.5244
objective = 7836.524
objective = 7836.5244
objective = 7836.5244
objective = 7836.523
objective = 7836.523
objective = 7836.5244
objective = 7836.5215
objective = 7836.521
objective = 7836.523
objective = 7836.523
objective = 7836.524
objective = 7836.523
objective = 7836.5244
objective = 7836.521
objective = 7836.521
objective = 7836.52
objective = 7836.5215
objective = 7836.5215
objective = 7836.5215
objective = 7836.5215
objective = 7836.52
objective = 7836.5244
objective = 7836.5215
objective = 7836.5195
objective = 7836.5186
objective = 7836.52
objective = 7836.5186
objective = 7836.5215
objective = 7836.523
objective = 7836.5244
objective = 7836.523
objective = 7836.523
objective = 7836.52
objective = 7836.5215
objective = 7836.523
objective = 7836.523
objective = 7836.5244
objective = 7836.524
objective = 7836.5254
objective = 7836.527
objective = 7836.528
objective = 7836.527
objective = 7836.528
objective = 7836.528
objective = 7836.523
objective = 7836.5215
objective = 7836.5215
objective = 7836.52
objective = 7836.521
objective = 7836.523
objective = 7836.521
objective = 7836.5195
objective = 7836.52
objective = 7836.5186
objective = 7836.5186
objective = 7836.5176
objective = 7836.517
objective = 7836.52
objective = 7836.521
objective = 7836.5176
objective = 7836.5195
objective = 7836.5186
objective = 7836.5186
objective = 7836.516
objective = 7836.5156
objective = 7836.5146
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.516
objective = 7836.5156
objective = 7836.5176
objective = 7836.5176
objective = 7836.5186
objective = 7836.5215
objective = 7836.52
objective = 7836.52
objective = 7836.5215
objective = 7836.5215
objective = 7836.5225
objective = 7836.52
objective = 7836.5186
objective = 7836.5176
objective = 7836.517
objective = 7836.5176
objective = 7836.517
objective = 7836.5156
objective = 7836.517
objective = 7836.5186
objective = 7836.52
objective = 7836.521
objective = 7836.521
objective = 7836.52
objective = 7836.52
objective = 7836.5186
objective = 7836.5195
objective = 7836.52
objective = 7836.5195
objective = 7836.52
objective = 7836.5186
objective = 7836.52
objective = 7836.521
objective = 7836.521
objective = 7836.5215
objective = 7836.52
objective = 7836.52
objective = 7836.521
objective = 7836.52
objective = 7836.521
objective = 7836.52
objective = 7836.5195
objective = 7836.5195
objective = 7836.5186
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.5156
objective = 7836.5146
objective = 7836.5146
objective = 7836.516
objective = 7836.516
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.5186
objective = 7836.5186
objective = 7836.517
objective = 7836.516
objective = 7836.5176
objective = 7836.5156
objective = 7836.5156
objective = 7836.517
objective = 7836.516
objective = 7836.516
objective = 7836.517
objective = 7836.5195
objective = 7836.52
objective = 7836.52
objective = 7836.52
objective = 7836.52
objective = 7836.52
objective = 7836.5186
objective = 7836.5176
objective = 7836.5176
objective = 7836.5186
objective = 7836.5186
objective = 7836.5195
objective = 7836.5186
objective = 7836.5195
objective = 7836.5186
objective = 7836.5195
objective = 7836.52
objective = 7836.52
objective = 7836.5195
objective = 7836.52
objective = 7836.5195
objective = 7836.5186
objective = 7836.5186
objective = 7836.5195
objective = 7836.5195
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5176
objective = 7836.5176
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5195
objective = 7836.5186
objective = 7836.5186
objective = 7836.517
objective = 7836.517
objective = 7836.516
objective = 7836.516
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.5176
objective = 7836.517
objective = 7836.5156
objective = 7836.517
objective = 7836.517
objective = 7836.516
objective = 7836.517
objective = 7836.516
objective = 7836.517
objective = 7836.517
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5176
objective = 7836.517
objective = 7836.5186
objective = 7836.517
objective = 7836.5186
objective = 7836.5176
objective = 7836.5176
objective = 7836.5176
objective = 7836.5186
objective = 7836.5176
objective = 7836.5186
objective = 7836.5195
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.5195
objective = 7836.5195
objective = 7836.52
objective = 7836.5186
objective = 7836.5186
objective = 7836.5186
objective = 7836.517
objective = 7836.5186
objective = 7836.5176
objective = 7836.517
objective = 7836.5186
objective = 7836.5176
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.5176
objective = 7836.517
objective = 7836.5186
objective = 7836.5186
objective = 7836.5176
objective = 7836.517
objective = 7836.5176
objective = 7836.5186
objective = 7836.5186
objective = 7836.5176
objective = 7836.5176
objective = 7836.5176
objective = 7836.517
objective = 7836.5176
objective = 7836.517
objective = 7836.5176
objective = 7836.5176
objective = 7836.5176
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.516
objective = 7836.516
objective = 7836.5156
objective = 7836.516
objective = 7836.516
objective = 7836.5156
objective = 7836.517
objective = 7836.516
objective = 7836.517
objective = 7836.516
objective = 7836.517
objective = 7836.516
objective = 7836.517
objective = 7836.517
objective = 7836.516
objective = 7836.517
objective = 7836.517
objective = 7836.517
objective = 7836.516
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.516
objective = 7836.517
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.516
objective = 7836.517
objective = 7836.516
objective = 7836.5156
objective = 7836.517
objective = 7836.5156
objective = 7836.516
objective = 7836.516
objective = 7836.516
objective = 7836.516
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.5146
objective = 7836.5146
objective = 7836.5156
objective = 7836.5156
objective = 7836.5156
objective = 7836.514
objective = 7836.5156
objective = 7836.5146
objective = 7836.5146
objective = 7836.5146
objective = 7836.5146
objective = 7836.514
objective = 7836.5146
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.514
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.512
objective = 7836.513
objective = 7836.512
objective = 7836.513
objective = 7836.512
objective = 7836.513
objective = 7836.513
objective = 7836.513
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.512
objective = 7836.5107
objective = 7836.5107
objective = 7836.512
objective = 7836.5117
objective = 7836.5117
objective = 7836.5107
objective = 7836.5117
objective = 7836.5117
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5103
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5107
objective = 7836.5103
objective = 7836.5103
objective = 7836.5093
objective = 7836.5093
objective = 7836.5093
objective = 7836.5093
objective = 7836.5093
objective = 7836.5093

Once gradient descent has converged, we can print out the results (i.e., theta)

In [65]:
with sess.as_default():
  print(MSE(X, y, theta).eval())
  print(theta.eval())

7836.5093
[[ 0.23223479]
 [-0.89481604]
 [ 0.11925128]
 [-0.4959688 ]]