{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# CSE 252B: Computer Vision II, Winter 2021 – Assignment 5\n", "### Instructor: Ben Ochoa\n", "### Due: Wednesday, March 17, 2021, 11:59 PM" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Instructions\n", "* Review the academic integrity and collaboration policies on the course website.\n", "* This assignment must be completed individually.\n", "* This assignment contains both math and programming problems.\n", "* All solutions must be written in this notebook\n", "* Math problems must be done in Markdown/LATEX.\n", "* You must show your work and describe your solution.\n", "* Programming aspects of this assignment must be completed using Python in this notebook.\n", "* Your code should be well written with sufficient comments to understand, but there is no need to write extra markdown to describe your solution if it is not explictly asked for.\n", "* This notebook contains skeleton code, which should not be modified (This is important for standardization to facilate effecient grading).\n", "* You may use python packages for basic linear algebra, but you may not use packages that directly solve the problem. If you are uncertain about using a specific package, then please ask the instructional staff whether or not it is allowable.\n", "* You must submit this notebook exported as a pdf. You must also submit this notebook as an .ipynb file.\n", "* Your code and results should remain inline in the pdf (Do not move your code to an appendix).\n", "* You must submit both files (.pdf and .ipynb) on Gradescope. You must mark each problem on Gradescope in the pdf.\n", "* It is highly recommended that you begin working on this assignment early." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 1 (Math): Point on Line Closest to the Origin (5 points)\n", " Given a line $\\boldsymbol{l} = (a, b, c)^\\top$, show that the point on\n", " $\\boldsymbol{l}$ that is closest to the origin is the point $\\boldsymbol{x} =\n", " (-ac, -bc, a^2+b^2)^\\top$ (Hint: this calculation is needed in the\n", " two-view optimal triangulation method used below)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"\"\"Write your solution here.\"\"\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 2 (Programming): Feature Detection (20 points)\n", "Download input data from the course website. The file IMG_5030.jpeg\n", "contains image 1 and the file IMG_5031.jpeg contains image 2. \n", "\n", "For each input image, calculate an image where each pixel value is the minor\n", "eigenvalue of the gradient matrix\n", "\n", "$N=\\left[ \\begin{array}{cc}\n", "\\sum\\limits_w I_x^2 & \\sum\\limits_w I_x I_y\\\\\n", "\\sum\\limits_w I_x I_y & \\sum\\limits_w I_y^2\n", "\\end{array} \\right]$\n", "\n", "where w is the window about the pixel, and $I_x$ and $I_y$ are the gradient images in\n", "the x and y direction, respectively. Calculate the gradient images using the fivepoint\n", "central difference operator. Set resulting values that are below a specified\n", "threshold value to zero (hint: calculating the mean instead of the sum in N allows\n", "for adjusting the size of the window without changing the threshold value). Apply\n", "an operation that suppresses (sets to 0) local (i.e., about a window) nonmaximum\n", "pixel values in the minor eigenvalue image. Vary these parameters such that\n", "around 1350–1400 features are detected in each image. For resulting nonzero pixel\n", "values, determine the subpixel feature coordinate using the Forstner corner point\n", "operator.\n", "\n", "#### Report your final values for:\n", " * the size of the feature detection window (i.e. the size of the window used to calculate the elements in the gradient matrix N)\n", " * the minor eigenvalue threshold value \n", " * the size of the local nonmaximum suppression window \n", " * the resulting number of features detected (i.e. corners) in each image.\n", "\n", "#### Display figures for:\n", " * original images with detected features, where the detected features are indicated by a square window (the size of the detection window) about the features\n", " \n", "A typical implementation takes around 30 seconds. If yours takes more than 60, you may lose points." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%matplotlib inline\n", "import numpy as np\n", "import matplotlib.pyplot as plt\n", "import matplotlib.patches as patches\n", "from scipy.signal import convolve2d as conv2d\n", "import scipy.ndimage\n", "\n", "def ImageGradient(I):\n", " # inputs: \n", " # I is the input image (may be mxn for Grayscale or mxnx3 for RGB)\n", " #\n", " # outputs:\n", " # Ix is the derivative of the magnitude of the image w.r.t. x\n", " # Iy is the derivative of the magnitude of the image w.r.t. y\n", " \n", " m, n = I.shape[:2]\n", " \n", " \"\"\"your code here\"\"\"\n", "\n", " \n", " return Ix, Iy\n", " \n", "\n", "def MinorEigenvalueImage(Ix, Iy, w):\n", " # Calculate the minor eigenvalue image J\n", " #\n", " # inputs:\n", " # Ix is the derivative of the magnitude of the image w.r.t. x\n", " # Iy is the derivative of the magnitude of the image w.r.t. y\n", " # w is the size of the window used to compute the gradient matrix N\n", " #\n", " # outputs:\n", " # J0 is the mxn minor eigenvalue image of N before thresholding\n", "\n", " m, n = Ix.shape[:2]\n", " J0 = np.zeros((m,n))\n", "\n", " #Calculate your minor eigenvalue image J0.\n", " \"\"\"your code here\"\"\"\n", "\n", "\n", " return J0\n", " \n", "def NMS(J, w_nms):\n", " # Apply nonmaximum supression to J using window w_nms\n", " #\n", " # inputs: \n", " # J is the minor eigenvalue image input image after thresholding\n", " # w_nms is the size of the local nonmaximum suppression window\n", " # \n", " # outputs:\n", " # J2 is the mxn resulting image after applying nonmaximum suppression\n", " # \n", " \n", " J2 = J.copy()\n", " \"\"\"your code here\"\"\"\n", "\n", " \n", " return J2\n", " \n", " \n", "def ForstnerCornerDetector(Ix, Iy, w, t, w_nms):\n", " # Calculate the minor eigenvalue image J\n", " # Threshold J\n", " # Run non-maxima suppression on the thresholded J\n", " # Gather the coordinates of the nonzero pixels in J \n", " # Then compute the sub pixel location of each point using the Forstner operator\n", " #\n", " # inputs:\n", " # Ix is the derivative of the magnitude of the image w.r.t. x\n", " # Iy is the derivative of the magnitude of the image w.r.t. y\n", " # w is the size of the window used to compute the gradient matrix N\n", " # t is the minor eigenvalue threshold\n", " # w_nms is the size of the local nonmaximum suppression window\n", " #\n", " # outputs:\n", " # C is the number of corners detected in each image\n", " # pts is the 2xC array of coordinates of subpixel accurate corners\n", " # found using the Forstner corner detector\n", " # J0 is the mxn minor eigenvalue image of N before thresholding\n", " # J1 is the mxn minor eigenvalue image of N after thresholding\n", " # J2 is the mxn minor eigenvalue image of N after thresholding and NMS\n", "\n", " m, n = Ix.shape[:2]\n", " J0 = np.zeros((m,n))\n", " J1 = np.zeros((m,n))\n", "\n", " #Calculate your minor eigenvalue image J0 and its thresholded version J1.\n", " \"\"\"your code here\"\"\"\n", "\n", " \n", " #Run non-maxima suppression on your thresholded minor eigenvalue image.\n", " J2 = NMS(J1, w_nms)\n", " \n", " #Detect corners.\n", " \"\"\"your code here\"\"\"\n", "\n", " \n", " return C, pts, J0, J1, J2\n", "\n", "\n", "# feature detection\n", "def RunFeatureDetection(I, w, t, w_nms):\n", " Ix, Iy = ImageGradient(I)\n", " C, pts, J0, J1, J2 = ForstnerCornerDetector(Ix, Iy, w, t, w_nms)\n", " return C, pts, J0, J1, J2" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from PIL import Image\n", "import time\n", "\n", "# input images\n", "I1 = np.array(Image.open('IMG_5030.jpeg'), dtype='float')/255.\n", "I2 = np.array(Image.open('IMG_5031.jpeg'), dtype='float')/255.\n", "\n", "# parameters to tune\n", "w = 15\n", "t = 1\n", "w_nms = 1\n", "\n", "tic = time.time()\n", "\n", "# run feature detection algorithm on input images\n", "C1, pts1, J1_0, J1_1, J1_2 = RunFeatureDetection(I1, w, t, w_nms)\n", "C2, pts2, J2_0, J2_1, J2_2 = RunFeatureDetection(I2, w, t, w_nms)\n", "toc = time.time() - tic\n", "\n", "print('took %f secs'%toc)\n", "\n", "# display results\n", "plt.figure(figsize=(14,24))\n", "\n", "# show corners on original images\n", "ax = plt.subplot(1,2,1)\n", "plt.imshow(I1)\n", "for i in range(C1): # draw rectangles of size w around corners\n", " x,y = pts1[:,i]\n", " ax.add_patch(patches.Rectangle((x-w/2,y-w/2),w,w, fill=False))\n", "# plt.plot(pts1[0,:], pts1[1,:], '.b') # display subpixel corners\n", "plt.title('Found %d Corners'%C1)\n", "\n", "ax = plt.subplot(1,2,2)\n", "plt.imshow(I2)\n", "for i in range(C2):\n", " x,y = pts2[:,i]\n", " ax.add_patch(patches.Rectangle((x-w/2,y-w/2),w,w, fill=False))\n", "# plt.plot(pts2[0,:], pts2[1,:], '.b')\n", "plt.title('Found %d Corners'%C2)\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Final values for parameters\n", "* w =\n", "* t =\n", "* w_nms =\n", "* C1 = \n", "* C2 = " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 3 (Programming): Feature matching (15 points)\n", "Determine the set of one-to-one putative feature correspondences by performing\n", "a brute-force search for the greatest correlation coefficient value (in the range\n", "[-1, 1]) between the detected features in image 1 and the detected features in\n", "image 2. Only allow matches that are above a specified correlation coefficient\n", "threshold value (note that calculating the correlation coefficient allows for adjusting\n", "the size of the matching window without changing the threshold value).\n", "Further, only allow matches that are above a specified distance ratio threshold\n", "value, where distance is measured to the next best match for a given feature.\n", "Vary these parameters such that around 300 putative feature correspondences are\n", "established. Optional: constrain the search to coordinates in image 2 that are\n", "within a proximity of the detected feature coordinates in image 1.\n", "\n", "**Note: Do center each window at the sub-pixel co-ordinate while computing normalized cross correlation. You may lose points otherwise.**\n", "\n", "#### Report your final values for:\n", "* the size of the matching window\n", "* the correlation coefficient threshold\n", "* the distance ratio threshold \n", "* the size of the proximity window (if used)\n", "* the resulting number of putative feature correspondences (i.e. matched features)\n", "\n", "#### Display figures for:\n", "* pair of images, where the matched features are indicated by a square window (the size of the matching window) about the feature and a line segment is drawn from the feature to the coordinates of the corresponding feature in the other image\n", "\n", "A typical implementation takes around 40 seconds. If yours takes more than 80 seconds, you may lose points." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def NCC(I1, I2, pts1, pts2, w, p):\n", " # compute the normalized cross correlation between image patches I1, I2\n", " # result should be in the range [-1,1]\n", " # \n", " # Do ensure that windows are centered at the sub-pixel co-ordinates \n", " # while computing normalized cross correlation.\n", " #\n", " # inputs:\n", " # I1, I2 are the input images\n", " # pts1, pts2 are the point to be matched\n", " # w is the size of the matching window to compute correlation coefficients\n", " # p is the size of the proximity window\n", " #\n", " # output:\n", " # normalized cross correlation matrix of scores between all windows in \n", " # image 1 and all windows in image 2\n", " \n", " \"\"\"your code here\"\"\"\n", " \n", " return scores\n", "\n", "\n", "def Match(scores, t, d):\n", " # perform the one-to-one correspondence matching on the correlation coefficient matrix\n", " # \n", " # inputs:\n", " # scores is the NCC matrix\n", " # t is the correlation coefficient threshold\n", " # d distance ration threshold\n", " #\n", " # output:\n", " # 2xM array of the feature coordinates in image 1 and image 2,\n", " # where M is the number of matches.\n", " \n", " \"\"\"your code here\"\"\"\n", " inds = []\n", " \n", " return inds\n", "\n", "\n", "\n", "def RunFeatureMatching(I1, I2, pts1, pts2, w, t, d, p):\n", " # inputs:\n", " # I1, I2 are the input images\n", " # pts1, pts2 are the point to be matched\n", " # w is the size of the matching window to compute correlation coefficients\n", " # t is the correlation coefficient threshold\n", " # d distance ration threshold\n", " # p is the size of the proximity window\n", " #\n", " # outputs:\n", " # inds is a 2xk matrix of matches where inds[0,i] indexs a point pts1 \n", " # and inds[1,i] indexs a point in pts2, where k is the number of matches\n", " \n", " scores = NCC(I1, I2, pts1, pts2, w)\n", " inds = Match(scores, t, d, p)\n", " return inds" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# parameters to tune\n", "w = 15\n", "t = 1\n", "d = 1\n", "p = np.inf\n", "\n", "tic = time.time()\n", "# run the feature matching algorithm on the input images and detected features\n", "inds = RunFeatureMatching(I1, I2, pts1, pts2, w, t, d, p)\n", "toc = time.time() - tic\n", "\n", "print('took %f secs'%toc)\n", "\n", "# create new matrices of points which contain only the matched features \n", "match1 = pts1[:,inds[0,:]]\n", "match2 = pts2[:,inds[1,:]]\n", "\n", "# # display the results\n", "plt.figure(figsize=(14,24))\n", "ax1 = plt.subplot(1,2,1)\n", "ax2 = plt.subplot(1,2,2)\n", "ax1.imshow(I1)\n", "ax2.imshow(I2)\n", "plt.title('Found %d Putative Matches'%match1.shape)\n", "for i in range(match1.shape):\n", " x1,y1 = match1[:,i]\n", " x2,y2 = match2[:,i]\n", " ax1.plot([x1, x2],[y1, y2],'-r')\n", " ax1.add_patch(patches.Rectangle((x1-w/2,y1-w/2),w,w, fill=False))\n", " ax2.plot([x2, x1],[y2, y1],'-r')\n", " ax2.add_patch(patches.Rectangle((x2-w/2,y2-w/2),w,w, fill=False))\n", "\n", "plt.show()\n", "\n", "print('unique points in image 1: %d'%np.unique(inds[0,:]).shape)\n", "print('unique points in image 2: %d'%np.unique(inds[1,:]).shape)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Final values for parameters\n", "* w =\n", "* t =\n", "* d =\n", "* p =\n", "* num_matches = " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 4 (Programming): Outlier Rejection (20 points)\n", " \n", " The resulting set of putative point correspondences should contain both inlier\n", "and outlier correspondences (i.e., false matches). Determine the set of inlier point\n", "correspondences using the M-estimator Sample Consensus (MSAC) algorithm,\n", "where the maximum number of attempts to find a consensus set is determined\n", "adaptively. For each trial, you must use the 7-point algorithm (as described in\n", "lecture) to estimate the fundamental matrix, resulting in 1 or 3 solutions. Calculate \n", "the (squared) Sampson error as a first order approximation to the geometric error.\n", "\n", "Hint: this problem has codimension 1\n", "\n", "Also: fix a random seed in your MSAC. If I cannot reproduce your results, you will lose points.\n", "\n", "#### Report your values for:\n", " * the probability $p$ that as least one of the random samples does not contain any outliers\n", " * the probability $\\alpha$ that a given point is an inlier\n", " * the resulting number of inliers\n", " * the number of attempts to find the consensus set\n", " * the tolerance for inliers\n", " * the cost threshold\n", " * random seed\n", " \n", "#### Display figures for:\n", "* pair of images, where the inlier features in each of the images are indicated by a square window about the feature and a line segment is drawn from the feature to the coordinates of the corresponding feature in the other image" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from scipy.stats import chi2\n", "\n", "def DisplayResults(F, title):\n", " print(title+' =')\n", " print(F/np.linalg.norm(F)*np.sign(F[-1,-1]))\n", "\n", "def MSAC(pts1, pts2, thresh, tol, p):\n", " # Inputs:\n", " # pts1 - matched feature correspondences in image 1\n", " # pts2 - matched feature correspondences in image 2\n", " # thresh - cost threshold\n", " # tol - reprojection error tolerance \n", " # p - probability that as least one of the random samples does not contain any outliers \n", " #\n", " # Output:\n", " # consensus_min_cost - final cost from MSAC\n", " # consensus_min_cost_model - fundamental matrix F\n", " # inliers - list of indices of the inliers corresponding to input data\n", " # trials - number of attempts taken to find consensus set\n", " \n", " \"\"\"your code here\"\"\"\n", " \n", " \n", " \n", " trials = 0\n", " max_trials = np.inf\n", " consensus_min_cost = np.inf\n", " consensus_min_cost_model = np.zeros((3,4))\n", " inliers = np.random.randint(0, 200, size=100)\n", " return consensus_min_cost, consensus_min_cost_model, inliers, trials\n", "\n", "\n", "# MSAC parameters \n", "thresh = 0\n", "tol = 0\n", "p = 0\n", "alpha = 0\n", "\n", "tic=time.time()\n", "\n", "cost_MSAC, F_MSAC, inliers, trials = MSAC(match1, match2, thresh, tol, p)\n", "\n", "# choose just the inliers\n", "xin1 = match1[:,inliers]\n", "xin2 = match2[:,inliers]\n", "\n", "toc=time.time()\n", "time_total=toc-tic\n", "\n", "# display the results\n", "print('took %f secs'%time_total)\n", "print('%d iterations'%trials)\n", "print('inlier count: ',len(inliers))\n", "print('inliers: ',inliers)\n", "print('MSAC Cost = %.9f'%cost_MSAC)\n", "DisplayResults(F_MSAC, 'F_MSAC')\n", "\n", "# display the figures\n", "plt.figure(figsize=(14,8))\n", "ax1 = plt.subplot(1,2,1)\n", "ax2 = plt.subplot(1,2,2)\n", "ax1.imshow(I1)\n", "ax2.imshow(I2)\n", "\n", "for i in range(xin1.shape):\n", " x1,y1 = xin1[:,i]\n", " x2,y2 = xin2[:,i]\n", " ax1.plot([x1, x2],[y1, y2],'-r')\n", " ax1.add_patch(patches.Rectangle((x1-w/2,y1-w/2),w,w, fill=False))\n", " ax2.plot([x2, x1],[y2, y1],'-r')\n", " ax2.add_patch(patches.Rectangle((x2-w/2,y2-w/2),w,w, fill=False))\n", "\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Final values for parameters\n", "* random seed = \n", "* $p$ =\n", "* $\\alpha$ =\n", "* tolerance = \n", "* threshold = \n", "* num_inliers =\n", "* num_attempts = \n", "* consensus_min_cost = " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 5 (Programming): Linear Estimation of the Fundamental Matrix (15 points)\n", " Estimate the fundamental matrix $\\boldsymbol{F}_\\text{DLT}$ from the\n", " resulting set of inlier correspondences using the direct linear\n", " transformation (DLT) algorithm (with data normalization). Include\n", " the numerical values of the resulting $\\boldsymbol{F}_\\text{DLT}$, scaled\n", " such that $||\\boldsymbol{F}_\\text{DLT}||_\\text{Fro} = 1$" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def Homogenize(x):\n", " # converts points from inhomogeneous to homogeneous coordinates \n", " return np.vstack((x,np.ones((1,x.shape))))\n", "\n", "def Dehomogenize(x):\n", " # converts points from homogeneous to inhomogeneous coordinates \n", " return x[:-1]/x[-1]\n", "\n", "def DLT(x1, x2, normalize=True):\n", " # Inputs:\n", " # x1 - inhomogeneous inlier correspondences in image 1\n", " # x2 - inhomogeneous inlier correspondences in image 2\n", " # normalize - if True, apply data normalization to x1 and x2\n", " #\n", " # Outputs:\n", " # F - the DLT estimate of the fundamental matrix \n", " \n", " \"\"\"your code here\"\"\"\n", " \n", " # data normalization\n", " if normalize:\n", " x1 = x1\n", " x2 = x2\n", " \n", " # data denormalization\n", " if normalize:\n", " x1 = x1\n", " x2 = x2\n", " \n", " \n", " F = np.eye(3)\n", " return F\n", "\n", "# Uncomment the following lines to use sample inliers. Please comment these before submitting.\n", "# As always, you may lose points otherwise\n", "\n", "# xin1 = np.loadtxt(\"hw5_pts1.txt\").T\n", "# xin2 = np.loadtxt(\"hw5_pts2.txt\").T\n", "\n", "# compute the linear estimate with data normalization\n", "print ('DLT with Data Normalization')\n", "time_start=time.time()\n", "F_DLT = DLT(xin1, xin2, normalize=True)\n", "time_total=time.time()-time_start\n", "\n", "# display the resulting F_DLT, scaled with its frobenius norm\n", "DisplayResults(F_DLT, 'F_DLT')" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 6 (Programming): Nonlinear Estimation of the Fundamental Matrix (70 points) \n", " Retrieve the camera projection matrices $\\boldsymbol{P} = [\\boldsymbol{I} \\,|\\,\n", " \\boldsymbol{0}]$ and $\\boldsymbol{P}' = [\\boldsymbol{M} \\,|\\, \\boldsymbol{v}]$, where $\\boldsymbol{M}$\n", " is full rank, from $\\boldsymbol{F}_\\text{DLT}$. Use the resulting camera\n", " projection matrix $\\boldsymbol{P}'$ associated with the second image and\n", " the triangulated 3D points as an initial estimate to an iterative\n", " estimation method, specifically the sparse Levenberg-Marquardt\n", " algorithm, to determine the Maximum Likelihood estimate of the\n", " fundamental matrix $\\boldsymbol{F} = [\\boldsymbol{v}]_\\times \\boldsymbol{M}$ that\n", " minimizes the reprojection error. The initial estimate of the 3D\n", " points must be determined using the two-view optimal triangulation\n", " method described in lecture (algorithm 12.1 in the Hartley \\&\n", " Zisserman book, but use the ray-plane intersection method for the\n", " final step instead of the homogeneous method). Additionally, you\n", " must parameterize the camera projection matrix $\\boldsymbol{P}'$ associated\n", " with the second image and the homogeneous 3D scene points that are\n", " being adjusted using the parameterization of homogeneous vectors\n", " (see section A6.9.2 (page 624) of the textbook, and the corrections\n", " and errata).\n", " \n", " Report the initial cost (i.e. cost at iteration 0) and the cost at the end\n", " of each successive iteration. Show the numerical values for the final \n", " estimate of the fundamental matrix $\\boldsymbol{F}_\\text{LM}$, scaled\n", " such that $||\\boldsymbol{F}_\\text{LM}||_\\text{Fro} = 1$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from scipy.linalg import block_diag\n", "\n", "def LM(F, x1, x2, max_iters, lam):\n", " # Input:\n", " # F - DLT estimate of the fundamental matrix\n", " # x1 - inhomogeneous inlier points in image 1\n", " # x2 - inhomogeneous inlier points in image 2\n", " # max_iters - maximum number of iterations\n", " # lam - lambda parameter\n", " # Output:\n", " # F - Final fundamental matrix obtained after convergence\n", " \n", " \n", " \n", " \"\"\"your code here\"\"\"\n", " \n", " cost = np.inf\n", " print ('iter %03d Cost %.9f'%(0, cost))\n", " \n", " for i in range(max_iters): \n", " print ('iter %03d Cost %.9f'%(i+1, cost))\n", " \n", " \n", " return F\n", "\n", "# LM hyperparameters\n", "lam = .001\n", "max_iters = 10\n", "\n", "# Run LM initialized by DLT estimate\n", "print ('Sparse LM')\n", "time_start=time.time()\n", "F_LM = LM(F_DLT, xin1, xin2, max_iters, lam)\n", "time_total=time.time()-time_start\n", "print('took %f secs'%time_total)\n", "\n", "# display the resulting F_LM, scaled with its frobenius norm\n", "DisplayResults(F_LM, 'F_LM')" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "## Problem 7 (Programming): Point to Line Mapping (10 points)\n", " Qualitatively determine the accuracy of $\\boldsymbol{F}_\\text{LM}$ by\n", " mapping points in image 1 to epipolar lines in image 2. Identify\n", " three distinct corners distributed in image 1 that are \n", " not in the set of inlier correspondences, visually approximate \n", " their pixel coordinates $\\boldsymbol{x}_{\\{1,2,3\\}}$, and map them to\n", " epipolar lines $\\boldsymbol{l'}_{\\{1,2,3\\}} = \\boldsymbol{F}_\\text{LM}\n", " \\boldsymbol{x}_{\\{1,2,3\\}}$ in the second image under the fundamental\n", " matrix $\\boldsymbol{F}_\\text{LM}$.\n", "\n", " Include a figure containing the pair of images,\n", " where the three points in image 1 are indicated by a square (or\n", " circle) about the feature and the corresponding epipolar lines are\n", " drawn in image 2. Comment on the qualitative accuracy of the mapping. (Hint: each line \n", " $\\boldsymbol{l'}_i$ should pass through the point $\\boldsymbol{x'}_i$ in image 2 that \n", " corresponds to the point $\\boldsymbol{x}_i$ in image 1)." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Store your three points in image 1 in variable xchosen1\n", "# Store the corresponding epipolar lines in variable epi_lines\n", "\n", "# You can modify the code to display the figures, to highlight the corresponding point in image 2.\n", "# You will have to find the pixel co-ordinates of the \n", "# corresponding point in image 2 manually, as we are explicitly choosing outliers(find the real matching point\n", "# and not the one your code outputs). The epipolar lines should\n", "# pass close by or through these points.\n", "# \n", "\n", "\"\"\"your code here\"\"\"\n", "\n", "\n", "# display the figures\n", "plt.figure(figsize=(28,16))\n", "ax1 = plt.subplot(1,2,1)\n", "ax2 = plt.subplot(1,2,2)\n", "ax1.imshow(I1)\n", "ax2.imshow(I2)\n", "im_height, im_width = I1.shape[:2]\n", "x_ax = np.linspace(0, im_width, im_width*10)\n", "colors = ['red', 'blue', 'yellow']\n", "for i in range(xchosen1.shape):\n", " a, b, c = epi_lines[:, i]\n", " xx, yy = [], []\n", " for xval in x_ax:\n", " yval = -(a/b)*xval - c/b\n", " if yval > 0 and yval < im_width:\n", " xx.append(xval)\n", " yy.append(yval)\n", " x1,y1 = xchosen1[:,i]\n", " ax1.add_patch(patches.Rectangle((x1-w/2,y1-w/2),w,w, fill=True, color=colors[i]))\n", " ax2.plot(xx,yy,'-r', color=colors[i])\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\"\"\"Comment on your results here.\"\"\"" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.5" }, "toc": { "base_numbering": 1, "nav_menu": {}, "number_sections": false, "sideBar": false, "skip_h1_title": true, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 2 }