Final Report (Filling in Template).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CS/ECE/ISyE 524 &mdash; Introduction to Optimization &mdash; Fall 2018 ###\n",
    "\n",
    "# Financial Factor Analysis #\n",
    "\n",
    "### Tyler Behle\n",
    "\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: tbehle@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9072599682  \n",
    "### Mitchel Berg\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: mberg9@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9072092852  \n",
    "### Christian Colomb\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: ccolumb@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9074912750  \n",
    "### Mike Osmian\n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Email: osmian@wisc.edu  \n",
    "   &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp; Student ID Number: 9073922677  \n",
    "\n",
    "### Table of Contents\n",
    "\n",
    "1. [Introduction](#1.-Introduction)\n",
    "1. [Mathematical Model](#2.-Mathematical-model)\n",
    "1. [Solution](#3.-Solution)\n",
    "1. [Results and Discussion](#4.-Results-and-discussion)\n",
    "1. [Optional Subsection](#4.A.-Feel-free-to-add-subsections)\n",
    "1. [Conclusion](#5.-Conclusion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction ##\n",
    "\n",
    "Our project aims to determine what financial factors are most likely to indicate the success of a portfolio.  Similar to the classic portfolio optimization problem, our project goes one step above determining which companies an individual should invest in and instead decides which factors are most important when selecting a company to put into a portfolio.  Doing so would allow an individual to set constraints on these factors that have been deemed \"most important\" and cross reference it with a list of companies to determine which would be best to invest in.\n",
    "\n",
    "To accomplish this goal, we created synthetically generated data of factor values over time to establish our model, which we then tested against another subset of similarly generated data.  The generated data we used for our factor values followed a normal distribution centered at zero.  The mathematical model we created uses infinite regularization, lasso regression, and ridge regression to determine which factors correlate most strongly with high returns.  \n",
    "\n",
    "\n",
    "\n",
    "### Steve's Comments\n",
    "\n",
    "The first few sentences should give a quick overview of the entire project. Then, elaborate with a description of the problem that will be solved, a brief history (with [citations](https://en.wikipedia.org/wiki/Citation)) of how the problem came about, why it's important/interesting, and any other interesting facts you'd like to talk about. You should address and explain where the problem data is coming from (research? the internet? synthetically generated?) Also give an outline of the rest of the report.\n",
    "\n",
    "This section should be 300-600 words long, and **should be accessible to a general audience** (don't assume your reader has taken the class!). Feel free to include images if you think it'll be helpful:\n",
    "![fixit flowchart][flow]\n",
    "\n",
    "For more help on using Markdown, see [this reference](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet).\n",
    "\n",
    "[flow]: https://s-media-cache-ak0.pinimg.com/736x/f5/75/c5/f575c53b93724808c6f0211890a54900.jpg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Mathematical model ##\n",
    "\n",
    "\n",
    "### 2.1. Model Overview\n",
    "\n",
    "Our problem's model is linear.  It analyzes 100 financial factors to establish the best combination of factors to observe when investing in a company.  In order to most effectively track our model's results we generated and minipulated parts of our data set to see how the model reacts to changes and relationships between factor data.\n",
    "\n",
    "\n",
    "\n",
    "### 2.2. Data Creation\n",
    "\n",
    "Because most financial data access requires expensive subscriptions, we chose to create our own data using a python script that prints to a CSV file.  Creating our own data also gave us more control over the data's attributes, allowing us to manipulate certain fields to see how the model reacts to these changes.  Some of these changes include:\n",
    "\n",
    "\n",
    "### 2.3. Parameters\n",
    "\n",
    "$$\\begin{aligned}\n",
    "    \\text{Parameters:}\\qquad& \\lambda = 0.01 && \\text{Regularization Tradeoff Parameter}\\\\\n",
    "    & \\mu = 0.00001 && \\text{Regularization Tradeoff Parameter}\\\\\n",
    "    & n = 100 && \\text{Number of Factors}\\\\\n",
    "    & d = 10000 && \\text{Number of Data Points Per Factor}\\\\\n",
    "    & data_{n,d} && \\text{n by d Matrix of Each Data Point}\\\\\n",
    "    & rets_d && \\text{Array containing the returns at each data point}\\\\\n",
    "    \\end{aligned}$$\n",
    "\n",
    "### 2.4. Variables\n",
    "\n",
    "$$\\begin{aligned}\n",
    "    \\text{Variables:}\\qquad& b_i && i=1,\\dots,n && \\text{Add Description}\\\\\n",
    "    & yt_i && i=1,\\dots,d && \\text{Add Description}\\\\\n",
    "    & t_i && i=1,\\dots,n && \\text{Add Description}\\\\\n",
    "    & maxt && && \\text{Add Description}\n",
    "    \\end{aligned}$$\n",
    "\n",
    "\n",
    "### 2.5. Constraints\n",
    "\n",
    "$$\\begin{aligned}\n",
    "    \\text{All Objective Functions Are Subject To:}\\qquad& yt_i = b \\cdot data[i,:] && i=1,\\dots,d && \\text{Add Description}\\\\\n",
    "    & t_i \\ge b_i && i=1,\\dots,n && \\text{Add Description}\\\\\n",
    "    & t_i \\ge -b_i && i=1,\\dots,n && \\text{Add Description}\\\\\n",
    "    & maxt \\ge t_i && i=1,\\dots,n && \\text{Add Description}\\\\\n",
    "    \\end{aligned}$$\n",
    "\n",
    "\n",
    "### 2.6. Objective Functions\n",
    "\n",
    "$$\\begin{aligned}\n",
    " & \\underset{x \\in \\mathbb{R^n}}{\\text{Minimize}}\\qquad& f_0(x) = \\sum_{i=1}^{d}(rets - yt_i)^2 + \\lambda * maxt + \\mu * \\sum_{i=1}^{n}(t_i) && \\text{LASSO with L-Infinite Regularization} \\\\\n",
    " & \\underset{x \\in \\mathbb{R^n}}{\\text{Minimize}}\\qquad& f_1(x) = \\sum_{i=1}^{d}(rets - yt_i)^2 + \\mu * maxt && \\text{L-Infinite Regularization} \\\\\n",
    " & \\underset{x \\in \\mathbb{R^n}}{\\text{Minimize}}\\qquad& f_2(x) = \\sum_{i=1}^{d}(rets - yt_i)^2 + \\lambda * \\sum_{t=i}^{n}(t_i) && \\text{LASSO} \\\\\n",
    " & \\underset{x \\in \\mathbb{R^n}}{\\text{Minimize}}\\qquad& f_3(x) = \\sum_{i=1}^{d}(rets - yt_i)^2 + \\lambda * \\sum_{b=i}^{n}(b_1^2) && \\text{L2 Regularization}\\\\\n",
    " & \\underset{x \\in \\mathbb{R^n}}{\\text{Minimize}}\\qquad& f_4(x) = \\sum_{i=1}^{d}(rets - yt_i)^2  && \\text{Least Squares} \\\\\n",
    "    \\end{aligned}$$\n",
    "\n",
    "### Steve's Notes\n",
    "\n",
    "A discussion of the modeling assumptions made in the problem (e.g. is it from physics? economics? something else?). Explain the decision variables, the constraints, and the objective function. Finally, show the optimization problem written in standard form. Discuss the model type (LP, QP, MIP, etc.). Equations should be formatted in $\\\\LaTeX$ within the IJulia notebook. For this section you may **assume the reader is familiar with the material covered in class**.\n",
    "\n",
    "Here is an example of an equation:\n",
    "\n",
    "$$\\begin{bmatrix}\n",
    "      1 & 2 \\\\\n",
    "       3 & 4\n",
    "    \\end{bmatrix}\n",
    "    \\begin{bmatrix} x \\\\ y \\end{bmatrix} =\n",
    "    \\begin{bmatrix} 5 \\\\ 6 \\end{bmatrix}$$\n",
    "\n",
    "And here is an example of an optimization problem in standard form:\n",
    "$$\\begin{aligned}\n",
    "  \\underset{x \\in \\mathbb{R^n}}{\\text{maximize}}\\qquad& f_0(x) \\\\\n",
    "    \\text{subject to:}\\qquad& f_i(x) \\le 0 && i=1,\\dots,m\\\\\n",
    "    & h_j(x) = 0 && j=1,\\dots,r\n",
    "    \\end{aligned}$$\n",
    "\n",
    "For some quick tips on using $\\LaTeX$, see [this cheat sheet](http://users.dickinson.edu/~richesod/latex/latexcheatsheet.pdf)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Solution ##\n",
    "\n",
    "\n",
    "### Steve's Notes\n",
    "Here, you should code up your model in Julia + JuMP and solve it. Your code should be clean, easy to read, well annotated and commented, and it should compile! You are not allowed to use other programming languages or DCP packages such as `convex.jl`. **I will be running your code**. I suggest having multiple code blocks separated by text blocks that explain the various parts of your solution. You may also solve several versions of your problem with different models/assumptions.\n",
    "\n",
    "It's fine to call external packages such as `Gurobi`, but try to minimize the use of exotic libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Our Code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1 Parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "using JuMP, Gurobi\n",
    "\n",
    "lambda = 0.01\n",
    "mu = 0.00001\n",
    "\n",
    "rets = readcsv(\"./rets.csv\")\n",
    "rets = rets[2:length(rets[:,1]), 2]\n",
    "\n",
    "data = readcsv(\"./full_norm_pct.csv\")\n",
    "dates = data[2:length(data[:,1]), 1]   #Extract Date Column\n",
    "data = data[:, 2:length(data[1,:])]\n",
    "cols = data[1,:]                       #Extract Column Titles\n",
    "data = data[2:length(data[:,1]),:]     #Remove Column Titles\n",
    "\n",
    "data = convert(Array{Float64},data)\n",
    "rets = convert(Array{Float64},rets)\n",
    "\n",
    "num_factors = length(data[1,:])\n",
    "num_dates = length(dates);\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2 Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Academic license - for non-commercial use only\n",
      "0.001175924781466682\tdebt\n",
      "0.001176379876447029\tdebtusd\n",
      "0.0011763428882229532\tebitdausd\n",
      "0.0011760615672569486\tepsusd\n",
      "0.00117567451945085\tequityusd\n",
      "0.0011763651023670068\tevebit\n",
      "0.0011763909143901509\tgrossmargin\n",
      "0.00047536625290461675\tinvestmentsc\n",
      "0.0011762452646425403\tncfcommon\n",
      "0.0011755777254313419\tncfi\n",
      "0.00038442906607985056\tncfinv\n",
      "0.0009771962461347118\tnetincdis\n",
      "0.0011761911436200718\tnetincnci\n",
      "0.0003095008901513142\tpayables\n",
      "0.0011763990948827534\tpe\n",
      "0.00018843827921669152\tprefdivis\n",
      "0.0011675827525512285\trevenueusd\n",
      "4.163285033462269e-7\tsharefactor\n",
      "0.0005278349385322946\tsps\n",
      "0.0011754952358400995\ttaxexp\n",
      "20\n",
      "Academic license - for non-commercial use only\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "0.001785158488166129\t\n",
      "0.003674232877735314\t\n",
      "0.002254814301724906\t\n",
      "0.0020480713305107098\t\n",
      "0.00011698191037164822\t\n",
      "0.0010645910523629798\t\n",
      "0.005740175467731956\t\n",
      "0.004745915100783196\t\n",
      "0.0007455390998545319\t\n",
      "0.0023025906519179838\t\n",
      "1.6288406720170137e-5\t\n",
      "0.000787090686563297\t\n",
      "0.005478266307146211\t\n",
      "0.00016719391353101044\t\n",
      "14"
     ]
    }
   ],
   "source": [
    "m = Model(solver = GurobiSolver(OutputFlag = 0))\n",
    "\n",
    "@variable(m, b[1:num_factors])  \n",
    "@variable(m, yt[1:num_dates])\n",
    "@variable(m, t[1:num_factors])  # Avsolute value of variable b\n",
    "@variable(m, maxt)\n",
    "@constraint(m, [i in 1:num_dates], yt[i] == dot(b, data[i, :]))\n",
    "@constraint(m, t .>= b)\n",
    "@constraint(m, t .>= -b)\n",
    "@constraint(m, [i in 1:num_factors], maxt >= t[i])\n",
    "@objective(m, Min, sum( (rets - yt).^2 ) + lambda * maxt + mu * sum(t))  # LASSO with L-Infinite Regularization\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ) + mu * maxt)                  # L-Infinite Regularization\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ) + lambda * sum(t))            # LASSO\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ) + lambda * sum(b.^2))         # L2 Regularization\n",
    "# @objective(m, Min, sum( (rets - yt).^2 ))                              # Least Squares\n",
    "\n",
    "solve(m)\n",
    "\n",
    "\n",
    "# Here we selected which factors made our requirements\n",
    "selection = []\n",
    "\n",
    "bs = getvalue(b)\n",
    "count = 0\n",
    "for i in 1:length(bs)\n",
    "    if bs[i] > 0.0000001\n",
    "        print(bs[i], \"\\t\", cols[i],\"\\n\")\n",
    "        count += 1\n",
    "        push!(selection, i)\n",
    "    end\n",
    "end\n",
    "print(count,\"\\n\")\n",
    "\n",
    "sel_data = data[:, selection]\n",
    "num_factors = length(sel_data[1,:])\n",
    "\n",
    "\n",
    "\n",
    "# Second model takes top factors selected and runs regression on just those to establish which are the best\n",
    "m2 = Model(solver = GurobiSolver(OutputFlag = 0))\n",
    "\n",
    "@variable(m2, b2[1:num_factors])\n",
    "@variable(m2, yt2[1:num_dates])\n",
    "@constraint(m2, [i in 1:num_dates], yt2[i] == dot(b2, sel_data[i, :]))\n",
    "@objective(m2, Min, sum( (rets - yt2).^2 ) + lambda * sum(b2.^2))\n",
    "\n",
    "solve(m2)\n",
    "\n",
    "print(\"\\n\\n\\n\\n\")\n",
    "bs2 = getvalue(b2)\n",
    "count = 0\n",
    "for i in 1:length(bs2)\n",
    "    if bs2[i] > 0.0000001\n",
    "        print(bs2[i], \"\\t\\n\")\n",
    "        count += 1\n",
    "    end\n",
    "end\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.001175924781466682\tdebt\n",
      "0.001176379876447029\tdebtusd\n",
      "0.0011763428882229532\tebitdausd\n",
      "0.0011760615672569486\tepsusd\n",
      "0.00117567451945085\tequityusd\n",
      "0.0011763651023670068\tevebit\n",
      "0.0011763909143901509\tgrossmargin\n",
      "0.00047536625290461675\tinvestmentsc\n",
      "0.0011762452646425403\tncfcommon\n",
      "0.0011755777254313419\tncfi\n",
      "0.00038442906607985056\tncfinv\n",
      "0.0009771962461347118\tnetincdis\n",
      "0.0011761911436200718\tnetincnci\n",
      "0.0003095008901513142\tpayables\n",
      "0.0011763990948827534\tpe\n",
      "0.00018843827921669152\tprefdivis\n",
      "0.0011675827525512285\trevenueusd\n",
      "4.163285033462269e-7\tsharefactor\n",
      "0.0005278349385322946\tsps\n",
      "0.0011754952358400995\ttaxexp\n",
      "20"
     ]
    }
   ],
   "source": [
    "bs = getvalue(b)\n",
    "count = 0\n",
    "for i in 1:length(bs)\n",
    "    if bs[i] > 0.0000001\n",
    "        print(bs[i], \"\\t\", cols[i],\"\\n\")\n",
    "        count += 1\n",
    "    end\n",
    "end\n",
    "print(count)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×20 Array{Float64,2}:\n",
       " -0.00103938   4.06405    0.156267    …  -1.14286  -4.03807    2.05337  \n",
       "  0.30437      6.46051    0.00650794      0.0       0.245311   0.498002 \n",
       " -0.634001    -0.279962  -3.18678         0.0      -1.1883    -2.03745  \n",
       " -2.86162     -0.494445  -1.13603         0.0       2.1999     0.89586  \n",
       " -0.676859    -2.1649    -1.48907         0.0       1.08407   -0.244727 \n",
       "  3.36258      1.34137   -0.886255    …   0.0      -0.399574  -1.8382   \n",
       "  0.0807311   -0.166374  47.4322          0.0      -2.67365    0.0910096"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sel_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "20"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "num_factors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.2888055051188396e-5"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "getobjectivevalue(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7×3 Array{Float64,2}:\n",
       "  0.0434216   0.106106   -0.716398\n",
       " -0.673924   -0.2201      4.23729 \n",
       " -2.1544     -0.597726   -0.476165\n",
       "  1.75225    -1.36557     0.574577\n",
       " -0.70383     6.28081    -4.20583 \n",
       "  0.141441   -0.0405702  -0.926255\n",
       " -0.386225    0.556266    0.91608 "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[:,[1,20,37]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Results and discussion ##\n",
    "\n",
    "Here, you display and discuss the results. Show figures, plots, images, trade-off curves, or whatever else you can think of to best illustrate your results. The discussion should explain what the results mean, and how to interpret them. You should also explain the limitations of your approach/model and how sensitive your results are to the assumptions you made.\n",
    "\n",
    " Use plots (see `PyPlot` examples from class), or you can display results in a table like this:\n",
    "\n",
    "| Tables        | Are          | Cool  |\n",
    "| ------------- |:-------------| -----:|\n",
    "| col 3 is      |right-aligned |\\$1600 |\n",
    "|  colons       | align columns|  \\$12 |\n",
    "| zebra stripes |    are neat  |   \\$1 |\n",
    "\n",
    "### 4.A. Feel free to add subsections\n",
    "\n",
    "#### 4.A.a. or subsubsections"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Conclusion ##\n",
    "\n",
    "Summarize your findings and your results, and talk about at least one possible future direction; something that might be interesting to pursue as a follow-up to your project."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Julia 0.6.4",
   "language": "julia",
   "name": "julia-0.6"
  },
  "language_info": {
   "file_extension": ".jl",
   "mimetype": "application/julia",
   "name": "julia",
   "version": "0.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}