mle_implementation-solidification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LaMassa S82 (x-ray) catalog *versus* UKIDSS\n",
    "\n",
    "In this notebook the cross-matching procedure applied by [LaMassa et al. 2016] (LM hereafter) to multi-wavelength catalogs will  be reproduced. Particularly, we will cross-match the x-ray data from XMM-Newton AO 13 cycle to UKIDSS near-infrared. The goal here is to verify whether we can recover the very same results therein, showing us our algorithm for Maximum Likelihood Estimator to be correct.\n",
    "\n",
    "The UKIDSS catalog here in use is the same used in LM: UKIDSS-LAS Data Release 8, *primary* objects, cleaned from spurious/noisy detections; \"?*apermag3*\" magnitude measurements and accordingly errors were retrieved. The table file `ukidss.fits` here used can be taken from [my github repository][github_ukidss].\n",
    "\n",
    "The LaMassa catalog --from where we'll take the x-ray sources-- was downloaded from [CDS].\n",
    "\n",
    "[LaMassa et al. 2016]: http://arxiv.org/abs/1510.00852\n",
    "[github_ukidss]: https://github.com/chbrandt/uks82\n",
    "[CDS]: http://vizier.u-strasbg.fr/viz-bin/VizieR?-source=J%2FApJ%2F817%2F172"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Maximum Likelihood Estimator (MLE)\n",
    "\n",
    "MLE is applied by LM to find the correct --or most possible-- counterpart to their x-ray sources. MLE was first proposed by [Sutherland & Saunders in 1992] and is being adopted as a better alternative to the simplistic *nearest-neighbour* algorithm.\n",
    "\n",
    "What MLE does is to estimate how probable a given counterpart candidate is to be real counterpart from a source in its vicinity. The method was developed having in mind that multiple candidates can be nearby in the (RA,Dec) sky-projected plan. Accordingly, the method includes the ancillary magnitudes as a third component to help differentiating background objects from candidate(s).\n",
    "\n",
    "Consider the situation where there is a source \"S\" (which was observed by instrument \"A\") and in the vicinities, within a distance \"da\", of S there are $N$ objects (\"N1\", \"N2\", ..., \"NN\") that were observed by a different instrument (\"B\"). Also observed using \"B\", but distant a bit further from \"S\". there are $M$ objects (\"M1\", \"M2\", ..., \"MM\") that can not be related to \"S\", but will be of our help further on. The \"M\" objects lie beyond the distance \"da\" and before distance \"db\", and $db > da$. *We want now to answer the following question: which of the objects observed by \"B\" is in fact \"S\" (observed by a different instrument)?* Before coming with any answer, we are taught that instruments \"A\" and \"B\" suffer from different physical effects that lead to uncorrelated errors and different image resolutions when registering the pictures; which means that \"S\" and its (true) counterpart may *not* be one over the other, but shifted by some amount.\n",
    "\n",
    "The distance \"da\" from \"S\" is considered to be \"vicinity\", and objects inside this distance are considered, *a priori*, candidates to the (true) counterpart. Such objects will be called *ancillary*  objects. The objects from sample `M` will be called *background* objects, they compose the sample of objects observed by \"B\" definitely *not* candidate to be \"S\" counterpart.\n",
    "\n",
    "The MLE method will eventually give a score called *Reliability* ($R$) to each of the candidates. Such score --reliability-- is the probability of being the true counterpart, and is given by:\n",
    "$$\n",
    "R_j = \\frac{LR_j}{\\sum_j{LR_j}+(1-Q)}\n",
    "$$\n",
    "\n",
    "The central figure in MLE is the likelihood ratio, $LR$:\n",
    "$$\n",
    "LR_j = \\frac{q(m) f(r)}{n(m)}\n",
    "$$\n",
    ". \n",
    "\n",
    "$f(r)$ is the prior regarding the position of the candidate object relative to the source. Typically, $f(r)$ is modelled as a bidimensional Gaussian with $\\sigma$ being the quadrature sum of source's positional error and objects' average positional error:\n",
    "$$\n",
    "f(r) = \\frac{1}{2 \\pi \\sigma} exp^{-r^2/2\\sigma^2} ; \n",
    "$$\n",
    "$$\n",
    "\\sigma = \\frac{1}{2}\\Big[\\sqrt{\\sigma^2_{\\alpha_S} + \\sigma^2_{\\delta_S}} + \\sqrt{\\sigma^2_{\\alpha_O} + \\sigma^2_{\\delta_O}}\\Big]\n",
    "$$\n",
    "\n",
    "$q(m)$ is the likelihood of the object being a (good) candidate given its magnitude. It is computed by drawing the ancillary objects normalized magnitude distribution and subtracting from it the normalized background magnitude distribution.\n",
    "\n",
    "Finally, $n(m)$ is the surface density of background objects with magnitude $m$.It is computed by counting the number of background objects per magnitude bin per square-degree; normalized by the number of objects.\n",
    "\n",
    "\n",
    "[Sutherland & Saunders in 1992]: http://mnras.oxfordjournals.org/cgi/doi/10.1093/mnras/259.3.413\n",
    "\n",
    "\n",
    "## The algorithm\n",
    "\n",
    "Let's put it all together to build an algorithm.\n",
    "\n",
    "To compute MLE quantities we need to define the background and ancillary samples. \n",
    "To do that we have to define the *search radius* ($r_s$) --from where the *ancillary* sample will come out-- and the *inner & outer radii* ($r_i$, $r_o$) for the background sample.\n",
    "\n",
    "### Search radii\n",
    "\n",
    "There are different ways to estimate the (best) *search radius*. \n",
    "Typically, the instrument's (nominal) error radius, systematic plus statistical, is used, as [LaMassa et al. 2016]. \n",
    "[Timlin et al. 2016] have used the Rayleigh Criterion to estimate such radius, considering then a physical limitation on resolving close by objects; similarly, the overall PSF (FWHM) is a valid estimator. \n",
    "Another way of estimating $r_s$, data driven, is by directly estimating the typical distance between the objects in each catalog.\n",
    "\n",
    "Analogously, we have to define the *inner and outer radii*, from the primary source, of the annulus defining the background region. \n",
    "Trully speaking, the background region does *not* need to be drawn as an annulus centered centered in the source, but that is a straightforward, generic choice for sampling background sources.\n",
    "It is important to notice that the background region should avoid other sources' ancillary sample, which is to say that the (annulus) region should not intersect with another source's search area.\n",
    "\n",
    "* Estimate samples radii\n",
    "  * (ancillary) search radius: $r_{s}$\n",
    "  * background (annulus) radii: $r_{i} \\lt r_{o}$\n",
    "\n",
    "\n",
    "### Samples definition\n",
    "\n",
    "Once we have the radii defined we cross-match the catalogs to define the *ancillary* and *background* samples; At this point, each source has two lists of objects related to it:\n",
    "\n",
    "* Source\n",
    "  * ancillary sample (within Rs)\n",
    "  * background sample (between Ri and Ro)\n",
    "  \n",
    "But before looping through each primary source, we may define $q(m)$ and $n(m)$ as they are globally defined functions. And after we have $q(m)$ we may estimate $Q$.\n",
    "\n",
    "* Estimate magnitude distributions\n",
    "  * $n(m)$: background surface brightness distribution\n",
    "  * $q(m)$: ancillary brightness distribution\n",
    "  * $Q$: expected counterpart recover rate\n",
    "  \n",
    "  \n",
    "#### Radial prior\n",
    "\n",
    "The radial profile $f(r) \\propto \\sigma^{-1} \\exp^{-r^2/\\sigma^2}$ is ideally defined for each source, for $\\sigma$ is a function of the source' and ancillary objects' positional errors, $\\sigma_s$ and $\\sigma_o$, resp.:\n",
    "$$\n",
    "\\sigma = \\sqrt{\\frac{\\sigma_s^2 + \\sigma_o^2}{2}}\n",
    "$$\n",
    "\n",
    "If the positional errors are well behaved --i.e, their dispersion is small--, we may approximate $f(r)$ as a global function.\n",
    "We may consider $\\sigma_s$ and $\\sigma_o$ as the mean of the respective positional errors.\n",
    "\n",
    "* Compute mean positional errors\n",
    "  * primary sources catalog\n",
    "  * ancillary objects\n",
    "* define $f(r)$\n",
    "\n",
    "\n",
    "#### Likelood Ratio threshold\n",
    "\n",
    "The LR-threshold, $LR_{th}$, is the minimum value an ancillary object may score to be considered a counterpart candidate.\n",
    "There are different ways to compute $LR_{th}$, the simplest one is based on the reliability parameter in a assintotic case: consider there is only *one* ancillary object within the *search radius* around a source; in this case we would expect such object to be the true source' counterpart.\n",
    "Considering the Reliability parameter, $R$ a probability score, $R_j=0.5$ is the minimal (reasonable) value for such parameter so that the object can be considered a candidate.\n",
    "Using the definition of $R$ above we should have:\n",
    "$$\n",
    "0.5 = \\frac{LR_{th}}{LR_{th} + (1-Q)}\n",
    "$$\n",
    "\n",
    "$$\n",
    "LR_{th} = \\frac{0.5(Q-1)}{-0.5}\n",
    "$$\n",
    "\n",
    "$$\n",
    "LR_{th} = 1-Q\n",
    "$$\n",
    "\n",
    "\n",
    "### Counterpart evaluation\n",
    "\n",
    "Now that we have all the ingredients in place we may visit each primary source' neighbourhood and evaluate each ancillary object.\n",
    "\n",
    "For each source,\n",
    "* Loop over the respective ancillary sample:\n",
    "  * evaluate each object's $LR$\n",
    "  * remove objects with $LR_j < LR_{th}$\n",
    "* Sum all ancillaries' $LR_j$\n",
    "* Loop over all candidates:\n",
    "  * compute $R_j$\n",
    "  \n",
    "The highest $R_j$ is said to be the true counterpart.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "[Timlin et al. 2016]: http://arxiv.org/abs/1603.08488"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Coding\n",
    "\n",
    "The workflow has being worked out in `mle_implementation`. Looks like we have a close enough to LaMassa 2016+ implementation: approximately 5% are mismatched (see `mle_implementation_verification`).\n",
    "\n",
    "We now go ahead to design code block, a module, for general use of this algorithm.\n",
    "Eventually, we will merge this method to `booq/xmatch`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### X-Ray (Lamassa-XMM-AO13)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: UnitsWarning: '[10-7W]' did not parse as unit: Invalid character at col 0 [astropy.units.core]\n",
      "WARNING:astropy:UnitsWarning: '[10-7W]' did not parse as unit: Invalid character at col 0\n"
     ]
    }
   ],
   "source": [
    "import booq\n",
    "from booq import table\n",
    "\n",
    "tab_lm = table.ATable.read('lamassa/cds/xmmao13.dat',readme='lamassa/cds/ReadMe',format='cds')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seq</th>\n",
       "      <th>ObsID</th>\n",
       "      <th>RAdeg</th>\n",
       "      <th>DEdeg</th>\n",
       "      <th>e_Pos</th>\n",
       "      <th>DistNN</th>\n",
       "      <th>ExtFlag</th>\n",
       "      <th>InXMM</th>\n",
       "      <th>InChandra</th>\n",
       "      <th>FSoft</th>\n",
       "      <th>...</th>\n",
       "      <th>rH</th>\n",
       "      <th>F250</th>\n",
       "      <th>e_F250</th>\n",
       "      <th>F350</th>\n",
       "      <th>e_F350</th>\n",
       "      <th>F500</th>\n",
       "      <th>e_F500</th>\n",
       "      <th>XMMAO10CP</th>\n",
       "      <th>ChCP</th>\n",
       "      <th>CPCoord</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2359.0</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.097</td>\n",
       "      <td>0.166</td>\n",
       "      <td>4.7</td>\n",
       "      <td>515.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>1.61</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2360.0</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.115</td>\n",
       "      <td>-0.353</td>\n",
       "      <td>3.5</td>\n",
       "      <td>171.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>1.68</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2361.0</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.115</td>\n",
       "      <td>-0.160</td>\n",
       "      <td>5.8</td>\n",
       "      <td>291.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>2.02</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2362.0</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.142</td>\n",
       "      <td>-0.442</td>\n",
       "      <td>3.4</td>\n",
       "      <td>49.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>0.95</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2363.0</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.154</td>\n",
       "      <td>-0.448</td>\n",
       "      <td>2.1</td>\n",
       "      <td>49.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>7.25</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      Seq        ObsID   RAdeg  DEdeg  e_Pos  DistNN  ExtFlag InXMM InChandra  \\\n",
       "0  2359.0  742830101.0  14.097  0.166    4.7   515.8      0.0    no        no   \n",
       "1  2360.0  742830101.0  14.115 -0.353    3.5   171.7      0.0    no        no   \n",
       "2  2361.0  742830101.0  14.115 -0.160    5.8   291.7      0.0    no        no   \n",
       "3  2362.0  742830101.0  14.142 -0.442    3.4    49.1      0.0    no        no   \n",
       "4  2363.0  742830101.0  14.154 -0.448    2.1    49.1      0.0    no        no   \n",
       "\n",
       "   FSoft   ...     rH  F250  e_F250  F350  e_F350  F500  e_F500  XMMAO10CP  \\\n",
       "0   1.61   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "1   1.68   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "2   2.02   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "3   0.95   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "4   7.25   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "\n",
       "   ChCP  CPCoord  \n",
       "0   NaN      0.0  \n",
       "1   NaN      0.0  \n",
       "2   NaN      0.0  \n",
       "3   NaN      0.0  \n",
       "4   NaN      0.0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_lm = tab_lm.to_dataframe()\n",
    "del tab_lm\n",
    "df_lm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Seq</th>\n",
       "      <th>ObsID</th>\n",
       "      <th>RAdeg</th>\n",
       "      <th>DEdeg</th>\n",
       "      <th>e_Pos</th>\n",
       "      <th>DistNN</th>\n",
       "      <th>ExtFlag</th>\n",
       "      <th>InXMM</th>\n",
       "      <th>InChandra</th>\n",
       "      <th>FSoft</th>\n",
       "      <th>...</th>\n",
       "      <th>rH</th>\n",
       "      <th>F250</th>\n",
       "      <th>e_F250</th>\n",
       "      <th>F350</th>\n",
       "      <th>e_F350</th>\n",
       "      <th>F500</th>\n",
       "      <th>e_F500</th>\n",
       "      <th>XMMAO10CP</th>\n",
       "      <th>ChCP</th>\n",
       "      <th>CPCoord</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2359</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.097</td>\n",
       "      <td>0.166</td>\n",
       "      <td>4.7</td>\n",
       "      <td>515.8</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>1.61</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2360</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.115</td>\n",
       "      <td>-0.353</td>\n",
       "      <td>3.5</td>\n",
       "      <td>171.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>1.68</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2361</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.115</td>\n",
       "      <td>-0.160</td>\n",
       "      <td>5.8</td>\n",
       "      <td>291.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>2.02</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2362</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.142</td>\n",
       "      <td>-0.442</td>\n",
       "      <td>3.4</td>\n",
       "      <td>49.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>0.95</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2363</td>\n",
       "      <td>742830101.0</td>\n",
       "      <td>14.154</td>\n",
       "      <td>-0.448</td>\n",
       "      <td>2.1</td>\n",
       "      <td>49.1</td>\n",
       "      <td>0.0</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>7.25</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 121 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    Seq        ObsID   RAdeg  DEdeg  e_Pos  DistNN  ExtFlag InXMM InChandra  \\\n",
       "0  2359  742830101.0  14.097  0.166    4.7   515.8      0.0    no        no   \n",
       "1  2360  742830101.0  14.115 -0.353    3.5   171.7      0.0    no        no   \n",
       "2  2361  742830101.0  14.115 -0.160    5.8   291.7      0.0    no        no   \n",
       "3  2362  742830101.0  14.142 -0.442    3.4    49.1      0.0    no        no   \n",
       "4  2363  742830101.0  14.154 -0.448    2.1    49.1      0.0    no        no   \n",
       "\n",
       "   FSoft   ...     rH  F250  e_F250  F350  e_F350  F500  e_F500  XMMAO10CP  \\\n",
       "0   1.61   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "1   1.68   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "2   2.02   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "3   0.95   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "4   7.25   ...    NaN   NaN     NaN   NaN     NaN   NaN     NaN        NaN   \n",
       "\n",
       "   ChCP  CPCoord  \n",
       "0   NaN      0.0  \n",
       "1   NaN      0.0  \n",
       "2   NaN      0.0  \n",
       "3   NaN      0.0  \n",
       "4   NaN      0.0  \n",
       "\n",
       "[5 rows x 121 columns]"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ID_column_lm = 'Seq'\n",
    "df_lm[ID_column_lm] = df_lm[ID_column_lm].astype(int).astype(str)\n",
    "\n",
    "df_lm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### IR (UKIDSS-DR8)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "  file: ukidss/ukidss.fits\n",
      "  extension: 1\n",
      "  type: BINARY_TBL\n",
      "  rows: 3501552\n",
      "  column info:\n",
      "    SOURCEID            i8  \n",
      "    RA                  f8  \n",
      "    DEC                 f8  \n",
      "    SIGRA               f4  \n",
      "    SIGDEC              f4  \n",
      "    EPOCH               f8  \n",
      "    EBV                 f4  \n",
      "    AY                  f4  \n",
      "    AJ                  f4  \n",
      "    AH                  f4  \n",
      "    AK                  f4  \n",
      "    MERGEDCLASS         i2  \n",
      "    MERGEDCLASSSTAT     f4  \n",
      "    YHALLMAG            f4  \n",
      "    YHALLMAGERR         f4  \n",
      "    YPETROMAG           f4  \n",
      "    YPETROMAGERR        f4  \n",
      "    YPSFMAG             f4  \n",
      "    YPSFMAGERR          f4  \n",
      "    YAPERMAG3           f4  \n",
      "    YAPERMAG3ERR        f4  \n",
      "    J_1HALLMAG          f4  \n",
      "    J_1HALLMAGERR       f4  \n",
      "    J_1PETROMAG         f4  \n",
      "    J_1PETROMAGERR      f4  \n",
      "    J_1PSFMAG           f4  \n",
      "    J_1PSFMAGERR        f4  \n",
      "    J_1APERMAG3         f4  \n",
      "    J_1APERMAG3ERR      f4  \n",
      "    HHALLMAG            f4  \n",
      "    HHALLMAGERR         f4  \n",
      "    HPETROMAG           f4  \n",
      "    HPETROMAGERR        f4  \n",
      "    HPSFMAG             f4  \n",
      "    HPSFMAGERR          f4  \n",
      "    HAPERMAG3           f4  \n",
      "    HAPERMAG3ERR        f4  \n",
      "    KHALLMAG            f4  \n",
      "    KHALLMAGERR         f4  \n",
      "    KPETROMAG           f4  \n",
      "    KPETROMAGERR        f4  \n",
      "    KPSFMAG             f4  \n",
      "    KPSFMAGERR          f4  \n",
      "    KAPERMAG3           f4  \n",
      "    KAPERMAG3ERR        f4  \n"
     ]
    }
   ],
   "source": [
    "from booq.io import fits\n",
    "uk = fits.open('ukidss/ukidss.fits')\n",
    "uk.info"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# cat_uk = table.read('ukidss/ukidss.fits',\n",
    "#                     columns=['SOURCEID','RA','DEC','SIGRA','SIGDEC',\n",
    "#                              'YAPERMAG3','J_1APERMAG3','HAPERMAG3','KAPERMAG3']\n",
    "#                    )\n",
    "# cat_uk['SOURCEID'] = cat_uk['SOURCEID'].astype(int).astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from booq import table\n",
    "tab_uk_pos = table.ATable.read('ukidss/ukidss.fits',columns=['RA','DEC'])\n",
    "#TODO: at init/reading be able to specific specific region(s) to load\n",
    "#cat_uk = table.ATable.read('ukidss/ukidss.fits',conesearch=(ra,dec,radius),\n",
    "#                          columns={'ra':'RA2000','dec':'DE2000'})\n",
    "#cat_uk = table.ATable.read('ukidss/ukidss.fits',around=(cat_lm,radius),\n",
    "#                          columns={'ra':'RA2000','dec':'DE2000'})\n",
    "#cat_uk = table.ATable.read('ukidss/ukidss.fits',region=booq.region.Region,\n",
    "#                          columns={'ra':'RA2000','dec':'DE2000'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RA</th>\n",
       "      <th>DEC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6.021207</td>\n",
       "      <td>0.021641</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>6.022647</td>\n",
       "      <td>0.021643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>6.021880</td>\n",
       "      <td>0.021649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>6.021893</td>\n",
       "      <td>0.021659</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.022186</td>\n",
       "      <td>0.021660</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         RA       DEC\n",
       "0  6.021207  0.021641\n",
       "1  6.022647  0.021643\n",
       "2  6.021880  0.021649\n",
       "3  6.021893  0.021659\n",
       "4  6.022186  0.021660"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_uk_pos = tab_uk_pos.to_dataframe()\n",
    "del tab_uk_pos\n",
    "df_uk_pos.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**For some reason, UKIDSS positional columns (RA,Dec) are in radians; transform to degrees**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                 RA           DEC\n",
      "count  3.501552e+06  3.501552e+06\n",
      "mean   1.841558e+02 -3.185555e-02\n",
      "std    1.530424e+02  7.262582e-01\n",
      "min    1.004211e-05 -1.249998e+00\n",
      "25%    3.243559e+01 -6.691321e-01\n",
      "50%    3.134817e+02 -5.326323e-02\n",
      "75%    3.349581e+02  6.023953e-01\n",
      "max    3.600000e+02  1.250000e+00\n",
      "\n",
      "Total number of rows: 3501552 (3.5e+06)\n",
      "\n",
      "-> Has Nil? (How many?)\n",
      "RA     0\n",
      "DEC    0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "from astropy.coordinates import Angle\n",
    "ra = Angle(df_uk_pos['RA'].values,'rad').to('deg')\n",
    "dec= Angle(df_uk_pos['DEC'].values,'rad').to('deg')\n",
    "df_uk_pos['RA'] = ra\n",
    "df_uk_pos['DEC'] = dec\n",
    "\n",
    "df_uk_pos.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Define a sub sample\n",
    "\n",
    "Because the UKIDSS catalog is quite big, we should define a sub-sample to better explorer the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                  RA            DEC\n",
      "count  231338.000000  231338.000000\n",
      "mean       21.215457       0.013751\n",
      "std         4.365869       0.389857\n",
      "min        13.597006      -0.721999\n",
      "25%        17.474631      -0.301291\n",
      "50%        21.748381       0.031961\n",
      "75%        24.765722       0.342271\n",
      "max        28.543990       0.703997\n",
      "\n",
      "Total number of rows: 231338 (2.3e+05)\n",
      "\n",
      "-> Has Nil? (How many?)\n",
      "RA     0\n",
      "DEC    0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "ra_min = df_lm.RAdeg.min() - 0.5\n",
    "ra_max = df_lm.RAdeg.max() + 0.5\n",
    "dec_min = df_lm.DEdeg.min() - 0.1\n",
    "dec_max = df_lm.DEdeg.max() + 0.1\n",
    "df_uk_sample = df_uk_pos.crop(RA=(ra_min,ra_max),\n",
    "                           DEC=(dec_min,dec_max))\n",
    "del df_uk_pos\n",
    "\n",
    "df_uk_sample.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>RA</th>\n",
       "      <th>DEC</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>601783</th>\n",
       "      <td>23.367877</td>\n",
       "      <td>0.687888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601784</th>\n",
       "      <td>23.460531</td>\n",
       "      <td>0.688232</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601785</th>\n",
       "      <td>23.414002</td>\n",
       "      <td>0.687667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601786</th>\n",
       "      <td>23.280570</td>\n",
       "      <td>0.687302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>601787</th>\n",
       "      <td>23.474184</td>\n",
       "      <td>0.687547</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               RA       DEC\n",
       "601783  23.367877  0.687888\n",
       "601784  23.460531  0.688232\n",
       "601785  23.414002  0.687667\n",
       "601786  23.280570  0.687302\n",
       "601787  23.474184  0.687547"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_uk_sample.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we get the positional-index of those selected lines to load the whole UKIDSS catalog."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "&lt;ATable length=231338&gt;\n",
       "<table id=\"table140091361958376\" class=\"table-striped table-bordered table-condensed\">\n",
       "<thead><tr><th>SOURCEID</th><th>RA</th><th>DEC</th><th>SIGRA</th><th>SIGDEC</th><th>EPOCH</th><th>EBV</th><th>AY</th><th>AJ</th><th>AH</th><th>AK</th><th>MERGEDCLASS</th><th>MERGEDCLASSSTAT</th><th>YHALLMAG</th><th>YHALLMAGERR</th><th>YPETROMAG</th><th>YPETROMAGERR</th><th>YPSFMAG</th><th>YPSFMAGERR</th><th>YAPERMAG3</th><th>YAPERMAG3ERR</th><th>J_1HALLMAG</th><th>J_1HALLMAGERR</th><th>J_1PETROMAG</th><th>J_1PETROMAGERR</th><th>J_1PSFMAG</th><th>J_1PSFMAGERR</th><th>J_1APERMAG3</th><th>J_1APERMAG3ERR</th><th>HHALLMAG</th><th>HHALLMAGERR</th><th>HPETROMAG</th><th>HPETROMAGERR</th><th>HPSFMAG</th><th>HPSFMAGERR</th><th>HAPERMAG3</th><th>HAPERMAG3ERR</th><th>KHALLMAG</th><th>KHALLMAGERR</th><th>KPETROMAG</th><th>KPETROMAGERR</th><th>KPSFMAG</th><th>KPSFMAGERR</th><th>KAPERMAG3</th><th>KAPERMAG3ERR</th></tr></thead>\n",
       "<thead><tr><th>int64</th><th>float64</th><th>float64</th><th>float32</th><th>float32</th><th>float64</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>int16</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th><th>float32</th></tr></thead>\n",
       "<tr><td>433827929919</td><td>0.407846390194</td><td>0.0120059039264</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0220508</td><td>0.0267036</td><td>0.0196032</td><td>0.0127454</td><td>0.0079383</td><td>1</td><td>4.93475</td><td>19.8162</td><td>0.339393</td><td>19.7927</td><td>0.352423</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.6622</td><td>0.129669</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.5194</td><td>0.341595</td><td>18.6227</td><td>0.444583</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.5062</td><td>0.151259</td><td>16.8393</td><td>0.166297</td><td>17.0132</td><td>0.244477</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.5441</td><td>0.124013</td></tr>\n",
       "<tr><td>433827929920</td><td>0.409463516869</td><td>0.012011917063</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0236491</td><td>0.028639</td><td>0.021024</td><td>0.0136692</td><td>0.00851367</td><td>-1</td><td>0.421079</td><td>17.9835</td><td>0.0401669</td><td>17.9463</td><td>0.0584219</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.8783</td><td>0.0279219</td><td>17.4947</td><td>0.0500438</td><td>17.5421</td><td>0.0748066</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.3798</td><td>0.0304339</td><td>16.9268</td><td>0.0543857</td><td>16.9449</td><td>0.0858596</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>16.8934</td><td>0.035704</td><td>16.618</td><td>0.0808161</td><td>16.6613</td><td>0.118318</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>16.5653</td><td>0.0519861</td></tr>\n",
       "<tr><td>433827929921</td><td>0.408651420325</td><td>0.0120020568056</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0226826</td><td>0.0274686</td><td>0.0201648</td><td>0.0131105</td><td>0.00816573</td><td>1</td><td>2.94201</td><td>19.834</td><td>0.366881</td><td>19.834</td><td>0.366881</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.2877</td><td>0.228542</td><td>19.2812</td><td>0.234339</td><td>19.2812</td><td>0.234339</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.6216</td><td>0.228677</td><td>18.7383</td><td>0.383572</td><td>18.7283</td><td>0.389533</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.6331</td><td>0.170696</td><td>17.6925</td><td>0.398679</td><td>17.6925</td><td>0.398679</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.7641</td><td>0.152736</td></tr>\n",
       "<tr><td>433827929922</td><td>0.406322596785</td><td>0.0119956787088</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0225056</td><td>0.0272542</td><td>0.0200075</td><td>0.0130082</td><td>0.00810201</td><td>1</td><td>16.0753</td><td>17.4584</td><td>0.0395172</td><td>17.2767</td><td>0.0590367</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.841</td><td>0.0268707</td><td>16.9499</td><td>0.0415769</td><td>16.7474</td><td>0.0635827</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.2757</td><td>0.0275967</td><td>16.1115</td><td>0.0399464</td><td>16.0146</td><td>0.0600487</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>16.5572</td><td>0.0260672</td><td>15.3375</td><td>0.0384352</td><td>15.0854</td><td>0.0602332</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>15.8079</td><td>0.0255751</td></tr>\n",
       "<tr><td>433827929923</td><td>0.409701804291</td><td>0.0119999505893</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0239274</td><td>0.0289761</td><td>0.0212715</td><td>0.0138301</td><td>0.00861388</td><td>1</td><td>0.782843</td><td>20.1834</td><td>0.387817</td><td>20.1834</td><td>0.387817</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.3265</td><td>0.239728</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433827929924</td><td>0.409192913378</td><td>0.0120002894351</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0233148</td><td>0.0282342</td><td>0.0207268</td><td>0.0134759</td><td>0.00839331</td><td>1</td><td>15.8572</td><td>18.3361</td><td>0.11912</td><td>18.4298</td><td>0.200747</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.941</td><td>0.0686888</td><td>17.7032</td><td>0.105149</td><td>17.7141</td><td>0.127878</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.4453</td><td>0.0785382</td><td>17.0186</td><td>0.134257</td><td>17.1364</td><td>0.218883</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.5933</td><td>0.0666682</td><td>16.3074</td><td>0.130962</td><td>16.4882</td><td>0.237619</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>16.8477</td><td>0.0668407</td></tr>\n",
       "<tr><td>433827929925</td><td>0.407144874736</td><td>0.0119879010813</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0219588</td><td>0.0265921</td><td>0.0195214</td><td>0.0126922</td><td>0.00790517</td><td>1</td><td>10.203</td><td>18.5988</td><td>0.166848</td><td>18.6272</td><td>0.239986</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.2887</td><td>0.0926362</td><td>18.2093</td><td>0.111538</td><td>18.1836</td><td>0.116781</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.0402</td><td>0.133803</td><td>17.1965</td><td>0.157632</td><td>17.3195</td><td>0.225639</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.0303</td><td>0.097468</td><td>16.7118</td><td>0.220821</td><td>16.7333</td><td>0.258675</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.5643</td><td>0.125003</td></tr>\n",
       "<tr><td>433827929926</td><td>0.409655117277</td><td>0.011988087937</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0238767</td><td>0.0289147</td><td>0.0212264</td><td>0.0138007</td><td>0.00859561</td><td>1</td><td>3.84407</td><td>19.8644</td><td>0.403661</td><td>20.653</td><td>0.987575</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.6256</td><td>0.127254</td><td>18.9018</td><td>0.18113</td><td>18.8652</td><td>0.185378</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.2267</td><td>0.160163</td><td>17.9615</td><td>0.274414</td><td>17.9615</td><td>0.274414</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.3873</td><td>0.137669</td><td>17.5568</td><td>0.221735</td><td>17.4939</td><td>0.304165</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.6057</td><td>0.133668</td></tr>\n",
       "<tr><td>433827929927</td><td>0.406068068638</td><td>0.0119687939775</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.0228351</td><td>0.0276533</td><td>0.0203004</td><td>0.0131987</td><td>0.00822064</td><td>1</td><td>1.64003</td><td>19.6853</td><td>0.295783</td><td>19.7383</td><td>0.269282</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.8526</td><td>0.155415</td><td>19.2255</td><td>0.223597</td><td>19.2255</td><td>0.223597</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.4793</td><td>0.201637</td><td>18.274</td><td>0.283236</td><td>18.2174</td><td>0.28653</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.6206</td><td>0.167925</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433827929928</td><td>0.406560901188</td><td>0.0119668732539</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2005.90215116</td><td>0.022154</td><td>0.0268285</td><td>0.0196949</td><td>0.012805</td><td>0.00797543</td><td>1</td><td>2.52627</td><td>19.6839</td><td>0.340545</td><td>19.6585</td><td>0.304635</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.6698</td><td>0.129848</td><td>19.0392</td><td>0.185266</td><td>19.1449</td><td>0.245873</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.0803</td><td>0.138229</td><td>17.9178</td><td>0.193481</td><td>17.9172</td><td>0.199938</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.1842</td><td>0.111608</td><td>17.4144</td><td>0.194213</td><td>17.4593</td><td>0.292663</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.5409</td><td>0.121203</td></tr>\n",
       "<tr><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td><td>...</td></tr>\n",
       "<tr><td>433837305044</td><td>0.34025866932</td><td>-0.0104056767725</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0342745</td><td>0.0415064</td><td>0.03047</td><td>0.0198106</td><td>0.0123388</td><td>-2</td><td>0.709851</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.8969</td><td>0.223847</td><td>19.8969</td><td>0.223847</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.808</td><td>0.186296</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.4788</td><td>0.227545</td><td>17.4788</td><td>0.227545</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>17.8522</td><td>0.164637</td></tr>\n",
       "<tr><td>433837305050</td><td>0.340506321416</td><td>-0.0109826421143</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0346866</td><td>0.0420055</td><td>0.0308364</td><td>0.0200489</td><td>0.0124872</td><td>-1</td><td>-3.11877</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.7356</td><td>0.366901</td><td>20.7356</td><td>0.366901</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.414</td><td>0.325283</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305053</td><td>0.340841774497</td><td>-0.0121659783303</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0352021</td><td>0.0426297</td><td>0.0312946</td><td>0.0203468</td><td>0.0126727</td><td>-1</td><td>-1.70577</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.5132</td><td>0.29812</td><td>20.5132</td><td>0.29812</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.2891</td><td>0.289107</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.0736</td><td>0.329767</td><td>18.1151</td><td>0.366375</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.0793</td><td>0.202633</td></tr>\n",
       "<tr><td>433837305055</td><td>0.341219413239</td><td>-0.0104824705603</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0359505</td><td>0.043536</td><td>0.03196</td><td>0.0207794</td><td>0.0129422</td><td>-1</td><td>-1.43004</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>21.152</td><td>0.71656</td><td>21.152</td><td>0.71656</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.7076</td><td>0.429333</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305057</td><td>0.341218704583</td><td>-0.0112323250807</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.035303</td><td>0.042752</td><td>0.0313844</td><td>0.0204052</td><td>0.0127091</td><td>1</td><td>1.08339</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.4278</td><td>0.156925</td><td>19.4278</td><td>0.156925</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.4976</td><td>0.141521</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305058</td><td>0.341219993198</td><td>-0.0109830742323</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0354352</td><td>0.0429121</td><td>0.0315019</td><td>0.0204816</td><td>0.0127567</td><td>-1</td><td>-3.96188</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>24.3294</td><td>13.8604</td><td>23.1295</td><td>5.28247</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>20.3296</td><td>0.303181</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305060</td><td>0.341247091418</td><td>-0.0120141150356</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0353353</td><td>0.0427911</td><td>0.0314131</td><td>0.0204238</td><td>0.0127207</td><td>1</td><td>2.99593</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.493</td><td>0.132775</td><td>19.3929</td><td>0.131533</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.5533</td><td>0.14854</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305062</td><td>0.34221406482</td><td>-0.0123296423311</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0361872</td><td>0.0438227</td><td>0.0321704</td><td>0.0209162</td><td>0.0130274</td><td>1</td><td>4.67382</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>18.4883</td><td>0.0883277</td><td>18.4883</td><td>0.0883277</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.1478</td><td>0.103815</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305063</td><td>0.342633797848</td><td>-0.0104163954124</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0368925</td><td>0.0446768</td><td>0.0327974</td><td>0.0213239</td><td>0.0132813</td><td>1</td><td>2.10527</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.5584</td><td>0.124152</td><td>19.5276</td><td>0.124712</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.3396</td><td>0.125078</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "<tr><td>433837305064</td><td>0.342778182631</td><td>-0.0113454301968</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>2006.96990376</td><td>0.0366883</td><td>0.0444295</td><td>0.0326159</td><td>0.0212058</td><td>0.0132078</td><td>-1</td><td>-1.25256</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.959</td><td>0.20353</td><td>19.9632</td><td>0.206217</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>19.606</td><td>0.158622</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td><td>-9.99999e+08</td></tr>\n",
       "</table>"
      ],
      "text/plain": [
       "<ATable length=231338>\n",
       "  SOURCEID         RA             DEC        ...  KAPERMAG3   KAPERMAG3ERR\n",
       "   int64        float64         float64      ...   float32      float32   \n",
       "------------ -------------- ---------------- ... ------------ ------------\n",
       "433827929919 0.407846390194  0.0120059039264 ...      17.5441     0.124013\n",
       "433827929920 0.409463516869   0.012011917063 ...      16.5653    0.0519861\n",
       "433827929921 0.408651420325  0.0120020568056 ...      17.7641     0.152736\n",
       "433827929922 0.406322596785  0.0119956787088 ...      15.8079    0.0255751\n",
       "433827929923 0.409701804291  0.0119999505893 ... -9.99999e+08 -9.99999e+08\n",
       "433827929924 0.409192913378  0.0120002894351 ...      16.8477    0.0668407\n",
       "433827929925 0.407144874736  0.0119879010813 ...      17.5643     0.125003\n",
       "433827929926 0.409655117277   0.011988087937 ...      17.6057     0.133668\n",
       "433827929927 0.406068068638  0.0119687939775 ... -9.99999e+08 -9.99999e+08\n",
       "433827929928 0.406560901188  0.0119668732539 ...      17.5409     0.121203\n",
       "         ...            ...              ... ...          ...          ...\n",
       "433837305044  0.34025866932 -0.0104056767725 ...      17.8522     0.164637\n",
       "433837305050 0.340506321416 -0.0109826421143 ... -9.99999e+08 -9.99999e+08\n",
       "433837305053 0.340841774497 -0.0121659783303 ...      18.0793     0.202633\n",
       "433837305055 0.341219413239 -0.0104824705603 ... -9.99999e+08 -9.99999e+08\n",
       "433837305057 0.341218704583 -0.0112323250807 ... -9.99999e+08 -9.99999e+08\n",
       "433837305058 0.341219993198 -0.0109830742323 ... -9.99999e+08 -9.99999e+08\n",
       "433837305060 0.341247091418 -0.0120141150356 ... -9.99999e+08 -9.99999e+08\n",
       "433837305062  0.34221406482 -0.0123296423311 ... -9.99999e+08 -9.99999e+08\n",
       "433837305063 0.342633797848 -0.0104163954124 ... -9.99999e+08 -9.99999e+08\n",
       "433837305064 0.342778182631 -0.0113454301968 ... -9.99999e+08 -9.99999e+08"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 'rows' contains the positions of our interest (the ones in df_uk_sample aget df_uk)\n",
    "rows = df_uk_sample.index.values\n",
    "del df_uk_sample\n",
    "\n",
    "tab_uk_sample = table.ATable.read('ukidss/ukidss.fits',rows=rows)\n",
    "tab_uk_sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "           SOURCEID             RA            DEC        SIGRA       SIGDEC  \\\n",
      "count  2.313380e+05  231338.000000  231338.000000     231338.0     231338.0   \n",
      "mean   4.338322e+11      21.215457       0.013751 -999999488.0 -999999488.0   \n",
      "std    2.958270e+06       4.365869       0.389857          0.0          0.0   \n",
      "min    4.338279e+11      13.597006      -0.721999 -999999488.0 -999999488.0   \n",
      "25%    4.338291e+11      17.474631      -0.301291 -999999488.0 -999999488.0   \n",
      "50%    4.338325e+11      21.748381       0.031961 -999999488.0 -999999488.0   \n",
      "75%    4.338345e+11      24.765722       0.342271 -999999488.0 -999999488.0   \n",
      "max    4.338373e+11      28.543990       0.703997 -999999488.0 -999999488.0   \n",
      "\n",
      "               EPOCH            EBV             AY             AJ  \\\n",
      "count  231338.000000  231338.000000  231338.000000  231338.000000   \n",
      "mean     2006.341197       0.032063       0.038828       0.028504   \n",
      "std         0.630836       0.005339       0.006466       0.004747   \n",
      "min      2005.686836       0.020811       0.025202       0.018501   \n",
      "25%      2005.896787       0.028523       0.034542       0.025357   \n",
      "50%      2005.924034       0.031697       0.038385       0.028179   \n",
      "75%      2006.890816       0.034540       0.041828       0.030706   \n",
      "max      2008.774788       0.062102       0.075205       0.055209   \n",
      "\n",
      "                  AH      ...          HAPERMAG3  HAPERMAG3ERR      KHALLMAG  \\\n",
      "count  231338.000000      ...       2.313380e+05  2.313380e+05  2.313380e+05   \n",
      "mean        0.018532      ...      -4.116270e+08 -4.116270e+08 -2.973526e+08   \n",
      "std         0.003086      ...       4.921291e+08  4.921291e+08  4.570939e+08   \n",
      "min         0.012029      ...      -9.999995e+08 -9.999995e+08 -9.999995e+08   \n",
      "25%         0.016486      ...      -9.999995e+08 -9.999995e+08 -9.999995e+08   \n",
      "50%         0.018321      ...       1.625615e+01  2.266635e-02  1.654052e+01   \n",
      "75%         0.019964      ...       1.814290e+01  1.224565e-01  1.752031e+01   \n",
      "max         0.035895      ...       1.983346e+01  5.110809e-01  2.880482e+01   \n",
      "\n",
      "        KHALLMAGERR     KPETROMAG  KPETROMAGERR      KPSFMAG   KPSFMAGERR  \\\n",
      "count  2.313380e+05  2.313380e+05  2.313380e+05     231338.0     231338.0   \n",
      "mean  -2.973526e+08 -2.972316e+08 -2.972316e+08 -999999488.0 -999999488.0   \n",
      "std    4.570939e+08  4.570402e+08  4.570402e+08          0.0          0.0   \n",
      "min   -9.999995e+08 -9.999995e+08 -9.999995e+08 -999999488.0 -999999488.0   \n",
      "25%   -9.999995e+08 -9.999995e+08 -9.999995e+08 -999999488.0 -999999488.0   \n",
      "50%    1.029659e-01  1.648322e+01  1.356286e-01 -999999488.0 -999999488.0   \n",
      "75%    2.378411e-01  1.751610e+01  2.668378e-01 -999999488.0 -999999488.0   \n",
      "max    6.197548e+03  2.535833e+01  2.745251e+02 -999999488.0 -999999488.0   \n",
      "\n",
      "          KAPERMAG3  KAPERMAG3ERR  \n",
      "count  2.313380e+05  2.313380e+05  \n",
      "mean  -2.964405e+08 -2.964405e+08  \n",
      "std    4.566884e+08  4.566884e+08  \n",
      "min   -9.999995e+08 -9.999995e+08  \n",
      "25%   -9.999995e+08 -9.999995e+08  \n",
      "50%    1.687404e+01  6.536197e-02  \n",
      "75%    1.775938e+01  1.459568e-01  \n",
      "max    2.030012e+01  1.511550e+00  \n",
      "\n",
      "[8 rows x 45 columns]\n",
      "\n",
      "Total number of rows: 231338 (2.3e+05)\n",
      "\n",
      "-> Has Nil? (How many?)\n",
      "SOURCEID           0\n",
      "RA                 0\n",
      "DEC                0\n",
      "SIGRA              0\n",
      "SIGDEC             0\n",
      "EPOCH              0\n",
      "EBV                0\n",
      "AY                 0\n",
      "AJ                 0\n",
      "AH                 0\n",
      "AK                 0\n",
      "MERGEDCLASS        0\n",
      "MERGEDCLASSSTAT    0\n",
      "YHALLMAG           0\n",
      "YHALLMAGERR        0\n",
      "YPETROMAG          0\n",
      "YPETROMAGERR       0\n",
      "YPSFMAG            0\n",
      "YPSFMAGERR         0\n",
      "YAPERMAG3          0\n",
      "YAPERMAG3ERR       0\n",
      "J_1HALLMAG         0\n",
      "J_1HALLMAGERR      0\n",
      "J_1PETROMAG        0\n",
      "J_1PETROMAGERR     0\n",
      "J_1PSFMAG          0\n",
      "J_1PSFMAGERR       0\n",
      "J_1APERMAG3        0\n",
      "J_1APERMAG3ERR     0\n",
      "HHALLMAG           0\n",
      "HHALLMAGERR        0\n",
      "HPETROMAG          0\n",
      "HPETROMAGERR       0\n",
      "HPSFMAG            0\n",
      "HPSFMAGERR         0\n",
      "HAPERMAG3          0\n",
      "HAPERMAG3ERR       0\n",
      "KHALLMAG           0\n",
      "KHALLMAGERR        0\n",
      "KPETROMAG          0\n",
      "KPETROMAGERR       0\n",
      "KPSFMAG            0\n",
      "KPSFMAGERR         0\n",
      "KAPERMAG3          0\n",
      "KAPERMAG3ERR       0\n",
      "dtype: int64\n"
     ]
    }
   ],
   "source": [
    "df_uk = tab_uk_sample.to_dataframe()\n",
    "del tab_uk_sample\n",
    "\n",
    "from astropy.coordinates import Angle\n",
    "ra = Angle(df_uk['RA'].values,'rad').to('deg')\n",
    "dec= Angle(df_uk['DEC'].values,'rad').to('deg')\n",
    "df_uk['RA'] = ra\n",
    "df_uk['DEC'] = dec\n",
    "\n",
    "df_uk.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Clean table null values\n",
    "\n",
    "I assume every negative value in *semi-positive* columns to be `Null/NaN`.\n",
    "Such columns are the positional (RA,Dec) standard deviations and the magnitude columns.\n",
    "(Physically speaking, magnitudes are/can be negative; It just does not happen in our case --galaxies.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cols = ['SIGRA','SIGDEC']\n",
    "cols.extend( [ c for c in df_uk.filter(like='MAG',axis=1).columns ] )\n",
    "for col in cols:\n",
    "    idx = df_uk[col] < 0\n",
    "    df_uk.loc[idx,col] = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SOURCEID</th>\n",
       "      <th>RA</th>\n",
       "      <th>DEC</th>\n",
       "      <th>SIGRA</th>\n",
       "      <th>SIGDEC</th>\n",
       "      <th>EPOCH</th>\n",
       "      <th>EBV</th>\n",
       "      <th>AY</th>\n",
       "      <th>AJ</th>\n",
       "      <th>AH</th>\n",
       "      <th>...</th>\n",
       "      <th>HAPERMAG3</th>\n",
       "      <th>HAPERMAG3ERR</th>\n",
       "      <th>KHALLMAG</th>\n",
       "      <th>KHALLMAGERR</th>\n",
       "      <th>KPETROMAG</th>\n",
       "      <th>KPETROMAGERR</th>\n",
       "      <th>KPSFMAG</th>\n",
       "      <th>KPSFMAGERR</th>\n",
       "      <th>KAPERMAG3</th>\n",
       "      <th>KAPERMAG3ERR</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>433827929919</td>\n",
       "      <td>23.367877</td>\n",
       "      <td>0.687888</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2005.902151</td>\n",
       "      <td>0.022051</td>\n",
       "      <td>0.026704</td>\n",
       "      <td>0.019603</td>\n",
       "      <td>0.012745</td>\n",
       "      <td>...</td>\n",
       "      <td>18.506187</td>\n",
       "      <td>0.151259</td>\n",
       "      <td>16.839336</td>\n",
       "      <td>0.166297</td>\n",
       "      <td>17.013243</td>\n",
       "      <td>0.244477</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17.544065</td>\n",
       "      <td>0.124013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>433827929920</td>\n",
       "      <td>23.460531</td>\n",
       "      <td>0.688232</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2005.902151</td>\n",
       "      <td>0.023649</td>\n",
       "      <td>0.028639</td>\n",
       "      <td>0.021024</td>\n",
       "      <td>0.013669</td>\n",
       "      <td>...</td>\n",
       "      <td>16.893356</td>\n",
       "      <td>0.035704</td>\n",
       "      <td>16.618038</td>\n",
       "      <td>0.080816</td>\n",
       "      <td>16.661270</td>\n",
       "      <td>0.118318</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16.565256</td>\n",
       "      <td>0.051986</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>433827929921</td>\n",
       "      <td>23.414002</td>\n",
       "      <td>0.687667</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2005.902151</td>\n",
       "      <td>0.022683</td>\n",
       "      <td>0.027469</td>\n",
       "      <td>0.020165</td>\n",
       "      <td>0.013111</td>\n",
       "      <td>...</td>\n",
       "      <td>18.633087</td>\n",
       "      <td>0.170696</td>\n",
       "      <td>17.692490</td>\n",
       "      <td>0.398679</td>\n",
       "      <td>17.692490</td>\n",
       "      <td>0.398679</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>17.764069</td>\n",
       "      <td>0.152736</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>433827929922</td>\n",
       "      <td>23.280570</td>\n",
       "      <td>0.687302</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2005.902151</td>\n",
       "      <td>0.022506</td>\n",
       "      <td>0.027254</td>\n",
       "      <td>0.020007</td>\n",
       "      <td>0.013008</td>\n",
       "      <td>...</td>\n",
       "      <td>16.557207</td>\n",
       "      <td>0.026067</td>\n",
       "      <td>15.337543</td>\n",
       "      <td>0.038435</td>\n",
       "      <td>15.085359</td>\n",
       "      <td>0.060233</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15.807901</td>\n",
       "      <td>0.025575</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>433827929923</td>\n",
       "      <td>23.474184</td>\n",
       "      <td>0.687547</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>2005.902151</td>\n",
       "      <td>0.023927</td>\n",
       "      <td>0.028976</td>\n",
       "      <td>0.021271</td>\n",
       "      <td>0.013830</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 45 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       SOURCEID         RA       DEC  SIGRA  SIGDEC        EPOCH       EBV  \\\n",
       "0  433827929919  23.367877  0.687888    NaN     NaN  2005.902151  0.022051   \n",
       "1  433827929920  23.460531  0.688232    NaN     NaN  2005.902151  0.023649   \n",
       "2  433827929921  23.414002  0.687667    NaN     NaN  2005.902151  0.022683   \n",
       "3  433827929922  23.280570  0.687302    NaN     NaN  2005.902151  0.022506   \n",
       "4  433827929923  23.474184  0.687547    NaN     NaN  2005.902151  0.023927   \n",
       "\n",
       "         AY        AJ        AH      ...       HAPERMAG3  HAPERMAG3ERR  \\\n",
       "0  0.026704  0.019603  0.012745      ...       18.506187      0.151259   \n",
       "1  0.028639  0.021024  0.013669      ...       16.893356      0.035704   \n",
       "2  0.027469  0.020165  0.013111      ...       18.633087      0.170696   \n",
       "3  0.027254  0.020007  0.013008      ...       16.557207      0.026067   \n",
       "4  0.028976  0.021271  0.013830      ...             NaN           NaN   \n",
       "\n",
       "    KHALLMAG  KHALLMAGERR  KPETROMAG  KPETROMAGERR  KPSFMAG  KPSFMAGERR  \\\n",
       "0  16.839336     0.166297  17.013243      0.244477      NaN         NaN   \n",
       "1  16.618038     0.080816  16.661270      0.118318      NaN         NaN   \n",
       "2  17.692490     0.398679  17.692490      0.398679      NaN         NaN   \n",
       "3  15.337543     0.038435  15.085359      0.060233      NaN         NaN   \n",
       "4        NaN          NaN        NaN           NaN      NaN         NaN   \n",
       "\n",
       "   KAPERMAG3  KAPERMAG3ERR  \n",
       "0  17.544065      0.124013  \n",
       "1  16.565256      0.051986  \n",
       "2  17.764069      0.152736  \n",
       "3  15.807901      0.025575  \n",
       "4        NaN           NaN  \n",
       "\n",
       "[5 rows x 45 columns]"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ID_column_uk = 'SOURCEID'\n",
    "df_uk[ID_column_uk] = df_uk[ID_column_uk].astype(int).astype(str)\n",
    "\n",
    "df_uk.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot table distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from bokeh.io import output_notebook\n",
    "# output_notebook()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from booq.plot import PlotHisto\n",
    "# p = PlotHisto()\n",
    "# for col in ['YAPERMAG3','J_1APERMAG3','HAPERMAG3','KAPERMAG3']:\n",
    "#     ds_label = p.add_dataset(df_uk,label=col,x=col)\n",
    "#     p.plot(ds_label)\n",
    "# p.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from booq.plot import PlotHisto\n",
    "# p = PlotHisto()\n",
    "# for col in ['YAPERMAG3','J_1APERMAG3','HAPERMAG3','KAPERMAG3']:\n",
    "#     ds_label = p.add_dataset(df_uk_sample,label=col,x=col)\n",
    "#     p.plot(ds_label)\n",
    "# p.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plot sky distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from booq.plot import PlotScatter\n",
    "# scatter = PlotScatter()\n",
    "\n",
    "# # uk_label = scatter.add_dataset(cat_uk,x='RA',y='DEC',label='scatter_uk')\n",
    "# # scatter.plot(uk_label)\n",
    "\n",
    "# uks_label = scatter.add_dataset(df_uk,x='RA',y='DEC',label='scatter_uk_sample')\n",
    "# scatter.plot(uks_label)\n",
    "\n",
    "# lm_label = scatter.add_dataset(df_lm,x='RAdeg',y='DEdeg',label='scatter_lm')\n",
    "# scatter.plot(lm_label)\n",
    "\n",
    "# scatter.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Radii definition"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from sklearn.neighbors import NearestNeighbors\n",
    "\n",
    "X = df_uk[['RA','DEC']].values\n",
    "nbrs = NearestNeighbors(n_neighbors=2, algorithm='ball_tree').fit(X)\n",
    "distances_uk, indices_uk = nbrs.kneighbors(X)\n",
    "distances_uk = distances_uk.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X = df_lm[['RAdeg','DEdeg']].values\n",
    "nbrs = NearestNeighbors(n_neighbors=2, algorithm='ball_tree').fit(X)\n",
    "distances_lm, indices_lm = nbrs.kneighbors(X)\n",
    "distances_lm = distances_lm.transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# from booq.plot import PlotHisto\n",
    "# p = PlotHisto()\n",
    "# ds_label = p.add_dataset(distances_uk,label='1st-neighbour ukidss',x=1)\n",
    "# p.plot(ds_label)\n",
    "# ds_label = p.add_dataset(distances_lm,label='1st-neighbour lamassa',x=1)\n",
    "# p.plot(ds_label)\n",
    "# p.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "##### Nearest neighbours\n",
    "\n",
    "Let's take the nearest neighbours plot above as a guide to define the search radii when cross-matching. The typical distance among \"LaMassa\" (x-ray) sources -- *i.e.*, the peak of the distribution, is around 90 arcsec; beggining at $\\approx 30\"$. On \"UKIDSS\", if we consider the first peak, it is around $7.2\"$.\n",
    "\n",
    "We may then take $r_s = 7 arcsec$ as the search radius. If we sextuple this radius, the corresponding circle area would be 36 times bigger; we may consider such radius as the background area outer radius. By defining the inner and outer radii as $r_i = 7\"$ and $r_o = 42 arcsec$ the annular region is 35 times the ancillary area, which should give us a good background sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from astropy.units import arcsec\n",
    "\n",
    "rs = 7 * arcsec\n",
    "ri = rs\n",
    "ro = 6 * ri"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Cross-Match\n",
    "\n",
    "Defined the corresponding radii for ancillary and background samples search, we proceed now with the corresponding cross-matchings as the first step of samples definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "from booq.pipelines import mle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "cols_A = {'ra':'RAdeg',\n",
    "          'dec':'DEdeg',\n",
    "          'id':'Seq',\n",
    "          'pos_err':'e_Pos'}\n",
    "\n",
    "cols = list(cols_A.values())\n",
    "\n",
    "cat_A = df_lm[cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "magnitude_columns = 'YAPERMAG3 J_1APERMAG3 HAPERMAG3 KAPERMAG3'.split()\n",
    "\n",
    "feature_column = magnitude_columns[0]\n",
    "\n",
    "cols_B = {'ra':'RA',\n",
    "          'dec':'DEC',\n",
    "          'id':'SOURCEID'}\n",
    "\n",
    "cols = list(cols_B.values())\n",
    "\n",
    "cols.append(feature_column)\n",
    "\n",
    "cat_B = df_uk[cols]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from importlib import reload\n",
    "reload(mle)\n",
    "\n",
    "mle_match = mle.mle(cat_A, cat_B,\n",
    "                    cols_A, cols_B,\n",
    "                    feature_column,\n",
    "                    ancillary_radius=rs,\n",
    "                    background_radii=[ri,ro],\n",
    "                    inner_join=False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2862\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style>\n",
       "    .dataframe thead tr:only-child th {\n",
       "        text-align: right;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: left;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th colspan=\"4\" halign=\"left\">A</th>\n",
       "      <th colspan=\"4\" halign=\"left\">B</th>\n",
       "      <th colspan=\"5\" halign=\"left\">AB_YAPERMAG3</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>RAdeg</th>\n",
       "      <th>DEdeg</th>\n",
       "      <th>Seq</th>\n",
       "      <th>e_Pos</th>\n",
       "      <th>SOURCEID</th>\n",
       "      <th>RA</th>\n",
       "      <th>DEC</th>\n",
       "      <th>YAPERMAG3</th>\n",
       "      <th>Reliability</th>\n",
       "      <th>LR</th>\n",
       "      <th>duplicates</th>\n",
       "      <th>duplicates_LR</th>\n",
       "      <th>duplicates_R</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2359</th>\n",
       "      <td>14.097</td>\n",
       "      <td>0.166</td>\n",
       "      <td>2359</td>\n",
       "      <td>4.7</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2360</th>\n",
       "      <td>14.115</td>\n",
       "      <td>-0.353</td>\n",
       "      <td>2360</td>\n",
       "      <td>3.5</td>\n",
       "      <td>433834423123</td>\n",
       "      <td>14.114893</td>\n",
       "      <td>-0.352297</td>\n",
       "      <td>17.924706</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.073829</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2361</th>\n",
       "      <td>14.115</td>\n",
       "      <td>-0.160</td>\n",
       "      <td>2361</td>\n",
       "      <td>5.8</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2362</th>\n",
       "      <td>14.142</td>\n",
       "      <td>-0.442</td>\n",
       "      <td>2362</td>\n",
       "      <td>3.4</td>\n",
       "      <td>433836362467</td>\n",
       "      <td>14.141094</td>\n",
       "      <td>-0.440343</td>\n",
       "      <td>18.888762</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.019703</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2363</th>\n",
       "      <td>14.154</td>\n",
       "      <td>-0.448</td>\n",
       "      <td>2363</td>\n",
       "      <td>2.1</td>\n",
       "      <td>433836362494</td>\n",
       "      <td>14.154556</td>\n",
       "      <td>-0.447174</td>\n",
       "      <td>18.222660</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.070513</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2364</th>\n",
       "      <td>14.162</td>\n",
       "      <td>-0.357</td>\n",
       "      <td>2364</td>\n",
       "      <td>5.2</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2365</th>\n",
       "      <td>14.162</td>\n",
       "      <td>0.038</td>\n",
       "      <td>2365</td>\n",
       "      <td>4.0</td>\n",
       "      <td>433832563084</td>\n",
       "      <td>14.162007</td>\n",
       "      <td>0.039508</td>\n",
       "      <td>18.878496</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.033257</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2366</th>\n",
       "      <td>14.163</td>\n",
       "      <td>-0.400</td>\n",
       "      <td>2366</td>\n",
       "      <td>3.8</td>\n",
       "      <td>433836363120</td>\n",
       "      <td>14.164038</td>\n",
       "      <td>-0.399393</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2367</th>\n",
       "      <td>14.170</td>\n",
       "      <td>-0.244</td>\n",
       "      <td>2367</td>\n",
       "      <td>2.7</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2368</th>\n",
       "      <td>14.172</td>\n",
       "      <td>-0.400</td>\n",
       "      <td>2368</td>\n",
       "      <td>4.3</td>\n",
       "      <td>433836362527</td>\n",
       "      <td>14.171665</td>\n",
       "      <td>-0.398114</td>\n",
       "      <td>20.279314</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.012236</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2369</th>\n",
       "      <td>14.186</td>\n",
       "      <td>-0.389</td>\n",
       "      <td>2369</td>\n",
       "      <td>4.4</td>\n",
       "      <td>433836363145</td>\n",
       "      <td>14.185372</td>\n",
       "      <td>-0.389149</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2370</th>\n",
       "      <td>14.196</td>\n",
       "      <td>-0.166</td>\n",
       "      <td>2370</td>\n",
       "      <td>3.6</td>\n",
       "      <td>433834437127</td>\n",
       "      <td>14.195810</td>\n",
       "      <td>-0.164875</td>\n",
       "      <td>20.141270</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.036455</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2371</th>\n",
       "      <td>14.197</td>\n",
       "      <td>-0.577</td>\n",
       "      <td>2371</td>\n",
       "      <td>3.4</td>\n",
       "      <td>433836363155</td>\n",
       "      <td>14.196392</td>\n",
       "      <td>-0.576760</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2372</th>\n",
       "      <td>14.207</td>\n",
       "      <td>0.456</td>\n",
       "      <td>2372</td>\n",
       "      <td>4.7</td>\n",
       "      <td>433829058905</td>\n",
       "      <td>14.206796</td>\n",
       "      <td>0.455432</td>\n",
       "      <td>20.627474</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.023225</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2373</th>\n",
       "      <td>14.215</td>\n",
       "      <td>-0.012</td>\n",
       "      <td>2373</td>\n",
       "      <td>1.3</td>\n",
       "      <td>433832563245</td>\n",
       "      <td>14.215556</td>\n",
       "      <td>-0.011784</td>\n",
       "      <td>18.467182</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.078296</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2374</th>\n",
       "      <td>14.217</td>\n",
       "      <td>0.068</td>\n",
       "      <td>2374</td>\n",
       "      <td>3.4</td>\n",
       "      <td>433832563006</td>\n",
       "      <td>14.217423</td>\n",
       "      <td>0.068694</td>\n",
       "      <td>20.095844</td>\n",
       "      <td>0.586907</td>\n",
       "      <td>0.047625</td>\n",
       "      <td>433832563005</td>\n",
       "      <td>0.0335210589771</td>\n",
       "      <td>0.413093071155</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2375</th>\n",
       "      <td>14.231</td>\n",
       "      <td>-0.452</td>\n",
       "      <td>2375</td>\n",
       "      <td>3.1</td>\n",
       "      <td>433836363033</td>\n",
       "      <td>14.231218</td>\n",
       "      <td>-0.451879</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2376</th>\n",
       "      <td>14.234</td>\n",
       "      <td>-0.034</td>\n",
       "      <td>2376</td>\n",
       "      <td>2.9</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2377</th>\n",
       "      <td>14.239</td>\n",
       "      <td>0.147</td>\n",
       "      <td>2377</td>\n",
       "      <td>2.9</td>\n",
       "      <td>433830897005</td>\n",
       "      <td>14.239424</td>\n",
       "      <td>0.147334</td>\n",
       "      <td>20.111305</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.055311</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2378</th>\n",
       "      <td>14.239</td>\n",
       "      <td>0.004</td>\n",
       "      <td>2378</td>\n",
       "      <td>4.6</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "      <td>None</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           A                                B                                  \\\n",
       "       RAdeg  DEdeg   Seq e_Pos      SOURCEID         RA       DEC  YAPERMAG3   \n",
       "2359  14.097  0.166  2359   4.7          None        NaN       NaN        NaN   \n",
       "2360  14.115 -0.353  2360   3.5  433834423123  14.114893 -0.352297  17.924706   \n",
       "2361  14.115 -0.160  2361   5.8          None        NaN       NaN        NaN   \n",
       "2362  14.142 -0.442  2362   3.4  433836362467  14.141094 -0.440343  18.888762   \n",
       "2363  14.154 -0.448  2363   2.1  433836362494  14.154556 -0.447174  18.222660   \n",
       "2364  14.162 -0.357  2364   5.2          None        NaN       NaN        NaN   \n",
       "2365  14.162  0.038  2365   4.0  433832563084  14.162007  0.039508  18.878496   \n",
       "2366  14.163 -0.400  2366   3.8  433836363120  14.164038 -0.399393        NaN   \n",
       "2367  14.170 -0.244  2367   2.7          None        NaN       NaN        NaN   \n",
       "2368  14.172 -0.400  2368   4.3  433836362527  14.171665 -0.398114  20.279314   \n",
       "2369  14.186 -0.389  2369   4.4  433836363145  14.185372 -0.389149        NaN   \n",
       "2370  14.196 -0.166  2370   3.6  433834437127  14.195810 -0.164875  20.141270   \n",
       "2371  14.197 -0.577  2371   3.4  433836363155  14.196392 -0.576760        NaN   \n",
       "2372  14.207  0.456  2372   4.7  433829058905  14.206796  0.455432  20.627474   \n",
       "2373  14.215 -0.012  2373   1.3  433832563245  14.215556 -0.011784  18.467182   \n",
       "2374  14.217  0.068  2374   3.4  433832563006  14.217423  0.068694  20.095844   \n",
       "2375  14.231 -0.452  2375   3.1  433836363033  14.231218 -0.451879        NaN   \n",
       "2376  14.234 -0.034  2376   2.9          None        NaN       NaN        NaN   \n",
       "2377  14.239  0.147  2377   2.9  433830897005  14.239424  0.147334  20.111305   \n",
       "2378  14.239  0.004  2378   4.6          None        NaN       NaN        NaN   \n",
       "\n",
       "     AB_YAPERMAG3                                                           \n",
       "      Reliability        LR    duplicates    duplicates_LR    duplicates_R  \n",
       "2359          NaN       NaN          None             None            None  \n",
       "2360     1.000000  0.073829          None             None            None  \n",
       "2361          NaN       NaN          None             None            None  \n",
       "2362     1.000000  0.019703          None             None            None  \n",
       "2363     1.000000  0.070513          None             None            None  \n",
       "2364          NaN       NaN          None             None            None  \n",
       "2365     1.000000  0.033257          None             None            None  \n",
       "2366          NaN       NaN          None             None            None  \n",
       "2367          NaN       NaN          None             None            None  \n",
       "2368     1.000000  0.012236          None             None            None  \n",
       "2369          NaN       NaN          None             None            None  \n",
       "2370     1.000000  0.036455          None             None            None  \n",
       "2371          NaN       NaN          None             None            None  \n",
       "2372     1.000000  0.023225          None             None            None  \n",
       "2373     1.000000  0.078296          None             None            None  \n",
       "2374     0.586907  0.047625  433832563005  0.0335210589771  0.413093071155  \n",
       "2375          NaN       NaN          None             None            None  \n",
       "2376          NaN       NaN          None             None            None  \n",
       "2377     1.000000  0.055311          None             None            None  \n",
       "2378          NaN       NaN          None             None            None  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(len(mle_match))\n",
    "mle_match.head(20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# mle_match.to_csv('mle_match.csv')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}