电脑/python/网络抓取/js相关内容抓取问题.html

<!DOCTYPE html>
<html>
<head>
<!-- 2016-07-30 六 23:04 -->
<meta  charset="utf-8">
<meta  name="viewport" content="width=device-width, initial-scale=1">
<title>js相关内容抓取问题</title>
<meta  name="generator" content="Org-mode">
<meta  name="author" content="万泽(德山书生)">
<meta  name="description" content="制作者邮箱：a358003542@gmail.com"
>
<style type="text/css">
body {
    margin-right: auto;
    margin-left: auto;
}

.title {
    display: block;
    text-align: center;
    padding: 10px;
}

.center-block {
    display: block;
    margin-left: auto;
    margin-right: auto;
}

.underline{
    text-decoration: underline;
}
   html, body {
    margin: 0;
    padding: 0;
    font-family:"Microsoft YaHei",Arial,Helvetica,sans-serif,"宋体";
    font-size: 1em;
}

#content {
    display: block;
    margin-left: auto;
    margin-right: auto;
    background:#fcfcfc;
    height:100%;
    margin-left:300px;
    max-width:800px;
    min-height:100%;
    padding:1.618em 3.236em;
}

code {
    padding: 2px 4px;
    color: #c7254e;
    background-color: #f9f2f4;
    border-radius: 4px;
}

pre {
    max-width: 100%;
    display: block;
    padding: 9.5px;
    margin: 0 0 10px;
    font-size: 13px;
    line-height: 1.42857143;
    color: #333;
    background-color: #f5f5f5;
    border: 1px solid #ccc;
    border-radius: 4px;
    overflow : auto;
}


blockquote {
    padding: 10px 20px;
    margin: 0 0 20px;
    font-size: 17.5px;
    border-left: 5px solid #eee;
}

li{
    margin-top: 0.5em;
    margin-bottom: 0.5em;
}

p{
    text-indent:2em;
    line-height:150%;
}

/* dl */
dl {
    margin-top: 0;
    margin-bottom: 20px;
}
dt {
    font-weight: 700;
    line-height: 150%;
    margin-top: 0.5em;
    margin-bottom: 0.5em;
}
dd {
    line-height: 150%;
}

@media (min-width: 768px) {
  dt {
    white-space: nowrap;
  }
}

video{
    max-width: 100%;
    margin-left: auto;
    margin-right: auto;
}

figure p{
    text-indent:0em;
}
img{
    max-width: 100%;
}

embed{
    max-width: 100%;
    margin-left: auto;
    margin-right: auto;
}

figure{
    text-align: center;
}

/*  class  */
.FRAMED{
    max-width:100%;
    border:1px solid ;
    padding: 1em;
    -webkit-box-sizing: border-box;
    -moz-box-sizing: border-box;
    box-sizing: border-box;
}

.NOTECARD{
    width: 35%;
    position:relative;
    right: -30%;
    padding: 1em;
    margin:0 auto;
    border: solid 1px;
    -webkit-box-sizing: border-box;
    -moz-box-sizing: border-box;
    box-sizing: border-box;
}

/*
table
*/
table {
    border-collapse: collapse;
    border-spacing: 0;
    margin: 16px 0;
    empty-cells: show;
    border: 1px solid #ccc;
    width: 100%;
    display: table;
}

table tr {
    border-bottom: 1px solid #ddd;
}
table th,table td{
    padding:8px;
}

table tr:nth-child(even) {
    background-color: #f1f1f1;
}


ul,ol,dl{
    line-height:24px;
    list-style-image:none;
    /* list-style:none; */
    margin:0px 0px 24px 0px;
    padding:0;
}

li{
    margin-left: 24px;
}

dd{
    margin:0;
}

#content .section ul,#content .toctree-wrapper ul,article ul{
    list-style:disc;
    line-height:24px;
    margin-bottom:24px}

#content .section ul li,#content .toctree-wrapper ul li,article ul li{
    list-style:disc;
    margin-left:24px}

#content .section ul li p:last-child,#content .toctree-wrapper ul li p:last-child,article ul li p:last-child{
                                                                                                                                  margin-bottom:0}

#content .section ul li ul,#content .toctree-wrapper ul li ul,article ul li ul{
    margin-bottom:0}

#content .section ul li li,#content .toctree-wrapper ul li li,article ul li li{
    list-style:circle}

#content .section ul li li li,#content .toctree-wrapper ul li li li,article ul li li li{
    list-style:square}

#content .section ul li ol li,#content .toctree-wrapper ul li ol li,article ul li ol li{
    list-style:decimal}

#content .section ol,#content ol,article ol{
    list-style:decimal;
    line-height:24px;
    margin-bottom:24px}

#content .section ol li,#content ol li,article ol li{
    list-style:decimal;
    margin-left:24px}

#content .section ol li p:last-child,#content ol li p:last-child,article ol li p:last-child{
                                                                                                                           margin-bottom:0}

#content .section ol li ul,#content ol li ul,article ol li ul{
    margin-bottom:0}

#content .section ol li ul li,#content ol li ul li,article ol li ul li{
    list-style:disc}

dl dt{
    font-weight:bold;
}

dl p,dl table,dl ul,dl ol{
    margin-bottom:12px !important;
}

dl dd{
    margin:0 0 12px 24px;
}

@media print{
    .codeblock,pre.src{
        white-space:pre.src-wrap}
}

@media print{
    html,body,section{
        background:none !important}

    *{
        box-shadow:none !important;
        text-shadow:none !important;
        filter:none !important;
        -ms-filter:none !important}

    a,a:visited{
          text-decoration:underline}

    pre.src,blockquote{
        page-break-inside:avoid}

    thead{
        display:table-header-group}

    tr,img{
        page-break-inside:avoid}

    img{
        max-width:100% !important}

    @page{
        margin:0.5cm}

    p,h2,h3{
        orphans:3;
        widows:3}

    h2,h3{
        page-break-after:avoid}
}

@media print{
    #postamble{
        display:none}

    #content{
        margin-left:0}
}

@media print{
    #table-of-contents{
        display:none}

    @page{
        size: auto;
        margin: 25mm 25mm 25mm 25mm;}

    body {
        margin: 0px;}
}

@media screen and (max-width: 768px){
}

@media only screen and (max-width: 480px){
}

@media screen and (max-width: 768px){
    .tablet-hide{
        display:none}
}

@media screen and (max-width: 480px){
    .mobile-hide{
        display:none}
}

@media screen and (max-width: 480px){
}

@media screen and (max-width: 768px){
    #content{
        margin-left:0;
        padding: 1em 1em;}

    #content #content{
        padding:1.618em}

    #content.shift{
        position:fixed;
        min-width:100%;
        left:85%;
        top:0;
        height:100%;
        overflow:hidden}
}

@media screen and (min-width: 1400px){
    #content{
        background:rgba(0,0,0,0.05)}

    #content{
        background:#fcfcfc}
}

@media screen and (max-width: 768px){
    #copyright{
        width:85%;
        display:none}

    #copyright.shift{
        display:block}

    img{
        width:100%;
        height:auto}
}

@media screen and (max-width: 480px){
    #content .sidebar{
        width:100%}
}


.figure p{
    color:#000;
    font:italic 85%/1 arial,sans-serif;
    padding:1em 0;
}

.rotate-90{
    -webkit-transform:rotate(90deg);
    -moz-transform:rotate(90deg);
    -ms-transform:rotate(90deg);
    -o-transform:rotate(90deg);
    transform:rotate(90deg);
}

.rotate-270{
    -webkit-transform:rotate(270deg);
    -moz-transform:rotate(270deg);
    -ms-transform:rotate(270deg);
    -o-transform:rotate(270deg);
    transform:rotate(270deg);
}

#toggle-sidebar,
#table-of-contents .close-sidebar {
    display: none;
}

@media screen and (max-width: 768px) {
    #table-of-contents {
        display: none;
        width: 60%;
    }

    #table-of-contents h2 a {
        display: block;
    }

    #table-of-contents:target {
        display: block;
    }

    #copyright, #postamble {
        display: none;
    }

    #toggle-sidebar {
        background-color: #2980B9;
        display: block;
        margin-bottom: 1.6em;
        padding: 0.6em;
        text-align: center;
    }

    #toggle-sidebar h2 {
        color: white;
        font-size: 100%;
        line-height: 50px;
        margin: 0;
        padding: 0;
    }

    #table-of-contents .close-sidebar {
       color: rgba(255, 255, 255, 0.3);
       display: inline-block;
       margin: 0px 10px 0px 45px;
       padding: 10px;
    }
}

*{
    -webkit-box-sizing:border-box;
    -moz-box-sizing:border-box;
    box-sizing:border-box;
}

figcaption,figure,footer,header,hgroup,nav{
    display:block}

ins{
    background:#ff9;
    color:#000;
    text-decoration:none}

mark{
    background:#ff0;
    color:#000;
    font-style:italic;
    font-weight:bold}

small{
    font-size:85%}

sub,sup{
    font-size:75%;
    line-height:0;
    position:relative;
    vertical-align:baseline}

sup{
    top:-0.5em}

sub{
    bottom:-0.25em}

img{
    -ms-interpolation-mode:bicubic;
    vertical-align:middle;
    max-width:100%}

svg:not(:root){
    overflow:hidden}

figure{
    margin:0}

label{
    cursor:pointer}

legend{
    border:0;
    margin-left:-7px;
    padding:0;
    white-space:normal}

#content .danger,#content .error{
    background:#fdf3f2}

legend{
    display:block;
    width:100%;
    border:0;
    padding:0;
    white-space:normal;
    margin-bottom:24px;
    font-size:150%;
    *margin-left:-7px}

label{
    display:block;
    margin:0 0 0.3125em 0;
    color:#333;
    font-size:90%}

a{
    color:#2980B9;
    text-decoration:none;
    cursor:pointer}


a:hover,a:active{
    outline:0;
}

a:hover{
    color:#3091d1}

a:visited{
    color:#9B59B6}

.left{
    text-align:left}

.center{
    text-align:center}

.right{
    text-align:right}

hr{
    display:block;
    height:1px;
    border:0;
    border-top:1px solid #e1e4e5;
    margin:24px 0;
    padding:0}

#table-of-contents li{
    list-style:none;
    margin-left: 0px;
}

#table-of-contents header{
    height:32px;
    display:inline-block;
    line-height:32px;
    padding:0 1.618em;
    display:block;
    font-weight:bold;
    text-transform:uppercase;
    font-size:80%;
    color:#2980B9;
    white-space:nowrap}

#table-of-contents ul{
    margin-bottom:0}

#table-of-contents li.divide-top{
    border-top:solid 1px #404040}

#table-of-contents li.divide-bottom{
    border-bottom:solid 1px #404040}

#table-of-contents li.current{
    background:#e3e3e3}

#table-of-contents li.current a{
    color:gray;
    border-right:solid 1px #c9c9c9;
    padding:0.4045em 2.427em}

#table-of-contents li.current a:hover{
    background:#d6d6d6}

#table-of-contents li a{
    /* color:#404040; */
    padding:0.4045em 1.618em;
    position:relative;
    /* background:#fcfcfc; */
    border:none;
    /* border-bottom:solid 1px #c9c9c9; */
    /* border-top:solid 1px #c9c9c9; */
    padding-left:1.618em -4px}

#table-of-contents li.on a:hover,#table-of-contents li.current>a:hover{
    background:#fcfcfc}

#table-of-contents li ul li a{
    /* background:#c9c9c9; */
    padding:0.4045em 2.427em}

#table-of-contents li ul li ul li a{
    padding:0.4045em 3.236em}

#table-of-contents li.current ul{
    display:block}

/* #table-of-contents li ul{ */
/*     margin-bottom:0; */
/*     display:none} */

#table-of-contents .local-toc li ul{
    display:block}

#table-of-contents li ul li a{
    margin-bottom:0;
    color:#b3b3b3;
    font-weight:normal}

#table-of-contents a{
    display:inline-block;
    line-height:18px;
    padding:0.4045em 1.618em;
    display:block;
    position:relative;
    font-size:90%;
    color:#b3b3b3;
    direction: ltr;
}

#table-of-contents a:hover{
    background-color:#4e4a4a;
    cursor:pointer}

/* #text-table-of-contents { */
/*     overflow:scroll; */
/* } */

#table-of-contents{
    position:fixed;
    top:0;
    left:0;
    width:300px;
    overflow-x:hidden;
    overflow-y:scroll;
    height:100%;
    background:#343131;
    z-index:200;
    scrollbar-base-color: #1F1D1D;
    scrollbar-arrow-color: #b3b3b3;
    scrollbar-shadow-color: #1F1D1D;
    scrollbar-track-color : #343131;
}

#table-of-contents h2{
    z-index:200;
    background-color:#2980B9;
    text-align:center;
    padding:0.809em;
    display:block;
    color:#fcfcfc;
    font-size: 100%;
    margin-bottom:0.809em}

ul.nav li ul li {
    display: none;
}

ul.nav li ul li ul li {
    display: none;
}

ul.nav li.active ul li {
    display: inline;
}

ul.nav li.active ul li ul li {
    display: inline;
}

ul.nav li.active ul li a {
    background-color: #E3E3E3;
    color: #8099B0;
    border-right:solid 1px #c9c9c9 !important;
}

ul.nav li.active ul li.active a {
    background-color: #C9C9C9;
    color: black !important;
    font-weight: bold !important;
}

ul.nav li.active ul li.active ul li.active a {
    color: black !important;
    font-weight: bold !important;
    display: block !important;
}

ul.nav li.active ul li.active ul li a {
    color: #808080 !important;
    font-weight: normal !important;
    display: block !important;
}

ul.nav li.active ul li ul li a {
    display: none !important;
}


ul.nav li ul li ul li ul li {
    display: none !important; /* as long as nav is on multiple levels of ul */
    /* display: none; /* as long as nav is on multiple levels of ul *\/ */
}

ul.nav li.active > a {
    border-bottom:solid 1px #c9c9c9 !important; /* XXX Restrict it to 2nd level */
    border-right:solid 1px #c9c9c9 !important;
}

ul.nav li.active a {
    color: gray !important;
    font-weight:bold;
    background-color: white;
    border-right:solid 0px white !important;
}

ul.nav > li.active > a {
    color: black !important;
}

footer{
    color:#999}

footer p{
    margin-bottom:12px}

#copyright, #postamble{
    position:fixed;
    bottom:0;
    left:0;
    width:300px;
    color:#fcfcfc;
    background:#1f1d1d;
    border-top:solid 10px #343131;
    font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;
    font-size: 90%;
    z-index:400;
    padding:12px;
}

#postamble .author {
    font-size: 100%;
    margin-bottom: 0px;
}

#postamble .date {
    font-size: 90%;
    margin-bottom: 0px;
    color: #27AE60;
}

#postamble .creator,#postamble .validation {
    display:none;
}

#copyright a{
    color:#2980B9;
    text-decoration:none}

#copyright .rst-current-version{
    padding:12px;
    background-color:#272525;
    display:block;
    text-align:right;
    font-size:90%;
    cursor:pointer;
    color:#27AE60;
    *zoom:1}

#content img{
    max-width:100%;
    height:auto !important}

#content div.figure{
    margin-bottom:24px}

#content div.figure.align-center{
    text-align:center}

#content .section>img,#content .section>a>img{
    margin-bottom:24px}

.verse{
    border-left:5px solid #6AB0DE;
    background-color: #E7F2FA;
    padding: 6px 20px;
    font-style:italic;
}

#content .note .last,#content .attention .last,#content .caution .last,#content .danger .last,#content .error .last,#content .hint .last,#content .important .last,#content .tip .last,#content .warning .last,#content .seealso .last,#content .admonitiontodo .last{
    margin-bottom:0}

#content .admonition-title:before{
    margin-right:4px}

#content .section ol p,#content .section ul p{
    margin-bottom:12px}

#content h1 .headerlink,#content h2 .headerlink,#content h3 .headerlink,#content h4 .headerlink,#content h5 .headerlink,#content h6 .headerlink,#content dl dt .headerlink{
    display:none;
    visibility:hidden;
    font-size:14px}

#content h1 .headerlink:after,#content h2 .headerlink:after,#content h3 .headerlink:after,#content h4 .headerlink:after,#content h5 .headerlink:after,#content h6 .headerlink:after,#content dl dt .headerlink:after{
    visibility:visible;
    content:"";
    font-family:FontAwesome;
    display:inline-block}

#content h1:hover .headerlink,#content h2:hover .headerlink,#content h3:hover .headerlink,#content h4:hover .headerlink,#content h5:hover .headerlink,#content h6:hover .headerlink,#content dl dt:hover .headerlink{
    display:inline-block}

#content .sidebar{
    float:right;
    width:40%;
    display:block;
    margin:0 0 24px 24px;
    padding:24px;
    background:#f3f6f6;
    border:solid 1px #e1e4e5}

#content .sidebar p,#content .sidebar ul,#content .sidebar dl{
    font-size:90%}

#content .sidebar .last{
    margin-bottom:0}

#content .sidebar .sidebar-title{
    display:block;
    font-family:"Roboto Slab","ff-tisa-web-pro","Georgia",Arial,sans-serif;
    font-weight:bold;
    background:#e1e4e5;
    padding:6px 12px;
    margin:-24px;
    margin-bottom:24px;
    font-size:100%}

#content .footnote-reference,#content .citation-reference{
    vertical-align:super;
    font-size:90%}

span[id*='MathJax-Span']{
    color:#404040}

.math{
    text-align:center}

#footnotes{
    border-top:1px solid #e1e4e5;
    padding-top: 36px;
}

h2.footnotes{
    display:none;
}


.footnum:before, .footref:before{
    content:"[";
}

.footnum:after, .footref:after{
    content:"]";
}

.footpara {
    color: #999;
    font-size: 90%;
    font-family:"Lato","proxima-nova","Helvetica Neue",Arial,sans-serif;
    padding-bottom: 8px;
    padding-left: 16px;
    padding-right: 16px;
    padding-top: 8px;
    line-height: 1.25em;
    /* display: inline; */
}
.highlight .hll { background-color: #ffffcc }
.highlight  { background: #f8f8f8; }
.highlight .c { color: #8f5902; font-style: italic } /* Comment */
.highlight .err { color: #a40000; border: 1px solid #ef2929 } /* Error */
.highlight .g { color: #000000 } /* Generic */
.highlight .k { color: #204a87; font-weight: bold } /* Keyword */
.highlight .l { color: #000000 } /* Literal */
.highlight .n { color: #000000 } /* Name */
.highlight .o { color: #ce5c00; font-weight: bold } /* Operator */
.highlight .x { color: #000000 } /* Other */
.highlight .p { color: #000000; font-weight: bold } /* Punctuation */
.highlight .ch { color: #8f5902; font-style: italic } /* Comment.Hashbang */
.highlight .cm { color: #8f5902; font-style: italic } /* Comment.Multiline */
.highlight .cp { color: #8f5902; font-style: italic } /* Comment.Preproc */
.highlight .cpf { color: #8f5902; font-style: italic } /* Comment.PreprocFile */
.highlight .c1 { color: #8f5902; font-style: italic } /* Comment.Single */
.highlight .cs { color: #8f5902; font-style: italic } /* Comment.Special */
.highlight .gd { color: #a40000 } /* Generic.Deleted */
.highlight .ge { color: #000000; font-style: italic } /* Generic.Emph */
.highlight .gr { color: #ef2929 } /* Generic.Error */
.highlight .gh { color: #000080; font-weight: bold } /* Generic.Heading */
.highlight .gi { color: #00A000 } /* Generic.Inserted */
.highlight .go { color: #000000; font-style: italic } /* Generic.Output */
.highlight .gp { color: #8f5902 } /* Generic.Prompt */
.highlight .gs { color: #000000; font-weight: bold } /* Generic.Strong */
.highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
.highlight .gt { color: #a40000; font-weight: bold } /* Generic.Traceback */
.highlight .kc { color: #204a87; font-weight: bold } /* Keyword.Constant */
.highlight .kd { color: #204a87; font-weight: bold } /* Keyword.Declaration */
.highlight .kn { color: #204a87; font-weight: bold } /* Keyword.Namespace */
.highlight .kp { color: #204a87; font-weight: bold } /* Keyword.Pseudo */
.highlight .kr { color: #204a87; font-weight: bold } /* Keyword.Reserved */
.highlight .kt { color: #204a87; font-weight: bold } /* Keyword.Type */
.highlight .ld { color: #000000 } /* Literal.Date */
.highlight .m { color: #0000cf; font-weight: bold } /* Literal.Number */
.highlight .s { color: #4e9a06 } /* Literal.String */
.highlight .na { color: #c4a000 } /* Name.Attribute */
.highlight .nb { color: #204a87 } /* Name.Builtin */
.highlight .nc { color: #000000 } /* Name.Class */
.highlight .no { color: #000000 } /* Name.Constant */
.highlight .nd { color: #5c35cc; font-weight: bold } /* Name.Decorator */
.highlight .ni { color: #ce5c00 } /* Name.Entity */
.highlight .ne { color: #cc0000; font-weight: bold } /* Name.Exception */
.highlight .nf { color: #000000 } /* Name.Function */
.highlight .nl { color: #f57900 } /* Name.Label */
.highlight .nn { color: #000000 } /* Name.Namespace */
.highlight .nx { color: #000000 } /* Name.Other */
.highlight .py { color: #000000 } /* Name.Property */
.highlight .nt { color: #204a87; font-weight: bold } /* Name.Tag */
.highlight .nv { color: #000000 } /* Name.Variable */
.highlight .ow { color: #204a87; font-weight: bold } /* Operator.Word */
.highlight .w { color: #f8f8f8; text-decoration: underline } /* Text.Whitespace */
.highlight .mb { color: #0000cf; font-weight: bold } /* Literal.Number.Bin */
.highlight .mf { color: #0000cf; font-weight: bold } /* Literal.Number.Float */
.highlight .mh { color: #0000cf; font-weight: bold } /* Literal.Number.Hex */
.highlight .mi { color: #0000cf; font-weight: bold } /* Literal.Number.Integer */
.highlight .mo { color: #0000cf; font-weight: bold } /* Literal.Number.Oct */
.highlight .sb { color: #4e9a06 } /* Literal.String.Backtick */
.highlight .sc { color: #4e9a06 } /* Literal.String.Char */
.highlight .sd { color: #8f5902; font-style: italic } /* Literal.String.Doc */
.highlight .s2 { color: #4e9a06 } /* Literal.String.Double */
.highlight .se { color: #4e9a06 } /* Literal.String.Escape */
.highlight .sh { color: #4e9a06 } /* Literal.String.Heredoc */
.highlight .si { color: #4e9a06 } /* Literal.String.Interpol */
.highlight .sx { color: #4e9a06 } /* Literal.String.Other */
.highlight .sr { color: #4e9a06 } /* Literal.String.Regex */
.highlight .s1 { color: #4e9a06 } /* Literal.String.Single */
.highlight .ss { color: #4e9a06 } /* Literal.String.Symbol */
.highlight .bp { color: #3465a4 } /* Name.Builtin.Pseudo */
.highlight .vc { color: #000000 } /* Name.Variable.Class */
.highlight .vg { color: #000000 } /* Name.Variable.Global */
.highlight .vi { color: #000000 } /* Name.Variable.Instance */
.highlight .il { color: #0000cf; font-weight: bold } /* Literal.Number.Integer.Long */
</style>
</head>
<body>
<div id="content">
<header>
<h1 class="title">js相关内容抓取问题</h1>
</header><nav id="table-of-contents">
<h2>Table of Contents</h2>
<div id="text-table-of-contents">
<ul>
<li><a href="#orgheadline1">1. 到底该不该管js代码</a></li>
<li><a href="#orgheadline5">2. selenium模块</a>
<ul>
<li><a href="#orgheadline2">2.1. 安装</a></li>
<li><a href="#orgheadline3">2.2. 一个简单例子</a></li>
<li><a href="#orgheadline4">2.3. 选中某个标签元素</a></li>
</ul>
</li>
<li><a href="#orgheadline6">3. 后台虚拟窗口运行</a></li>
</ul>
</div>
</nav>


<div class="outline-2">
<h2 id="orgheadline1">到底该不该管js代码</h2>
<div class="outline-text-2" id="text-1">
<p>
把这个问题先提出来就是要提醒读者，只有确定你要抓取的内容很重要而且必须把js展开内容才能获取，那么才考虑展开js代码。这倒不是实现上有多困难，而是成本太昂贵了。如下面所述及的，我们需要建立一种虚拟后台显示窗口，然后在这个窗口内运行如FireFox这样的浏览器。读者可以看到这种展开工作成本是极其昂贵的。所以如果你的程序一旦决定具体展开js代码了，那么你要清楚决定你程序性能的短板就在这里了，很多常规的程序优化技术和手段可能反倒是杯水车薪了，而围绕着这个具体展开，尽量减少具体的展开工作量，就成了程序性能优化的关键了。
</p>
</div>
</div>


<div class="outline-2">
<h2 id="orgheadline5">selenium模块</h2>
<div class="outline-text-2" id="text-2">
<p>
selenium模块的官方文档在 <a href="http://selenium-python.readthedocs.org/index.html">这里</a> 。下面就模块的最核心内容做出一些说明。
</p>
</div>

<div class="outline-3">
<h3 id="orgheadline2">安装</h3>
<div class="outline-text-3" id="text-2-1">
<pre>
sudo pip3 install selenium
</pre>
</div>
</div>

<div class="outline-3">
<h3 id="orgheadline3">一个简单例子</h3>
<div class="outline-text-3" id="text-2-2">
<div class="highlight"><pre><span></span><span class="kn">from</span> <span class="nn">selenium</span> <span class="kn">import</span> <span class="n">webdriver</span>

<span class="n">browser</span> <span class="o">=</span> <span class="n">webdriver</span><span class="o">.</span><span class="n">Firefox</span><span class="p">()</span>
<span class="n">browser</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s1">&#39;http://www.163.com&#39;</span><span class="p">)</span>

<span class="n">html</span> <span class="o">=</span> <span class="n">browser</span><span class="o">.</span><span class="n">page_source</span>
<span class="k">print</span><span class="p">(</span><span class="n">html</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">browser</span><span class="o">.</span><span class="n">title</span><span class="p">)</span>
<span class="k">print</span><span class="p">(</span><span class="n">browser</span><span class="o">.</span><span class="n">current_url</span><span class="p">)</span>
<span class="n">browser</span><span class="o">.</span><span class="n">quit</span><span class="p">()</span>
</pre></div>

<p>
这里调用的是火狐后台 <code>webdriver.Firefox</code> ，你还可以调用其他后台，这里略过讨论了。
</p>

<p>
通过browser的 <code>get</code> 方法来在模拟浏览器中输入一个网址，然后通过 <code>page_source</code> 方法来获取目标网址源码， browser还有属性 <code>title</code> 当前网页标题； <code>current_url</code> 当前网址。然后通过 <code>quit</code> 方法关闭该浏览器。
</p>

<p>
一般情况下获取目标网页的html解析之后的源码就可以了，通过其他工具如beautifulsoup等来进一步提取信息。selenium的browser还有很多方法来获取标签对象，某一两个信息提取任务的时候可以用一下，其主要是用于选中某个标签元素，然后进行额外的模拟人手工的如点击输入文本等动作的。
</p>
</div>
</div>


<div class="outline-3">
<h3 id="orgheadline4">选中某个标签元素</h3>
<div class="outline-text-3" id="text-2-3">
<p>
前面提及的browser对象有很多选中某个或某些标签元素的方法:
</p>
<ul class="org-ul">
<li>find_element_by_id</li>
<li>find_element_by_name</li>
<li>find_element_by_xpath</li>
<li>find_element_by_link_text</li>
<li>find_element_by_partial_link_text</li>
<li>find_element_by_tag_name</li>
<li>find_element_by_class_name</li>
<li>find_element_by_css_selector</li>
</ul>

<p>
然后这些方法若element改为elements则会选中多个标签元素，如: <code>find_elements_by_id</code> 。
</p>

<p>
具体请参看官方文档，这里就不一一说明了。 <a href="https://saucelabs.com/resources/selenium/css-selectors">这篇网页</a> 介绍了Xpath和css选定的一些细节，介绍的很好。
</p>

<p>
这些函数返回的是标签元素对象e，你可以通过 <code>e.get_attribute('src')</code> 来获取该标签的某个属性的值。然后还有其他一些方法，这里先略过讨论了。正如前所述及的，selenium的重点是模拟浏览器的各个动作，而不是要做类似beautifulsoup这样的api工作。所以其提供了 <code>e.click()</code> 就模拟了鼠标点击在该标签元素上的行为；还有 <code>e.send_keys(Keys.RETURN)</code> 就模拟了在该标签元素上按下回车键的行为。等等这些才是selenium的关注重点，更多相关细节请参看官方文档的相关 Action Chains 部分。
</p>
</div>
</div>
</div>


<div class="outline-2">
<h2 id="orgheadline6">后台虚拟窗口运行</h2>
<div class="outline-text-2" id="text-3">
<p>
经过测试行为一切OK了，你就不希望再看到弹出的浏览器窗口了，你可以通过一种虚拟显示技术来做到。首先安装 <code>xvfb</code>:
</p>
<pre>
sudo apt-get install xvfb
</pre>

<p>
然后安装 <code>pyvirtualdisplay</code> 模块:
</p>

<pre>
sudo pip3 install pyvirtualdisplay
</pre>

<p>
这里参考了 <a href="http://coreygoldberg.blogspot.com/2011/06/python-headless-selenium-webdriver.html">这篇网页</a> 。大体例子如其所示:
</p>
<pre>
def wget_forcejs(url):
    from pyvirtualdisplay import Display
    from selenium import webdriver

    display = Display(visible=0, size=(800, 600))
    display.start()

    # now Firefox will run in a virtual display.
    # you will not see the browser.
    browser = webdriver.Firefox()
    browser.get(url)

    html = browser.page_source
    print(browser.title)

    browser.quit()
    display.stop()
    return html
</pre>
</div>
</div>
</div>
</body>
</html>