Not encoding UTF-8 correctly #18

shtse8 · 2017-02-27T13:58:51Z

Code to reproduce:

use \Wa72\HtmlPageDom\HtmlPageCrawler;
$html = <<<EOF
<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
<body>
网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
</body>
</html>
EOF;
$document = new HtmlPageCrawler($html);
echo $document->saveHTML();

Result:

<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8"><title>&#32593;&#21451;&#32456;&#20110;&#32905;&#25628;&#20986;&#12300;&#33539;&#20912;&#20912;&#12301;&#23478;&#26063;&#29031;&#29255;&#65292;&#27809;&#24819;&#21040;&#30475;&#35265;&#22905;&#22902;&#22902;&#25165;&#21457;&#29616;&#12300;&#33539;&#20912;&#20912;&#26159;&#20840;&#23478;&#26368;&#38590;&#30475;&#30340;&#12301;&#65281;</title></head><body>
&#32593;&#21451;&#32456;&#20110;&#32905;&#25628;&#20986;&#12300;&#33539;&#20912;&#20912;&#12301;&#23478;&#26063;&#29031;&#29255;&#65292;&#27809;&#24819;&#21040;&#30475;&#35265;&#22905;&#22902;&#22902;&#25165;&#21457;&#29616;&#12300;&#33539;&#20912;&#20912;&#26159;&#20840;&#23478;&#26368;&#38590;&#30475;&#30340;&#12301;&#65281;
</body></html>

Expected Result:

<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
<body>
网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
</body></html>

It is a known bug of PHP DomDocument. Here is the reference:
http://stackoverflow.com/questions/8218230/php-domdocument-loadhtml-not-encoding-utf-8-correctly

The text was updated successfully, but these errors were encountered:

shtse8 · 2017-02-27T14:10:18Z

I have tried DomCrawler, but it is fine without any problem when encoding Utf-8.

Code:

use Symfony\Component\DomCrawler\Crawler;
$html = <<<EOF
<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#"><head><meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
<body>
网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
</body>
</html>
EOF;
$crawler = new Crawler($html);
echo $crawler->html();

Result:

<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
</head>
<body>
网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
</body>

And I find they have fixed this bug already.
symfony/dom-crawler#4

shtse8 · 2017-02-27T22:15:45Z

I am trying to fix this problem. #19

kukungkung · 2017-10-21T17:00:00Z

I think it can help you
curl_setopt($ch, CURLOPT_ENCODING, 'UTF-8');

yellow1912 · 2019-02-20T04:17:37Z

Any update on this issue, do we still have problem with UTF-8 because this is a huge problem if it does exist, most sites use UTF-8 anyhow.

havran · 2019-03-04T10:55:55Z

Same problem here, in v1.3.

glensc · 2019-03-04T11:13:49Z

I just decode entities after the save:

        $html = html_entity_decode($html, ENT_NOQUOTES, 'UTF-8');

glensc · 2019-03-04T11:17:49Z

seems the underlying problem is that symfony/dom-crawler switches to entities to avoid some other bugs:

https://github.com/symfony/dom-crawler/blob/v4.2.4/Crawler.php#L195

havran · 2019-03-04T11:52:16Z

I just decode entities after the save:
        $html = html_entity_decode($html, ENT_NOQUOTES, 'UTF-8');

I think this is not good idea - because this decode all entities from HTML (for example i have bigger document where can by for example used > or  ).

I found solution which work for me - in line:

htmlpagedom/src/HtmlPageCrawler.php

Line 887 in 563bc7a

return $this->getDOMDocument()->saveHTML();

i change from:

return $this->getDOMDocument()->saveHTML();

to:

return $this->getDOMDocument()->saveHTML($this->getDOMDocument()->documentElement);

based by https://stackoverflow.com/a/20675396

havran · 2019-03-04T12:05:38Z

But my solution for some reason remove DOCTYPE in this test script:

<?php

$html = <<<EOF
<!DOCTYPE html>
<html prefix="og: http://ogp.me/ns#">
  <head>
    <meta charset="UTF-8">
    <title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
  </head>
  <body>
    网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
  </body>
</html>
EOF;

use \Wa72\HtmlPageDom\HtmlPageCrawler;
$document = new HtmlPageCrawler($html);
echo "--- HtmlPageDom -----------------------------------------------------------" .PHP_EOL.PHP_EOL;
echo $document->saveHTML();
echo PHP_EOL;

use Symfony\Component\DomCrawler\Crawler;
$crawler = new Crawler($html);
echo "--- DomCrawler ------------------------------------------------------------" .PHP_EOL.PHP_EOL;
echo $crawler->html();

This is output:

vagrant@d8:/data/uniweb/uniweb-cms/cms3[feature/UCMS-313-content-migration-blogs *]$ drush @lv scr test.php
--- HtmlPageDom -----------------------------------------------------------

<html prefix="og: http://ogp.me/ns#">
<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
</head>
<body>
    网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
  </body>
</html>
--- DomCrawler ------------------------------------------------------------

<head>
<meta charset="UTF-8">
<title>网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！</title>
</head>
<body>
    网友终于肉搜出「范冰冰」家族照片，没想到看见她奶奶才发现「范冰冰是全家最难看的」！
  </body>

glensc · 2019-03-04T12:06:46Z

html_entity_decode is perfectly valid, if you wanted to have putput as & then it should be in source document as &amp;.

havran · 2019-03-09T16:29:20Z

I make small test script for compare various DOM parsers - https://github.com/havran/php-html-parsers-test

Old simplehtmldom seems still best :-).

shtse8 mentioned this issue Feb 27, 2017

Fix utf-8 encoding problem #19

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Not encoding UTF-8 correctly #18

Not encoding UTF-8 correctly #18

shtse8 commented Feb 27, 2017 •

edited

Loading

shtse8 commented Feb 27, 2017 •

edited

Loading

shtse8 commented Feb 27, 2017 •

edited

Loading

kukungkung commented Oct 21, 2017

yellow1912 commented Feb 20, 2019

havran commented Mar 4, 2019

glensc commented Mar 4, 2019

glensc commented Mar 4, 2019

havran commented Mar 4, 2019 •

edited

Loading

havran commented Mar 4, 2019 •

edited

Loading

glensc commented Mar 4, 2019

havran commented Mar 9, 2019

Not encoding UTF-8 correctly #18

Not encoding UTF-8 correctly #18

Comments

shtse8 commented Feb 27, 2017 • edited Loading

shtse8 commented Feb 27, 2017 • edited Loading

shtse8 commented Feb 27, 2017 • edited Loading

kukungkung commented Oct 21, 2017

yellow1912 commented Feb 20, 2019

havran commented Mar 4, 2019

glensc commented Mar 4, 2019

glensc commented Mar 4, 2019

havran commented Mar 4, 2019 • edited Loading

havran commented Mar 4, 2019 • edited Loading

glensc commented Mar 4, 2019

havran commented Mar 9, 2019

shtse8 commented Feb 27, 2017 •

edited

Loading

shtse8 commented Feb 27, 2017 •

edited

Loading

shtse8 commented Feb 27, 2017 •

edited

Loading

havran commented Mar 4, 2019 •

edited

Loading

havran commented Mar 4, 2019 •

edited

Loading