Production: reviews for -0 and -1 cause server error

[codersquid]

While testing reviews, I found that ranks of -0 and -1 cause a server error. log excerpts:

DEBUG POST /scipy2014/reviews/review/10/
/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/django/db/backends/mysql/base.py
:114: Warning: Incorrect string value: '\xE2\x88\x921' for column 'vote' at row 1
  return self.cursor.execute(query, args)

ERROR Internal Server Error: /scipy2014/reviews/review/10/
Traceback (most recent call last):
  File "/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/django/core/handlers/bas
e.py", line 111, in get_response
    response = callback(request, *callback_args, **callback_kwargs)
  File "/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/django/contrib/auth/deco
rators.py", line 20, in _wrapped_view
    return view_func(request, *args, **kwargs)
  File "/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/symposion/reviews/views.
py", line 190, in review_detail
    review.save()
  File "/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/symposion/reviews/models
.py", line 124, in save
    self.proposal.result.update_vote(self.vote)
  File "/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/symposion/reviews/models
.py", line 272, in update_vote

...

  File "/home/scipy/venvs/99ddd7a7d38fb1b6bd5b14dda291143329a3a87e/local/lib/python2.7/site-packages/MySQLdb/connections.py",
 line 36, in defaulterrorhandler
    raise errorclass, errorvalue
DatabaseError: (1690, "BIGINT UNSIGNED value is out of range in '(((3 * `scipy2014`.`reviews_proposalresult`.`plus_one`) + `s
cipy2014`.`reviews_proposalresult`.`plus_zero`) - (`scipy2014`.`reviews_proposalresult`.`minus_zero` + (3 * `scipy2014`.`revi
ews_proposalresult`.`minus_one`)))'")

[codersquid]

@katyhuff if this also happens in prod, I'll change the title to remove staging.

[katyhuff]

yep, this is happening in prod.

[jlaura]

Is this an issue with utf8 support in the backend on those columns? Can the values be remapped to some other int value to avoid this error? I would guess that the provided unicode is a mapping to a negative sign ('\xE2\x88\x921') which is causing the issue...maybe?

[codersquid]

I just checked and the vote column is of a latin1 encoding so perhaps that is the cause.

[codersquid]

(This also probably means we will break people's names)

[katyhuff]

Indeed, we have been, and @aterrel suggested we'd fix it before we announced speakers (see email below from @certik .

So, you can fix two birds with this one UTF8 stone!

Goooooooooo Sheila!!!!!

On Mon, Mar 31, 2014 at 10:33 AM, Ondřej Čertík [email protected] wrote:
Hi,

I put in my name as Ondřej Čertík, but the website shows my name as:

Ond?ej ?ertík [email protected]

Looks like some unicode errors. So for now I fixed it by just using
ascii as Ondrej Certik. It's not a big deal, but I figured to report
the bug. These days I just use my correct name online and it works
pretty much everywhere.

Ondřej

---

Scipy-organizers mailing list
[email protected]
http://mail.scipy.org/mailman/listinfo/scipy-organizers

[codersquid]

argargarg I haven't fixed it yet because I am at pycon and did everything yesterday. Perhaps also today too! Hoping today!

[kthyng]

A fix would be nice for tutorial reviews but not mandatory. We have a small
number of people involved, so it makes that less critical for us.

On Sun, Apr 13, 2014 at 6:53 AM, Sheila Miguez [email protected]:

argargarg I haven't fixed it yet because I am at pycon and did everything
yesterday. Perhaps also today too! Hoping today!

Reply to this email directly or view it on GitHubhttps://github.com//issues/78#issuecomment-40305737
.

Kristen M. Thyng
Postdoctoral Research Associate
Department of Oceanography
Texas A&M University
http://kristenthyng.com

[polarbeardesign]

I can take a look - but don't see the database - I guess it's using different credentials than the ones I am using...

Jim

On Apr 13, 2014, at 6:53 AM, Sheila Miguez wrote:

argargarg I haven't fixed it yet because I am at pycon and did everything yesterday. Perhaps also today too! Hoping today!

—
Reply to this email directly or view it on GitHub.

[codersquid]

Hi all, I emailed more of a braindump, but here is a shorter recap:

Before doing anything in prod I wanted to check locally, and did a mysqldump. Well, this was silly of me since mysqldump does not dump exactly to a file with the encoding I would want. Now I would like to have a way to get this without first serializing to a text file so that I can try the alter commands with some verisimilitude before wacking the prod data willy-nilly.

I've passed along the key to Jim!

I hope we get this done soon!

The beast of character encoding is not yet slain.

[codersquid]

I went ahead and posted to the symposion mailing list to ask for more help. https://groups.google.com/forum/#!topic/pinax-symposion/OJxw2vkzzX0

And while writing up a description I realized I had not explicitly stated here that votes are varchar(2) and not int. So take that in to consideration while reviewing this bug.

[asl2]

The traceback on https://groups.google.com/forum/#!topic/pinax-symposion/OJxw2vkzzX0 ('BIGINT UNSIGNED value is out of range') looks different from the one which started this thread ('Incorrect string value:'). But I concur with jlaura: '\xE2\x88\x921' is the utf-8 encoding of '−1', which doesn't fit in 2 latin-1 characters. '−1' is u'\u2212' which doesn't seem to be transcodable losslessly to latin-1: would it be possible to work around in the short term by doing .replace('−', '-') when updating, or using '-' more globally?

[katyhuff]

Hi folks. I'm getting an email about this about once a day. Just wanted to bump it up. If it can't be fixed, maybe it can be removed?