feature request: unicode in source #95

alashworth · 2019-03-12T09:36:31Z

Issue by tpapp
Monday May 09, 2016 at 17:29 GMT
Originally opened as https://github.com/stan-dev/stan/issues/1888

Introduction

Some languages now support Unicode (mostly UTF8) for writing source code. It would be great if one could also use Unicode in Stan source. (Note that comments in UTF8, or any superset that embeds ASCII, are already supported in the sense the parser just ignores them.)

Broadly, there are two possible levels of support:

in variable and function names (eg ϕ), and
in operators (eg ≤), which provide synonyms for existing ones (eg <=)

Example

This is how the 8 schools example would look like in unicode:

data {
  int<lower=0> J;             // number of schools
  real y[J];                  // estimated treatment effect (school j)
  real<lower=0> σ[J];         // std err of effect estimate (school j)
}
parameters {
  real μ;
  real θ[J];
  real<lower=0> τ;
}
model {
  θ ~ normal(μ, τ); 
  y ~ normal(θ, σ);
}

Possible benefits

more compact source code
better mapping to equations in papers

Possible downsides

editor/entry support
font support
possibly corrupted files

The first two are mitigated by the fact that ASCII is a subset of UTF8, so using the feature is optional.

UTF8 support in various languages which have interfaces for Stan

language	literals	identifiers	operators	would UTF8 variables work for interfacing with Stan?
R	yes	yes	no	yes
Python	yes	only from version 3	no	yes, even in Python 2, as they are used as literal keys
Julia	yes	yes	yes	yes
Matlab	yes	yes, but needs to be enabled	no	yes
Stata	yes	yes, from version 14	no	probably?

Editor support

Emacs

See this list for various UTF8 implementations using autocomplete, company-mode, and quail.

On May 11, 2016, at 1:24 PM, Allen Riddell [email protected] wrote:

Here's a link to the exact section: https://docs.python.org/2/howto/unicode.html#unicode-literals-in-python-source-code

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

alashworth · 2019-03-12T09:36:47Z

Comment by ariddell
Wednesday May 11, 2016 at 20:07 GMT

You're right. In Python 2.7 you can't have unicode variables. In Python 3 you can. But why does that matter? We only need unicode in the Stan program code. (Parameter lookups aren't affected since keys are and always were strings.)

alashworth · 2019-03-12T09:36:49Z

Comment by bob-carpenter
Wednesday May 11, 2016 at 20:48 GMT

It looks like you use a dictionary structure for variable
names.

schools_dat = {
'J': 8,
'y': [28, 8, -3, 7, -1, 1, 18, 12],
'sigma': [15, 10, 16, 11, 9, 11, 10, 18]
}

Can the keys be unicode?

RStan can read data values out of the environment if they're
named after variables in the Stan program. And it can attach
the resulting draws as variables in the environment.

Bob

On May 11, 2016, at 4:07 PM, Allen Riddell [email protected] wrote:

You're right. In Python 2.7 you can't have unicode variables. In Python 3 you can. But why does that matter? We only need unicode in the Stan program code. (Parameter lookups aren't affected since keys are and always were strings.)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

alashworth · 2019-03-12T09:36:51Z

Comment by ariddell
Thursday May 12, 2016 at 12:20 GMT

Python 2 has no problems with unicode dictionary keys. In fact, it can
have unicode variables in the environment but you have to reference them
via strings indirectly. For example, this works in Python 2:

>>> locals()[u'é'] = 9
>>> locals()[u'é']

(locals is something like baseenv or .GlobalEnv in R)

Bref, there is nothing Python 2 can't do that's relevant to supporting
unicode in Stan code. The table above is inaccurate.

On 05/11, Bob Carpenter wrote:

It looks like you use a dictionary structure for variable
names.

schools_dat = {
'J': 8,
'y': [28, 8, -3, 7, -1, 1, 18, 12],
'sigma': [15, 10, 16, 11, 9, 11, 10, 18]
}

Can the keys be unicode?

RStan can read data values out of the environment if they're
named after variables in the Stan program. And it can attach
the resulting draws as variables in the environment.

Bob

On May 11, 2016, at 4:07 PM, Allen Riddell [email protected] wrote:

You're right. In Python 2.7 you can't have unicode variables. In Python 3 you can. But why does that matter? We only need unicode in the Stan program code. (Parameter lookups aren't affected since keys are and always were strings.)

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub:
stan-dev/stanc3#1406

alashworth · 2019-03-12T09:36:53Z

Comment by bob-carpenter
Thursday May 12, 2016 at 15:33 GMT

On May 12, 2016, at 8:20 AM, Allen Riddell [email protected] wrote:

Python 2 has no problems with unicode dictionary keys. In fact, it can
have unicode variables in the environment but you have to reference them
via strings indirectly. For example, this works in Python 2:
>>> locals()[u'é'] = 9
>>> locals()[u'é']
(locals is something like baseenv or .GlobalEnv in R)

Bref, there is nothing Python 2 can't do that's relevant to supporting
unicode in Stan code. The table above is inaccurate.

You should have edit permission on the issues.

Bob

alashworth · 2019-03-12T09:36:55Z

Comment by ariddell
Thursday May 12, 2016 at 16:18 GMT

I was just recording my thought on the matter. I appreciate @tpapp putting work into drafting the issue text and would prefer to leave any edits to him.

alashworth · 2019-03-12T09:36:57Z

Comment by tpapp
Friday May 13, 2016 at 06:54 GMT

@ariddell: The table was accurate, but since not all Stan interfaces work the way that R/Julia does, I extended it with the information that is probably most relevant: whether the interfaces, in the way they currently operate, would support UTF8 variables for (1) passing data to Stan and (2) extracting MCMC results. Thanks for pointing this out, this is much more important than the details of UTF8 support in those languages per se.

Not being a STATA user, I am reluctant to make a definitive statement about it. If someone could help with that it would be great.

alashworth · 2019-03-12T09:37:00Z

Comment by ariddell
Saturday May 14, 2016 at 23:28 GMT

One use of unicode in Stan Program code which should definitely be supported is in comments. Leaving code comments in one's native language is fairly routine in Python/Java/etc. We should at least support that in Stan.

alashworth · 2019-03-12T09:37:02Z

Comment by bob-carpenter
Saturday May 14, 2016 at 23:51 GMT

Unicode in comments is OK now.

Bob

On May 14, 2016, at 7:28 PM, Allen Riddell [email protected] wrote:

One use of unicode in Stan Program code which should definitely be supported is in comments. Leaving code comments in one's native language is fairly routine in Python/Java/etc. We should at least support that in Stan.

—
You are receiving this because you commented.
Reply to this email directly or view it on GitHub

alashworth · 2019-03-12T09:37:04Z

Comment by tpapp
Sunday May 15, 2016 at 07:02 GMT

Indeed UTF8 comments work fine, and I have been using them for a while. Made a clarification in the issue.

alashworth · 2019-03-12T09:37:06Z

Comment by ariddell
Sunday May 15, 2016 at 14:16 GMT

UTF8 comments aren't supported in PyStan right now (non-ASCII characters will generate an error). I'll fix this. stan-dev/pystan#223

alashworth added this to the Future milestone Mar 12, 2019

alashworth added feature language labels Mar 12, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feature request: unicode in source #95

feature request: unicode in source #95

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

feature request: unicode in source #95

feature request: unicode in source #95

Comments

alashworth commented Mar 12, 2019

Introduction

Example

Possible benefits

Possible downsides

UTF8 support in various languages which have interfaces for Stan

Editor support

Emacs

See also

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019

alashworth commented Mar 12, 2019