in quarto convert failing to convert R cells?

[MichelNivard]

Bug description

HI I think quite convert is failing when sed iwht r cells?

I am trying to cover .qmd to .ipynb on Mac, quarto: 1.6.40, r: .4.4.1

this is my qmd file"

---
title: 'test conversion'
---

```{r}
library(tinyplot)
```

# Introduction

This document is a test...

```{r}
a <- rnorm(100)
hist(a)
```

Does it work? I don't know?

which generates this .ipynb (pasting raw .json but basically the entire thing is in one big markdown cell):

{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"title: 'test conversion'\n",
"---\n",
"\n",
"{r}\n", "library(tinyplot)\n", "\n",
"\n",
"# Introduction\n",
"\n",
"This document is a test...\n",
"\n",
"{r}\n", "a <- rnorm(100)\n", "hist(a)\n", "\n",
"\n",
"Does it work? I don't know?\n"
]
}
],
"metadata": {
"kernelspec": {
"name": "python3",
"language": "python",
"display_name": "Python 3 (ipykernel)",
"path": "/Users/michelnivard/miniconda3/share/jupyter/kernels/python3"
}
},
"nbformat": 4,
"nbformat_minor": 4
}

the expected behaviour here would be a series of:

1 raw cells with the yaml
1 code cell: library(tinyplot)
1 markdown cell:
Introduction

This document is a test...
1 code cell:
a <- rnorm(100)
hist(a)
1 markdown cell:
Does it work? I don't know?

I think its failing because the conversion expects only python cells?

Conversion work fine if I change the r cells to python cells...

Steps to reproduce

No response

Expected behavior

No response

Actual behavior

No response

Your environment

No response

Quarto check output

quarto check
Quarto 1.6.40
[✓] Checking environment information...
Quarto cache location: /Users/michelnivard/Library/Caches/quarto
[✓] Checking versions of quarto binary dependencies...
Pandoc version 3.4.0: OK
Dart Sass version 1.70.0: OK
Deno version 1.46.3: OK
Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
Version: 1.6.40
Path: /Applications/quarto/bin

[✓] Checking tools....................OK
TinyTeX: v2023.01
Chromium: (not installed)

[✓] Checking LaTeX....................OK
Using: TinyTex
Path: /Users/michelnivard/Library/TinyTeX/bin/universal-darwin
Version: 2022

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
Version: 3.10.8 (Conda)
Path: /Users/michelnivard/miniconda3/bin/python
Jupyter: 5.7.1
Kernels: ir, python3

[✓] Checking Jupyter engine render....OK

[✓] Checking R installation...........OK
Version: 4.4.1
Path: /Library/Frameworks/R.framework/Resources
LibPaths:
- /Library/Frameworks/R.framework/Versions/4.4-arm64/Resources/library
knitr: 1.48
rmarkdown: 2.27

[✓] Checking Knitr engine render......OK

(base) michelnivard@michels-MacBook-Air resolve-demo-doc %

[mcanouil]

You have to set kernels, etc.
Default Jupyter kernel does not know anything about R thus it's converted as a code block of unknown language and not as a code cell. Take a look at the kernel in your produced Jupyter Notebook.

As a reminder, the default engine for R in Quarto is via knitr/R which is not a Jupyter kernel.

How do you expect Quarto to know how you execute R code in Jupyter if you don't tell it?

[MichelNivard]

okay fair enough, I assumed this was an agnostic translation function, neither quarto convert help, nor the quarto documentation mentioned anywhere that conversion requires this yaml flag is set.

BTW the r code isnt converted to a code cell of unknown language, it's all wrapped into a single markdown cell which still seems like unexpected behaviour?

[venpopov]

I just had the same confusion. I am also confused by @mcanouil response. @MichelNivard is talking about the CLI

quarto convert mynotebook.qmd

which according to the documentation should convert a Quarto notebook to a Jupyter notebook. Jupyter is not involved in the conversion, and there is no step that let's you specify a Jupyter kernel. As @MichelNivard noted, running the convert command produces a Jupyter notebook with a single cell that also includes the yaml metadata. So in this case quarto convert literally just wraps the content of the qmd file in a single cell of a jupyter notebook.

[mcanouil]

See the documentation on specifying a kernel: https://quarto.org/docs/computations/python.html#kernel-selection

If you don't, Quarto will set knitr as the engine thus not a Jupyter kernel.
Then quarto convert will have to pick a kernel for Jupyter and will use the default which does not know anything about R leading to the result reported.

For a markdown code cell to be converted correctly to a Jupyter code cell, the kernel MUST know about it.

[venpopov]

I was about to post this reproducible example before I saw your response: https://github.com/venpopov/quarto-convert-demo

In any case, this strikes me as a design issue. quarto convert should be able to convert a qmd notebook from the wild, without having to change the kernel inside the notebook. IMO, specifying any options about how to treat executable qmd chunks should be a responsibility of the quarto convert API, not that of the notebook.

[mcanouil]

You are converting a Quarto document with R which won't use Jupyter at all to a Jupyter notebook which must use Jupyter by definition.

If you wanted to use Jupyter as the engine for a Quarto document, you would have specify the kernel as the default is Python.

How Quarto could know what kernel to use?

This would mean Quarto has to assume the use of one of the many kernels possible to evaluate R code regardless of what the users actually have.

Same applies for Julia where you have to set explicitly the kernel: https://quarto.org/docs/computations/julia.html#kernel-selection

Note that the issue is the same in a plain Jupyter Notebook. If you don't choose/set a kernel yourself in it, it is Python by default.

[venpopov]

I get all of this. My point is this. I have a regular .qmd document that uses R chunks with no kernel specified in the yaml, so it uses knitr by default when rendering the document. When I apply quarto convert on this document, I should be able to specify in the convert command what kernel should be used. The responsibility should not be to change the input file, but is for the function that does the conversion to ask for this. Otherwise you end up with unexpected behaviors like the one that prompted this thread. Also, this isn't documented anywhere! The docs simply state that you can use quarto convert to switch from one file format to another and that's it.

[mcanouil]

You are asking for something different than the original post.

To me, the original report is not a bug but a lack of kernel specification in the document.

This being said, as a feature request, adding a flag to the quarto convert to allow the specification of the kernel when converting is a good idea, but it won't change the fact that the user will still have to specify the kernel regardless of the method.

I let the maintainers address the issue / feature request.

[MichelNivard]

I really appreciate the time take n to respond here @mcanouil. I think it's really fine for the 'Jupyter:' tag to be required in the yaml for the cells in the ipynb to be assigned the right language.

What I dont understand is why you dont simply amend/change the typescript logic that is in place to detect code fences (https://github.com/quarto-dev/quarto-cli/blob/main/src/command/convert/jupyter.ts line 164):

const md: string[] = [backticks + "{" + language + "}\n"];

with a saver fallback like THIS IS PEEUDO CODE, I DONT TYPESCRIPT VERY WELL:

const md: string[] = [backticks + "{" + * + "}\n"];

where * can be a variable will all languages you detect in the file?

you could always then send a warning that the kernel space and language use is out of whack?

my point is, the entire field as one markdown cell, where not even the yaml header is recognised as a raw cell (see my example) doesn't seem like a great, or expected, outcome? If the code cells were in the markdown, or flagged as out of spec or unrecognised code cells but the yaml was good, that would still be a valid quarto renderable ipynb?

so the bug, to my mind, is that if I get the kernel spec wrong, it doesn't only fail to recognised the code cells (which is a defensible choice!) but that now also the yaml header isn't in a raw cell, but part of the markdown.

[mcanouil]

As I said "I let the maintainers address the issue / feature request.".
One of the main reasons is that I'm not familiar with that part of the codebase.

I am quite sure your suggestion won't work because that's for Jupyter Notebook to Markdown which uses the kernel information to write the code cells as markdown.

Here is the right part:

• quarto-cli/src/core/jupyter/jupyter.ts

  Lines 312 to 315 in a2046c0

  	const startCodeCellRegEx = new RegExp(
  	"^(\\s*)(```+)\\s*\\{" + kernelspec.language.toLowerCase() +
  	"( *[ ,].*)?\\}\\s*$",
  	);

As you can see it also uses the kernel information to detect code cells.

[venpopov]

It appears this was already discussed here so linking it for context: #9463