index_files not copied to project.output-dir with certain markdown formats

[mikedecr]

Bug description

Hi folks.

I am rendering a document with project.output-dir specified in _quarto.yml

When some index.qmd contains a code block to generate some index_files, the index_files directory is not being copied to the output-dir when certain output formats (like hugo-md) are used.
The index_files are correctly copied with other output formats (like pdf or commonmark), so this is somehow format-specific.

Steps to reproduce

note: this example was edited to reflect advice below about project.type = hugo

The code below replicates the issue for me on the following quarto version:

quarto --version

shows 1.6.40.

First let's make a .qmd that we will use.
I am using a small mermaid diagram as a code chunk that should generate dependencies in index_files.
We put in page sub-directory.

mkdir ./page
echo -e "---\ntitle: my title\n---\n\n\`\`\`{mermaid}\nflowchart LR\nA[start] --> B[end]\n\`\`\`" > page/index.qmd

I will then use _quarto.yml files to vary the render behavior.
We save the output to output sub-directory.

md output behaves, as I understand, correctly.

echo "project:\n    output-dir: output\nformat: md" > _quarto.yml
quarto render page/index.qmd
ls output/page

The ls output shows:

index.html      index_files

Here is an example with gfm output.

# gfm
rm -rf ./output
echo "project:\n    output-dir: output\nformat: gfm" > _quarto.yml
quarto render page/index.qmd
ls output/page

and now the ls command only shows index.md.

And here is an example with hugo-md which also sets project.type to hugo.

rm -rf ./output
echo "project:\n    output-dir: output\n    type: hugo\nformat: hugo-md" > _quarto.yml
quarto render page/index.qmd
ls output/page

again ls only shows index.md

Expected behavior

As I understand, we should always expect index_files to appear in the output dir.

Actual behavior

On my side, the following outputs place index_files in the output-dir:

• pdf
• typst
• html
• md
• markdown
• hugo-markdown
• commonmark
• hugo-commonmark

And these formats do not plaace index_files in the output dir:

• hugo-md
• gfm
• hugo-gfm

and that's all I have tested.

Your environment

• OS: MacOS Sonoma 14.5
• quarto 1.6.40 installed as homebrew cask

Quarto check output

quarto check   

Quarto 1.6.40
[✓] Checking environment information...
      Quarto cache location: /Users/michaeldecrescenzo/Library/Caches/quarto
[✓] Checking versions of quarto binary dependencies...
      Pandoc version 3.4.0: OK
      Dart Sass version 1.70.0: OK
      Deno version 1.46.3: OK
      Typst version 0.11.0: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
      Version: 1.6.40
      Path: /Applications/quarto/bin

[✓] Checking tools....................OK
      TinyTeX: v2024.12
      Chromium: (not installed)

[✓] Checking LaTeX....................OK
      Using: TinyTex
      Path: /Users/michaeldecrescenzo/Library/TinyTeX/bin/universal-darwin
      Version: 2024

[✓] Checking basic markdown render....OK

[✓] Checking Python 3 installation....OK
      Version: 3.13.0
      Path: /opt/homebrew/opt/python@3.13/bin/python3.13
      Jupyter: (None)

      Jupyter is not available in this Python installation.
      Install with python3 -m pip install jupyter

[✓] Checking R installation...........OK
      Version: 4.3.3
      Path: /Library/Frameworks/R.framework/Resources
      LibPaths:
        - /Library/Frameworks/R.framework/Versions/4.3-arm64/Resources/library
      knitr: (None)
      rmarkdown: (None)

      The knitr package is not available in this R installation.
      Install with install.packages("knitr")
      The rmarkdown package is not available in this R installation.
      Install with install.packages("rmarkdown")

[mcanouil]

Why would index_files be copied? You are supposed to use Hugo to create the HTML files.

You are also suppsed to use type: hugo as project type not the default which your example implies.

See https://quarto.org/docs/output-formats/hugo.html.

It’s important to note that if you do provide an explicit _quarto.yml file you need to explicitly specify the project type (type: hugo) as shown above.

[mikedecr]

Why would index_files be copied?

Sure, quarto injects code into the output that assumes that the index_files are going to be there:

cat output/page/index.md 

---
title: my title
---

<script  src="index_files/libs/quarto-diagram/mermaid.min.js"></script>
<script  src="index_files/libs/quarto-diagram/mermaid-init.js"></script>
<link  href="index_files/libs/quarto-diagram/mermaid.css" rel="stylesheet" />

<figure class=''>

<pre class="mermaid mermaid-js">flowchart LR
A[start] --&gt; B[end]
</pre>

</figure>

The other markdown formats inject this same code but do the additional step of copying the index_files, so it seems like it is the expected behavior. Furthermore hugo-md output copies image files to the output, so it copying some of these "render-time" deps but just not index_files. If it's incorrect to expect index_files to be copied, can you let me in on why the other markdown formats do it / why hugo-md copies images but not index_files, so I can understand why this subset of markdown formats should be an exception?

You are supposed to use Hugo to create the HTML files.
You are also suppsed to use type: hugo as project type not the default which your example implies.

edit: I went updated my reprex code to set project type: hugo just to rule this out.
I think the rest of this reply is no longer relevant.

---

I am unsure if the Hugo project details will turn out to be critical here. We see the same index_files behavior with gfm as we do with hugo-md. So my hunch would be that quarto derives the hugo outputs from gfm in some way and there is some reason why gfm differs from the other formats. I could be wrong!

But if some details about my use case with Hugo might be helpful, I can provide some more context.

Hugo is indeed doing the HTML rendering for me as you suggest.

The reason why I don't have project-type hugo is because I do not intend to manage a hugo blog with quarto, simply put. I am just using quarto to render individual files in some directory project-root/my-blog-posts/post-a, post-b etc, to project-root/hugo/content/blog/.... My "quarto step" and my "hugo step" aren't aware of one another. It's also done w/ separate quarto render steps per file (for environment mgmt reasons) so I am not calling quarto render . to render an entire project's worth of qmd files. I am in the most minimal sense just trying to render a single file to a single output, one at a time, and move the output to some other location with its render-time dependencies.

If I need project-type hugo to support the hugo-md output on a standalone basis then I am happy to change my ways. But I am unsure how the project type would bear on the index_files behavior. Other dependencies, such as image files generated from R code for example, are migrated to the output-dir so this seems to be different behavior only for index_files. Furthermore specifying the project-type as hugo in the above reprex doesn't change the behavior in a way that I can detect.

[mcanouil]

The documentation says:

It’s important to note that if you do provide an explicit _quarto.yml file you need to explicitly specify the project type (type: hugo) as shown above.

If you don't, it's like asking things to go wrong.

To properly use output-dir (#2171), a project is needed, i.e., _quarto.yml.

To be clear, I am not saying there is no bug. I am saying your example is by definition wrong, at least for hugo-md.

[mcanouil]

For reference as it fits in:

• Epic: Execution Planner #6518

[mikedecr]

Thanks, I will examine this link

[mikedecr]

@mcanouil I updated the reprex in my original post to add project.type = hugo following your feedback. Apologies for being difficult about it before.

[cderv]

I believe the behavior you encounter may be specific to Hugo project.

Let me try to explain.

When working with Quarto and Hugo, Quarto is only meant to be used to generate .md version of the .qmd source. Hugo will still be the one to create the website following Hugo website structure.

This sentence sums it up in our doc:

The basic concept of using Quarto with Hugo is that you take computational markdown documents (.qmd) or Jupyter notebooks (.ipynb) and use them to generate plain markdown files (.md) that are rendered to HTML by Hugo.

Quarto is not meant to move the files around as it is important to follow the Hugo expected structure of Leaf Bundle

Our doc says

The computational files are located in the same place you would also locate ordinary markdown files (e.g. the blog directory).

After rendering, a plain .md file is written right alongside the computational document. This markdown file is then processed by Hugo.

So in your example the expected results is

project
- _quarto.yml
- page/
  |-- index.qmd
  |-- index.md
  |-- index_files/**

This way when hugo build will be run, all the resources for index.md rendering to html will be available.

Though there is an "External Directory" support where output-dir could be used to have Quarto project and Hugo Project into two different directory and we can reproduce the index_files not being moved.

So I think output-dir for Hugo project never really worked as expected when resource files is created.

[mikedecr]

Though there is an "External Directory" support where output-dir could be used to have Quarto project and Hugo Project into two different directory and we can reproduce the index_files not being moved.

Thanks for the reply @cderv. The "External directory" section is all I am trying to accomplish, nothing too fancy. Just rendering the qmds to .md and relocating the outputs according to the project variable. Hugo already controls rendering to HTML, as you say.

So I think output-dir for Hugo project never really worked as expected when resource files is created.

Just to clarify, are we in agreement that the current behavior (not moving the index_files) is a bug? Everything else is working predictably for me; other resource dirs are correctly located to the output dir (such as generated figures), it's just index_files that are not moved.

---

Separate question... as I understand it, an output format like hugo-md is really just an alias to a set of options for rendering markdown through pandoc. Would you be able to point me to where the hugo-md format is mapped to those options, so I might experiment w/ some of my own option-passing to learn more about the behavior locally? if it's a tall ask then no worries.

[cderv]

Just to clarify, are we in agreement that the current behavior (not moving the index_files) is a bug?

Yes we document that external dir should work with quarto project type hugo, and it is missing moving the index_files that clearly are necessary and should be considered resource files. I don't think it was ever working so this is a real bug and not just a regression. Unless it was working for you ?

an output format like hugo-md is really just an alias to a set of options for rendering markdown through pandoc.

Hugo support is quite specific in quarto. hugo-md is defined as a custom format same that you could define using an extension https://quarto.org/docs/extensions/formats.html
So it is extending the generic format: md

Internaly, there are also some Lua processing associated to hugo-md specific format.

To be clear, format: hugo-md is defined. But hugo-gfm and hugo-commonmark are not, and they should not be used. It seems nothing protect against those because they are valid format syntax for custom format, but they are not really defined and I don't think specific Lua processing for hugo would apply

Maybe we should try warn or disallow to be sure hugo-md is the only format used. 🤔

[mcanouil]

So it is extending the generic format: md

To clarify, md is also a custom Quarto format, the Pandoc format is markdown (this is the format to use, not md).

[mikedecr]

Unless it was working for you ?

@cderv Just to confirm, it's not a regression afaict. It wasn't working on 1.5 where I first encountered it, and I did an extra step of upgrading to 1.6 before opening the issue just to see if it was fixed since 1.5