@unpublished creates spurious extradate letters

[ebouna]

I have got a bibliography with several documents from archives and unpublished sources. I am using the @unpublished entry type to include them in my bibliography. Since these documents have no proper publication date, I have basically two kinds of entries, viz. @unpublished, usually without a date indicated and with nodate being suppressed vs. all other entry types where instead of nodate the German equivalent "o. J." is output.
Unfortunately, now, @unpublished interferes with extradate. So, in the MWE given below, even if GB-dt belonging to the @unpublished entry type does not have a date and even if nodate is later on suppressed in the bibliography output of the entries, we find a second entry BT-dt in the bibliography being output as BT-dt Aron Bell (o. J.[a]). .... The expected result should rather be BT-dt Aron Bell (o. J.). .... Apparently, the problem is that GB-dt has the same author as BT-dt and is also undated. So removing the author from all @unpublished entry types with \DeclareSourcemap helps produce BT-dt Aron Bell (o. J.). .... But, unfortunately, then the author information of @unpublished entries is lost, which is not what I would like to have in the end, either.

Could you help me find a better solution without the spurious extradate letters in non-@unpublished entry types when these are undated?

MWE

\documentclass{article}
\usepackage[german,ngerman]{babel}
\usepackage[
  backend=biber,
  style=ext-authoryear,
 citexref=true,
  innamebeforetitle=true,
  mergedate=basic,
  alldates=short,
  labeldate=year,
  dateuncertain=true
]{biblatex}
\usepackage{csquotes}
\usepackage{hyperref}


\DeclareDelimFormat{finalnamedelim}{\addslash\space}

\makeatletter
\DeclareFieldFormat{origdate}{\mkbibbrackets{#1}}
\renewbibmacro*{cite:labeldate+extradate}{%
 \ifentrytype{unpublished}% <--- does not work, no influence
    {}
 {\iffieldundef{labelyear}
    {}
    {\printtext[extblx@inner\blx@delimcontext delims]{%
       \printtext[bibhyperref]{%
         \printorigdate
         \setunit*{\addspace}%
         \printtext[\blx@delimcontext labeldate]{%
         \iflabeldateisdate
       {\printlabeldateextra}
         {\iffieldundef{pubstate}{\printtext{o.\addnbspace J.\@}\printfield{extradate}}%
         {\printfield{pubstate}\printfield{extradate}}%
         }}}}}}}
        

\renewbibmacro*{date+extradate}{%
    \printorigdate
  \setunit*{\addspace}
   \iffieldundef{labelyear}
    {}
    {\printtext[\blx@delimcontext labeldate]{%
       \iflabeldateisdate
       {\printlabeldateextra}
         {\iffieldundef{pubstate}{\printtext{o.\addnbspace J.\@}\printfield{extradate}}%
         {\printfield{pubstate}\printfield{extradate}}}}}}%

\renewbibmacro*{addendum+pubstate}{%
  \printfield{addendum}%
  \iffieldequalstr{labeldatesource}{pubstate}{}
  {\newunit\newblock\printfield{pubstate}}}

\makeatother

\defbibcheck{noshorthand}{%
  \iffieldundef{shorthand}{}{\skipentry}%
}

\DeclareDelimFormat{editortypedelim}{\addspace}
\DeclareFieldFormat{editortype}{\mkbibparens{#1}}

\DeclareLabeldate{%
 \field{date}
  \field{year}
 \field{eventdate}
  \field{origdate}
  \field{urldate}
  \field{pubstate}
  \literal{nodate}
  }

\DeclareLabeldate[unpublished]{% <-- make sure "nodate" is not output
  \field{date}
  \field{year}
\field{eventdate}
  \field{origdate}
  \field{urldate}
  \field{pubstate}
  }

\begin{filecontents}{\jobname.bib}
@unpublished{GB-dt,
  title = {[Volkswagen am Abgrund]},
  author = {Bell, Aron},  
  location = {{National- und Universitätsbibliothek Hobbingen, ID: 000889748}},
  howpublished = {Manuskript},
  langid = {ngerman},
  shorthand = {GB-dt},
  note = {Anonyme Übersetzung},  
}

@book{BT-dt,
  title = {Herzen im Feuer: Schauspiel},
  author = {Bell, Aron},
  translator = {Alten, Egon},
  publisher = {{Die Resterampe}},
  location = {{Berlin}},
  langid = {ngerman},
  shorthand = {BT-dt}
}

@incollection{alexander,
  title = {Language and identity},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},  
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english}
}

@incollection{alexandera,
  title = {Language1 and identity},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},  
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english}
}
\end{filecontents}
\addbibresource{\jobname.bib}



\begin{document}


Test \cite{alexander, alexandera,}

as well as \cite{BT-dt} and \cite{GB-dt} and 

\printbibheading[heading=bibintoc,title={Literaturverzeichnis}]
\printshorthands[heading=subbibliography, title={Literarische und anonym erschienene Quellen sowie Sprachwörterbücher}]
\printbibliography[heading=subbibliography,title={Sonstige Quellen und Fachliteratur},check=noshorthand]
\end{document}

As I mentioned the only way to somehow solve the issue for the moment is to remove the author from @unpublished:

\DeclareSourcemap{
  \maps[datatype=bibtex]{
    \map{
      \pertype{unpublished}
       %\step[fieldset=date, null]% <-- no influence
       %\step[fieldset=year, null]%<-- no influence
       %\step[fieldset=labelyear, null]% <-- no influence
       \step[fieldset=author, null]
    }
  }
}

[plk]

Can you post this to tex.stackexchange.com as you will get better help for a question like this there as this isn't confirmed as an issue with biblatex per se?

[ebouna]

It appears this is nontheless an issue with biblatex: Is it not possible to modify @unpublished in a way that extradate ignores @unpublished's data fields when there is no information in them?

[moewew]

Reducing the code a bit, I agree that the result in

\documentclass{article}
\usepackage[ngerman]{babel}
\usepackage{csquotes}
\usepackage[
  backend=biber,
  style=authoryear,
]{biblatex}
\usepackage{hyperref}

\defbibcheck{noshorthand}{%
  \iffieldundef{shorthand}{}{\skipentry}%
}

\begin{filecontents}{\jobname.bib}
@unpublished{GB-dt,
  title = {[Volkswagen am Abgrund]},
  author = {Bell, Aron},
  location = {{National- und Universitätsbibliothek Hobbingen, ID: 000889748}},
  howpublished = {Manuskript},
  langid = {ngerman},
  shorthand = {GB-dt},
  note = {Anonyme Übersetzung},
}
@book{BT-dt,
  title = {Herzen im Feuer: Schauspiel},
  author = {Bell, Aron},
  translator = {Alten, Egon},
  publisher = {{Die Resterampe}},
  location = {{Berlin}},
  langid = {ngerman},
  shorthand = {BT-dt}
}
@incollection{alexander,
  title = {Language and identity},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english}
}
@incollection{alexandera,
  title = {Language1 and identity},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english}
}
\end{filecontents}
\addbibresource{\jobname.bib}

\begin{document}
Test \cite{alexander, alexandera}

as well as \cite{BT-dt} and \cite{GB-dt} and

\printshorthands[heading=subbibliography, title={Shorthands}]
\printbibliography[heading=subbibliography,title={No shorthands},check=noshorthand]
\end{document}

is a bit unexpected: Since the entries with shorthand are only ever referred to by their shorthand, there is no need for the extrdate to be calculated and to appear.

I understand that at the moment Biber does not take into account the presence of shorthand for extradate calculations. The question is if it should. In this example we would probably agree that the extradate need not be calculated since we only use shorthand in citations. But the fact that for entries with shorthand author and year are never used in citations and extradate is not needed is up to the citation style. It's not something Biber can know in advance.

[plk]

biber would need to know if the style used shorthands (and that this hadn't been overridden in the document) in order to automate this. The only way I can see to do this would be to have an explicit biblatex option that controls the calculation of extradate - it isn't about the appearance of extradate - we don't want to calculate it at all for some entries, just in case some entries don't have shorthands.

[ebouna]

Thank you very much. The fact (cf. @moewew's remark) that extradate letters are superfluous for all entries with shorthand is something I have not realized so far. But yes, you are right.

[plk]

The problem is, they are only superfluous if the style always uses the shorthand.

[ebouna]

On the other hand, is it then perhaps feasible to construct an entry type that has an author-like entry field and maybe a date-like one, but that is nevertheless discarded by extradate? I mean, as a workaround.

[ebouna]

I think I found the solution, viz. how to switch off extradate letters for shorthands. I had to modify \DeclareExtradateContext by inserting \field{shorthand}. I do not completely understand why this does not produce any extradate letters for the shorthand-part of the bibliography, but it does what I need, as it seems:

\documentclass{article}
\usepackage[german,ngerman]{babel}
\usepackage[
  backend=biber,
  style=authoryear
]{biblatex}
\usepackage{csquotes}
\usepackage{hyperref}        


\defbibcheck{noshorthand}{%
  \iffieldundef{shorthand}{}{\skipentry}%
}

\DeclareExtradateContext{%
\field{shorthand}
\field{labelname}
\field{labeltitle}
}

\DeclareLabeldate{%
 \field{date}
  \field{year}
 \field{eventdate}
  \field{origdate}
  \field{urldate}
  \field{pubstate}
  \literal{nodate}
  }

\DeclareLabeldate[unpublished]{% <-- make sure "nodate" is not output
  \field{date}
  \field{year}
\field{eventdate}
  \field{origdate}
  \field{urldate}
  \field{pubstate}
  }





\begin{filecontents}{\jobname.bib}
@unpublished{AGB-dt,
  title = {[Volkswagen am Abgrund]},
  author = {Bell, Aron},  
  %shortname = {Bell, Aron1},
  location = {{National- und Universitätsbibliothek Hobbingen, ID: 000889748}},
  howpublished = {Manuskript},
  langid = {ngerman},
  shorthand = {AGB-dt},
  note = {Anonyme Übersetzung},  
}

@unpublished{GB-dt,
  title = {[Volkswagen am Abgrund]},
  author = {Bell, Aron},  
  %shortname = {Bell, Aron1},
  location = {{National- und Universitätsbibliothek Hobbingen, ID: 000889748}},
  howpublished = {Manuskript},
  langid = {ngerman},
  shorthand = {GB-dt},
  note = {Anonyme Übersetzung},  
}

@book{BT-dt,
  title = {Herzen im Feuer: Schauspiel},
  author = {Bell, Aron},
  translator = {Alten, Egon},
  publisher = {{Die Resterampe}},
  location = {{Berlin}},
  langid = {ngerman},
  shorthand = {BT-dt}
}

@incollection{ron,
  title = {Language},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},  
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english},
  shorthand = {ron}
}

@incollection{alexander,
  title = {Language1},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},  
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english}
}

@incollection{alexandera,
  title = {Language2},
  booktitle = {Entangled histories of the {{Balkans}}},
  author = {Alexander, Ronelle},
  editor = {Daskalov, Roumen and Marinov, Tchavdar},  
  volume = {1},
  pages = {341--417},
  location = {{Leiden, Boston}},
  langid = {english}
}

@book{book,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings},
  year      = {2013},
  shorthand = {PR2013}
}

@book{book2,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings},
  year      = {2013},
  shorthand = {PR2013-a}
}

@book{book1,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings},
  year      = {2013},
 
}

@book{book3,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My2 Proceedings},
  year      = {2013},
 
}


\end{filecontents}
\addbibresource{\jobname.bib}



\begin{document}


Test \cite{alexander, alexandera,}

as well as \cite{BT-dt} and \cite{GB-dt} and \cite{AGB-dt} and  \cite{ron, book, book1, book2, book3}

\printshorthands[heading=subbibliography, title={Shorthands}]
\printbibliography[heading=subbibliography,title={No shorthands},check=noshorthand]
\end{document}

[ebouna]

Note also that removing

\DeclareExtradateContext{%
\field{shorthand}
\field{labelname}
\field{labeltitle}
}

reveals what seems to be a bug. Compare the erroneous alphabetical order of extradate letters in the marked cased in the output below:

[plk]

Well, deleting the entire extradatacontext will always result in strange things as at least the default is needed.
However, this is an interesting use of the extradatecontext since it will date-disambiguate every entry with a shorthand against all other entries with shorthands based on repeated shorthands (which will usually never be the case) so this may well be a fix for this situation.

[ebouna]

Well, deleting the entire extradatacontext will always result in strange things as at least the default is needed.

I think here it appears you misunderstand me. By removing

\DeclareExtradateContext{%
\field{shorthand}
\field{labelname}
\field{labeltitle}
}

from the above code, I meant not including those lines of code. This should then result in having the default. The default as per the specs (p. 241) is:

\DeclareExtradateContext{%
\field{labelname}
\field{labeltitle}
}

But with a shorthand list in the bibliography, this default obviously does not produce the expected output, i.e., it yields the erroneous alphabetical order of extradate letters I described.

[plk]

Ah, sorry, I see. This is expected with the default settings as both biblists are in the same refsection and there are four entries with the same labelname in the refsection. They are just artificially split up into two lists.

[moewew]

biber would need to know if the style used shorthands (and that this hadn't been overridden in the document) in order to automate this. The only way I can see to do this would be to have an explicit biblatex option that controls the calculation of extradate - it isn't about the appearance of extradate - we don't want to calculate it at all for some entries, just in case some entries don't have shorthands.

Exactly, that's what I was thinking. I'm just not too sure if we want another option or if this could better be integrated into \DeclareExtradate, \DeclareExtradateContext and friends.

On the other hand, is it then perhaps feasible to construct an entry type that has an author-like entry field and maybe a date-like one, but that is nevertheless discarded by extradate? I mean, as a workaround.

I don't think this would be a good idea for the biblatex core. Whether or not shorthand-ed entries are tracked for extradate should not depend on the entry type. It should be a choice of the style that should then apply to all entries alike.

[moewew]

Hmm. I think in the following MWE something more is at play

\documentclass{article}
\usepackage[german,ngerman]{babel}
\usepackage[
  backend=biber,
  style=authoryear
]{biblatex}
\usepackage{csquotes}
\usepackage{hyperref}

\defbibcheck{noshorthand}{%
  \iffieldundef{shorthand}{}{\skipentry}%
}

\begin{filecontents}{\jobname.bib}
@book{book,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings},
  year      = {2013},
  shorthand = {PR2013}
}
@book{book2,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings},
  year      = {2013},
  shorthand = {PR2013-a}
}
@book{book1,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings},
  year      = {2013},
}
@book{book3,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My2 Proceedings},
  year      = {2013},
}
\end{filecontents}
\addbibresource{\jobname.bib}

\begin{document}
\cite{book, book1, book2, book3}

\printshorthands[heading=subbibliography, title={Shorthands}]
\printbibliography[heading=subbibliography,title={No shorthands},check=noshorthand]
\end{document}

generates

PR2013; Editor und Editora 2013b; PR2013-a; Editor und Editora 2013d

for me.

In particular the entries are all different, but "2013b" is assigned twice. As far as I can see the problem is that the shorthand biblist is handled separately from the overall biblist and the assignment of extradate is different in the two lists.

[plk]

Well, this is a very odd example since book, book1 and book2 are the same entry, just with different shorthands - is that realistic?

[moewew]

I can offer

\documentclass{article}
\usepackage[german,ngerman]{babel}
\usepackage[
  backend=biber,
  style=authoryear
]{biblatex}
\usepackage{csquotes}
\usepackage{hyperref}

\defbibcheck{noshorthand}{%
  \iffieldundef{shorthand}{}{\skipentry}%
}

\begin{filecontents}{\jobname.bib}
@book{book,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings D},
  year      = {2013},
  shorthand = {PR2013}
}
@book{book2,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings B},
  year      = {2013},
  shorthand = {PR2013-a}
}
@book{book1,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings C},
  year      = {2013},
}
@book{book3,
  editor    = {Senor Editor and Senora Editora},
  publisher = {Any Publisher},
  title     = {My Proceedings A},
  year      = {2013},
}
\end{filecontents}
\addbibresource{\jobname.bib}

\begin{document}
\cite{book, book1, book2, book3}

\printshorthands[heading=subbibliography, title={Shorthands}]
\printbibliography[heading=subbibliography,title={No shorthands},check=noshorthand]
\end{document}

which gives

Here "2013a" is used twice for different works.

[plk]

Hmm, what is happening here is that extra* information is calculated per-datalist. Shorthands are a special case as they usually accompany a normal bibliography. extradate is being calculated for all four entries in both lists but the filters are run after biber. This is a general issue with post-biber filters.

[ebouna]

Here "2013a" is used twice for different works

This and similar irregularities is precisely the unexpected behaviour I have been observing, too.

[plk]

I'm not sure what to do here. You have to calculate extra* for any datalists separately since they can have different sorting. The issue here is specific to having the same labelname list in both datalists, which confuses things. Is it really a problem to have "2013a" for two different works since they will usually (always?) be by a different list of authors?

[moewew]

Hmmm. While some of the situations we create here might be artificial (or at least highly unlikely), I think at least some of the issues discussed here are a symptom of the general principle that different biblists and refcontexts (e.g. sorting) are given as completely separate datalists in the .bbl. I think usually the expectation of a user would be that there is exactly one set of data for each entry (generated once based on a global sorting/uniqueness scheme) and that all other biblists within the same refsection should use the same data and should only apply different filtering and sorting based on that data - no context-dependent data need to be recalculated, it can just be taken as-is from the global context.

I mentioned something like this before in #1009 (comment).

Maybe I need to draw up a more concrete plan of what that should look like and we can discuss its advantages and disadvantages in more detail and see if it can be implemented in biblatex and Biber.

[plk]

I don't think we can do this since sorting is specifiable at a biblist level and therefore extra* labels can differ between lists. This is only really an issue when you have more than one list for the same data (which is common with shorthand lists and they are always sorted by shorthand so very different sorting from standard biblists). Need to think a bit about what can be done here.

[ebouna]

"2013a" for two different works since they will usually (always?) be by a different list of authors?

The problem is not only about having twice "2013a". The real or more disturbing problem is rather the irregular alphabetical order or omissions in the alphabetical assignment of extradate letters that may occur: cf. "2013b" and then "2013d" in the no-shorthands-list!

[moewew]

True, but I don't think people always want sorting based on recalculated extra* data. In some settings having one set of extra* data as "ground truth" and then sorting (with other schemes) based on that is fine. Take for example the common case of sorting citations differently from the bibliography. There we have one set of extra* data in the bib-sorting, say nyt. It is not unreasonable to want the citation sorting, say to be based only on the visible bits of the citation label, which would be name, year and extradate from the nyt context. We wouldn't want to recalculate a new extradate here, because that would not correspond to the relevant data in the bibliography anyway.

[ebouna]

Thanks a lot for this additional comment and your explanation as well as thanks, @plk, for the other useful pieces of information.