[WIP] bpo-31589: Build PDF using xelatex for better UTF8 support.

[JulienPalard]

Trying to fix: https://bugs.python.org/issue31589

I sucessfully built locally PDF for english and french, still have to try japanese, and still have to "diff" (at least visually) PDFs from pdflatex vs pdf from xelatex.

https://bugs.python.org/issue31589

[JulienPalard]

If someone want to test it locally, after checking-out my branch and installing texlive-xetex, texlive-fonts-recommended, texlive-lang-all, creating a venv with an up-to-date sphinx-doc and blurb, one can do:

git clone --depth 1 --no-single-branch https://github.com/python/python-docs-fr.git
cd python-docs-fr/
git checkout 3.6
mkdir -p /tmp/$USER/locales/fr
ln -nfs $(readlink -f .) /tmp/$USER/locales/fr/LC_MESSAGES
make -C your/cpython/clone/Doc SPHINXOPTS="-D locale_dirs=/tmp/$USER/locales/ -D language=fr -D gettext_compact=0" autobuild-dev

[vstinner]

@methane: Are you interested to check if it works with the Japanese translation?

[methane]

I don't know about PDF and latex much.
But I'm asking question about recommended way to generate Japanese PDF in Japanese
Sphinx community. (Most active maintainers of Sphinx are Japanese.)

[JulienPalard]

They sent a lot of feedback on sphinx-doc/sphinx#4136

I'm trying to build locally for japanese currently.

[JulienPalard]

Looks like build is failing for japanese:

------------
Running 'xelatex   -recorder  "howto-curses.tex"'
------------
This is XeTeX, Version 3.14159265-2.6-0.99998 (TeX Live 2017/Debian) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(./howto-curses.tex
LaTeX2e <2017-04-15>
Babel <3.11> and hyphenation patterns for 84 language(s) loaded.
(./sphinxhowto.cls
Document Class: sphinxhowto 2017/03/26 v1.6 Document class (Sphinx HOWTO)
(/usr/share/texlive/texmf-dist/tex/platex/base/jreport.cls

! LaTeX Error: This file needs format `pLaTeX2e'
               but this is `LaTeX2e'.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.16 \NeedsTeXFormat{pLaTeX2e}

[methane]

Generating Japanese PDF via Latex is very hard way.
Please ignore Japanese for now.

[tk0miya]

Usually, platex is used to build Japanese TeX documents. So Sphinx depends on it by default.
In other words, Sphinx does not support xelatex to build Japanese document.
(If you're TeX specialist, you might be able to configure your Sphinx project to do that!)

IMO, it would be better to use xelatex only if the document is not Japanese. How about this?

if language != 'ja':
    latex_engine = 'xelatex'

I believe this will work well.

[JulienPalard]

@tk0miya Thanks for the recommendation, I'll try this. I'm no latex expert though.

[JulienPalard]

Tests for today:

• Using pdflatex, with the current set of inputenc and fontenc, english build is OK, even the Є symbol in the changelog is rendered correcly.
• Using xelatex and an empty latex_elements, english still builds OK
• Using pdflatex, with the current set of inputenc and fontenc, french fails (latexmk fail as documented in https://bugs.python.org/issue31589#msg303502)
• Using xelatex and an empty latex_elements, french produces PDFs
• Usinx xelatex to build japanese gives: LaTeX Error: This file needs format 'pLaTeX2e' but this is 'LaTeX2e'.
• Using platex, japanese fails to build:

Japanese errors using platex:

! Text line contains an invalid character.
l.66796 ...de{subprocess.Popen()}}}} の stream ^^P
                                                  引数など、ファイル記述子つ...

Looks like the problem is from the Japanese translation:

#: ../../library/socket.rst:1072
msgid ""
"On Windows, the file-like object created by :meth:`makefile` cannot be used "
"where a file object with a file descriptor is expected, such as the stream "
"arguments of :meth:`subprocess.Popen`."
msgstr ""
"Windows では、 :meth:`subprocess.Popen` の stream ^P引数など、ファイル記述子つき file "
"オブジェクトが期待されている場所では、 :meth:`makefile` によって作成される file-like オブジェクトは使用できません。"

I'm opening an issue on their side.

[JulienPalard]

ref: python/python-docs-ja#2

[JulienPalard]

Fixing the \x10 found in japanese translation yields to another error:

$ xelatex   -recorder  howto-curses.tex
This is XeTeX, Version 3.14159265-2.6-0.99998 (TeX Live 2017/Debian) (preloaded format=xelatex)
 restricted \write18 enabled.
entering extended mode
(./howto-curses.tex
LaTeX2e <2017-04-15>
Babel <3.13> and hyphenation patterns for 84 language(s) loaded.
(./sphinxhowto.cls
Document Class: sphinxhowto 2017/03/26 v1.6 Document class (Sphinx HOWTO)
(/usr/share/texlive/texmf-dist/tex/platex/base/jreport.cls

! LaTeX Error: This file needs format `pLaTeX2e'
               but this is `LaTeX2e'.

See the LaTeX manual or LaTeX Companion for explanation.
Type  H <return>  for immediate help.
 ...                                              
                                                  
l.16 \NeedsTeXFormat{pLaTeX2e}

? H
The current input file will not be processed further,
because it was written for some other flavor of TeX.
You're in trouble here.  Try typing  <return>  to proceed.
If that doesn't work, type  X <return>  to quit.
                          
? x
No pages of output.
Transcript written on howto-curses.log.

I still don't get this one, as I explicitly asked sphinx to use xelatex.

[JulienPalard]

But building with platex works!

So xelatex works for english and french and platex works for japanese, as told by @tk0miya thanks!

Still have to find a clean way to jump between those two implementations, as the 'language' is not yet initialized when parsing conf.py.

[miss-islington]

Thanks @JulienPalard for the PR, and @ned-deily for merging it 🌮🎉.. I'm working now to backport this PR to: 3.6.
🐍🍒⛏🤖

[bedevere-bot]

GH-4683 is a backport of this pull request to the 3.6 branch.

[miss-islington]

Thanks @JulienPalard for the PR, and @ned-deily for merging it 🌮🎉.. I'm working now to backport this PR to: 2.7.
🐍🍒⛏🤖

[bedevere-bot]

GH-16636 is a backport of this pull request to the 2.7 branch.