[v2,0/8] vt: more Unicode handling changes

Message ID	20250507141535.40655-1-nico@fluxnic.net
Headers	show Received: from fhigh-a4-smtp.messagingengine.com (fhigh-a4-smtp.messagingengine.com [103.168.172.155]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6BCFA28935B; Wed, 7 May 2025 14:15:43 +0000 (UTC) Feedback-ID: i58514971:Fastmail From: Nicolas Pitre <nico@fluxnic.net> To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>, Jiri Slaby <jirislaby@kernel.org> Cc: Nicolas Pitre <npitre@baylibre.com>, linux-serial@vger.kernel.org, linux-kernel@vger.kernel.org Subject: [PATCH v2 0/8] vt: more Unicode handling changes Date: Wed, 7 May 2025 10:13:15 -0400 Message-ID: <20250507141535.40655-1-nico@fluxnic.net> Precedence: bulk MIME-Version: 1.0 Content-Transfer-Encoding: 8bit
Series	vt: more Unicode handling changes \| expand [v2,0/8] vt: more Unicode handling changes [v2,1/8] vt: ucs.c: fix misappropriate in_range() usage [v2,2/8] vt: make sure displayed double-width characters are remembered as such [v2,3/8] vt: move glyph determination to a separate function [v2,4/8] vt: introduce gen_ucs_fallback_table.py to create ucs_fallback_table.h [v2,5/8] vt: create ucs_fallback_table.h_shipped with gen_ucs_fallback_table.py [v2,6/8] vt: add ucs_get_fallback() [v2,7/8] vt: make use of ucs_get_fallback() when glyph is unavailable [v2,8/8] vt: process the full-width ASCII fallback range programmatically

Message ID

20250507141535.40655-1-nico@fluxnic.net

Headers

Feedback-ID: i58514971:Fastmail
From: Nicolas Pitre <nico@fluxnic.net>
To: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Jiri Slaby <jirislaby@kernel.org>
Cc: Nicolas Pitre <npitre@baylibre.com>,
	linux-serial@vger.kernel.org,
	linux-kernel@vger.kernel.org
Subject: [PATCH v2 0/8] vt: more Unicode handling changes
Date: Wed,  7 May 2025 10:13:15 -0400
Message-ID: <20250507141535.40655-1-nico@fluxnic.net>
Precedence: bulk
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit

Series

vt: more Unicode handling changes | expand

Message

Nicolas Pitre May 7, 2025, 2:13 p.m. UTC

The Linux VT console has many problems with regards to proper Unicode
handling. A first set of patches was submitted here:

https://lore.kernel.org/all/20250417184849.475581-1-nico@fluxnic.net/

Those patches are currently in Greg's tty-next branch.

The first 2 patches in the following series contain fixes for those
already-applied patches.

Remaining patches introduce tables that map complex Unicode characters
to simpler fallback characters for terminal display when corresponding
glyphs are unavailable. Only the subset of Unicode that can reasonably
be substituted by ASCII/Latin-1 characters is covered. Substitution may
not be as good as the actual glyphs but still way more helpful than squared
question marks.

This applies on top of tty-next currently at commit 5ee558c5d9e9.

Changes from v1 (https://lore.kernel.org/all/20250505170021.29944-1-nico@fluxnic.net/):

- Reworked the gen_ucs_fallback_table.py script to be based on unidecode.
  This creates a much more comprehensive fallback table with a much smaller
  script.

- Changed the fallback table encoding to be more space efficient.

- Addressed style comments from Jiri.

diffstat:
 drivers/tty/vt/.gitignore                   |    1 +
 drivers/tty/vt/Makefile                     |    8 +-
 drivers/tty/vt/gen_ucs_fallback_table.py    |  360 +++
 drivers/tty/vt/ucs.c                        |  102 +-
 drivers/tty/vt/ucs_fallback_table.h_shipped | 3346 +++++++++++++++++++++
 drivers/tty/vt/vt.c                         |   93 +-
 include/linux/consolemap.h                  |    6 +
 7 files changed, 3875 insertions(+), 41 deletions(-)