fix(bengali): version-safe collation check, accurate \w guidance

The pg_collation check referenced colllocale unconditionally. That column
only exists on PostgreSQL 17+; 15-16 call it colliculocale and <= 14 keep
the ICU locale in collcollate, so the query failed with "column does not
exist" on anything older than 17. Read the columns through to_jsonb(c),
which tolerates missing keys. Verified against postgres:14, :16 and :17 —
all return icu_provider = t, locale = bn-BD.

The reviewer's regex rule flagged `\w` unconditionally. In Python str
patterns `\w` does match Bengali letters, but not vowel signs or hasanta
(category M), so `re.findall(r'\w+', 'রহিম')` gives ['রহ', 'ম']. Say
that precisely, note where `\w` is ASCII-only (JavaScript, re.ASCII), and
point at [ঀ-৿] or \p{Bengali} instead.

Co-Authored-By: Kishor Morol <kishormorol@gmail.com>
This commit is contained in:
Md. Kishor MorolandKishor Morol committed 2026-09-17 08:28:24 -04:00
1 parent bcb23704a8
commit 6f967dcee4
2 files changed
+16 -8

No files matched your search

+6 -2
View File
@@ -117,8 +117,12 @@ first_char = graphemes[0] # Returns 'ক্ষ' as a complete unit
from Unicode code point order.
- **Tokenization splitting on whitespace only** — Bengali has compound words and
postposition attachments that require linguistic tokenization.
- **Regex not covering Bengali range** — Character class patterns like `\w` or `[a-zA-Z]`
that exclude Bengali characters when Bengali input is expected.
- **Regex not covering Bengali range** — Character classes like `[a-zA-Z]` that
exclude Bengali characters when Bengali input is expected, and `\w` used to match
Bengali words. `\w` is ASCII-only in JavaScript (with or without the `u` flag) and
in Python with `re.ASCII`. In Python `str` patterns it matches Bengali letters but
not vowel signs or hasanta (category M), so `re.findall(r'\w+', 'রহিম')` returns
`['রহ', 'ম']`. Recommend `[ঀ-৿]` or the `regex` module's `\p{Bengali}`.
- **Bengali digits not handled** — Validation or parsing that rejects Bengali digits
(০-৯) when users may input them.
+10 -6
View File
@@ -339,12 +339,16 @@ CREATE TABLE bengali_content (
-- FROM pg_database WHERE datname = current_database();
--
-- 2. The exact collation must exist with the right provider and locale.
-- `collcollate` holds the locale on PG < 15; `colllocale` holds it on PG 15+,
-- so check both and require a non-null match:
-- SELECT collprovider = 'i' AS icu_provider,
-- coalesce(colllocale, collcollate) AS locale
-- FROM pg_collation
-- WHERE collname = 'bn-BD-x-icu';
-- The ICU locale column moved between releases: `collcollate` on PG <= 14,
-- `colliculocale` on PG 15-16, `colllocale` on PG 17+. Naming a column the
-- server lacks is an error, so read them through to_jsonb(), which works on
-- every version, and require a non-null match:
-- SELECT c.collprovider = 'i' AS icu_provider,
-- coalesce(to_jsonb(c) ->> 'colllocale',
-- to_jsonb(c) ->> 'colliculocale',
-- c.collcollate) AS locale
-- FROM pg_collation c
-- WHERE c.collname = 'bn-BD-x-icu';
--
-- A name-only `CREATE COLLATION IF NOT EXISTS` is not enough: it silently keeps
-- an existing collation that may be bound to a different provider or locale.