mirror of
https://github.com/affaan-m/ECC.git
synced 2026-10-26 18:35:05 +00:00
fix(bengali): version-safe collation check, accurate \w guidance
The pg_collation check referenced colllocale unconditionally. That column
only exists on PostgreSQL 17+; 15-16 call it colliculocale and <= 14 keep
the ICU locale in collcollate, so the query failed with "column does not
exist" on anything older than 17. Read the columns through to_jsonb(c),
which tolerates missing keys. Verified against postgres:14, :16 and :17 —
all return icu_provider = t, locale = bn-BD.
The reviewer's regex rule flagged `\w` unconditionally. In Python str
patterns `\w` does match Bengali letters, but not vowel signs or hasanta
(category M), so `re.findall(r'\w+', 'রহিম')` gives ['রহ', 'ম']. Say
that precisely, note where `\w` is ASCII-only (JavaScript, re.ASCII), and
point at [ঀ-] or \p{Bengali} instead.
Co-Authored-By: Kishor Morol <kishormorol@gmail.com>
This commit is contained in:
1 parent
bcb23704a8
commit
6f967dcee4
2 files changed
+16
-8
No files matched your search
@@ -117,8 +117,12 @@ first_char = graphemes[0] # Returns 'ক্ষ' as a complete unit
|
||||
from Unicode code point order.
|
||||
- **Tokenization splitting on whitespace only** — Bengali has compound words and
|
||||
postposition attachments that require linguistic tokenization.
|
||||
- **Regex not covering Bengali range** — Character class patterns like `\w` or `[a-zA-Z]`
|
||||
that exclude Bengali characters when Bengali input is expected.
|
||||
- **Regex not covering Bengali range** — Character classes like `[a-zA-Z]` that
|
||||
exclude Bengali characters when Bengali input is expected, and `\w` used to match
|
||||
Bengali words. `\w` is ASCII-only in JavaScript (with or without the `u` flag) and
|
||||
in Python with `re.ASCII`. In Python `str` patterns it matches Bengali letters but
|
||||
not vowel signs or hasanta (category M), so `re.findall(r'\w+', 'রহিম')` returns
|
||||
`['রহ', 'ম']`. Recommend `[ঀ-]` or the `regex` module's `\p{Bengali}`.
|
||||
- **Bengali digits not handled** — Validation or parsing that rejects Bengali digits
|
||||
(০-৯) when users may input them.
|
||||
|
||||
|
||||
@@ -339,12 +339,16 @@ CREATE TABLE bengali_content (
|
||||
-- FROM pg_database WHERE datname = current_database();
|
||||
--
|
||||
-- 2. The exact collation must exist with the right provider and locale.
|
||||
-- `collcollate` holds the locale on PG < 15; `colllocale` holds it on PG 15+,
|
||||
-- so check both and require a non-null match:
|
||||
-- SELECT collprovider = 'i' AS icu_provider,
|
||||
-- coalesce(colllocale, collcollate) AS locale
|
||||
-- FROM pg_collation
|
||||
-- WHERE collname = 'bn-BD-x-icu';
|
||||
-- The ICU locale column moved between releases: `collcollate` on PG <= 14,
|
||||
-- `colliculocale` on PG 15-16, `colllocale` on PG 17+. Naming a column the
|
||||
-- server lacks is an error, so read them through to_jsonb(), which works on
|
||||
-- every version, and require a non-null match:
|
||||
-- SELECT c.collprovider = 'i' AS icu_provider,
|
||||
-- coalesce(to_jsonb(c) ->> 'colllocale',
|
||||
-- to_jsonb(c) ->> 'colliculocale',
|
||||
-- c.collcollate) AS locale
|
||||
-- FROM pg_collation c
|
||||
-- WHERE c.collname = 'bn-BD-x-icu';
|
||||
--
|
||||
-- A name-only `CREATE COLLATION IF NOT EXISTS` is not enough: it silently keeps
|
||||
-- an existing collation that may be bound to a different provider or locale.
|
||||
|
||||
Reference in new issue
Block a user