From pgsql-patches-owner+M16987@postgresql.org Thu Aug  4 17:35:52 2005
Return-path: <pgsql-patches-owner+M16987@postgresql.org>
Received: from svr1.postgresql.org (svr1.postgresql.org [200.46.204.71])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j74LZot06577
	for <pgman@candle.pha.pa.us>; Thu, 4 Aug 2005 17:35:50 -0400 (EDT)
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id ED433529DB;
	Thu,  4 Aug 2005 18:35:44 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 70225-08; Thu,  4 Aug 2005 21:35:44 +0000 (GMT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by svr1.postgresql.org (Postfix) with ESMTP id 0ECB452998;
	Thu,  4 Aug 2005 18:35:44 -0300 (ADT)
X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org
Received: from localhost (unknown [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id AFAF752840
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>; Thu,  4 Aug 2005 18:33:25 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 70920-01
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>;
	Thu,  4 Aug 2005 21:33:13 +0000 (GMT)
Received: from mx-2.sollentuna.net (mx-2.sollentuna.net [195.84.163.199])
	by svr1.postgresql.org (Postfix) with ESMTP id E1C935280D
	for <pgsql-patches@postgresql.org>; Thu,  4 Aug 2005 18:33:11 -0300 (ADT)
Received: from ALGOL.sollentuna.se (janus.sollentuna.se [62.65.68.67])
	by mx-2.sollentuna.net (Postfix) with ESMTP id 7F5A08F289
	for <pgsql-patches@postgresql.org>; Thu,  4 Aug 2005 23:33:13 +0200 (CEST)
content-class: urn:content-classes:message
MIME-Version: 1.0
Content-Type: multipart/mixed;
	boundary="----_=_NextPart_001_01C5993C.1E1CB100"
Subject: [PATCHES] FW: Win32 unicode vs ICU
X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0
Date: Thu, 4 Aug 2005 23:33:12 +0200
Message-ID: <6BCB9D8A16AC4241919521715F4D8BCE094656@algol.sollentuna.se>
X-MS-Has-Attach: yes
Thread-Topic: Win32 unicode vs ICU
Thread-Index: AcWVyKf6M9GrCqZAQGeSvCZNg9n/jgDc1WwQ
From: "Magnus Hagander" <mha@sollentuna.net>
To: "PostgreSQL-patches" <pgsql-patches@postgresql.org>
X-Virus-Scanned: by amavisd-new at hub.org
X-Mailing-List: pgsql-patches
List-Archive: <http://archives.postgresql.org/pgsql-patches>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-patches.postgresql.org>
List-Owner: <mailto:pgsql-patches-owner@postgresql.org>
List-Post: <mailto:pgsql-patches@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-patches>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-patches>
Precedence: bulk
Sender: pgsql-patches-owner@postgresql.org
X-Virus-Scanned: by amavisd-new at hub.org
Status: ORr

This is a multi-part message in MIME format.

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: text/plain;
	charset="us-ascii"
Content-Transfer-Encoding: quoted-printable

I just realised this mail didn't go through. Probably because it was too
large for -hackers. So: repost to -patches. Sorry about that. If it's a
duplicate, even more sorry, but I couldn't find it in the archives.

(This may explain that nobody answered me :P)

//Magnus
=20

> -----Original Message-----
> From: Magnus Hagander=20
> Sent: Sunday, July 31, 2005 2:09 PM
> To: PostgreSQL-development
> Cc: pgsql-hackers-win32@postgresql.org
> Subject: Win32 unicode vs ICU
>=20
> Hi!
>=20
> I've been working with Palles ICU patch to make it work on=20
> win32, and I believe I have it done. While doing it I noticed=20
> that ICU basically converts to UTF16 and back - I previously=20
> thought it worked on UTF8 strings. Based on this I also tried=20
> out an implementation for the win32-unicode problem that does=20
> *not* require ICU. It uses the win32 native functions to map=20
> to utf16 and back, and then to process the text there. And I=20
> got through with much less code than the ICU version, while=20
> doing the same thing.
>=20
> I am unsure of how to proceed. As I see it there are three paths:
> 1) Use native win32 functionality only on win32
> 2) Use ICU functionality only on win32
> 3) Allow both ICU and native functionality, compile time=20
> switch --with-icu (same as unix with the ICU patch)
>=20
>=20
> The main downsides of ICU vs the native ones are:
> * ICU does not accept win32 locale names. When doing=20
> setlocale("sv_se"), for example, win32 will return this in=20
> later calls as "Swedish_Sweden.1252". To get around this in=20
> the ICU patch, I had to implement a lookup map that converts=20
> it back to sv_se for ICU.
>=20
> * ICU is yet another build and runtime dependency, and a=20
> large one (comes in at 11Mb for the DLL files alone in the=20
> win32 download)
>=20
>=20
> I guess that the main upside of it is that we'd get=20
> constistent behaviour - in case there are issues with either=20
> ICU or win32 native they'd otherwise differ. And only one new=20
> codepath. But we already live with the platform-inconsistency today...
>=20
> Another upside is that it handles more encodings in ICU - my=20
> native implementation does *only* UTF8 and relies on existing=20
> functionality to deal with other encodings. It could of=20
> course be extended if necessary, but from what I can tell=20
> UTF8 is the big one.
>=20
>=20
>=20
> I have attached both patches. For the native version, only=20
> win32_utf8.patch is required.  For the ICU version,=20
> icu_win32.patch is needed and also the files=20
> localemap.c,localemap.pl, iso639 and iso3166 needs to go in=20
> src/backend/port/win32. (the localemap needs to be updated to=20
> do a better-than-linear search, but I wanted to include an example)
>=20
>=20
> Thoughts on the options?
>=20
>=20
> And anohter question - my native patch touches the same=20
> functions as the ICU patch. Can somebody who knows the=20
> internals confirm or deny that these are all the required=20
> locations, or do we need to modify more?
>=20
> (I have run simple tests in swedish locale and both behave=20
> the same and correct, but I'm unsure of exactly how much=20
> would be affected)
>=20
> Finally, the win32 patch also changes the normal path to use=20
> strncoll(). The comment above the function states that we'd=20
> like to use strncoll but it's not available. Well, on win32=20
> it is, so it should provide a speedup on win32. It is=20
> currently not included in the ICU patch, but should probably=20
> be included whichever path we'd chose.
>=20
>=20
> //Magnus
>=20

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="win32_utf8.patch"
Content-Transfer-Encoding: base64
Content-Description: win32_utf8.patch
Content-Disposition: attachment;
	filename="win32_utf8.patch"

[encoded data, 8113 characters]
KyAxKTsNCg==

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="icu_win32.patch"
Content-Transfer-Encoding: base64
Content-Description: icu_win32.patch
Content-Disposition: attachment;
	filename="icu_win32.patch"

[encoded data, 30256 characters]
MzJfbG9jYWxlbWFwKGNoYXIgKndpbmxvY2FsZSk7DQo=

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="localemap.pl"
Content-Transfer-Encoding: base64
Content-Description: localemap.pl
Content-Disposition: attachment;
	filename="localemap.pl"

IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMNCiMgbG9jYWxlbWFwLnBs
IC0tIEJ1aWxkIGxvY2FsZSBtYXAgYmFzZWQgb24gc291cmNlIGRhdGEgZmls
ZXMNCiMNCiMgQ29weXJpZ2h0IChjKSAyMDA1LCBQb3N0Z3JlU1FMIEdsb2Jh
bCBEZXZlbG9wbWVudCBHcm91cA0KIw0KIyAkUG9zdGdyZVNRTCQNCiMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMj
IyMjIyMjIyMjIyMjIyMjIyMjIyMjIyMjDQojDQojIFRoZSBmaWxlIGlzbzYz
OSBjb250YWlucyBJU08tNjM5IGxhbmd1YWdlIGNvZGVzLCBjdXQtYW5kLXBh
c3RlIGZyb20NCiMgaHR0cDovL3d3dy53My5vcmcvV0FJL0VSL0lHL2VydC9p
c282MzkuaHRtLg0KIw0KIyBUaGUgZmlsZSBpc28zMTY2IGNvbnRhaW5zIElT
Ty0zMTY2IGNvdW50cnkgY29kZXMsIGN1dC1hbmQtcGFzdGUgZnJvbQ0KIyBo
dHRwOi8vd3d3Lm9hc2lzLW9wZW4ub3JnL2NvdmVyL2NvdW50cnkzMTY2Lmh0
bWwsDQojDQoNCm9wZW4oSSwiaXNvNjM5IikgfHwgZGllICJDb3VsZCBub3Qg
b3BlbiBpc282MzlcbiI7DQp3aGlsZSAoPEk+KSB7DQoJY2hvbXA7DQoJZGll
ICJNYWxmb3JtYXR0ZWQgbGluZSBpbiBpc282MzkiIHVubGVzcyAvXihbQS1a
XXsyfSkgXCIoLispXCIkLzsNCgkkY29kZSA9IGxjICQxOw0KCSRsYW5nID0g
bGMgJDI7DQoJaWYgKCRsYW5nID1+IC9eKC4rKVwiIC8pIHsgJGxhbmcgPSAk
MTsgfQ0KCSRpc282Mzl7JGxhbmd9PSRjb2RlOw0KfQ0KY2xvc2UoSSk7DQoN
Cm9wZW4oSSwiaXNvMzE2NiIpIHx8IGRpZSAiQ291bGQgbm90IG9wZW4gaXNv
MzE2NlxuIjsNCndoaWxlICg8ST4pIHsNCgljaG9tcDsNCglkaWUgIk1hbGZv
cm1hdHRlZCBsaW5lIGluIGlzbzMxNjYiIHVubGVzcyAvXihbQS1aXXsyfSlc
cysoLispJC87DQoJJGNvZGUgPSBsYyAkMTsNCgkkY291bnRyeSA9IGxjICQy
Ow0KCSRpc28zMTY2eyRjb3VudHJ5fSA9ICRjb2RlOw0KfQ0KY2xvc2UoSSk7
DQoNCnByaW50IDw8RU9GOw0KdHlwZWRlZiBzdHJ1Y3Qgew0KCWNoYXIgKmtl
eTsNCgljaGFyICp2YWw7DQp9IF9rZXl2YWw7DQoNCnN0YXRpYyBfa2V5dmFs
IGlzbzYzOVtdID0gew0KRU9GDQpmb3IgJGsgKHNvcnQga2V5cyAlaXNvNjM5
KSB7DQoJcHJpbnQgIiB7XCIka1wiLFwiJGlzbzYzOXska31cIn0sXG4iOw0K
fQ0KcHJpbnQgIiB7TlVMTCxOVUxMfVxufTtcblxuIjsNCg0KcHJpbnQgInN0
YXRpYyBfa2V5dmFsIGlzbzMxNjZbXSA9IHtcbiI7DQpmb3IgJGsgKHNvcnQg
a2V5cyAlaXNvMzE2Nikgew0KCXByaW50ICIge1wiJGtcIixcIiRpc28zMTY2
eyRrfVwifSxcbiI7DQp9DQoNCnByaW50ICIge05VTEwsTlVMTH1cbn07XG5c
biI7DQo=

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: application/octet-stream;
	name="localemap.c"
Content-Transfer-Encoding: base64
Content-Description: localemap.c
Content-Disposition: attachment;
	filename="localemap.c"

LyoNCiAqIGxvY2FsZW1hcCAtIG1hcCB3aW4zMiBsb2NhbGUgbmFtZXMgdG8g
SUNVIHN0YW5kYXJkIG5hbWVzDQogKg0KICogQ29weXJpZ2h0IChDKSAyMDA1
LCBQb3N0Z3JlU1FMIEdsb2JhbCBEZXZlbG9wbWVudCBHcm91cA0KICoNCiAq
ICRQb3N0Z3JlU1FMJA0KICovDQojaW5jbHVkZSAicG9zdGdyZXMuaCIgDQoN
CiNpbmNsdWRlICJsb2NhbGVtYXAuaCINCg0KDQpzdGF0aWMgY2hhciAqbWF0
Y2hfbGFuZ3VhZ2UoY2hhciAqd2lubGFuZ3VhZ2UpDQp7DQoJX2tleXZhbCAq
a3Y7DQoNCglmb3IgKGt2ID0gaXNvNjM5OyBrdi0+a2V5OyBrdisrKSB7DQoJ
CWlmICghc3RyY2FzZWNtcChrdi0+a2V5LCB3aW5sYW5ndWFnZSkpDQoJCQly
ZXR1cm4ga3YtPnZhbDsNCgl9DQoJZXJlcG9ydChFUlJPUiwNCgkJCShlcnJt
c2coImNvdWxkIG5vdCBtYXRjaCBsYW5ndWFnZSBmb3IgJXMiLCB3aW5sYW5n
dWFnZSkpKTsNCglyZXR1cm4gTlVMTDsNCn0NCg0Kc3RhdGljIGNoYXIgKm1h
dGNoX2NvdW50cnkoY2hhciAqd2luY291bnRyeSkNCnsNCglfa2V5dmFsICpr
djsNCg0KCWZvciAoa3YgPSBpc28zMTY2OyBrdi0+a2V5OyBrdisrKSB7DQoJ
CWlmICghc3RyY2FzZWNtcChrdi0+a2V5LCB3aW5jb3VudHJ5KSkNCgkJCXJl
dHVybiBrdi0+dmFsOw0KCX0NCglyZXR1cm4gTlVMTDsNCn0NCg0KLyoNCiAq
IEF0dGVtcHQgdG8gY29udmVydCBhIHdpbjMyIGxvY2FsZSAoU3dlZGlzaF9T
d2VkZW4uMTI1MikNCiAqIHRvIElDVSBmb3JtYXQgKHN2X3NlKQ0KICovDQpz
dGF0aWMgY2hhciBvdXRidWZbMTZdOw0KY2hhciAqcGd3aW4zMl9sb2NhbGVt
YXAoY2hhciAqd2lubG9jYWxlKQ0Kew0KCWNoYXIgYnVmWzI1Nl07DQoJY2hh
ciAqdW5kZXJzY29yZTsNCgljaGFyICpkb3Q7DQoJY2hhciAqY291bnRyeTsN
Cg0KCWlmIChzdHJsZW4od2lubG9jYWxlKT4yNTUpIA0KCQllcmVwb3J0KEVS
Uk9SLA0KCQkJCShlcnJtc2dfaW50ZXJuYWwoImxvY2FsZSBuYW1lIHRvbyBs
b25nIikpKTsNCglzdHJjcHkoYnVmLHdpbmxvY2FsZSk7DQoNCgl1bmRlcnNj
b3JlID0gc3RyY2hyKGJ1ZiwnXycpOw0KCWlmICghdW5kZXJzY29yZSkNCgkJ
LyogT25seSBsYW5ndWFnZSBuYW1lICovDQoJCXJldHVybiBtYXRjaF9sYW5n
dWFnZShidWYpOw0KDQoJKnVuZGVyc2NvcmUgPSAwOw0KCXVuZGVyc2NvcmUr
KzsNCg0KCWRvdCA9IHN0cmNocih1bmRlcnNjb3JlLCcuJyk7DQoJaWYgKGRv
dCkNCgkJLyogSWYgY29kZXBhZ2UgaXMgaW5jbHVkZWQsIHdlIGp1c3QgaWdu
b3JlIGl0LiAqLw0KCQkqZG90ID0gMDsNCgkNCgljb3VudHJ5ID0gbWF0Y2hf
Y291bnRyeSh1bmRlcnNjb3JlKTsNCglpZiAoIWNvdW50cnkpDQoJCXJldHVy
biBtYXRjaF9sYW5ndWFnZShidWYpOw0KDQoJc3ByaW50ZihvdXRidWYsIiVz
XyVzIixtYXRjaF9sYW5ndWFnZShidWYpLGNvdW50cnkpOw0KCXJldHVybiBv
dXRidWY7DQp9DQo=

------_=_NextPart_001_01C5993C.1E1CB100
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org

------_=_NextPart_001_01C5993C.1E1CB100--

From pgsql-hackers-owner+M72011@postgresql.org Wed Aug 24 15:37:17 2005
Return-path: <pgsql-hackers-owner+M72011@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j7OJbH103190
	for <pgman@candle.pha.pa.us>; Wed, 24 Aug 2005 15:37:17 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id D5152B21F6B;
	Wed, 24 Aug 2005 19:37:12 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 34C03D7357
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Wed, 24 Aug 2005 15:53:55 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 85477-03
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Wed, 24 Aug 2005 18:53:52 +0000 (GMT)
Received: from rodrick.geeknet.com.au (ns1.geeknet.com.au [220.244.63.182])
	by svr1.postgresql.org (Postfix) with ESMTP id 7C774D78F9
	for <pgsql-hackers@postgresql.org>; Wed, 24 Aug 2005 15:53:48 -0300 (ADT)
Subject: Re: [HACKERS] FreeBSD ICU  was Win32 unicode vs ICU 
MIME-Version: 1.0
Date: Thu, 25 Aug 2005 04:53:47 +1000
Content-Type: text/plain;
	charset="us-ascii"
Message-ID: <5066E5A966339E42AA04BA10BA706AE50A938F@rodrick.geeknet.com.au>
content-class: urn:content-classes:message
X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0
Thread-Topic: [HACKERS] FreeBSD ICU  was Win32 unicode vs ICU 
Thread-Index: AcWo0l6W575cyph2Twi7Fhfaa+w/tAACptMA
From: "John Hansen" <john@geeknet.com.au>
To: "Kevin McArthur" <postgresql-list@stormtide.ca>,
   <pgsql-hackers@postgresql.org>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.017 required=5 tests=[AWL=0.017]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Content-Transfer-Encoding: 8bit
X-MIME-Autoconverted: from quoted-printable to 8bit by candle.pha.pa.us id j7OJbH103190
Status: OR

Kevin McArthur Wrote:

> Should the postgresql project also be looking at CLDR for 
> cross-platform unicode support?

Afaict, from the ICU website, ICU too uses CLDR.
Why reinvent the wheel?

... John

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72581@postgresql.org Sat Sep  3 16:47:42 2005
Return-path: <pgsql-hackers-owner+M72581@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Klg129281
	for <pgman@candle.pha.pa.us>; Sat, 3 Sep 2005 16:47:42 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id CA484B21F00;
	Sat,  3 Sep 2005 20:47:36 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 354A7D8B35
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sat,  3 Sep 2005 17:35:06 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 55971-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sat,  3 Sep 2005 20:34:58 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 4AFECD8AE7
	for <pgsql-hackers@postgresql.org>; Sat,  3 Sep 2005 17:34:56 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1EBeim-0002xI-00; Sun, 04 Sep 2005 06:34:40 +1000
Date: Sat, 3 Sep 2005 22:34:40 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Greg Stark <gsstark@mit.edu>, pgsql-hackers@postgresql.org
Subject: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
Message-ID: <20050903203434.GA4281@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="mP3DRpeJDSE+ciuQ"
Content-Disposition: inline
In-Reply-To: <14696.1125675741@sss.pgh.pa.us>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--mP3DRpeJDSE+ciuQ
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Fri, Sep 02, 2005 at 11:42:21AM -0400, Tom Lane wrote:
> The objection is fundamentally that a platform-specific implementation
> cannot be our long-term goal, and so expending effort on creating one
> seems like a diversion.  If there were a plan put forward showing how
> this is just a useful way-station, and we could see how we'd later get
> rid of the glibc dependency without throwing away the work already done,
> then it would be a different story.

Well, my patch showed that useful locale work can be acheived with
precisely two functions: newlocale and strxfrm_l.

I'm going to talk about two things: one, the code from Apple. Two, how
we present locale support to users.
---
Now, it would be really nice to take Apple's implementation in Darwin
and use that. What I don't understand is the licence of the code in
Darwin. My interpretation is that stuff in:

http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/

is Apple stuff under APSL, useless to us. And that stuff in:

http://darwinsource.opendarwin.org/10.4.2/Libc-391/locale/FreeBSD/

are just patches to FreeBSD and this under the normal BSD license (no
big header claiming the licence change). The good news is that the
majority of what we need is in patch form. The bad news is that the hub
of the good stuff (newlocale, duplocale, freelocale) is under a big fat
APSL licence.

Does anyone know if this code can be used at all by BSD projects or did
they blanket relicence everything?
---
Now, I want to bring up some points relating to including a locale
library in PostgreSQL. Given that none of the BSDs seem really
interested in fixing the issue we'll have to do it ourselves (I don't
see anyone else doing it). We can save ourselves effort by basing it on
FreeBSDs locale code, because then we can use their datafiles, which we
*definitly* don't want to maintain ourselves. Now:

1. FreeBSDs locale list is short, some 48 compared with glibc's 217.
Hopefully Apple can expand on that in a way we can use. But given the
difference we should probably give people a way of falling back to the
system libraries in case there's a locale we don't support.

On the other hand, lots of locales are similar so maybe people can find
ones close enough to work. No, glibc and FreeBSD use different file
formats, so you can't copy them.

Do we want this locale data just for collation, or do we want to be
able to use it for formatting monetary amounts too? This is even more
info to store. Lots of languages use ISO/IEC 14651 for order.

2. Locale data needs to be combined with a charset and compiled to work
with the library. PostgreSQL supports at least 15 charsets but we don't
want to ship compiled versions of all of these (Debian learnt that the
hard way). So, how do we generate the files people need.

  a. Auto-compile on demand. First time a locale is referenced spawn
the compiler to create the locale, then continue. (Ugh)
  b. Add a CREATE LOCALE english AS 'en_US' WITH CHARSET 'utf8'. Then
require the COLLATE clause to refer to this identifier. This has some
appeal, seperating the system names from the PostgreSQL names. It also
gives some info regarding charsets.
  c. Should users be allowed to define new locales?
  d. Should admins be required to create the external files using a
program, say pg_createlocale.

Remember, if you use a latin1 locale to sort utf8 you'll get the wrong
result, so we want to avoid that.

3. Compiled locale files are large. One UTF-8 locale datafile can
exceed a megabyte. Do we want the option of disabling it for small
systems?

4. Do we want the option of running system locale in parallel with the
internal ones?

5. I think we're going to have to deal with the very real possibility
that our locale database will not be as good as some of the system
provided ones. The question is how. This is quite unlike timezones
which are quite standardized and rarely change. That database is quite
well maintained.

Would people object to a configure option that selected:
  --with-locales=3Dinternal     (use pg database)
  --with-locales=3Dsystem       (use system database for win32, glibc or Ma=
cOS X)
  --with-locales=3Dnone         (what we support now, which is neither)

I don't think it will be much of an issue to support this, all the
functions take the same parameters and have almost the same names.

6. Locales for SQL_ASCII. Seems to me you have two options, either
reject COLLATE altogether unless they specify a charset, or don't care
and let the user shoot themselves in the foot if they wish...

BTW, this MacOS locale supports seems to be new for 10.4.2 according to
the CVS log info, can anyone confirm this?

Anyway, I hope this post didn't bore too much. Locale support has been
one of those things that has bugged me for a long time and it would be
nice if there could be some real movement.

Have a nice weekend,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--mP3DRpeJDSE+ciuQ
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDGgjYIB7bNG8LQkwRAr6OAJ9uqNDKDQKWUAY4KiPAazHJ1TsVWwCeJ7sq
7hcILjdgZTQ4LjyPAhWnJwQ=
=5MT9
-----END PGP SIGNATURE-----

--mP3DRpeJDSE+ciuQ--

From pgsql-hackers-owner+M72583@postgresql.org Sat Sep  3 17:54:58 2005
Return-path: <pgsql-hackers-owner+M72583@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Lsv108133
	for <pgman@candle.pha.pa.us>; Sat, 3 Sep 2005 17:54:57 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 3325AB21F00;
	Sat,  3 Sep 2005 21:54:51 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id DA9AED8B14
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sat,  3 Sep 2005 18:44:52 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 88603-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sat,  3 Sep 2005 21:44:51 +0000 (GMT)
Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227])
	by svr1.postgresql.org (Postfix) with ESMTP id 01864D8B50
	for <pgsql-hackers@postgresql.org>; Sat,  3 Sep 2005 18:44:50 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=stark.xeocode.com)
	by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian))
	id 1EBfog-0004U8-00; Sat, 03 Sep 2005 17:44:50 -0400
To: Martijn van Oosterhout <kleptog@svana.org>
cc: Tom Lane <tgl@sss.pgh.pa.us>, Greg Stark <gsstark@mit.edu>,
   pgsql-hackers@postgresql.org
Subject: Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
References: <20050902130420.GA15466@svana.org>
	<87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us>
	<20050903203434.GA4281@svana.org>
In-Reply-To: <20050903203434.GA4281@svana.org>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 03 Sep 2005 17:44:50 -0400
Message-ID: <87oe79ygfh.fsf@stark.xeocode.com>
Lines: 39
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


Martijn van Oosterhout <kleptog@svana.org> writes:

> 2. Locale data needs to be combined with a charset and compiled to work
> with the library. PostgreSQL supports at least 15 charsets but we don't
> want to ship compiled versions of all of these (Debian learnt that the
> hard way). So, how do we generate the files people need.

That's just one of many lessons learned the hard way by distributions. Nor
will it be the last innovation in this area.

I really find this instinct of wanting to reimplement large swaths of the OS
inside Postgres (and annoying detail-ridden swaths that are hard to get right
and continually evolving too) to be a bad idea.

I can't believe it's harder to maintain an 

#ifdef HAVE_STRCOL_L
#else
#endif

than it is to try to maintain an entire independent locale library. Nor is it
simpler for sysadmins to have to maintain an entirely separate set of locales
independently from the system locales.

If you really are unhappy enough with OS setlocale implementations to want to
try to do this then it would be more helpful to do it outside of Postgres.
Package up the Apple setlocale library as a separate package that anyone can
install on Solaris, BSD, Linux or whatever. Then Postgres can just say "it
works fine with your OS library but your OS library might be very slow. Here's
a third-party library that you can install that is fast and may relieve any
problems you have with collation performance."

But I think that's getting ahead of things. Until Postgres even supports
collations using the OS libraries you won't even know if that's even
necessary.

-- 
greg


---------------------------(end of broadcast)---------------------------
TIP 4: Have you searched our list archives?

               http://archives.postgresql.org

From pgsql-hackers-owner+M72585@postgresql.org Sun Sep  4 08:44:01 2005
Return-path: <pgsql-hackers-owner+M72585@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84Ci0127486
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 08:44:01 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 8ED1DB21F00;
	Sun,  4 Sep 2005 12:43:55 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id D87D4D6E38
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 09:31:31 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 80362-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 12:31:29 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id C0F8DD6D8D
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 09:31:24 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1EBteM-0005wf-00; Sun, 04 Sep 2005 22:31:06 +1000
Date: Sun, 4 Sep 2005 14:31:05 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Greg Stark <gsstark@mit.edu>
cc: Tom Lane <tgl@sss.pgh.pa.us>, pgsql-hackers@postgresql.org
Subject: Re: Locale implementation questions (was: [HACKERS] Proof of concept COLLATE support with patch)
Message-ID: <20050904123103.GA21198@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <87oe79ygfh.fsf@stark.xeocode.com>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="EeQfGwPcQSOJBaQU"
Content-Disposition: inline
In-Reply-To: <87oe79ygfh.fsf@stark.xeocode.com>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--EeQfGwPcQSOJBaQU
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sat, Sep 03, 2005 at 05:44:50PM -0400, Greg Stark wrote:
> [...] Nor is it
> simpler for sysadmins to have to maintain an entirely separate set of loc=
ales
> independently from the system locales.

Indeed, I was already coming up with mechanisms to determine what
locales the system uses and try to autogenerate them. I agree though,
it's not useful for systems that already have complete locale support.
Why add to the burden?

Anyway, my reading of the specs says that we must support the syntax.
It doesn't say we need to support any orderings other than the default
(ie what we do now).

> If you really are unhappy enough with OS setlocale implementations to wan=
t to
> try to do this then it would be more helpful to do it outside of Postgres.
> Package up the Apple setlocale library as a separate package that anyone =
can
> install on Solaris, BSD, Linux or whatever. Then Postgres can just say "it
> works fine with your OS library but your OS library might be very slow. H=
ere's
> a third-party library that you can install that is fast and may relieve a=
ny
> problems you have with collation performance."

That's why I asked about the patches and files that Apple wrote. What
are the licence restrictions? Would we be able to download the, what,
20 files and distribute it as a library. Being APSL we couldn't include
it in the tarball, but it could be a pgfoundry project or something.

If somebody knows a reason why this could not be done, speak up now because
my reading of the APSL licence tells me it's fine.

> But I think that's getting ahead of things. Until Postgres even supports
> collations using the OS libraries you won't even know if that's even
> necessary.

Well, I added COLLATE support for ORDER BY and CREATE INDEX and it
worked in under 200 lines. I'm thinking ahead and I don't think the
COLLATE rules are that hard. Implementing them seems a bit fiddly. It
may be easiest to consider COLLATE a non-associative operator.

I'm still unsure if I should turn the string comparison operators into
three-argument functions.

Anyway, I'll look into the library issue first.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--EeQfGwPcQSOJBaQU
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDGukEIB7bNG8LQkwRAkeaAJ9rj5Tz7WPZpp+wWYWjTdjR68o8DwCfZzcZ
uB57noHmcnvefVoaw27bQ5Q=
=nLio
-----END PGP SIGNATURE-----

--EeQfGwPcQSOJBaQU--

From pgsql-hackers-owner+M72582@postgresql.org Sat Sep  3 17:46:57 2005
Return-path: <pgsql-hackers-owner+M72582@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j83Lkr106932
	for <pgman@candle.pha.pa.us>; Sat, 3 Sep 2005 17:46:57 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 6E192B21F00;
	Sat,  3 Sep 2005 21:46:47 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 7CF09D72B7
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sat,  3 Sep 2005 18:36:14 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 77080-09
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sat,  3 Sep 2005 21:36:13 +0000 (GMT)
Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227])
	by svr1.postgresql.org (Postfix) with ESMTP id 85FC0D6FEB
	for <pgsql-hackers@postgresql.org>; Sat,  3 Sep 2005 18:36:09 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=stark.xeocode.com)
	by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian))
	id 1EBfgA-0004RS-00; Sat, 03 Sep 2005 17:36:02 -0400
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Greg Stark <gsstark@mit.edu>, Martijn van Oosterhout <kleptog@svana.org>,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
References: <20050902130420.GA15466@svana.org>
	<87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us>
In-Reply-To: <14696.1125675741@sss.pgh.pa.us>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 03 Sep 2005 17:36:02 -0400
Message-ID: <87u0h1ygu5.fsf@stark.xeocode.com>
Lines: 36
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Tom Lane <tgl@sss.pgh.pa.us> writes:

> Greg Stark <gsstark@mit.edu> writes:
> > I still doesn't get where the hostility towards this functionality comes from.
> 
> We're not really willing to say "here is a piece of syntax REQUIRED
> BY THE SQL SPEC which we only support on some platforms".  readline,
> O_DIRECT, and the like are a completely inappropriate analogy, because
> those are inherently platform-dependent (and not in the spec).

But that's not the case at all. The syntax can be supported everywhere it
would just be somewhat faster on some platforms than others. It's already
reasonably fast on any platform that caches locale information which includes
glibc and presumably other free software libcs. It would be slightly faster if
there are _l functions. And much slower if the libc setlocale implementation
is braindead. But there's nothing wrong with saying "it's slow because your
libc is slow. Compile with this freely available library which has a better
implementation". The programming syntax would still be exactly 100% the same.

> The objection is fundamentally that a platform-specific implementation
> cannot be our long-term goal, and so expending effort on creating one
> seems like a diversion.  If there were a plan put forward showing how
> this is just a useful way-station, and we could see how we'd later get
> rid of the glibc dependency without throwing away the work already done,
> then it would be a different story.

It's not like the actual calls to setlocale are going to be much code. One day
presumably some variant of these _l functions will become entirely standard.
In which case you're talking about potentially "throwing away" 50 lines of
code. The bulk of the code is going to be parsing and implementing the actual
syntax and behaviour of the SQL spec. And in any case I wouldn't expect it to
ever get thrown away. There will be people compiling on RH9 or similar vintage
systems for a long time.

-- 
greg


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq

From pgsql-hackers-owner+M72589@postgresql.org Sun Sep  4 13:20:26 2005
Return-path: <pgsql-hackers-owner+M72589@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84HKQ121839
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 13:20:26 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id E4A85B21FBB;
	Sun,  4 Sep 2005 17:20:23 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id C4CA7D7AEA
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 14:07:02 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 63550-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 17:07:01 +0000 (GMT)
Received: from mail.gmx.net (mail.gmx.net [213.165.64.20])
	by svr1.postgresql.org (Postfix) with SMTP id 02A4FD7A6F
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 14:06:59 -0300 (ADT)
Received: (qmail invoked by alias); 04 Sep 2005 17:06:59 -0000
Received: from dsl-082-083-230-040.arcor-ip.net (EHLO colt.pezone.net) [82.83.230.40]
  by mail.gmx.net (mp033) with SMTP; 04 Sep 2005 19:06:59 +0200
X-Authenticated: #495269
From: Peter Eisentraut <peter_e@gmx.net>
To: pgsql-hackers@postgresql.org, Martijn van Oosterhout <kleptog@svana.org>
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
Date: Sun, 4 Sep 2005 19:06:57 +0200
User-Agent: KMail/1.8.1
References: <20050902130420.GA15466@svana.org>
In-Reply-To: <20050902130420.GA15466@svana.org>
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-ID: <200509041906.57721.peter_e@gmx.net>
X-Y-GMX-Trusted: 0
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.018 required=5 tests=[AWL=0.018]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Martijn van Oosterhout wrote:
> This is just a proof of concept patch. I didn't send it to -patches
> because as Tom pointed out, there's no hope of it getting in due to
> platform dependant behaviour.

I think it would be best if we defined an internal API for plugging in 
various kinds of locale support.  Then you can hook in this 
"newlocale", the Windows variant, ICU, or plain-old POSIX locale 
support for backward compatibility.  You already identified most of the 
API functions.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
       choose an index scan if your joining column's datatypes do not
       match

From pgsql-hackers-owner+M72592@postgresql.org Sun Sep  4 18:15:47 2005
Return-path: <pgsql-hackers-owner+M72592@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84MFk103432
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 18:15:46 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id E623DB21FBF;
	Sun,  4 Sep 2005 22:15:40 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id D671BD83B4
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 19:06:26 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 56030-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 22:06:23 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
	by svr1.postgresql.org (Postfix) with ESMTP id 44141D6F63
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 19:06:21 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84M6Boq008947;
	Sun, 4 Sep 2005 18:06:12 -0400 (EDT)
To: Greg Stark <gsstark@mit.edu>
cc: Martijn van Oosterhout <kleptog@svana.org>, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch 
In-Reply-To: <87u0h1ygu5.fsf@stark.xeocode.com> 
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com>
Comments: In-reply-to Greg Stark <gsstark@mit.edu>
	message dated "03 Sep 2005 17:36:02 -0400"
Date: Sun, 04 Sep 2005 18:06:11 -0400
Message-ID: <8946.1125871571@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Greg Stark <gsstark@mit.edu> writes:
> But there's nothing wrong with saying "it's slow because your
> libc is slow. Compile with this freely available library which has a better
> implementation".

The hole in that argument is the assumption that there *is* a freely
available library that can be used (where freely == BSD license).
We wouldn't be having this discussion if we knew of one.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72596@postgresql.org Sun Sep  4 19:27:26 2005
Return-path: <pgsql-hackers-owner+M72596@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84NRQ109968
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 19:27:26 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id A427211ED4E0;
	Sun,  4 Sep 2005 23:27:20 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 64BFED8B2F
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 20:15:22 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 08469-05
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 23:15:21 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
	by svr1.postgresql.org (Postfix) with ESMTP id CE6D9D8AD4
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 20:15:19 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84NFLVH009399;
	Sun, 4 Sep 2005 19:15:21 -0400 (EDT)
To: Peter Eisentraut <peter_e@gmx.net>
cc: pgsql-hackers@postgresql.org, Martijn van Oosterhout <kleptog@svana.org>
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch 
In-Reply-To: <200509041906.57721.peter_e@gmx.net> 
References: <20050902130420.GA15466@svana.org> <200509041906.57721.peter_e@gmx.net>
Comments: In-reply-to Peter Eisentraut <peter_e@gmx.net>
	message dated "Sun, 04 Sep 2005 19:06:57 +0200"
Date: Sun, 04 Sep 2005 19:15:21 -0400
Message-ID: <9398.1125875721@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Peter Eisentraut <peter_e@gmx.net> writes:
> I think it would be best if we defined an internal API for plugging in 
> various kinds of locale support.

Agreed ...

> Then you can hook in this 
> "newlocale", the Windows variant, ICU, or plain-old POSIX locale 
> support for backward compatibility.

If plain old POSIX actually did what we needed, we likely wouldn't be
having this discussion at all.  POSIX doesn't give us enough visibility
of the locale's properties (in particular, which character set encoding
it wants).  The performance penalties it imposes are pretty bad also,
though arguably secondary.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72599@postgresql.org Sun Sep  4 20:01:47 2005
Return-path: <pgsql-hackers-owner+M72599@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j8501k113218
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 20:01:46 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 5FCFD11ED4E1;
	Mon,  5 Sep 2005 00:01:40 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 3CE03D8A88
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 20:53:09 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 14738-01
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 23:53:08 +0000 (GMT)
Received: from gateway.pjmodos.bounceme.net (gprsplusa185.isp.t-mobile.cz [62.141.20.185])
	by svr1.postgresql.org (Postfix) with ESMTP id B5606D8A4E
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 20:53:03 -0300 (ADT)
Received: from modos ([10.12.0.96] ident=PJMODOS)
	by gateway.pjmodos.bounceme.net with esmtp (Exim 3.35 #1 (Debian))
	id 1EC4IE-0004Le-00; Mon, 05 Sep 2005 01:52:58 +0200
Message-ID: <431B88CD.8050904@seznam.cz>
Date: Mon, 05 Sep 2005 01:52:45 +0200
From: Petr Jelinek <pjmodos@seznam.cz>
User-Agent: Mozilla Thunderbird 1.0 (Windows/20041206)
X-Accept-Language: cs, en, en-us
MIME-Version: 1.0
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: PostgreSQL-development <pgsql-hackers@postgresql.org>
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> <8946.1125871571@sss.pgh.pa.us>
In-Reply-To: <8946.1125871571@sss.pgh.pa.us>
Content-Type: text/plain; charset=windows-1250; format=flowed
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=1.173 required=5 tests=[AWL=-0.253,
	DNS_FROM_RFC_POST=1.376, FORGED_RCVD_HELO=0.05]
X-Spam-Level: *
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Tom Lane wrote:
> 
> The hole in that argument is the assumption that there *is* a freely
> available library that can be used (where freely == BSD license).
> We wouldn't be having this discussion if we knew of one.

I see this discussion as another reason to use ICU, I mean complete 
rewrite of locale handling to use ICU on all platforms. I know it's big 
project but it's doable for 8.2 and it would virtually solve all locale 
problems and could be base for new unicode/locale features. I am not 
sure if this is the way postgres wants to go tho (having dependency on 
such a big and uncommon library).

-- 
Regards
Petr Jelinek (PJMODOS)


---------------------------(end of broadcast)---------------------------
TIP 6: explain analyze is your friend

From pgsql-hackers-owner+M72602@postgresql.org Mon Sep  5 04:02:37 2005
Return-path: <pgsql-hackers-owner+M72602@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j8582Z126563
	for <pgman@candle.pha.pa.us>; Mon, 5 Sep 2005 04:02:36 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 6640811EB014;
	Mon,  5 Sep 2005 08:02:28 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 1FC36D7841
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Mon,  5 Sep 2005 04:52:11 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 09506-02
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Mon,  5 Sep 2005 07:52:06 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 0DC42D77AE
	for <pgsql-hackers@postgresql.org>; Mon,  5 Sep 2005 04:52:06 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ECBlg-0001TS-00; Mon, 05 Sep 2005 17:51:52 +1000
Date: Mon, 5 Sep 2005 09:51:52 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Tom Lane <tgl@sss.pgh.pa.us>
cc: Greg Stark <gsstark@mit.edu>, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Proof of concept COLLATE support with patch
Message-ID: <20050905075152.GA5278@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <20050902130420.GA15466@svana.org> <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <87u0h1ygu5.fsf@stark.xeocode.com> <8946.1125871571@sss.pgh.pa.us>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="MGYHOYXEY6WxJCY8"
Content-Disposition: inline
In-Reply-To: <8946.1125871571@sss.pgh.pa.us>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--MGYHOYXEY6WxJCY8
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sun, Sep 04, 2005 at 06:06:11PM -0400, Tom Lane wrote:
> Greg Stark <gsstark@mit.edu> writes:
> > But there's nothing wrong with saying "it's slow because your
> > libc is slow. Compile with this freely available library which has a be=
tter
> > implementation".
>=20
> The hole in that argument is the assumption that there *is* a freely
> available library that can be used (where freely =3D=3D BSD license).
> We wouldn't be having this discussion if we knew of one.

1. Use the one in Darwin just for the *BSDs and Solaris at least. It's
not great but it would probably work.

2. Long term, transition to ICU (http://icu.sourceforge.net/) which is
the cross-platform internationalisation library used by Java. Looks
like Mono and Gnome/GTK are going to use this (or at least allow use
of) soon also. It uses the X licence AFAICS. It's a big pill right now
but it a year it could be installed standard on most linux systems.
It's at least avaiable everywhere now.

Note, it's not compatable with POSIX locales so if we go there it'll be
an all or nothing switch. But if we intend to go there eventually, it
makes fiddling on our own library a waste of time.

Incidently, I played with the code in Darwin and getting it to compile
on a system that already has extended locale support is, uh, tricky to
say the least. Lots of conflicting definitions.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--MGYHOYXEY6WxJCY8
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDG/kXIB7bNG8LQkwRAijSAJ4xv5JiigSwEe4jgBmxvqHN+GuOgQCfSEHp
C5tpavd58A8VEnf/BBUgfzo=
=gsNf
-----END PGP SIGNATURE-----

--MGYHOYXEY6WxJCY8--

From pgsql-hackers-owner+M72586@postgresql.org Sun Sep  4 09:37:44 2005
Return-path: <pgsql-hackers-owner+M72586@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84Dbh100551
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 09:37:43 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 824CCB21F00;
	Sun,  4 Sep 2005 13:37:37 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 413CCD7015
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 10:26:10 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 20529-10
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 13:26:07 +0000 (GMT)
Received: from sraigw.sra.co.jp (sraigw.sra.co.jp [202.32.10.2])
	by svr1.postgresql.org (Postfix) with ESMTP id 12EF2D6F75
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 10:26:06 -0300 (ADT)
Received: from srascb.sra.co.jp (srascb [133.137.8.65])
	by sraigw.sra.co.jp (Postfix) with ESMTP
	id 08EB262112; Sun,  4 Sep 2005 22:26:09 +0900 (JST)
Received: from srascb.sra.co.jp (localhost [127.0.0.1])
	by localhost.sra.co.jp (Postfix) with ESMTP
	id D1D9610CD06; Sun,  4 Sep 2005 22:26:09 +0900 (JST)
Received: from sranhm.sra.co.jp (sranhm [133.137.44.16])
	by srascb.sra.co.jp (Postfix) with ESMTP
	id ABF1210CD04; Sun,  4 Sep 2005 22:26:09 +0900 (JST)
Received: from localhost (sraihb-hub.sra.co.jp [133.137.8.6])
	by sranhm.sra.co.jp (8.9.3+3.2W/3.7W-srambox) with ESMTP id WAA12274;
	Sun, 4 Sep 2005 22:26:09 +0900
Date: Sun, 04 Sep 2005 22:25:36 +0900 (JST)
Message-ID: <20050904.222536.39155679.ishii@sraoss.co.jp>
To: kleptog@svana.org
cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions
From: Tatsuo Ishii <t-ishii@sra.co.jp>
In-Reply-To: <20050903203434.GA4281@svana.org>
References: <87k6hzzems.fsf@stark.xeocode.com>
	<14696.1125675741@sss.pgh.pa.us>
	<20050903203434.GA4281@svana.org>
X-Mailer: Mew version 2.3 on Emacs 20.7 / Mule 4.1
	=?iso-2022-jp?B?KBskQjAqGyhCKQ==?=
MIME-Version: 1.0
Content-Type: Text/Plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.028 required=5 tests=[AWL=0.028]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

> 3. Compiled locale files are large. One UTF-8 locale datafile can
> exceed a megabyte. Do we want the option of disabling it for small
> systems?

To avoid the problem, you could dynmically load the compiled
tables. The charset conversion tables are handled similar way.

Also I think it's important to allow user defined collate data. To
implement the CREATE COLLATE syntax, we need to have that capability
anyway.

> 4. Do we want the option of running system locale in parallel with the
> internal ones?
> 
> 5. I think we're going to have to deal with the very real possibility
> that our locale database will not be as good as some of the system
> provided ones. The question is how. This is quite unlike timezones
> which are quite standardized and rarely change. That database is quite
> well maintained.
> 
> Would people object to a configure option that selected:
>   --with-locales=internal     (use pg database)
>   --with-locales=system       (use system database for win32, glibc or MacOS X)
>   --with-locales=none         (what we support now, which is neither)
> 
> I don't think it will be much of an issue to support this, all the
> functions take the same parameters and have almost the same names.

To be honest, I don't understand why we have to rely on (often broken)
system locales. I don't think building our own locale data is too
hard, and once we make up it, the maintenace cost will be very small
since it should not be changed regularly. Moreover we could enjoy the
benefit that PostgreSQL handles collations in a corret manner on any
platform which PostgreSQL supports.

> 6. Locales for SQL_ASCII. Seems to me you have two options, either
> reject COLLATE altogether unless they specify a charset, or don't care
> and let the user shoot themselves in the foot if they wish...
> 
> BTW, this MacOS locale supports seems to be new for 10.4.2 according to
> the CVS log info, can anyone confirm this?
> 
> Anyway, I hope this post didn't bore too much. Locale support has been
> one of those things that has bugged me for a long time and it would be
> nice if there could be some real movement.

Right. We Japanese (and probably Chinese too) have been bugged by the
broken mutibyte locales for long time. Using C locale help us to a
certain extent, but for Unicode we need correct locale data, othewise
the sorted data will be completely chaos.
--
SRA OSS, Inc. Japan
Tatsuo Ishii

---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

From pgsql-hackers-owner+M72587@postgresql.org Sun Sep  4 11:13:07 2005
Return-path: <pgsql-hackers-owner+M72587@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84FD5107053
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 11:13:06 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id D1EC5B21F00;
	Sun,  4 Sep 2005 15:13:04 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 8823DD723A
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 12:01:47 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 65135-01
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 15:01:43 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 5896DD70F7
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 12:01:40 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1EBvzd-0006eP-00; Mon, 05 Sep 2005 01:01:13 +1000
Date: Sun, 4 Sep 2005 17:01:13 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: Tatsuo Ishii <t-ishii@sra.co.jp>
cc: tgl@sss.pgh.pa.us, gsstark@mit.edu, pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions
Message-ID: <20050904150055.GB21198@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
References: <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <20050904.222536.39155679.ishii@sraoss.co.jp>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="ZfOjI3PrQbgiZnxM"
Content-Disposition: inline
In-Reply-To: <20050904.222536.39155679.ishii@sraoss.co.jp>
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--ZfOjI3PrQbgiZnxM
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

On Sun, Sep 04, 2005 at 10:25:36PM +0900, Tatsuo Ishii wrote:
> > 3. Compiled locale files are large. One UTF-8 locale datafile can
> > exceed a megabyte. Do we want the option of disabling it for small
> > systems?
>=20
> To avoid the problem, you could dynmically load the compiled
> tables. The charset conversion tables are handled similar way.

That's not the point, ofcourse they are loaded dynamically. The
question is, when do we create the files in the first place. There are
48*15 =3D 750 combinations which would amount to tens of megabytes of
essentially useless data. *When* you create the files is an important
question. Compile time is out.

Charset conversion is completely different, there just arn't that many
combinations.

> Also I think it's important to allow user defined collate data. To
> implement the CREATE COLLATE syntax, we need to have that capability
> anyway.

Most OS's allow you to create collate data yourself anyway, why do we
need to implement this too?

> To be honest, I don't understand why we have to rely on (often broken)
> system locales. I don't think building our own locale data is too
> hard, and once we make up it, the maintenace cost will be very small
> since it should not be changed regularly. Moreover we could enjoy the
> benefit that PostgreSQL handles collations in a corret manner on any
> platform which PostgreSQL supports.

You say building our own locale data is not hard. I disagree, it's a
waste of time we can do without. Unless you know the language yourself
you cannot check changes made by anybody else. If there's an error in
locale ordering, take it up with your OS distributor.

I also think we open ourselves to questions like:

1. My locale is supported by the system but not by PostgreSQL, why?
2. My locale was supported last release but not this one, why?
3. Why does PostgreSQL sort differently from 'sort' or any other app on
my system?

> Right. We Japanese (and probably Chinese too) have been bugged by the
> broken mutibyte locales for long time. Using C locale help us to a
> certain extent, but for Unicode we need correct locale data, othewise
> the sorted data will be completely chaos.

Ok, is glibc still wrong or are they just implementing the unicode
standard and that's what's wrong.

All I'm saying is that we need to allow use of system locales until our
native locale support is mature. In the end something like ICU
(http://icu.sourceforge.net/) will end up obsoleting us. Nobody (in
free-software anyway) uses it yet, but eventually it may be viable to
require that to allow system independant locales.
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--ZfOjI3PrQbgiZnxM
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDGwwmIB7bNG8LQkwRAmKxAJ9MccMVTLVLqW7c9Lsa8FzqEMvNGgCdGFgg
nsvl/ZIAK/Qe1XpkMOItRFM=
=h0V5
-----END PGP SIGNATURE-----

--ZfOjI3PrQbgiZnxM--

From pgsql-hackers-owner+M72590@postgresql.org Sun Sep  4 15:15:31 2005
Return-path: <pgsql-hackers-owner+M72590@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84JFU107171
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 15:15:31 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 950F5B21FBE;
	Sun,  4 Sep 2005 19:15:26 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id AA4C5D85BF
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 16:02:50 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 51227-06
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 19:02:48 +0000 (GMT)
Received: from stark.xeocode.com (stark.xeocode.com [216.58.44.227])
	by svr1.postgresql.org (Postfix) with ESMTP id CC650D8580
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 16:02:46 -0300 (ADT)
Received: from localhost ([127.0.0.1] helo=stark.xeocode.com)
	by stark.xeocode.com with smtp (Exim 3.36 #1 (Debian))
	id 1EBzlN-0003LY-00; Sun, 04 Sep 2005 15:02:45 -0400
To: Tatsuo Ishii <t-ishii@sra.co.jp>
cc: kleptog@svana.org, tgl@sss.pgh.pa.us, gsstark@mit.edu,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions
References: <87k6hzzems.fsf@stark.xeocode.com>
	<14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org>
	<20050904.222536.39155679.ishii@sraoss.co.jp>
In-Reply-To: <20050904.222536.39155679.ishii@sraoss.co.jp>
From: Greg Stark <gsstark@mit.edu>
Organization: The Emacs Conspiracy; member since 1992
Date: 04 Sep 2005 15:02:45 -0400
Message-ID: <87fysky7u2.fsf@stark.xeocode.com>
Lines: 22
User-Agent: Gnus/5.09 (Gnus v5.9.0) Emacs/21.3
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.005 required=5 tests=[AWL=0.005]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


Tatsuo Ishii <t-ishii@sra.co.jp> writes:

> To be honest, I don't understand why we have to rely on (often broken)
> system locales. I don't think building our own locale data is too
> hard, and once we make up it, the maintenace cost will be very small
> since it should not be changed regularly. Moreover we could enjoy the
> benefit that PostgreSQL handles collations in a corret manner on any
> platform which PostgreSQL supports.

I think it's sheer madness to try to reproduce large swaths of the OS inside
Postgres because you're unhappy with the quality of the OS implementation. You
should be asking yourself why OS vendors have such a hard time getting this
stuff right and why would Postgres do any better. Wouldn't that work be better
spent improving the database functionality of Postgres?

Or at least better spent improving the locale support for the entire OS? It
would be positively awful if every application on my system had its own locale
database each of which had its own set of bugs and its own feature set.

-- 
greg


---------------------------(end of broadcast)---------------------------
TIP 3: Have you checked our extensive FAQ?

               http://www.postgresql.org/docs/faq

From pgsql-hackers-owner+M72597@postgresql.org Sun Sep  4 19:28:34 2005
Return-path: <pgsql-hackers-owner+M72597@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j84NSY110074
	for <pgman@candle.pha.pa.us>; Sun, 4 Sep 2005 19:28:34 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 7898611ED4E0;
	Sun,  4 Sep 2005 23:28:28 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id BE640D8910
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Sun,  4 Sep 2005 20:19:37 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 09430-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Sun,  4 Sep 2005 23:19:37 +0000 (GMT)
Received: from sss.pgh.pa.us (sss.pgh.pa.us [66.207.139.130])
	by svr1.postgresql.org (Postfix) with ESMTP id 7DE22D8864
	for <pgsql-hackers@postgresql.org>; Sun,  4 Sep 2005 20:19:34 -0300 (ADT)
Received: from sss2.sss.pgh.pa.us (tgl@localhost [127.0.0.1])
	by sss.pgh.pa.us (8.13.1/8.13.1) with ESMTP id j84NJclc009441;
	Sun, 4 Sep 2005 19:19:38 -0400 (EDT)
To: Greg Stark <gsstark@mit.edu>
cc: Tatsuo Ishii <t-ishii@sra.co.jp>, kleptog@svana.org,
   pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Locale implementation questions 
In-Reply-To: <87fysky7u2.fsf@stark.xeocode.com> 
References: <87k6hzzems.fsf@stark.xeocode.com> <14696.1125675741@sss.pgh.pa.us> <20050903203434.GA4281@svana.org> <20050904.222536.39155679.ishii@sraoss.co.jp> <87fysky7u2.fsf@stark.xeocode.com>
Comments: In-reply-to Greg Stark <gsstark@mit.edu>
	message dated "04 Sep 2005 15:02:45 -0400"
Date: Sun, 04 Sep 2005 19:19:38 -0400
Message-ID: <9440.1125875978@sss.pgh.pa.us>
From: Tom Lane <tgl@sss.pgh.pa.us>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.006 required=5 tests=[AWL=0.006]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

Greg Stark <gsstark@mit.edu> writes:
> I think it's sheer madness to try to reproduce large swaths of the OS
> inside Postgres because you're unhappy with the quality of the OS
> implementation. You should be asking yourself why OS vendors have such
> a hard time getting this stuff right

In the case of the *BSDs, it's pretty obviously because they don't care.

> and why would Postgres do any better

In the first place, we do care, and in the second place, having to deal
with only one set of locale bugs would in itself be a huge advance over
where we are now.

We went over to maintaining our own timezone code for more or less the
same reasons, and in hindsight that was obviously the right decision.
Locale support is a bigger chunk, no doubt about it, but we also have
a lot of motivation.

			regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 2: Don't 'kill -9' the postmaster

From pgsql-hackers-owner+M72619@postgresql.org Mon Sep  5 18:14:28 2005
Return-path: <pgsql-hackers-owner+M72619@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j85MER115087
	for <pgman@candle.pha.pa.us>; Mon, 5 Sep 2005 18:14:27 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 68D9F11EB015;
	Mon,  5 Sep 2005 22:14:19 +0000 (GMT)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 92268D6FB8
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Mon,  5 Sep 2005 18:59:42 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 29325-07
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Mon,  5 Sep 2005 21:59:38 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id 98C2CD7A97
	for <pgsql-hackers@postgresql.org>; Mon,  5 Sep 2005 18:59:33 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ECP02-0004SQ-00; Tue, 06 Sep 2005 07:59:34 +1000
Date: Mon, 5 Sep 2005 23:59:33 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] Install Darwin's locale library on your system :)
Message-ID: <20050905215928.GB5278@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="hHWLQfXTYDoKhP50"
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--hHWLQfXTYDoKhP50
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Well, it was pointed out the other day that the Darwin C library
supports the non-standard extensions to the POSIX locale interface and
that this might be ported to other systems so PostgreSQL could use it.

So, I have written a few scripts which download the libc and locale
library from darwinsource, shuffle some files around and build the
result into a library called libdummylocale.so. It basically completely
replaces your locale support on whatever system you use it on.

It's all under the APSL, though some parts may be BSD licenced.

Let me say right now, the locale support here sucks, no two ways about
it. It doesn't support a single UTF-8 locale. Oh, it lets you specify
them, but when you ask for the CHARSET it still says US-ASCII. It does
support a number of other different charsets. (Not for collation
though).

So my challenge to those people who think maintaining a locale library
is easy: make *one* locale in FreeBSD (or Darwin or this lib) support
full UTF-8 collation in whatever locale and/or charset you choose. It's
all downhill from there.

While it builds simple programs, I don't think it's totally safe. You'd
need to rename the headers at least. And building on Darwin will
probably blow up due to the way it plays fast and loose with Darwin
specific #defines. But it's a beginning if anyone is interested. It
builds in my glibc system.

I'm going to drop the idea of making a locale library, there's just
nothing good enough. glibc is the only thing that comes close. From here
on I'm going to work on COLLATE for systems that support xlocale, with
an eye on ICU if/when it becomes standard enough.

Download: http://svana.org/kleptog/pgsql/dummylocale.tar.gz

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--hHWLQfXTYDoKhP50
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDHL++IB7bNG8LQkwRAjvyAJ98Hwm6Ak7zSdEU7shrzxbvMJJp/QCfcJSq
ZseeCwJGWAs+3MEPEr8u2dM=
=rr4R
-----END PGP SIGNATURE-----

--hHWLQfXTYDoKhP50--

From pgsql-patches-owner+M17366@postgresql.org Wed Sep  7 12:16:02 2005
Return-path: <pgsql-patches-owner+M17366@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j87GG1104073
	for <pgman@candle.pha.pa.us>; Wed, 7 Sep 2005 12:16:01 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id D3FD711EB015;
	Wed,  7 Sep 2005 16:15:58 +0000 (GMT)
X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 326F8D7B55
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>; Wed,  7 Sep 2005 13:11:41 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 66946-04
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>;
	Wed,  7 Sep 2005 16:11:25 +0000 (GMT)
Received: from svana.org (svana.org [203.20.62.76])
	by svr1.postgresql.org (Postfix) with ESMTP id D88A6D6E3B
	for <pgsql-patches@postgresql.org>; Wed,  7 Sep 2005 13:11:20 -0300 (ADT)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ED2W5-00052g-00; Thu, 08 Sep 2005 02:11:17 +1000
Date: Wed, 7 Sep 2005 18:11:17 +0200
From: Martijn van Oosterhout <kleptog@svana.org>
To: pgsql-patches@postgresql.org
Subject: [PATCHES] For review: Initial support for COLLATE
Message-ID: <20050907161112.GA10273@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="ibTvN161/egqYuK8"
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.003 required=5 tests=[AWL=0.003]
X-Mailing-List: pgsql-patches
List-Archive: <http://archives.postgresql.org/pgsql-patches>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-patches.postgresql.org>
List-Owner: <mailto:pgsql-patches-owner@postgresql.org>
List-Post: <mailto:pgsql-patches@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-patches>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-patches>
Precedence: bulk
Sender: pgsql-patches-owner@postgresql.org
Status: OR


--ibTvN161/egqYuK8
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

[Please CC any replies, thanks]

This patch is the beginnings of support for COLLATE. I need to do some
other work for a few days so I'm posting here to get some initial
reviews. Various parts are marked [done] and [not done].

The steps involved are:

- Add COLLATE to grammer as part of expression tree. New CollateClause
  node stores the relevent facts. [done]

- parse_expr goes through tree and determines the appropriate COLLATE
  state for each node as per SQL spec. [done]

  Note: PostgreSQL doesn't really have a way to identify text-like
  types and there's no real need to anyway. The implementation allows
  any node to have a collate state. This can be used for other things,
  see below.

- CREATE COLLATE statement [not done, currently added on the fly]

- Two new datatypes, pg_locale_t and pg_localedata_t. The former
  represents a locale (and will eventually have an OID). The latter is
  an anonymous cookie that stores locale specific information and is
  passed to functions that need it. It may be that once done, this
  latter type will vanish again. [done]

- Several utility functions in the new file pg_xlocale.c for use by the
  rest of the system. [done]

- Add boolean column 'proislocalized' to pg_proc which indicates if the
  output of this function is affected by the LOCALE. eg textcat doesn't
  care but textle does. [done] This is for:

  a) So the parser can complain about functions that look at the
     locale/collate order but it's not clear from the arguments (state
     None per SQL spec) and it's not specified explicitly. [not done]

  b) So when a column or function is indexed, the index code knows if
     the locale is relevent to sorting order. This is particularly
     interesting for btree indexes on character strings. [not done]

  Currently I've marked 58 functions as being locale sensetive, but
  that list will need careful going over. To some extent it can be
  checked automatically by examining which backend functions use the
  new PG_GETLOCALE() macro.

- Check for correct encoding in loaded locales [not done]

- Check the partial indexes do the right thing when matching
  expressions. [not done]

- Docs, regression, etc...

- make check: right now I'm getting some regressions in the rules
  and plpgsql, very odd... Possibly due to the fact that rules get
  collate nodes with locales that don't persist across invokations.

Goals:

- This setup extends the SQL spec a bit, in the sense that COLLATE can
  be attached to anything. It is my intention to allow functions such
  as to_char() and to_timestamp() to be localized. eg:

test=3D# select cash_out('1.00'::money collate 'nl_NL'), cash_out('1.00'::m=
oney collate 'en_AU');
 cash_out | cash_out=20
----------+----------
 EUR1,00  | $1.00
(1 row)

- Should LOCALE be created as a synonym for COLLATE? It reads more
  naturally.

- Currently LC_COLLATE is fixed at initdb and LC_NUMERIC and
  LC_CURRENCY can be altered. The idea is that eventually even
  LC_COLLATE can be altered anytime (from the users point view anyway),
  as any objects that care will store the collate order they want and
  can execute functions as appropriate.

- Eventually once the transition to full locale support is complete,
  change the backend so it always runs under locale C and only
  functions on userdata are affected by locales. This means that
  unquoted identifiers when converted to lowercase will be lowered as
  per ASCII rules. Judging by [1] this is what people want, but
  feedback would be nice.

[1] http://www.mail-archive.com/pgsql-hackers@postgresql.org/msg39742.html

  Hence, right now, names compare using normal strcmp, but text,
  varchar, etc use strcoll.

- The patch as it currently stands won't compile at all without system
  xlocale support. The goal is to provide some level of backward
  compatability where the COLLATE clause can be used, but only to
  affect functions like to_char(). Changing COLLATE order would be
  forbidden (ie just like now).

- Eventually, once the parts relevent to locales are sufficiently
  abstracted, look into ICU to plug it in. Unfortunatly, it's a
  completely different model (all utf-16) so that's another phase
  altogether.

- The default type output functions should never be locale specific.
  This is to avoid issues with pgdump and frontends. Create a function
  'localize(anyelement)' to "do the obvious" to force it to happen.

Download:=20
  Compressed 57K, uncompressed >500K but that's due to rewriting the
  whole of pg_proc. The important code is not so big. Against todays
  CVS.

http://svana.org/kleptog/pgsql/collate2.patch.gz

Examples:

test=3D# SELECT text('a') < text('B') COLLATE 'C', text('a') < text('B') CO=
LLATE 'en_US.UTF-8';
 ?column? | ?column?=20
----------+----------
 f        | t
(1 row)

test=3D# SELECT text('A') < text('b') COLLATE 'C', text('A') < text('b') CO=
LLATE 'en_US.UTF-8';
 ?column? | ?column?=20
----------+----------
 t        | t
(1 row)

test=3D# SELECT text('A') COLLATE 'en_US.UTF-8' < text('b') COLLATE 'C';
ERROR:  Conflicting COLLATE clauses

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--ibTvN161/egqYuK8
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDHxEgIB7bNG8LQkwRAqTqAJ0TOH3RkxnYHJmCPhjjYha0idsEoACfTjY0
EP4a7CtIaZ0Ar0dyuPja3h8=
=8zan
-----END PGP SIGNATURE-----

--ibTvN161/egqYuK8--

From pgsql-patches-owner+M17368@postgresql.org Wed Sep  7 16:31:39 2005
Return-path: <pgsql-patches-owner+M17368@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id j87KVd101436
	for <pgman@candle.pha.pa.us>; Wed, 7 Sep 2005 16:31:39 -0400 (EDT)
Received: from postgresql.org (svr1.postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 847AF11EB01E;
	Wed,  7 Sep 2005 20:31:32 +0000 (GMT)
X-Original-To: pgsql-patches-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by svr1.postgresql.org (Postfix) with ESMTP id 22823D7D5C
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>; Wed,  7 Sep 2005 16:12:20 -0300 (ADT)
Received: from svr1.postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 19251-09
	for <pgsql-patches-postgresql.org@localhost.postgresql.org>;
	Wed,  7 Sep 2005 19:12:13 +0000 (GMT)
Received: from mail.gmx.net (mail.gmx.de [213.165.64.20])
	by svr1.postgresql.org (Postfix) with SMTP id 05BA7D7B7E
	for <pgsql-patches@postgresql.org>; Wed,  7 Sep 2005 16:12:13 -0300 (ADT)
Received: (qmail invoked by alias); 07 Sep 2005 19:12:14 -0000
Received: from dsl-082-083-195-234.arcor-ip.net (EHLO colt.pezone.net) [82.83.195.234]
  by mail.gmx.net (mp022) with SMTP; 07 Sep 2005 21:12:14 +0200
X-Authenticated: #495269
From: Peter Eisentraut <peter_e@gmx.net>
To: Martijn van Oosterhout <kleptog@svana.org>
Subject: Re: [PATCHES] For review: Initial support for COLLATE
Date: Wed, 7 Sep 2005 21:12:12 +0200
User-Agent: KMail/1.8.1
References: <20050907161112.GA10273@svana.org>
In-Reply-To: <20050907161112.GA10273@svana.org>
cc: pgsql-patches@postgresql.org
MIME-Version: 1.0
Content-Type: text/plain;
  charset="iso-8859-1"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-ID: <200509072112.12955.peter_e@gmx.net>
X-Y-GMX-Trusted: 0
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, hits=0.044 required=5 tests=[AWL=-0.006,
	FORGED_RCVD_HELO=0.05]
X-Mailing-List: pgsql-patches
List-Archive: <http://archives.postgresql.org/pgsql-patches>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-patches.postgresql.org>
List-Owner: <mailto:pgsql-patches-owner@postgresql.org>
List-Post: <mailto:pgsql-patches@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-patches>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-patches>
Precedence: bulk
Sender: pgsql-patches-owner@postgresql.org
Status: OR

Martijn van Oosterhout wrote:
> - Should LOCALE be created as a synonym for COLLATE? It reads more
>   naturally.

No, and in fact the terminology mixup in your patch and description 
concerns me.  If you are talking about collation, then the data types, 
system catalog columns, etc. should talk about collation, not about 
"locale", because that encompasses a number of other things that can be 
handled independent of the collation order.

-- 
Peter Eisentraut
http://developer.postgresql.org/~petere/

---------------------------(end of broadcast)---------------------------
TIP 9: In versions below 8.0, the planner will ignore your desire to
       choose an index scan if your joining column's datatypes do not
       match

From pgsql-hackers-owner+M77964=pgman=candle.pha.pa.us@postgresql.org Tue Dec 27 06:30:14 2005
Return-path: <pgsql-hackers-owner+M77964=pgman=candle.pha.pa.us@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id jBRDUET16504
	for <pgman@candle.pha.pa.us>; Tue, 27 Dec 2005 08:30:14 -0500 (EST)
Received: from postgresql.org (postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 3BCBA67A57F
	for <pgman@candle.pha.pa.us>; Tue, 27 Dec 2005 09:30:14 -0400 (AST)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by postgresql.org (Postfix) with ESMTP id 1DB3B9DC859
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Tue, 27 Dec 2005 09:29:47 -0400 (AST)
Received: from postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 89101-04
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Tue, 27 Dec 2005 09:29:50 -0400 (AST)
X-Greylist: from auto-whitelisted by SQLgrey-
Received: from svana.org (svana.org [203.20.62.76])
	by postgresql.org (Postfix) with ESMTP id 319839DC8A5
	for <pgsql-hackers@postgresql.org>; Tue, 27 Dec 2005 09:29:44 -0400 (AST)
Received: from kleptog by svana.org with local (Exim 3.35 #1 (Debian))
	id 1ErEte-0001lm-00; Wed, 28 Dec 2005 00:29:46 +1100
Date: Tue, 27 Dec 2005 14:29:46 +0100
From: Martijn van Oosterhout <kleptog@svana.org>
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] Proposed COLLATE implementation
Message-ID: <20051227132941.GA32404@svana.org>
Reply-To: Martijn van Oosterhout <kleptog@svana.org>
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha1;
	protocol="application/pgp-signature"; boundary="W/nzBZO5zC0uMSeA"
Content-Disposition: inline
User-Agent: Mutt/1.3.28i
X-PGP-Key-ID: Length=1024; ID=0x0DC67BE6
X-PGP-Key-Fingerprint: 295F A899 A81A 156D B522  48A7 6394 F08A 0DC6 7BE6
X-PGP-Key-URL: <http://svana.org/kleptog/0DC67BE6.pgp.asc>
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, score=0.08 required=5 tests=[AWL=0.080]
X-Spam-Score: 0.08
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR


--W/nzBZO5zC0uMSeA
Content-Type: text/plain; charset=us-ascii
Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable

Greetings all,

If you're not interested in COLLATE, operator classes or related
things, stop now, this is quite a long email.

Firstly, status. PostgreSQL doesn't really support collations at all.
The order of strings is defined at initdb time by the locale then and
cannot be changed later. We allow lists to be sorted in either
ascending or decending order but that's about it. Whatever order there
is is deduced from b-tree operator classes.

The purpose of this patch is to raise collations to (reasonably) first
class object. The idea is that you can define a collation across any
type and that you will then be able to ORDER BY, GROUP BY and INDEX
using that collation. A collation defines both order and equality.

The SQL standard does define COLLATE although they only apply that to
character strings. There are no predefined collations in the standard.
There are rules about how collations and collation states propegate
from the leaves of the parse tree all the way to the root. In its
simplest form, columns and constants have defined collations which
modify the behaviour of functions using these values. At any point in
the parse tree the user can override the collation with an explicit
<collate clause>. If there is ambiguity about what collation applies at
any point for a function that needs to know, this is a error.

All this is parse-time analysis.

Proposed Implementation:

NODES

To implement the above, two new node types are created: CollateClause
which represents the <collate clause> in SQL syntax, and CollateState
which represents the actual state at any node. Currently the only nodes
expected to require these are OpExpr, FuncExpr, Var and Const. Although
I guess it may apply to any node that can be used in an expression.

CATALOG CHANGES

To track collations requires a new table in the catalog, which I have
named pg_collations. It contains the following fields:

Oid oid;           -- OID for this collation
Name collname;     -- Name of the collation (for collate clause)
bool collasc;      -- Ascending or descending
Oid  collopclass;  -- Implementing Operator Class
int4 colltype;     -- Currently, 0=3Dsimple, 1=3Duses locale
Oid colllocale;    -- Locale in pg_locales

(Should we be identifying the type here? or is it ok to lookup the type
via the operator class).

The first few fields name the collation so it can be referred to by the
user. Then the collasc field determines how to use the operator class
as given in the collopclass field. If it indicates descending order, it
will invert the sense of the operator class. For example, asking for
the GT op for a reverse collation will actually return the LT operator
for the operator class.

The purpose of the colltype and colllocale fields are described further
down.

The important thing at this point is that by specifying a collation you
are also specifying an operator class. At the moment the ascending and
descending collations for each type are hard-coded for initdb. At the
moment they have been allocated OIDs starting at 2800, which is the
first large available block.

Each column of a table has a default collation, which defaults to the
default collation of the type but can be specified in the table
declaration. To store this requires an additional column in
pg_attribute (attcollate) which contains the OID of the collation for
that column. When it is referenced in a query, this collation is copied
to the CollateState node of the Var node, from whence it can affect the
query.

Finally, to allow the parser to complain about ambiguous CollateStates,
we need to indicate which functions actually need a senseble collate
state to function. To this end a single boolean field has been added to
pg_proc (proneedcollate). If this is true, the parser should error out
when the collation state is COLLATE_NONE.

INDEXES

Another place you will be allowed to use the collate clause is while
creating indexes. If you declare an index using a particular collation,
it can be used in queries that order by the same collation. Note, that
the collate clause indicates the operator class, so you can either
specify one or the other, but not both.

So each column of an index will also have a collation. However,
pg_attribute has already got an extra field to store the collation for
columns so it makes sense to store the collation here. In the process
the pg_index.indclass field becomes redundant as it can be inferred
from the pg_attribute rows associated with the index.

To make this work there also needs to be a notion of compatability
between collations. For example, two collations which are the reverse
of eachother are compatable in the sense that an index defined with one
collation would be usable for the other simply by scanning in reverse.

FUNCTIONS

In particular for string comparison but also possibly for user-defined
types, a function will need to know what collation it is operating
under. For this purpose an extra field (fn_collate) is added to
FmgrInfo which is filled in with the collation from the expression tree
(if any) or wherever relevent (eg. from the pg_attribute column when
doing statistics or creating indexes).

A PG_GETCOLLATE() macro is added to facilitate user-functions
retreiving this data. This function throws an error when no collation
has been defined. This shouldn't happen in practice as issues should
have been weeded out at parse-time.

This macro returns the OID of the collation but in many cases it will
not be necessary. In particular, functions should NOT invert their
result if the collation is inverted. It is considered the
responsibility of the caller to invert the result if necessary. The
reasons for this are:

1. In most cases that matter (order comparison) the issue can be dealt
with at parse time by the NEGATOR or COMMUTATOR options.
2. For index scans, we would just do a reverse scan instead (or forward
if the index is inverted)
3. Requiring every function to check the collation for inversion is
wasteful, since in many cases the case can be dealt with statically.

DEFAULT COLLATIONS

At this point I'm inclined to define a few collations to be built in or
specially handled:

COLLATE ASC    - default collation for type, ascending (ie, what we do now)
COLLATE DESC   - default collation for type, inverted
COLLATE POSIX  - For strings, define a simple bytewise string comparison.

Indeed, it is expected that by default, all columns involving strings
in the system catalog will always use COLLATE POSIX. Additionaly, type
"name" will always use that collation, even if the user changes the
default (by a method to be specified). This is straightforwardly done
at initdb time.

The purpose of COLLATE DESC is to simplify index declarations. Saying

CREATE INDEX foo ON bar( a COLLATE ASC, b COLLATE DESC );

would allow it to be used in a query using ORDER BY a, b DESC, without
the user having to lookup the name of the collation.

When it comes to naming collations, the question arises whether
ascending/descending collations need to have different names. Or
should there be two collations with the same name with ASC/DESC as a
modifier? Do collations have to be unique across different types; for
example, can varchar and text both have a collation "ignorecase"?

Another issue is that a column could be declared with a descending
collation by default. Say it was an integer column, then (a < 5) would
return FALSE for a =3D 1. While technically correct, I'm thinking of
ruling it out on the basis of being too confusing, and only allow
descending collations at query time or in index specifications.

Another strange point at the moment is how to determine the default
collation of a type. At the moment it is done by finding the default
operator class and looking up the ascending version of that. However,
we may want to provide the user a way of specifying it directly,
perhaps by:

ALTER TYPE text SET DEFAULT COLLATION ignorecase;

PATHKEYS

Currently during planning, pathkeys are indicated by an operator of the
operator class. Here we would simply replace that with the oid of the
collation, which can be matched directly with the collation defined by
the index.

USER DEFINED TYPES

None of this is interesting unless it can be applied to user-defined
types also. Fortunatly this is easy, when the user declares a b-tree
operator class, we can generate the collations automatically. We may
even allow the user to specify the name of the collation. However, if
the user wanted to create multiple collations based on the same
operator class (by using the PG_GETCOLLATE() macro above, we may want
to provide them a way of creating them directly.

COLLATIONS USING LOCALES

For strings, collation can be done in many different ways defined by
what is referred to as a locale. As indicated above in the definition
of pg_collations, there is a colltype field. For most collations this
will be 0 (simple collation). However, for strings the intention is to
use a type 1 (using locales). In this case the last column refers to
the OID of the locale, so you can many collations using the same
operator class, but different locale oids. On a system level it changes
nothing, but inside the functions implementing it, they should use
PG_GETLOCALE(). This will return an opaque pg_locale_t (see below)
handle which can then be used to implement the specifics.

In principle, user-defined types need to be able to use this also,
perhaps by using the clause COLLATE USING LOCALE in the operator class.
In theory there should a collation for each combination of
locale-dependant datatype, locale and order ascending/descending.
How/when these are created has not yet be determined.

MORE TYPES OF COLLATION

Another collation type I've speculated about but not thought about
implementing is a "mapping collation", in which you map the values
through a function and then collate that. The obvious example would be
a case-insensetive mapping where lower is applied before collation.

Implementation could be pretty much done by simply substituting the
functions into the parse tree. For example, if you defined something
like:

CREATE COLLATION ignorecase ON text USING lower($1) COLLATE defaulttext;

Then anytime you did a comparison with that collation, you would simply
insert those function calls into the parse-tree and then collate with
"defaulttext". When declaring an index you would just make it a
functional index. The rules for functional indexes should make it work
out-of-the-box.

OTHER TECHNICAL ISSUES

- Applying a COLLATE clause to an unknown literal causes it to be
coerced to the type that collation is based on. But what about if we
have something like COLLATE DESC?

- This requires some changes in the bootstrap procedures given that we
need to be able to do lookups on the operator class for each type
fairly early on. Some are predefined but it does require moving the
opclass setup further up the list. However, if we store a default
collation in pg_type, we wouldn't need to do that.

- Sorting arrays. Should they get their own collations, or should
they use the collations of their base types.

LOCALES

I've left this to the end because I don't want people distracted by
what is essentially a side-issue. While this would be needed to
implement COLLATE the way the SQL spec intended, it can actually be
implemented and dealt with as a seperate patch. The main reason a basic
implementation exists is that it provides a great way of finding places
that didn't define a collation, since any comparison involving "text"
requires one.

To deal with locales I created another table in the catalog,
pg_locales. This provides an OID which can be referenced from
elsewhere, such as the pg_collations table.

The design is intended to provide some pluggability, so locale
information can come from multiple sources. Also, each locale will be
referenced by an identifier which is unrelated to any external
identifier, so we're not bound by them.

The columns defined currently are:

Name	locname       - Identifier used by postgresql
Name	locsysname    - String identifying the locale for the locale provider
int4	locsource     - System providing this locale
int4	locencoding   - Encoding expected by provider

It is expected that the list of sources for locale data will be short,
probably hard-coded into the backend (currenty internal/system/icu).
The only locale defined at startup is POSIX, which is implemented
internally. The intention is for any other locales to be defined at the
end of initdb. The expected syntax is something like:

CREATE LOCALE hungarian AS 'hu_HU' USING glibc;

This should use the provider to check the locale exists and has a
conpatible encoding. If so it is entered into the table ready for use.

In the backend, there will be implementations of functions like
pg_strcoll_l, pg_localeconv_l, which work like the C system library
versions only they take an extra pg_locale_t argument. This is used to
dispatch the call to the right place. There will be a function to
quickly determine if a locale is C to shortcircuit complexity where it
is not needed.

STATUS

Implementation so far is available here:

http://svana.org/kleptog/temp/collate-current.patch.gz

This patch isn't "clean" and changes a few things that are not strictly
necessary. It won't finish initdb right now because it gets an error in
ANALYSE (the array issue above).

Feedback, help, comments: please reply.

Have a nice day,
--=20
Martijn van Oosterhout   <kleptog@svana.org>   http://svana.org/kleptog/
> Patent. n. Genius is 5% inspiration and 95% perspiration. A patent is a
> tool for doing 5% of the work and then sitting around waiting for someone
> else to do the other 95% so you can sue them.

--W/nzBZO5zC0uMSeA
Content-Type: application/pgp-signature
Content-Disposition: inline

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.0.6 (GNU/Linux)
Comment: For info see http://www.gnupg.org

iD8DBQFDsUHDIB7bNG8LQkwRAnh0AJ0YUNLkVaSY3u0jWBPdlaq+9dujZACfTmis
JR1mF60lKx14Ih850p3lpVk=
=/Ghs
-----END PGP SIGNATURE-----

--W/nzBZO5zC0uMSeA--

From pgsql-hackers-owner+M78869@postgresql.org Tue Jan 24 10:59:03 2006
Return-path: <pgsql-hackers-owner+M78869@postgresql.org>
Received: from ams.hub.org (ams.hub.org [200.46.204.13])
	by candle.pha.pa.us (8.11.6/8.11.6) with ESMTP id k0OGx0324883
	for <pgman@candle.pha.pa.us>; Tue, 24 Jan 2006 11:59:00 -0500 (EST)
Received: from postgresql.org (postgresql.org [200.46.204.71])
	by ams.hub.org (Postfix) with ESMTP id 6E4BE67B09D;
	Tue, 24 Jan 2006 12:58:56 -0400 (AST)
X-Original-To: pgsql-hackers-postgresql.org@localhost.postgresql.org
Received: from localhost (av.hub.org [200.46.204.144])
	by postgresql.org (Postfix) with ESMTP id 1C9E59DC9BE
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>; Tue, 24 Jan 2006 12:58:31 -0400 (AST)
Received: from postgresql.org ([200.46.204.71])
	by localhost (av.hub.org [200.46.204.144]) (amavisd-new, port 10024)
	with ESMTP id 43415-09
	for <pgsql-hackers-postgresql.org@localhost.postgresql.org>;
	Tue, 24 Jan 2006 12:58:29 -0400 (AST)
Received: from svr4.postgresql.org (svr4.postgresql.org [66.98.251.159])
	by postgresql.org (Postfix) with ESMTP id F05519DC86E
	for <pgsql-hackers@postgresql.org>; Tue, 24 Jan 2006 12:58:27 -0400 (AST)
Received: from alife.ru (alife.ru [82.146.44.110])
	by svr4.postgresql.org (Postfix) with ESMTP id 42D155AF193
	for <pgsql-hackers@postgresql.org>; Tue, 24 Jan 2006 16:58:27 +0000 (GMT)
Received: from [212.192.243.99] (elizabet.sai.msu.ru [212.192.243.99])
	(using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits))
	(No client certificate requested)
	by alife.ru (Postfix) with ESMTP id A54D554219D
	for <pgsql-hackers@postgresql.org>; Tue, 24 Jan 2006 19:58:20 +0300 (MSK)
Message-ID: <43D65CA3.4000402@tronet.ru>
Date: Tue, 24 Jan 2006 19:58:11 +0300
From: Alexey Slynko <slynko@tronet.ru>
User-Agent: Mozilla Thunderbird 1.0.7 (X11/20051005)
X-Accept-Language: ru-ru, ru
MIME-Version: 1.0
To: pgsql-hackers@postgresql.org
Subject: [HACKERS] TODO item: locale per database patch (new iteration)
Content-Type: multipart/mixed;
	boundary="------------060707070409000500060805"
X-Virus-Scanned: by amavisd-new at hub.org
X-Spam-Status: No, score=0 required=5 tests=[none]
X-Spam-Score: 0
X-Mailing-List: pgsql-hackers
List-Archive: <http://archives.postgresql.org/pgsql-hackers>
List-Help: <mailto:majordomo@postgresql.org?body=help>
List-Id: <pgsql-hackers.postgresql.org>
List-Owner: <mailto:pgsql-hackers-owner@postgresql.org>
List-Post: <mailto:pgsql-hackers@postgresql.org>
List-Subscribe: <mailto:majordomo@postgresql.org?body=sub%20pgsql-hackers>
List-Unsubscribe: <mailto:majordomo@postgresql.org?body=unsub%20pgsql-hackers>
Precedence: bulk
Sender: pgsql-hackers-owner@postgresql.org
Status: OR

This is a multi-part message in MIME format.
--------------060707070409000500060805
Content-Type: text/plain; charset=KOI8-R; format=flowed
Content-Transfer-Encoding: 7bit

Hi,

it's a renewed locale per database patch. Unfortunately, i've not found
clean way to rebuild database indexes automatically, if locale settings
of two databases (created and template) are differs. Now it's only
raises a NOTICE. So, if anyone has a right notion about it - let will
express. Comment and suggestions are highly appreciated


--------------060707070409000500060805
Content-Type: text/plain;
 name="locale_per_database.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
 filename="locale_per_database.patch"

Index: src/backend/access/transam/xlog.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/access/transam/xlog.c,v
retrieving revision 1.226
diff -u -r1.226 xlog.c
--- src/backend/access/transam/xlog.c	11 Jan 2006 08:43:12 -0000	1.226
+++ src/backend/access/transam/xlog.c	22 Jan 2006 16:41:02 -0000
@@ -3394,7 +3394,6 @@
 {
 	int			fd;
 	char		buffer[BLCKSZ]; /* need not be aligned */
-	char	   *localeptr;
 
 	/*
 	 * Initialize version and compatibility-check fields
@@ -3418,18 +3417,6 @@
 	ControlFile->enableIntTimes = FALSE;
 #endif
 
-	ControlFile->localeBuflen = LOCALE_NAME_BUFLEN;
-	localeptr = setlocale(LC_COLLATE, NULL);
-	if (!localeptr)
-		ereport(PANIC,
-				(errmsg("invalid LC_COLLATE setting")));
-	StrNCpy(ControlFile->lc_collate, localeptr, LOCALE_NAME_BUFLEN);
-	localeptr = setlocale(LC_CTYPE, NULL);
-	if (!localeptr)
-		ereport(PANIC,
-				(errmsg("invalid LC_CTYPE setting")));
-	StrNCpy(ControlFile->lc_ctype, localeptr, LOCALE_NAME_BUFLEN);
-
 	/* Contents are protected with a CRC */
 	INIT_CRC32(ControlFile->crc);
 	COMP_CRC32(ControlFile->crc,
@@ -3612,34 +3599,6 @@
 			   " but the server was compiled without HAVE_INT64_TIMESTAMP."),
 				 errhint("It looks like you need to recompile or initdb.")));
 #endif
-
-	if (ControlFile->localeBuflen != LOCALE_NAME_BUFLEN)
-		ereport(FATAL,
-				(errmsg("database files are incompatible with server"),
-				 errdetail("The database cluster was initialized with LOCALE_NAME_BUFLEN %d,"
-				  " but the server was compiled with LOCALE_NAME_BUFLEN %d.",
-						   ControlFile->localeBuflen, LOCALE_NAME_BUFLEN),
-				 errhint("It looks like you need to recompile or initdb.")));
-	if (pg_perm_setlocale(LC_COLLATE, ControlFile->lc_collate) == NULL)
-		ereport(FATAL,
-			(errmsg("database files are incompatible with operating system"),
-			 errdetail("The database cluster was initialized with LC_COLLATE \"%s\","
-					   " which is not recognized by setlocale().",
-					   ControlFile->lc_collate),
-			 errhint("It looks like you need to initdb or install locale support.")));
-	if (pg_perm_setlocale(LC_CTYPE, ControlFile->lc_ctype) == NULL)
-		ereport(FATAL,
-			(errmsg("database files are incompatible with operating system"),
-		errdetail("The database cluster was initialized with LC_CTYPE \"%s\","
-				  " which is not recognized by setlocale().",
-				  ControlFile->lc_ctype),
-			 errhint("It looks like you need to initdb or install locale support.")));
-
-	/* Make the fixed locale settings visible as GUC variables, too */
-	SetConfigOption("lc_collate", ControlFile->lc_collate,
-					PGC_INTERNAL, PGC_S_OVERRIDE);
-	SetConfigOption("lc_ctype", ControlFile->lc_ctype,
-					PGC_INTERNAL, PGC_S_OVERRIDE);
 }
 
 void
Index: src/backend/commands/dbcommands.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/commands/dbcommands.c,v
retrieving revision 1.175
diff -u -r1.175 dbcommands.c
--- src/backend/commands/dbcommands.c	22 Nov 2005 18:17:08 -0000	1.175
+++ src/backend/commands/dbcommands.c	22 Jan 2006 16:41:03 -0000
@@ -25,6 +25,10 @@
 #include <unistd.h>
 #include <sys/stat.h>
 
+#ifdef HAVE_LANGINFO_H
+#include <langinfo.h>
+#endif
+
 #include "access/genam.h"
 #include "access/heapam.h"
 #include "catalog/catalog.h"
@@ -49,6 +53,7 @@
 #include "utils/fmgroids.h"
 #include "utils/guc.h"
 #include "utils/lsyscache.h"
+#include "utils/pg_locale.h"
 #include "utils/syscache.h"
 
 
@@ -57,9 +62,11 @@
 			int *encodingP, bool *dbIsTemplateP, bool *dbAllowConnP,
 			Oid *dbLastSysOidP,
 			TransactionId *dbVacuumXidP, TransactionId *dbFrozenXidP,
-			Oid *dbTablespace);
+			Oid *dbTablespace, char **dbCollate, char **dbCtype);
 static bool have_createdb_privilege(void);
 static void remove_dbtablespaces(Oid db_id);
+static char * get_locale_encoding(const char *ctype);
+static int check_locale_encoding(int encid, const char *ctype);
 
 
 /*
@@ -73,6 +80,8 @@
 	Oid			src_dboid;
 	Oid			src_owner;
 	int			src_encoding;
+	char			*src_collate;
+	char			*src_ctype;
 	bool		src_istemplate;
 	bool		src_allowconn;
 	Oid			src_lastsysoid;
@@ -92,10 +101,14 @@
 	DefElem    *downer = NULL;
 	DefElem    *dtemplate = NULL;
 	DefElem    *dencoding = NULL;
+	DefElem    *dlc_collate = NULL;
+	DefElem    *dlc_ctype = NULL;
 	DefElem    *dconnlimit = NULL;
 	char	   *dbname = stmt->dbname;
 	char	   *dbowner = NULL;
 	const char *dbtemplate = NULL;
+	char *lc_collate = NULL;
+	char *lc_ctype = NULL;
 	volatile int encoding = -1;
 	volatile int dbconnlimit = -1;
 
@@ -139,6 +152,22 @@
 						 errmsg("conflicting or redundant options")));
 			dencoding = defel;
 		}
+		else if (strcmp(defel->defname, "lccollate") == 0)
+		{
+			if (dlc_collate)
+				ereport(ERROR,
+						(errcode(ERRCODE_SYNTAX_ERROR),
+						errmsg("conflicting or redundant options")));
+			dlc_collate = defel;
+		}
+		else if (strcmp(defel->defname, "lcctype") == 0)
+		{
+			if (dlc_ctype)
+				ereport(ERROR,
+						(errcode(ERRCODE_SYNTAX_ERROR),
+						errmsg("conflicting or redundant options")));
+			dlc_ctype = defel;
+		}
 		else if (strcmp(defel->defname, "connectionlimit") == 0)
 		{
 			if (dconnlimit)
@@ -192,6 +221,22 @@
 			elog(ERROR, "unrecognized node type: %d",
 				 nodeTag(dencoding->arg));
 	}
+	if (dlc_collate && dlc_collate->arg) {
+		lc_collate = strVal(dlc_collate->arg);
+		if ((locale_collate_assign(lc_collate, false, (GucSource)NULL)) == NULL)
+			ereport(ERROR,
+				(errcode(ERRCODE_UNDEFINED_OBJECT),
+				errmsg("%s is not a valid LC_COLLATE name",
+						lc_collate)));
+	}
+	if (dlc_ctype && dlc_ctype->arg) {
+		lc_ctype = strVal(dlc_ctype->arg);
+		if ((locale_collate_assign(lc_ctype, false, (GucSource)NULL)) == NULL)
+			ereport(ERROR,
+				(errcode(ERRCODE_UNDEFINED_OBJECT),
+				errmsg("%s is not a valid LC_CTYPE name",
+						lc_ctype)));
+	}
 	if (dconnlimit && dconnlimit->arg)
 		dbconnlimit = intVal(dconnlimit->arg);
 
@@ -224,7 +269,7 @@
 	 * grab the exclusive lock.
 	 */
 	if (get_db_info(dbname, NULL, NULL, NULL,
-					NULL, NULL, NULL, NULL, NULL, NULL))
+					NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL))
 		ereport(ERROR,
 				(errcode(ERRCODE_DUPLICATE_DATABASE),
 				 errmsg("database \"%s\" already exists", dbname)));
@@ -237,7 +282,8 @@
 
 	if (!get_db_info(dbtemplate, &src_dboid, &src_owner, &src_encoding,
 					 &src_istemplate, &src_allowconn, &src_lastsysoid,
-					 &src_vacuumxid, &src_frozenxid, &src_deftablespace))
+					 &src_vacuumxid, &src_frozenxid, &src_deftablespace,
+					 &src_collate, &src_ctype))
 		ereport(ERROR,
 				(errcode(ERRCODE_UNDEFINED_DATABASE),
 			 errmsg("template database \"%s\" does not exist", dbtemplate)));
@@ -277,6 +323,21 @@
 				(errcode(ERRCODE_WRONG_OBJECT_TYPE),
 				 errmsg("invalid server encoding %d", encoding)));
 
+	/* Set database lc_collate and lc_ctype */
+	if (!lc_collate)
+		lc_collate = src_collate;
+	if (!lc_ctype) 
+		lc_ctype = src_ctype;
+
+#if defined(HAVE_LANGINFO_H) && defined(CODESET)
+	if (encoding > 0 && check_locale_encoding(encoding, lc_ctype) == -1)
+		ereport(ERROR,
+				(errcode(ERRCODE_WRONG_OBJECT_TYPE),
+				 errmsg("encoding %s is not suitable for locale %s", 
+							pg_encoding_to_char(encoding), 
+							lc_ctype)));
+#endif
+
 	/* Resolve default tablespace for new database */
 	if (dtablespacename && dtablespacename->arg)
 	{
@@ -441,7 +502,7 @@
 
 		/* Check to see if someone else created same DB name meanwhile. */
 		if (get_db_info(dbname, NULL, NULL, NULL,
-						NULL, NULL, NULL, NULL, NULL, NULL))
+						NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL))
 			ereport(ERROR,
 					(errcode(ERRCODE_DUPLICATE_DATABASE),
 					 errmsg("database \"%s\" already exists", dbname)));
@@ -459,6 +520,11 @@
 			DirectFunctionCall1(namein, CStringGetDatum(dbname));
 		new_record[Anum_pg_database_datdba - 1] = ObjectIdGetDatum(datdba);
 		new_record[Anum_pg_database_encoding - 1] = Int32GetDatum(encoding);
+		new_record[Anum_pg_database_datcollate - 1] = 
+			DirectFunctionCall1(namein, CStringGetDatum(lc_collate));
+		new_record[Anum_pg_database_datctype - 1] = 
+			DirectFunctionCall1(namein, CStringGetDatum(lc_ctype));
+
 		new_record[Anum_pg_database_datistemplate - 1] = BoolGetDatum(false);
 		new_record[Anum_pg_database_datallowconn - 1] = BoolGetDatum(true);
 		new_record[Anum_pg_database_datconnlimit - 1] = Int32GetDatum(dbconnlimit);
@@ -527,6 +593,15 @@
 		 * Set flag to update flat database file at commit.
 		 */
 		database_file_update_needed();
+
+		/*
+		 * Message about reindexing new database
+		 */
+		if (lc_collate != src_collate || lc_ctype != src_ctype)
+			ereport(NOTICE,
+					(errmsg("database \"%s\" need to be reindexed manually (REINDEX DATABASE)",
+							dbname)));
+
 	}
 	PG_CATCH();
 	{
@@ -584,7 +659,7 @@
 	pgdbrel = heap_open(DatabaseRelationId, ExclusiveLock);
 
 	if (!get_db_info(dbname, &db_id, NULL, NULL,
-					 &db_istemplate, NULL, NULL, NULL, NULL, NULL))
+					 &db_istemplate, NULL, NULL, NULL, NULL, NULL, NULL, NULL))
 	{
 		if (!missing_ok)
 		{
@@ -1100,7 +1175,7 @@
 			int *encodingP, bool *dbIsTemplateP, bool *dbAllowConnP,
 			Oid *dbLastSysOidP,
 			TransactionId *dbVacuumXidP, TransactionId *dbFrozenXidP,
-			Oid *dbTablespace)
+			Oid *dbTablespace, char **dbCollate, char **dbCtype)
 {
 	Relation	relation;
 	ScanKeyData scanKey;
@@ -1155,6 +1230,11 @@
 		/* default tablespace for this database */
 		if (dbTablespace)
 			*dbTablespace = dbform->dattablespace;
+		/* default locale settings for this database */
+		if (dbCollate)
+			*dbCollate = NameStr(dbform->datcollate);
+		if (dbCtype)
+			*dbCtype = NameStr(dbform->datctype);
 	}
 
 	systable_endscan(scan);
@@ -1416,3 +1496,45 @@
 	else
 		strcat(buf, "UNKNOWN");
 }
+
+#if defined(HAVE_LANGINFO_H) && defined(CODESET)
+   
+static char *
+get_locale_encoding(const char *ctype)
+{
+	char       *save;
+	char       *sys;
+                        
+        save = setlocale(LC_CTYPE, NULL);
+        if (!save)
+                return NULL;
+        save = pstrdup(save);
+                                
+        setlocale(LC_CTYPE, ctype);
+        sys = nl_langinfo(CODESET);
+        sys = pstrdup(sys);
+                        
+        setlocale(LC_CTYPE, save);
+        pfree(save);
+            
+        return sys;
+}
+        
+static int
+check_locale_encoding(int encid, const char *ctype)
+{
+        char       *sys;
+        
+        sys = get_locale_encoding(ctype);
+        if (encid == pg_char_to_encoding(sys))
+        {
+                pfree(sys);
+                return 0;
+        }
+        
+        pfree(sys);
+        return -1;
+}
+
+#endif
+
Index: src/backend/parser/gram.y
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/parser/gram.y,v
retrieving revision 2.522
diff -u -r2.522 gram.y
--- src/backend/parser/gram.y	21 Jan 2006 02:16:19 -0000	2.522
+++ src/backend/parser/gram.y	22 Jan 2006 16:41:09 -0000
@@ -372,7 +372,7 @@
 
 	KEY
 
-	LANCOMPILER LANGUAGE LARGE_P  LAST_P LEADING LEAST LEFT LEVEL
+	LANCOMPILER LANGUAGE LARGE_P  LAST_P LCCOLLATE LCCTYPE LEADING LEAST LEFT LEVEL
 	LIKE LIMIT LISTEN LOAD LOCAL LOCALTIME LOCALTIMESTAMP LOCATION
 	LOCK_P LOGIN_P
 
@@ -4635,6 +4635,22 @@
 				{
 					$$ = makeDefElem("encoding", NULL);
 				}
+			| LCCOLLATE opt_equal name
+				{
+					$$ = makeDefElem("lccollate", (Node *)makeString($3));
+				}
+			| LCCOLLATE opt_equal DEFAULT
+				{
+					$$ = makeDefElem("lccollate", NULL);
+				}
+			| LCCTYPE opt_equal name
+				{
+					$$ = makeDefElem("lcctype", (Node *)makeString($3));
+				}
+			| LCCTYPE opt_equal DEFAULT
+				{
+					$$ = makeDefElem("lcctype", NULL);
+				}
 			| CONNECTION LIMIT opt_equal SignedIconst
 				{
 					$$ = makeDefElem("connectionlimit", (Node *)makeInteger($4));
@@ -8225,6 +8241,8 @@
 			| LANGUAGE
 			| LARGE_P
 			| LAST_P
+			| LCCOLLATE
+			| LCCTYPE
 			| LEVEL
 			| LISTEN
 			| LOAD
Index: src/backend/parser/keywords.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/parser/keywords.c,v
retrieving revision 1.170
diff -u -r1.170 keywords.c
--- src/backend/parser/keywords.c	27 Dec 2005 04:00:07 -0000	1.170
+++ src/backend/parser/keywords.c	22 Jan 2006 16:41:09 -0000
@@ -193,6 +193,8 @@
 	{"language", LANGUAGE},
 	{"large", LARGE_P},
 	{"last", LAST_P},
+	{"lccollate", LCCOLLATE},
+	{"lcctype", LCCTYPE},
 	{"leading", LEADING},
 	{"least", LEAST},
 	{"left", LEFT},
Index: src/backend/utils/adt/pg_locale.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/utils/adt/pg_locale.c,v
retrieving revision 1.34
diff -u -r1.34 pg_locale.c
--- src/backend/utils/adt/pg_locale.c	2 Jan 2006 20:25:45 -0000	1.34
+++ src/backend/utils/adt/pg_locale.c	22 Jan 2006 16:41:10 -0000
@@ -10,10 +10,8 @@
  */
 
 /*----------
- * Here is how the locale stuff is handled: LC_COLLATE and LC_CTYPE
- * are fixed by initdb, stored in pg_control, and cannot be changed.
- * Thus, the effects of strcoll(), strxfrm(), isupper(), toupper(),
- * etc. are always in the same fixed locale.
+ * Here is how the locale stuff is handled: 
+ * LC_COLLATE and LC_CTYPE are defined by createdb and stored in pg_database.
  *
  * LC_MESSAGES is settable at run time and will take effect
  * immediately.
@@ -208,6 +206,17 @@
 	return value;
 }
 
+const char *
+locale_collate_assign(const char *value, bool doit, GucSource source)
+{
+	return locale_xxx_assign(LC_COLLATE, value, doit, source);
+}
+
+const char *
+locale_ctype_assign(const char *value, bool doit, GucSource source)
+{
+	return locale_xxx_assign(LC_CTYPE, value, doit, source);
+}
 
 const char *
 locale_monetary_assign(const char *value, bool doit, GucSource source)
Index: src/backend/utils/init/postinit.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/backend/utils/init/postinit.c,v
retrieving revision 1.160
diff -u -r1.160 postinit.c
--- src/backend/utils/init/postinit.c	4 Jan 2006 21:06:32 -0000	1.160
+++ src/backend/utils/init/postinit.c	22 Jan 2006 16:41:10 -0000
@@ -138,6 +138,8 @@
 	ScanKeyData key;
 	HeapTuple	tup;
 	Form_pg_database dbform;
+	char		*lc_ctype;
+	char		*lc_collate;
 
 	/*
 	 * Because we grab RowShareLock here, we can be sure that dropdb() is not
@@ -225,6 +227,32 @@
 	SetConfigOption("client_encoding", GetDatabaseEncodingName(),
 					PGC_BACKEND, PGC_S_DEFAULT);
 
+	/* Set up database locale */
+	lc_collate = NameStr(dbform->datcollate);
+	lc_ctype = NameStr(dbform->datctype);
+
+	if (setlocale(LC_COLLATE, lc_collate) == NULL)
+		ereport(FATAL,
+			(errmsg("database locale is incompatible with operating system"),
+			errdetail("The database was initialized with LC_COLLATE \"%s\","
+					" which is not recognized by setlocale().",
+					lc_collate),
+			errhint("It looks like you need to recreate database or install locale support.")));
+	if (setlocale(LC_CTYPE, lc_ctype) == NULL)
+		ereport(FATAL,
+			(errmsg("database locale are incompatible with operating system"),
+			errdetail("The database was initialized with LC_CTYPE \"%s\","
+					" which is not recognized by setlocale().",
+					lc_ctype),
+			errhint("It looks like you need to recreate database or install locale support.")));
+
+	/* Record it as a GUC internal option, too */
+	SetConfigOption("lc_collate", lc_collate,
+					PGC_INTERNAL, PGC_S_DATABASE);
+	SetConfigOption("lc_ctype", lc_ctype,
+					PGC_INTERNAL, PGC_S_DATABASE);
+
+
 	/*
 	 * Lastly, set up any database-specific configuration variables.
 	 */
Index: src/bin/initdb/initdb.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/initdb/initdb.c,v
retrieving revision 1.106
diff -u -r1.106 initdb.c
--- src/bin/initdb/initdb.c	5 Jan 2006 10:07:46 -0000	1.106
+++ src/bin/initdb/initdb.c	22 Jan 2006 16:41:13 -0000
@@ -1377,6 +1377,10 @@
 
 	bki_lines = replace_token(bki_lines, "ENCODING", encodingid);
 
+	bki_lines = replace_token(bki_lines, "LC_COLLATE", lc_collate);
+
+	bki_lines = replace_token(bki_lines, "LC_CTYPE", lc_ctype);
+
 	/*
 	 * Pass correct LC_xxx environment to bootstrap.
 	 *
@@ -2617,7 +2621,7 @@
 		printf(_("The database cluster will be initialized with locale %s.\n"), lc_ctype);
 	else
 	{
-		printf(_("The database cluster will be initialized with locales\n"
+		printf(_("The database template1 will be initialized with locales\n"
 				 "  COLLATE:  %s\n"
 				 "  CTYPE:    %s\n"
 				 "  MESSAGES: %s\n"
Index: src/bin/pg_controldata/pg_controldata.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/pg_controldata/pg_controldata.c,v
retrieving revision 1.27
diff -u -r1.27 pg_controldata.c
--- src/bin/pg_controldata/pg_controldata.c	15 Oct 2005 02:49:37 -0000	1.27
+++ src/bin/pg_controldata/pg_controldata.c	22 Jan 2006 16:41:13 -0000
@@ -177,9 +177,5 @@
 	printf(_("Maximum columns in an index:          %u\n"), ControlFile.indexMaxKeys);
 	printf(_("Date/time type storage:               %s\n"),
 		   (ControlFile.enableIntTimes ? _("64-bit integers") : _("floating-point numbers")));
-	printf(_("Maximum length of locale name:        %u\n"), ControlFile.localeBuflen);
-	printf(_("LC_COLLATE:                           %s\n"), ControlFile.lc_collate);
-	printf(_("LC_CTYPE:                             %s\n"), ControlFile.lc_ctype);
-
 	return 0;
 }
Index: src/bin/pg_resetxlog/pg_resetxlog.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/pg_resetxlog/pg_resetxlog.c,v
retrieving revision 1.39
diff -u -r1.39 pg_resetxlog.c
--- src/bin/pg_resetxlog/pg_resetxlog.c	5 Jan 2006 03:01:37 -0000	1.39
+++ src/bin/pg_resetxlog/pg_resetxlog.c	22 Jan 2006 16:41:13 -0000
@@ -465,22 +465,6 @@
 #else
 	ControlFile.enableIntTimes = FALSE;
 #endif
-	ControlFile.localeBuflen = LOCALE_NAME_BUFLEN;
-
-	localeptr = setlocale(LC_COLLATE, "");
-	if (!localeptr)
-	{
-		fprintf(stderr, _("%s: invalid LC_COLLATE setting\n"), progname);
-		exit(1);
-	}
-	StrNCpy(ControlFile.lc_collate, localeptr, LOCALE_NAME_BUFLEN);
-	localeptr = setlocale(LC_CTYPE, "");
-	if (!localeptr)
-	{
-		fprintf(stderr, _("%s: invalid LC_CTYPE setting\n"), progname);
-		exit(1);
-	}
-	StrNCpy(ControlFile.lc_ctype, localeptr, LOCALE_NAME_BUFLEN);
 
 	/*
 	 * XXX eventually, should try to grovel through old XLOG to develop more
@@ -530,9 +514,6 @@
 	printf(_("Maximum columns in an index:          %u\n"), ControlFile.indexMaxKeys);
 	printf(_("Date/time type storage:               %s\n"),
 		   (ControlFile.enableIntTimes ? _("64-bit integers") : _("floating-point numbers")));
-	printf(_("Maximum length of locale name:        %u\n"), ControlFile.localeBuflen);
-	printf(_("LC_COLLATE:                           %s\n"), ControlFile.lc_collate);
-	printf(_("LC_CTYPE:                             %s\n"), ControlFile.lc_ctype);
 }
 
 
Index: src/bin/psql/describe.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/psql/describe.c,v
retrieving revision 1.130
diff -u -r1.130 describe.c
--- src/bin/psql/describe.c	22 Nov 2005 18:17:29 -0000	1.130
+++ src/bin/psql/describe.c	22 Jan 2006 16:41:15 -0000
@@ -360,6 +360,12 @@
 	appendPQExpBuffer(&buf,
 			",\n       pg_catalog.pg_encoding_to_char(d.encoding) as \"%s\"",
 					  _("Encoding"));
+	appendPQExpBuffer(&buf,
+		",\n       d.datcollate as \"%s\"",
+					 _("LC_COLLATE"));
+	appendPQExpBuffer(&buf,
+		",\n       d.datctype as \"%s\"",
+					 _("LC_CTYPE"));
 	if (verbose)
 		appendPQExpBuffer(&buf,
 						  ",\n       pg_catalog.obj_description(d.oid, 'pg_database') as \"%s\"",
Index: src/bin/scripts/createdb.c
===================================================================
RCS file: /projects/cvsroot/pgsql/src/bin/scripts/createdb.c,v
retrieving revision 1.15
diff -u -r1.15 createdb.c
--- src/bin/scripts/createdb.c	21 Jun 2005 04:02:33 -0000	1.15
+++ src/bin/scripts/createdb.c	22 Jan 2006 16:41:15 -0000
@@ -34,6 +34,8 @@
 		{"tablespace", required_argument, NULL, 'D'},
 		{"template", required_argument, NULL, 'T'},
 		{"encoding", required_argument, NULL, 'E'},
+		{"lc-collate", required_argument, NULL, 1},
+		{"lc-ctype", required_argument, NULL, 2},
 		{NULL, 0, NULL, 0}
 	};
 
@@ -53,6 +55,8 @@
 	char	   *tablespace = NULL;
 	char	   *template = NULL;
 	char	   *encoding = NULL;
+	char	   *lc_collate = NULL;
+	char	   *lc_ctype = NULL;
 
 	PQExpBufferData sql;
 
@@ -98,6 +102,12 @@
 			case 'E':
 				encoding = optarg;
 				break;
+			case 1:
+				lc_collate = optarg;
+				break;
+			case 2:
+				lc_ctype = optarg;
+				break;
 			default:
 				fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
 				exit(1);
@@ -155,7 +165,12 @@
 		appendPQExpBuffer(&sql, " ENCODING '%s'", encoding);
 	if (template)
 		appendPQExpBuffer(&sql, " TEMPLATE %s", fmtId(template));
+	if (lc_collate)
+		appendPQExpBuffer(&sql, " LCCOLLATE %s", fmtId(lc_collate));
+	if (lc_ctype)
+		appendPQExpBuffer(&sql, " LCCTYPE %s", fmtId(lc_ctype));
 	appendPQExpBuffer(&sql, ";\n");
+	
 
 	conn = connectDatabase(strcmp(dbname, "postgres") == 0 ? "template1" : "postgres",
 						   host, port, username, password, progname);
@@ -219,19 +234,20 @@
 	printf(_("Usage:\n"));
 	printf(_("  %s [OPTION]... [DBNAME] [DESCRIPTION]\n"), progname);
 	printf(_("\nOptions:\n"));
-	printf(_("  -D, --tablespace=TABLESPACE  default tablespace for the database\n"));
-	printf(_("  -E, --encoding=ENCODING      encoding for the database\n"));
-	printf(_("  -O, --owner=OWNER            database user to own the new database\n"));
-	printf(_("  -T, --template=TEMPLATE      template database to copy\n"));
-	printf(_("  -e, --echo                   show the commands being sent to the server\n"));
-	printf(_("  -q, --quiet                  don't write any messages\n"));
-	printf(_("  --help                       show this help, then exit\n"));
-	printf(_("  --version                    output version information, then exit\n"));
+	printf(_("  -D, --tablespace=TABLESPACE  	default tablespace for the database\n"));
+	printf(_("  -E, --encoding=ENCODING      	encoding for the database\n"));
+	printf(_("  --lc-collate, --lc-ctype=LOCALE	initialize database with given locale\n"));
+	printf(_("  -O, --owner=OWNER            	database user to own the new database\n"));
+	printf(_("  -T, --template=TEMPLATE      	template database to copy\n"));
+	printf(_("  -e, --echo                   	show the commands being sent to the server\n"));
+	printf(_("  -q, --quiet                  	don't write any messages\n"));
+	printf(_("  --help                       	show this help, then exit\n"));
+	printf(_("  --version                    	output version information, then exit\n"));
 	printf(_("\nConnection options:\n"));
-	printf(_("  -h, --host=HOSTNAME          database server host or socket directory\n"));
-	printf(_("  -p, --port=PORT              database server port\n"));
-	printf(_("  -U, --username=USERNAME      user name to connect as\n"));
-	printf(_("  -W, --password               prompt for password\n"));
+	printf(_("  -h, --host=HOSTNAME          	database server host or socket directory\n"));
+	printf(_("  -p, --port=PORT              	database server port\n"));
+	printf(_("  -U, --username=USERNAME      	user name to connect as\n"));
+	printf(_("  -W, --password               	prompt for password\n"));
 	printf(_("\nBy default, a database with the same name as the current user is created.\n"));
 	printf(_("\nReport bugs to <pgsql-bugs@postgresql.org>.\n"));
 }
Index: src/include/catalog/pg_control.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/include/catalog/pg_control.h,v
retrieving revision 1.26
diff -u -r1.26 pg_control.h
--- src/include/catalog/pg_control.h	22 Nov 2005 18:17:30 -0000	1.26
+++ src/include/catalog/pg_control.h	22 Jan 2006 16:41:15 -0000
@@ -137,11 +137,6 @@
 	/* flag indicating internal format of timestamp, interval, time */
 	uint32		enableIntTimes; /* int64 storage enabled? */
 
-	/* active locales */
-	uint32		localeBuflen;
-	char		lc_collate[LOCALE_NAME_BUFLEN];
-	char		lc_ctype[LOCALE_NAME_BUFLEN];
-
 	/* CRC of all above ... MUST BE LAST! */
 	pg_crc32	crc;
 } ControlFileData;
Index: src/include/catalog/pg_database.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/include/catalog/pg_database.h,v
retrieving revision 1.38
diff -u -r1.38 pg_database.h
--- src/include/catalog/pg_database.h	15 Oct 2005 02:49:42 -0000	1.38
+++ src/include/catalog/pg_database.h	22 Jan 2006 16:41:16 -0000
@@ -38,6 +38,8 @@
 	NameData	datname;		/* database name */
 	Oid			datdba;			/* owner of database */
 	int4		encoding;		/* character encoding */
+	NameData	datcollate;		/* locale LC_COLLATE */
+	NameData	datctype;		/* locale LC_CTYPE */
 	bool		datistemplate;	/* allowed as CREATE DATABASE template? */
 	bool		datallowconn;	/* new connections allowed? */
 	int4		datconnlimit;	/* max connections allowed (-1=no limit) */
@@ -60,21 +62,23 @@
  *		compiler constants for pg_database
  * ----------------
  */
-#define Natts_pg_database				12
+#define Natts_pg_database				14
 #define Anum_pg_database_datname		1
 #define Anum_pg_database_datdba			2
 #define Anum_pg_database_encoding		3
-#define Anum_pg_database_datistemplate	4
-#define Anum_pg_database_datallowconn	5
-#define Anum_pg_database_datconnlimit	6
-#define Anum_pg_database_datlastsysoid	7
-#define Anum_pg_database_datvacuumxid	8
-#define Anum_pg_database_datfrozenxid	9
-#define Anum_pg_database_dattablespace	10
-#define Anum_pg_database_datconfig		11
-#define Anum_pg_database_datacl			12
+#define Anum_pg_database_datcollate	4
+#define Anum_pg_database_datctype	5	
+#define Anum_pg_database_datistemplate	6
+#define Anum_pg_database_datallowconn	7
+#define Anum_pg_database_datconnlimit	8
+#define Anum_pg_database_datlastsysoid	9
+#define Anum_pg_database_datvacuumxid	10	
+#define Anum_pg_database_datfrozenxid	11	
+#define Anum_pg_database_dattablespace	12	
+#define Anum_pg_database_datconfig		13
+#define Anum_pg_database_datacl			14
 
-DATA(insert OID = 1 (  template1 PGUID ENCODING t t -1 0 0 0 1663 _null_ _null_ ));
+DATA(insert OID = 1 (  template1 PGUID ENCODING "LC_CTYPE" "LC_COLLATE" t t -1 0 0 0 1663 _null_ _null_ ));
 DESCR("Default template database");
 #define TemplateDbOid			1
 
Index: src/include/utils/pg_locale.h
===================================================================
RCS file: /projects/cvsroot/pgsql/src/include/utils/pg_locale.h,v
retrieving revision 1.21
diff -u -r1.21 pg_locale.h
--- src/include/utils/pg_locale.h	28 Dec 2005 23:22:51 -0000	1.21
+++ src/include/utils/pg_locale.h	22 Jan 2006 16:41:16 -0000
@@ -22,6 +22,10 @@
 extern char *locale_numeric;
 extern char *locale_time;
 
+extern const char *locale_collate_assign(const char *value,
+					   bool doit, GucSource source);
+extern const char *locale_ctype_assign(const char *value,
+					   bool doit, GucSource source);
 extern const char *locale_messages_assign(const char *value,
 					   bool doit, GucSource source);
 extern const char *locale_monetary_assign(const char *value,


--------------060707070409000500060805
Content-Type: text/plain
Content-Disposition: inline
Content-Transfer-Encoding: 8bit
MIME-Version: 1.0


---------------------------(end of broadcast)---------------------------
TIP 1: if posting/reading through Usenet, please send an appropriate
       subscribe-nomail command to majordomo@postgresql.org so that your
       message can get through to the mailing list cleanly

--------------060707070409000500060805--

