quetz-mod_python-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Earle Ady <ea...@bluelavatech.com>
Subject mod_python, unicode, utf-8, latin1
Date Fri, 11 Aug 2006 22:33:50 GMT
Aloha!

I've done some searching online regarding character encoding and  
UTF-8 support within mod_python, but haven't been able to get the  
proper functionality out of mod_python.

Here's the situation:  I have changed my site.py in Python 2.4.3 to  
use "utf-8" as the default encoding.   I have a database with correct  
unicode representations in it.  I execute routines from the  
interpreter and get correct unicode objects out of the database.   
When I run these exact routines from inside of a PSP page, the  
unicode object has now been latin1 decoded.  Please note that from  
the examples below that I am using identical MySQLdb connection  
settings.

I am still a bit unclear as to where exactly this is happening inside  
of mod_python, and any advice to a solution would be greatly  
appreciated.  It's pretty critical that a developer can provide UTF-8  
support in order for mod_python to gain traction in enterprise  
applications.

If this is a user error on my part, I'd greatly appreciate being  
pointed to a proper solution.

Best,
earle.

------  THIS WORKS FROM WITHIN THE INTERPRETER:
(conn, cursor) = util.DBConnect(MySQLdb.cursors.DictCursor)

cursor.execute("SELECT * from unicode_test")
items = cursor.fetchall()

for item in items:
         print item,

# RESULTS:  correct unicode:
# (earle@www-1 14:55 266) python utest.py
# {'data': u'\u9577\u5ca1', 'id': 35L}
# {'data': u'\u9577\u5ca1', 'id': 36L}


-------  THIS DOES NOT WORK FROM .PSP, it produces a latin1 decoded  
unicode object of the correct unicode (see below):

<%
req.content_type = 'text/html;charset=UTF-8;';
%>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"  
"http://www.w3.org/TR/xhtml1/DTD/xhtm
l1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" dir="ltr" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
</head>
<body>
<%@ include file="include/webglobals.psps" %>
<%
(conn, cursor) = util.DBConnect(MySQLdb.cursors.DictCursor)
req.write("MYSQL CONNECTION CHARSET: ")
req.write(conn.character_set_name())
req.write("<p/>")
req.write("SYS.DEFAULTENCODING: ")
req.write(sys.getdefaultencoding())
req.write("<p/>")

res = cursor.execute("SELECT * from unicode_test")
items = cursor.fetchall()

for i in items:
         #
         req.write("DATA: ")
         req.write(i['data'])
         req.write(", item: ")
%>
<%= i %>
<%
         req.write(",  BYTES: ")
         req.write(i['data'].encode('unicode_escape'))

         req.write("<p/>")
         #
# end: items

req.write("SHOULD LOOK LIKE THIS: %s" % ( u'\u9577\u5ca1', ))
%>
</body>
</html>

---- RESULTS:

MYSQL CONNECTION CHARSET: utf8

SYS.DEFAULTENCODING: utf-8

DATA: 長岡, item: {'data': u'\xe9\x95\xb7\xe5\xb2\xa1', 'id':  
35L} , BYTES: \xe9\x95\xb7\xe5\xb2\xa1

DATA: 長岡, item: {'data': u'\xe9\x95\xb7\xe5\xb2\xa1', 'id':  
36L} , BYTES: \xe9\x95\xb7\xe5\xb2\xa1

SHOULD LOOK LIKE THIS: 長岡


-------   Notice if I latin1 decode the -correct- unicode object, i  
get the exact
unicode object that is appearing inside of the PSP:

 >>> u'\u9577\u5ca1'.decode('latin1')
u'\xe9\x95\xb7\xe5\xb2\xa1'






Mime
View raw message