[FIXED] What exactly does Spyder do to Unicode strings?

Issue

Running Python in a standard GNU terminal emulator on Ubuntu 14.04, I get the expected behavior when typing interactively:

>>> len('tiθ')
4
>>> len(u'tiθ')
3

The same thing happens when running an explicitly utf8-encoded script in Spyder:

# -*- coding: utf-8 -*-
print(len('tiθ'))
print(len(u'tiθ'))

...gives the following output, regardless of whether I run it in a new dedicated interpreter, or run in a Spyder-default interpreter (shown here):

>>> runfile('/home/dan/Desktop/mwe.py', wdir=r'/home/dan/Desktop')
4
3

But when typing interactively in a Python console within Spyder:

>>> len('tiθ')
4
>>> len(u'tiθ')
4

This issue has been brought up elsewhere, but that question regards differences between Windows and Linux. Here, I'm getting different results in different consoles on the same system, and the Python startup message in the terminal emulator and in the console within Spyder are identical:

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 
[GCC 4.8.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.

What is going on here, and how can I get Python-within-Spyder to behave like Python-in-the-shell with regard to unicode strings? @martijn-pieters makes the comment on this question that

Spyder does all sorts of things to break normal a Python environment.

But I'm hoping there's a way to un-break this particular feature, since it makes it really hard to debug scripts in the IDE when I can't rely on my interactive typed commands to yield the same results as scripts run as a whole with their coding: utf-8 declaration.

UPDATES

In the GNU terminal:

>>> repr(u'tiθ')
"u'ti\\u03b8'"
>>> import sys
>>> sys.stdin.encoding
'UTF-8'
>>> sys.getdefaultencoding()
'ascii'

In Spyder console:

>>> repr(u'tiθ')
"u'ti\\xce\\xb8'"
>>> import sys
>>> sys.stdin.encoding  # returns None
>>> sys.getdefaultencoding()
'UTF-8'

So knowing that, can I convince Spyder to behave like the GNU terminal?

Solution

After a bit of research, it seems that the strange behavior is at least in part built into Python (see this discussion thread; briefly, Python sets sys.stdin.encoding to None as default, and only changes it if it detects that the host is tty and it can detect the tty's encoding).

That said, a hackish workaround was just to tell Spyder to use /usr/bin/python3 as its executable instead of the default (which was Python 2.7.6). When running a Python 3 console within Spyder or within a GNU terminal emulator, I get results different (better!) than before, but importantly the results are consistent, regardless of whether running a script or typing interactively, and regardless of using GNU terminal or Spyder console:

>>> len('tiθ')
3
>>> len(u'tiθ')
3
>>> import sys
>>> sys.stdin.encoding
'UTF-8'
>>> sys.getdefaultencoding()
'utf-8'

This leads to other problems within Spyder, however: its sitecustomize.py script is not Python 3 friendly, for example, so every new interpreter starts with

Error in sitecustomize; set PYTHONVERBOSE for traceback:
SyntaxError: invalid syntax (sitecustomize.py, line 432)

The interpreter seems to work okay anyway, but it makes me nervous enough that I'm not considering this an "acceptable" answer, so if anyone else has a better idea...

Answered By - drammock