[FIXED] Python - Issues with Unicode String from API Call

Issue

I'm using Python to call an API that returns the last name of some soccer players. One of the players has a "ć" in his name.

When I call the endpoint, the name prints out with the unicode attached to it:

>>> last_name = (json.dumps(response["response"][2]["player"]["lastname"]))

>>> print(last_name)

"Mitrovi\u0107"

>>> print(type(last_name))

<class 'str'>

If I were to take copy and paste that output and put it in a variable on its own like so:

>>> print("Mitrovi\u0107")

Mitrović

>>> print(type("Mitrovi\u0107"))

<class 'str'>

Then it prints just fine?

What is wrong with the API endpoint call and the string that comes from it?

Solution

Well, you serialise the string with json.dumps() before printing it, that's why you get a different output. Compare the following:

>>> print("Mitrović")
Mitrović

and

>>> print(json.dumps("Mitrović"))
"Mitrovi\u0107"

The second command adds double quotes to the output and escapes non-ASCII chars, because that's how strings are encoded in JSON. So it's possible that response["response"][2]["player"]["lastname"] contains exactly what you want, but maybe you fooled yourself by wrapping it in json.dumps() before printing.

Note: don't confuse Python string literals and JSON serialisation of strings. They share some common features, but they aren't the same (eg. JSON strings can't be single-quoted), and they serve a different purpose (the first are for writing strings in source code, the second are for encoding data for sending it accross the network).

Another note: You can avoid most of the escaping with ensure_ascii=False in the json.dumps() call:

>>> print(json.dumps("Mitrović", ensure_ascii=False))
"Mitrović"

Answered By - lenz