Working with UTF-8 in JSON: Encoding and Decoding

JSON (JavaScript Object Notation) is a widely used data format for data interchange. By default, the json.dumps() function in Python escapes non-ASCII characters, representing them as Unicode escape sequences (e.g., \u05d1). While valid, this can make the JSON less human-readable. This tutorial explains how to serialize and deserialize UTF-8 characters directly in your JSON output, ensuring both validity and readability.

Understanding the Issue

When dealing with text containing characters outside the basic ASCII range (like those in many languages other than English), Python’s json.dumps() function, by default, encodes these characters using Unicode escape sequences. This ensures that the JSON is universally compatible, as all systems can interpret Unicode escapes. However, the result is less readable for humans. For example, the Hebrew phrase "ברי צקלה" might be represented as "\u05d1\u05e8\u05d9 \u05e6\u05e7\u05dc\u05d4". Often, you want the original UTF-8 characters to be preserved directly in the JSON string.

The ensure_ascii Parameter

The key to controlling this behavior is the ensure_ascii parameter of the json.dumps() function. Setting ensure_ascii=False instructs the function to output UTF-8 characters directly instead of escaping them.

import json

text = "ברי צקלה"  # Hebrew for "Brian Checkle"
json_string = json.dumps(text, ensure_ascii=False)
print(json_string)  # Output: "ברי צקלה"

In this example, the output JSON string directly contains the Hebrew characters, making it more readable.

Serializing Dictionaries and Lists

The ensure_ascii=False parameter works seamlessly with dictionaries and lists as well:

import json

data = {
    "name": "ברי צקלה",
    "city": "תל אביב", # Tel Aviv in Hebrew
    "languages": ["עברית", "English"] # Hebrew and English
}

json_string = json.dumps(data, ensure_ascii=False, indent=2) # indent for readability
print(json_string)

This code produces a well-formatted JSON string with all non-ASCII characters preserved:

{
  "name": "ברי צקלה",
  "city": "תל אביב",
  "languages": [
    "עברית",
    "English"
  ]
}

Writing to Files

When writing JSON data to a file, it’s crucial to specify the encoding as UTF-8. This ensures that the file is saved with the correct character encoding. There are two main ways to achieve this:

• Using json.dump() with encoding='utf-8': This is the preferred approach.

  import json
  
  data = {"message": "xin chào việt nam"} # Vietnamese for "hello Vietnam"
  
  with open('data.json', 'w', encoding='utf-8') as f:
      json.dump(data, f, ensure_ascii=False)
  

• Using codecs.open(): This provides more control over the file encoding.

  import json
  import codecs
  
  data = {"message": "xin chào việt nam"}
  
  with codecs.open('data.json', 'w', encoding='utf-8') as f:
      json.dump(data, f, ensure_ascii=False)
  

Both methods achieve the same result: a JSON file containing UTF-8 characters.

Deserializing JSON with UTF-8 Characters

When reading JSON data containing UTF-8 characters, Python automatically handles the decoding correctly. The json.loads() function seamlessly converts the UTF-8 characters into Unicode strings.

import json

with open('data.json', 'r', encoding='utf-8') as f:
    data = json.load(f)

print(data["message"]) # Output: xin chào việt nam

Important Considerations

• Consistency: Always be consistent with your encoding. Use UTF-8 for both reading and writing JSON data to avoid unexpected errors.
• File Editors: Ensure your file editor is configured to use UTF-8 encoding when opening and saving JSON files.
• Web Servers and APIs: When sending JSON data over a network, make sure your web server and API are configured to use UTF-8 encoding and set the appropriate Content-Type header (application/json; charset=utf-8).