This page is for anyone writing expressions, macros or scripts in which you need to escape special characters.
When writing expressions, macros or scripts you may need to enter a character that has special meaning. For example, a string could include a double-quote, but in the expression PhixFlow will read this as the end of the string. To tell PhixFlow to ignore a special character, you can use the backslash character.
| Character | Type | Escape string |
|---|---|---|
backspace | Represented As | "\b" |
| Escaped | "\\\\b" | |
| form feed | Represented As | "\f" |
| Escaped | "\\\\f" | |
| tab | Represented As | "\t" |
| Escaped | "\\\\t" | |
| carraige return | Represented As | "\r" |
| Escaped | "\\\\r" | |
| newline | Represented As | "\n" |
| Escaped | "\\\\n" | |
| backslash | Represented As |
|
| Escaped | "\\\\\\\\" | |
| double quote | Represented As | '"' |
| Escaped | '\\\\"' |
Macro: Escaping Invalid Characters
If you are outputting stream data into JSON format, the data can include special characters that JSON will not accept. The following macro escapes these invalid characters.
By way of understanding: \b is standard notation for the backspace character. However, \ is also a special character in JSON, so we need to escape this with \\\\ to tell it that the \ should not be treated as a special character in this instance.
do(
$value = $args[1],
$value = replaceAll( $value , "\b","\\\\b"),
$value = replaceAll( $value , "\f","\\\\f"),
$value = replaceAll( $value , "\t","\\\\t"),
$value = replaceAll( $value , "\r","\\\\r"),
$value = replaceAll( $value , "\n","\\\\n"),
$value = replaceAll( $value , "\\\\","\\\\\\\\"),
$value = replaceAll( $value , '"','\\\\"')
)
JSON un-escape Macro. This is useful when importing JSON into PhixFlow from an external source. It converts an escaped sequence (eg \\\\t) back into a usable character. The double \\ ahead of the \t in the output is necessary to escape \t when the jep expression is passed and will be converted to simply \t (the tab character).
do(
$value= $args[1],
$value = replaceAll($value, "\\\\t","\\\t"),
$value = replaceAll($value, "\\\\b","\\\b"),
$value = replaceAll($value, "\\\\r","\\\r"),
$value = replaceAll($value, "\\\\n","\\\n"),
$value = replaceAll($value, "\\\\f","\\\f"),
$value
)
Check the format of the JSON you are importing to see what adjustments are required. Sometimes characters such as \\\\\\\\" may be helpfully un-escaped prior to import, making it unnecessary to unescape every special character.
Validating JSON Files
JSON data can include characters that cannot be written to a database that is using the UTF-8 character set. For example, 4-bit Emojis are not supported by UTF-8.
The following regular expression lists the characters expected in a JSON file and removes invalid characters from it.
replaceAll(JSON Code,"[^\\p{Space}0-9A-Za-z!:\\\"%&\\[*()\\],-/_\\\\{}\\.]","")
The backslash character escapes special characters such as [] , . and \
The UTF8mb4 character set supports a wider range of characters, such as Emojis. Migrating your database to use this character set would allow JSON to be used without needing to validate it using a regular expression.
If PhixFlow is running on a MariaDB database, it will already be using the UTF8mb4 character set.
See also: Regular Expressions