Utf-16 Hex Decode Nodejs

I am trying to decode a UTF-16 Hex (Hello 世界) to a String in NodeJS. I've tried doing so by making a buffer from the hex: let vari = new Buffer.from('00 48 00 65 00 6C 00 6C 00

Solution 1:

It's not working because you are creating a new buffer out of the representation of the buffer as a string. This will result in a buffer then when decoded, will be the string of the buffer '00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C', but because of hex the buffer is going to be empty. if you are to console.log(Buffer.from('00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C', 'hex') you will see an empty buffer.

Also, '00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C' is not a UTF-16 hex representation of "Hello 世界". when encoded as a string, it is this: 䠀攀氀氀漀 ᙎ䱵. 48 00 65 00 6c 00 6c 00 6f 00 20 00 16 4e 4c 75 is "Hello 世界" in UTF-16 hex, I got this from running console.log(Buffer.from('Hello 世界', 'utf16le'));.

To answer the question on how you can convert '48 00 65 00 6c 00 6c 00 6f 00 20 00 16 4e 4c 75' back to "Hello 世界" you can do the following:

let hexStrings = '48 00 65 00 6c 00 6c 00 6f 00 20 00 16 4e 4c 75'.split(' '); // split string chunkslet hex = hexStrings.map(x =>parseInt(x, 16)); // convert string chunks to hexadecimallet buffer = Buffer.from(hex); // create buffer from hexadecimal arraylet string = buffer.toString('utf16le'); // convert buffer to stringconsole.log(string); // output -> Hello 世界

Hope this helps!

Solution 2:

I came from node.js fastest way to decode hex string as utf-16

hex 48 00 65 00 6c 00 6c 00 6f 00 20 00 16 4e 4c 75 decoded as utf16le is "Hello 世界"

you have 00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C let's compare the 2: 48 00 65 00 6c 00 6c 00 6f 00 20 00 16 4e 4c 75 decoded as utf16le is 'Hello 世界' 00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C decoded as utf16be is 'Hello 世界' the only difference is that every 2 bytes are swapped

and it's not any every 2 bytes, they have opposite byte order

either you need to reverse the byte order manually OR have a function .toString('utf16be')be:Big Endian is the opposite of le:Little Endian

but node.js doesn't provide buf.toString('utf16be')

to "reverse the byte order manually" there's: buf.swap16()https://nodejs.org/api/buffer.html#buffer_buf_swap16

One convenient use of buf.swap16() is to perform a fast in-place conversion between UTF-16 little-endian and UTF-16 big-endian:

const myHexStr = '00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C'//your bytes shouldn't be whiteSpace separated for `Buffer.from`, it won't workconst removedSpaces = myHexStr.replace(/ /g,'') // '00480065006C006C006F00204E16754C'const buf = Buffer.from(removedSpaces,'hex') // <Buffer 00 48 00 65 00 6c 00 6c 00 6f 00 20 4e 16 75 4c>const reversedBuf = buf.swap16() // <Buffer 48 00 65 00 6c 00 6c 00 6f 00 20 00 16 4e 4c 75>const backToJavascriptString = reversedBuf.toString("utf16le") // 'Hello 世界'

this can be simplified to one line:

Buffer.from('00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C'.replace(/ /g,''),'hex').swap16().toString("utf16le")

---

Here is a better way to do it (performance-wise): thankfully node.js provides at least a function to read as big endian: buf.readUInt16BE() : but it only reads 2 bytes at a time. guess what else is 2 bytes ? UTF-16

my implementation: push all these readUInt16BE to array of fixed length

functionubeFunc(hexx4) {
  const buf = Buffer.from(hexx4, "hex"), len = buf.length/2, arr = newArray(len)
  for (let i = 0; i < len; i++) {
    arr[i]=buf.readUInt16BE(i*2)
  }
  returnString.fromCharCode(...arr)
}

how you'd use it

ubeFunc('00 48 00 65 00 6C 00 6C 00 6F 00 20 4E 16 75 4C'.replace(/ /g,'')) // 'Hello 世界'

this is overall fastest: read here: node.js fastest way to decode hex string as utf-16