How To Create A Firestore Safe Document Id Based On A String

After some discussion on my question about base64 not being safe for Firestore IDs here I would like to know how one can encode a string to a Firestore 'safe' Document ID. Here is

Solution 1:

Use encodeURI() followed by SHA256. This will constraint the document ID to

Must be valid UTF-8 characters
Must be no longer than 1,500 bytes
Cannot containa forward slash (/)
Cannot solely consist of a single period (.) or double periods (..)
Cannot match the regular expression __.*__

encodeURI is for valid UTF-8 characters.

SHA256 is fixed length at 256 bits (or 32 bytes) therefore not exceeding 1,500 bytes limit.

SHA256 characters are [a-fA-F0-9] according to https://stackoverflow.com/a/12618366/3073280.

Lastly, you mentioned that it will need entropy. SHA256 is well diffused.

Solution 2:

EDIT

In order to transform strings into uniqueIDs very fast use crypto.createHash() instead. The result will be the same for a given string input.

You can use MD5 or SHA256 as both takes the same time, 2.2s average to calculate 1 Million unique IDs.

Here is the code:

const crypto = require('crypto');

functionuniqueId(string, algorithm = 'md5') {
  return crypto.createHash(algorithm).update(string).digest('hex');
}

console.log('started');
console.time('generateIDsMD5')
for (let i = 0; i < 1000000; i++) {
  uniqueId('a string ' + i);
}
console.timeEnd('generateIDsMD5');

console.time('generateIDsSHA256')
for (let i = 0; i < 1000000; i++) {
  uniqueId('a string ' + i, 'sha256');
}
console.timeEnd('generateIDsSHA256');

// For instance, It will take around 2.2s average// to generate 1Million Unique IDs with MD5 or SHA256 encryptionconsole.log('MD5 string ', uniqueId('a string ' + 1));
console.log('MD5 sameString ', uniqueId('a string ' + 2));
console.log('MD5 sameString ', uniqueId('a string ' + 2));
console.log('SHA256 string ', uniqueId('a string ' + 1, 'sha256'));
console.log('SHA256 sameString ', uniqueId('a string ' + 2, 'sha256'));
console.log('SHA256 sameString ', uniqueId('a string ' + 2, 'sha256'));
console.log('finished');

PREVIOUS ANSWER

I adapted the code from Firebase and made it available directly on your node.js with some custom test for you. It takes up to 3s for 1 Million IDs, and only 300ms for 100.000 IDs which is your considered daily usage approach.

This uses crypto considered very safe if run in node.js environment.

here is the function wrapped with usage example:

const crypto = require('crypto');

functionautoId(bytesLength) {
  const chars =
    'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789';
  let autoId = '';
  while (autoId.length < bytesLength) {
    const bytes = crypto.randomBytes(40);
    bytes.forEach(b => {
      // Length of `chars` is 62. We only take bytes between 0 and 62*4-1// (both inclusive). The value is then evenly mapped to indices of `char`// via a modulo operation.const maxValue = 62 * 4 - 1;
      if (autoId.length < bytesLength && b <= maxValue) {
        autoId += chars.charAt(b % 62);
      }
    });
  }
  return autoId;
}

console.log('started');
console.time('generateIDs')
for (let i = 0; i < 1000000; i++) {
  autoId(20);
}
console.timeEnd('generateIDs');
// For instance, It will take around 3s average// to generate 1 Million Unique IDs with 20 bytes lengthconsole.log('example 20bytes ', autoId(20));
console.log('example 40bytes ', autoId(40));
console.log('example 60bytes ', autoId(60));
console.log('finished');

Simply use node thisfile.js and you will see your result.

Since firebase is mainly open source we can find the official uniqueId generator used in node.js to generate the IDs here: https://github.com/googleapis/nodejs-firestore/blob/4f4574afaa8cf817d06b5965492791c2eff01ed5/dev/src/util.ts#L52

IMPORTANT

If you are going to join 2 IDs, do not use any slash /, as you know it is not allowed, instead use underscore _ or nothing at all since you have control of the length of an ID, therefore you should know how to split the ID accordingly (40 bytes contain 2 IDs of 20 bytes for instance).

The limitation of firestore in Document Ids is 1500 bytes so you have plenty to play with.

More info: https://firebase.google.com/docs/firestore/quotas#limits

Solution 3:

I used Base58 and that was the most safe I could research for