How To Duplicate A Firebase Child With Many Records Without Downloading And Uploading The Data?

I have a Firebase child with nearly 500k records. I want to duplicate this child in order to create a backup of the data (within the same Firebase). So, if my child is called lines

Solution 1:

Trying to access/query a node with 500k child records is a bad idea. It's also not very likely that you need to synchronize those 500k lines to users. So please work hard on separating the active from the historical data. You current approach of creating a backup copy is a great start for that. Just also delete nodes while you're writing them to their new location.

To get this data is going to be "a bit involved". It indeed involves first getting a list of the child keys with a shallow REST call:

https://yours.firebaseio.com/path-to-list.json?shallow=true

Then you loop through the keys and access each child in turn, with a regular JavaScript client:

ref.child('path-to-list').child(key).once('value', function(snapshot) {
  ref.child('path-to-backup').child(key).set(snapshot.val(), function(error) {
    if (!error) {
      // Delete the original item, so that the list stays small
      snapshot.ref().remove();
    }
  });

});

This loop doesn't create problems, since you're not querying the entire list. Instead you are accessing specific child nodes directly, which is fine.

Be sure to throttle the loop, because you might otherwise swamp you database with small writes.