New Features in InfluxDB Open Source Backup and Restore
By Adam Anthony / Mar 15, 2018 / InfluxData, InfluxDB, Community, Developer
New in version 1.5, the open source backup utility provides the option to run both backup and restore functions on a live database. It also provides features to backup and restore single or multiple databases, along with optional filtering based on data point timestamps. Finally, the restore tool supports import of data from an InfluxDB Enterprise cluster, while the backup tool can now generate backup files that may be imported into an Enterprise database. The offline backup/restore functions provided in InfluxDB versions 1.4 and earlier are retained in version 1.5 without change, and are detailed in the open source documentation.
About File Formats
Prior to version 1.5, the open source backup tool created a different backup file format than the Enterprise version of InfluxDB. This legacy format remains fully supported, and in some cases may even be used as input to the new online restore functionality. For new users, we recommend using the new portable backup format, which uses less disk space and also provides a clear transfer path for data between the Enterprise and open source versions of InfluxDB.
The improved backup command is similar to previous versions of InfluxDB, except that it can optionally generate backups in a portable format and has some new filtering options to constrain the range of data points that are exported to the backup. It is invoked by the
influxd binary using the
influxd backup -portable [options] <path-to-backup>
-host <host:port>- The host to connect to and perform a snapshot of. Defaults to
-database <name>- The database to backup. Optional. If not given, all databases are backed up.
-retention <name>- The retention policy to backup. Optional.
-shard <id>- The shard id to backup. Optional. If specified,
-since <2015-12-24T08:12:13Z>- Do a file-level backup since the given time. The time needs to be in the RFC3339 format. Optional.
-start <2015-12-24T08:12:23Z>- All points earlier than this timestamp will be excluded from the export. Not compatible with
-end <2015-12-24T08:12:23Z>- All points later than this time stamp will be excluded from the export. Not compatible with
-portable- Generate backup files in the format used for InfluxDB Enterprise.
To backup all databases in an existing system:
[/tmp/backup_ex]$ influxd backup -portable /tmp/backup_ex 2018/03/15 12:55:26 backing up metastore to /tmp/backup_ex/meta.00 2018/03/15 12:55:26 No database, retention policy or shard ID given. Full meta store backed up. 2018/03/15 12:55:26 Backing up all databases in portable format 2018/03/15 12:55:26 backing up db= 2018/03/15 12:55:26 backing up db=collectd_db rp=autogen shard=1 to /tmp/backup_ex/collectd_db.autogen.00001.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=2 to /tmp/backup_ex/telegraf.autogen.00002.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=3 to /tmp/backup_ex/telegraf.autogen.00003.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=4 to /tmp/backup_ex/telegraf.autogen.00004.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=5 to /tmp/backup_ex/telegraf.autogen.00005.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=6 to /tmp/backup_ex/telegraf.autogen.00006.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=7 to /tmp/backup_ex/telegraf.autogen.00007.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=telegraf rp=autogen shard=8 to /tmp/backup_ex/telegraf.autogen.00008.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=mydb rp=forever shard=9 to /tmp/backup_ex/mydb.forever.00009.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=mydb rp=forever shard=10 to /tmp/backup_ex/mydb.forever.00010.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=tmp3 rp=autogen shard=11 to /tmp/backup_ex/tmp3.autogen.00011.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=tmp3 rp=autogen shard=12 to /tmp/backup_ex/tmp3.autogen.00012.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=tmp3 rp=autogen shard=13 to /tmp/backup_ex/tmp3.autogen.00013.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=tmp3 rp=autogen shard=14 to /tmp/backup_ex/tmp3.autogen.00014.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backing up db=_internal rp=monitor shard=15 to /tmp/backup_ex/_internal.monitor.00015.00 since 0001-01-01T00:00:00Z 2018/03/15 12:55:26 backup complete: 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.meta 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s1.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s2.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s3.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s4.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s5.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s6.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s7.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s8.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s9.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s10.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s11.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s12.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s13.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s14.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.s15.tar.gz 2018/03/15 12:55:26 /tmp/backup_ex/20180315T165526Z.manifest
The restore function is improved in two important ways. First, the restore process no longer requires system down time. Second, a data restore no longer erases the data currently on the target system. A brief technical explanation is that the old restore process deactivated the system and replaced the data folder with the backup data, while the new online process imports data through a streaming API provided by the
Regardless of whether you have existing backup automation that supports the legacy format, or you are a new user, you may wish to test the new online feature for legacy to gain the advantages described above. It is activated by using either the
-online flags. The flags indicate that the input is in either the new portable backup format (which is the same format that Enterprise InfluxDB uses), or the legacy backup format, respectively. It has the following options:
-host <host:port>- The host to connect to and perform a snapshot of. Defaults to
-db <name>- Identifies the database from the backup that will be restored.
-newdb <name>- The name of the database into which the archived data will be imported on the target system. If not given, then the value of
-dbis used. The new database name must be unique to the target system.
-rp <name>- Identifies the retention policy from the backup that will be restored. Requires that
-newrp <name>- The name of the retention policy that will be created on the target system. Requires that
-rpis set. If not given, the value of
-shard <id>- Optional. If given,
-rpare required. Will restore the single shard's data.
To restore the backup taken above to a new, empty instance of influxdb (Note: the _internal database is skipped by default. Though it would be uncommon to do so, it may be imported explicitly using the -db parameter as described above):
[/tmp/backup_ex]$ influxd restore -portable /tmp/backup_ex 2018/03/15 13:01:45 Restoring shard 13 live from backup 20180315T165526Z.s13.tar.gz 2018/03/15 13:01:45 Restoring shard 14 live from backup 20180315T165526Z.s14.tar.gz 2018/03/15 13:01:45 Restoring shard 5 live from backup 20180315T165526Z.s5.tar.gz 2018/03/15 13:01:45 Restoring shard 6 live from backup 20180315T165526Z.s6.tar.gz 2018/03/15 13:01:45 Restoring shard 11 live from backup 20180315T165526Z.s11.tar.gz 2018/03/15 13:01:45 Restoring shard 3 live from backup 20180315T165526Z.s3.tar.gz 2018/03/15 13:01:45 Restoring shard 10 live from backup 20180315T165526Z.s10.tar.gz 2018/03/15 13:01:45 Restoring shard 12 live from backup 20180315T165526Z.s12.tar.gz 2018/03/15 13:01:45 Meta info not found for shard 15 on database _internal. Skipping shard file 20180315T165526Z.s15.tar.gz 2018/03/15 13:01:45 Restoring shard 7 live from backup 20180315T165526Z.s7.tar.gz 2018/03/15 13:01:45 Restoring shard 8 live from backup 20180315T165526Z.s8.tar.gz 2018/03/15 13:01:46 Restoring shard 9 live from backup 20180315T165526Z.s9.tar.gz 2018/03/15 13:01:46 Restoring shard 1 live from backup 20180315T165526Z.s1.tar.gz 2018/03/15 13:01:46 Restoring shard 2 live from backup 20180315T165526Z.s2.tar.gz 2018/03/15 13:01:46 Restoring shard 4 live from backup 20180315T165526Z.s4.tar.gz