From Lars Kruse:
Thus the situation seems to be that obnam stores all relevant chunks, but due to the symlink pointing to something outside of the repository I cannot restore
From liw: This is obviously a bug. Obnam should either give an error if the backup root is a symlink, or back it up as a symlink, not recurse into the directory and then backup that and only then ruin everything by backing up the symlink as a symlink.
Ben Kelly reported on August 31, 2012, that he's seeing crashes due to file descriptor leaks. See list mail archive for logs and suggested patches. I have not been able to reproduce this, however. --liw
A small performance improvement on startup may be possible by replacing len(key('',0,0)) with struct.calcsize(self.fmt):
python -m timeit -s \
'import struct; checksum_length = 32; fmt = "!%dsQQ" % checksum_length'\
'struct.calcsize(fmt)'
1000000 loops, best of 3: 0.201 usec per loop
vs
python -m timeit -s \
'import struct; checksum_length = 32; fmt = "!%dsQQ" % checksum_length'\
'len(struct.pack(fmt,"",0,0))'
1000000 loops, best of 3: 0.753 usec per loop
I can't attach files, so I can't attach the bundle I think you prefer, but here's the diff, and I'll email you the bundle as an attachment.
=== modified file 'obnamlib/checksumtree.py'
--- obnamlib/checksumtree.py 2011-07-26 13:23:55 +0000
+++ obnamlib/checksumtree.py 2013-02-08 07:09:22 +0000
@@ -33,7 +33,7 @@
upload_queue_size, lru_size, hooks):
tracing.trace('new ChecksumTree name=%s' % name)
self.fmt = '!%dsQQ' % checksum_length
- key_bytes = len(self.key('', 0, 0))
+ key_bytes = struct.calcsize(self.fmt)
obnamlib.RepositoryTree.__init__(self, fs, name, key_bytes, node_size,
upload_queue_size, lru_size, hooks)
self.keep_just_one_tree = True
=== modified file 'obnamlib/chunklist.py'
--- obnamlib/chunklist.py 2011-07-26 13:23:55 +0000
+++ obnamlib/chunklist.py 2013-02-08 07:09:22 +0000
@@ -35,14 +35,15 @@
def __init__(self, fs, node_size, upload_queue_size, lru_size, hooks):
tracing.trace('new ChunkList')
- self.key_bytes = len(self.key(0))
+ self.fmt = '!Q'
+ self.key_bytes = struct.calcsize(self.fmt)
obnamlib.RepositoryTree.__init__(self, fs, 'chunklist', self.key_bytes,
node_size, upload_queue_size,
lru_size, hooks)
self.keep_just_one_tree = True
def key(self, chunk_id):
- return struct.pack('!Q', chunk_id)
+ return struct.pack(self.fmt, chunk_id)
def add(self, chunk_id, checksum):
tracing.trace('chunk_id=%s', chunk_id)
=== modified file 'obnamlib/clientlist.py'
--- obnamlib/clientlist.py 2011-07-26 13:23:55 +0000
+++ obnamlib/clientlist.py 2013-02-08 07:09:22 +0000
@@ -49,7 +49,7 @@
tracing.trace('new ClientList')
self.hash_len = len(self.hashfunc(''))
self.fmt = '!%dsQB' % self.hash_len
- self.key_bytes = len(self.key('', 0, 0))
+ self.key_bytes = struct.calcsize(self.fmt)
self.minkey = self.hashkey('\x00' * self.hash_len, 0, 0)
self.maxkey = self.hashkey('\xff' * self.hash_len, obnamlib.MAX_ID,
self.SUBKEY_MAX)
cliapp now allows overriding parts of the logging setup, and obnam should use that as soon as cliapp has been released.
Add an Obnam cmdtest to verify v6 repos can be restored from.
From mail:
I think there is problem in keep policy, see attached file, --keep=Nh always do same as --keep=Nd
Seems to be true. Need cmdtest test case for reproduction.
Obnam version 1.0
Host: Gentoo amd64
When excluding a file that already exists on the backup, the file is not removed from the last backup, but it is kept, without being updated.
The file excluded should not appear on generations created with the exclude filter.
How to reproduce:
#############################################
#!/bin/sh
rm -rf /tmp/test
mkdir -p /tmp/test/backup
touch /tmp/test/backup/file1
touch /tmp/test/backup/file2
# 1st run, exclude file1
# only file2 is present on backup
obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup
obnam -r /tmp/test/repo ls
# 2nd run
# none excluded, both files present on backup
obnam -r /tmp/test/repo backup /tmp/test/backup
obnam -r /tmp/test/repo ls
# 3rd run, modify files and backup with exclude=file1
# both files present, file2 is updated on backup, file1 is neither updated, or removed
echo "000" > /tmp/test/backup/file1
echo "000" > /tmp/test/backup/file2
obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup
obnam -r /tmp/test/repo ls
cd /
rm -rf /tmp/test/backup
obnam -r /tmp/test/repo restore /tmp/test/backup
ls -lha /tmp/test/backup
#############################################
OUTPUT:
+ rm -rf /tmp/test
+ mkdir -p /tmp/test/backup
+ touch /tmp/test/backup/file1
+ touch /tmp/test/backup/file2
# 1st run, exclude file1
# only file2 is present on backup
+ obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup
Backed up 2 files, uploaded 0.0 B in 0s at 0.0 B/s average speed
+ obnam -r /tmp/test/repo ls
Generation 2 (2012-06-20 09:23:41 - 2012-06-20 09:23:41)
drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 /
drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp
drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test
drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup
-rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file2
# 2nd run
# none excluded, both files present on backup
+ obnam -r /tmp/test/repo backup /tmp/test/backup
Backed up 3 files, uploaded 0.0 B in 0s at 0.0 B/s average speed
+ obnam -r /tmp/test/repo ls
Generation 5 (2012-06-20 09:23:41 - 2012-06-20 09:23:41)
drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 /
drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp
drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test
drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup
-rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file1
-rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file2
# 3rd run, modify files and backup with exclude=file1
# both files present, file2 is updated on backup, file1 is neither updated, or removed
+ obnam -r /tmp/test/repo --exclude=/tmp/test/backup/file1 backup /tmp/test/backup
Backed up 2 files, uploaded 4.0 B in 0s at 13.8 B/s average speed
+ obnam -r /tmp/test/repo ls
Generation 8 (2012-06-20 09:23:42 - 2012-06-20 09:23:42)
drwxr-xr-x 24 root root 4096 2012-06-20 05:27:42 /
drwxrwxrwx 19 root root 480 2012-06-20 07:23:40 /tmp
drwxrwx--- 4 jordi jordi 80 2012-06-20 07:23:41 /tmp/test
drwxrwx--- 2 jordi jordi 80 2012-06-20 07:23:40 /tmp/test/backup
-rw-rw---- 1 jordi jordi 0 2012-06-20 07:23:40 /tmp/test/backup/file1
-rw-rw---- 1 jordi jordi 4 2012-06-20 07:23:41 /tmp/test/backup/file2
# when restoring, they appear, the new file2, and the old file1.
+ rm -rf /tmp/test/backup
+ cd /
+ obnam -r /tmp/test/repo restore /tmp/test/backup
--h--m--s 4 files 4 B (100 %) 1.4 KiB/s /tmp/test/backup
+ ls -lha /tmp/test/backup
total 4,0K
drwxrwx--- 2 jordi jordi 80 jun 20 09:23 .
drwxrwx--- 4 jordi jordi 80 jun 20 09:23 ..
-rw-rw---- 1 jordi jordi 0 jun 20 09:23 file1
-rw-rw---- 1 jordi jordi 4 jun 20 09:23 file2