Thursday, June 10, 2010

how netflix runs their talent pool

netflix culture presentation

dirvish duplicity hashbackup oh my

thread and
post

That looks interesting from the standpoint of compressed backups. I
worry that the diff between one backup and the next will be so great
that the solution is invalidated (for this particular project) due to
the bandwidth requirements.

I can, however, give an update on where I'm going with my push backups
so far.

The key in my instance is going to be a file based loopback device that
can be encrypted. Follow the rabbit trail with me for a second.

*Use 'dd' to create a file. (100gigs let's say.)
*use losetup to assign that file to a loopback device.
*Encrypt the loopback device with the deprecated cryptsetup or the new
dm-crypt
*Format the loopback device
*Mount the loopback device as a dirvish bank.
*make your initial backup to the dirvish vaults on the encrypted device
*umount your encrypted loopback device
*perform your initial rsync of the file that is your encrypted loopback
device

----Now starts the nightly backup procedure--

*Mount the encrypted loopback device to your dirvish bank
*run dirvish to update your vaults
*dismount the encrypted loopback device
*rsync the updated loopback file with a -c (for checksum) so that only
the delta changes cross the wire.

TADA

I realize that this has the added step of having to find extra disk
space or convert existing disk space to a file for use as a loopback
--BUT-- it does allow for one to 'push' a secure dirvish bank with its
vaults to a remote location.

another idea

spin up an EC2 instance, rsync our host to an EBS volume that's appropriately sized... and then shut it down when rsync is done.

$0.10 /mo /GB + $0.10/million IO requests ( EBS + pricing )

volia - instant offsite backup.

aha - we're not the first to think of this:

http://www.johnandcailin.com/blog/john/never-lose-your-data-again%3A-backup-remotely-using-rsync-ssh-and-rdiff-backup

which references rsynccrypto - not a likely option but interesting.

ooh more interesting stuff with dirvish and duplicity to come next...

Wednesday, June 9, 2010

working from: build-a-rpm-of-python-25-on-centos-5-redhat-enterprise-rhel-5/

to get python2.5 installed on a Centos5 box.... so I can run google-docs-fs on FUSE

and to get started, we'll grab a src rpm from here .

today's potential fun:

==========================================================================================================================================================
Package Arch Version Repository Size
==========================================================================================================================================================
Installing:
bzip2-devel i386 1.0.3-4.el5_2 base 38 k
db4-devel i386 4.3.29-10.el5 base 1.9 M
expat-devel i386 1.95.8-8.3.el5_4.2 base 132 k
gcc-c++ i386 4.1.2-48.el5 base 3.4 M
gdbm-devel i386 1.8.0-26.2.1 base 36 k
gmp-devel i386 4.1.4-10.el5 base 522 k
tix-devel i386 1:8.4.0-11.fc6 base 135 k
tk-devel i386 8.4.13-5.el5_1.1 base 809 k
Updating:
cpp i386 4.1.2-48.el5 base 2.6 M
db4 i386 4.3.29-10.el5 base 909 k
expat i386 1.95.8-8.3.el5_4.2 base 77 k
findutils i386 1:4.2.27-6.el5 base 294 k
gcc i386 4.1.2-48.el5 base 5.2 M
glibc i686 2.5-49 base 5.2 M
glibc-common i386 2.5-49 base 16 M
glibc-devel i386 2.5-49 base 2.0 M
glibc-headers i386 2.5-49 base 601 k
libX11 i386 1.0.3-11.el5 base 796 k
libX11-devel i386 1.0.3-11.el5 base 661 k
libgcc i386 4.1.2-48.el5 base 95 k
libstdc++ i386 4.1.2-48.el5 base 362 k
mesa-libGL i386 6.5.1-7.8.el5 base 9.6 M
mesa-libGL-devel i386 6.5.1-7.8.el5 base 461 k
nscd i386 2.5-49 base 163 k
openssl i686 0.9.8e-12.el5_4.6 base 1.4 M
openssl-devel i386 0.9.8e-12.el5_4.6 base 1.9 M
popt i386 1.10.2.3-18.el5 base 74 k
readline i386 5.1-3.el5 base 223 k
readline-devel i386 5.1-3.el5 base 146 k
rpm i386 4.4.2.3-18.el5 base 1.2 M
rpm-build i386 4.4.2.3-18.el5 base 301 k
rpm-devel i386 4.4.2.3-18.el5 base 1.2 M
rpm-libs i386 4.4.2.3-18.el5 base 927 k
rpm-python i386 4.4.2.3-18.el5 base 59 k
sqlite i386 3.3.6-5 base 213 k
sqlite-devel i386 3.3.6-5 base 257 k
tar i386 2:1.15.1-30.el5 base 747 k
Installing for dependencies:
libstdc++-devel i386 4.1.2-48.el5 base 2.8 M
tix i386 1:8.4.0-11.fc6 base 329 k

Transaction Summary
==========================================================================================================================================================
Install 10 Package(s)
Update 29 Package(s)
Remove 0 Package(s)

Total download size: 64 M


uh... I think I'll update rpm separately first

and a good place to get a 3rd set of hands

http://www.instructables.com/id/Third-Hand-A-multi-use-helping-hand-for-electro/step1/Getting-Started/

build a bike

http://www.americanspeedster.com/index.htm

Tuesday, June 8, 2010

http://www.linuxfoundation.org/collaborate/workgroups/networking/bridge#Can_I_do_bridging_in_combination_with_netfilter.2Fiptables.3F

saved my butt yesterday:

# cd /proc/sys/net/bridge
# ls
bridge-nf-call-arptables bridge-nf-call-iptables
bridge-nf-call-ip6tables bridge-nf-filter-vlan-tagged
# for f in bridge-nf-*; do echo 0 > $f; done

Friday, June 4, 2010

radios - cheap and AM

http://sci-toys.com/scitoys/scitoys/radio/three_penny/three_penny.html
mmm radios
http://sci-toys.com/scitoys/scitoys/radio/am_transmitter.html
and more radios

#RSFtalks with Edward Snowden

What an intelligent, thoughtful individual. I find it difficult to forgive 44 for failing to pardon this patriot and instead pursuing him ...

Other Popular Posts: