#1 2015-09-13 09:36:56

curmudgeon
Trusted Member
Registered: 2014-12-31
Posts: 19

rsync - trying to moderate system load

hi guys, forgive a bit of a newbie question

I wanted to poll some ideas for how I could moderate the load of what I'm doing on the host system. One thing that I need to do to rsync a bunch (thousands) of small files with content between a database server and the children (which are the LES ones). This needs to be done daily per folder, but speed is not really important (within reason).

What I'm wondering and maybe I should just experiment, but wanted to get a handle on people's expertise and thoughts about whether there are other avenues to explore. Also, it depends on whether these servers are most tightly capacity constrained on CPU, network, or disk

The only ideas I had:
(1) using rsync wholefiles rather than using diff because so as not to hit the CPU too hard on the LES machines trading it off against more network and disk usage
(2) tarring everything up on the source, transmitting it to the receiver, untarring and then doing the rsync as a local job, but that may create much more disk IO as well as extra bandwidth
(3) changing the architecture so that it's a big file per folder, but I think this would actually make things worse because the whole thing would have to be diffed/read/written/transmitted and would probably make reading/writing the data needed on the fly could be worse. Sometimes the majority of the content changes day to day, sometimes only a little.
(4) is there a way to better nice the rsync server daemon so that it doesn't consume more than x amount of load ? For security reasons I want the parent to push on to the children rather than the other way around

Any other ideas ? Is there another tool than rsync to consider in such a context ?

Also, is there a tool anyone could suggest to monitor system load per time period and either log it or see it remotely? I notice often on the Solus that there's blanks in the graph which makes it harder to figure out whether I'm making things better or worse ! I suppose in worst case I could do a cron job to cat /proc/loadavg into a logfile with a time stamp every few minutes but that feels a bit less than optimal

Thx

Last edited by curmudgeon (2015-09-13 09:45:16)

Offline

#2 2015-09-13 17:31:53

yomero
Trusted Member
Registered: 2014-06-30
Posts: 222

Re: rsync - trying to moderate system load

Well, maybe this idea won't keep every resource capped to a limit, but will help.

You can use a combination of ioinice, nice and cpulimit. And for the bandwidth, there is some stuff too, like "trickle" (never used it) to cap your traffic usage. Anyway, rsync has options to do that too.

Offline

#3 2015-09-13 18:42:53

curmudgeon
Trusted Member
Registered: 2014-12-31
Posts: 19

Re: rsync - trying to moderate system load

hi Yomero

Thanks for the suggestion, but one thing about that if the rsync is running from the database server, will it not nice relative to its own processes ? It's got a bit more oomph than the LES boxes so even niced it might go pretty quickly... Or is there a daemon side way of doing nice on rsync ?

For security reasons, I'm wanting shell access to the database server to be as restricted as possible, so I don't want to go rsync from the LES box to the database and have it be niced that way...

Last edited by curmudgeon (2015-09-13 18:43:28)

Offline

#4 2015-09-13 19:07:59

yomero
Trusted Member
Registered: 2014-06-30
Posts: 222

Re: rsync - trying to moderate system load

I don't get it completely but, you are doing a... "normal" rsync via ssh?

If so, maybe you can write a simple script running all the time trying to locate any rsync process and doing renice on them.

Offline

#5 2015-09-13 19:10:31

curmudgeon
Trusted Member
Registered: 2014-12-31
Posts: 19

Re: rsync - trying to moderate system load

hi Yomero. I'll try that. Thank you.

Offline

#6 2015-09-13 19:14:45

yomero
Trusted Member
Registered: 2014-06-30
Posts: 222

Re: rsync - trying to moderate system load

Something like...

while true; do
  pgrep rsync | xargs renice....  #or something like that
  sleep 3;
done

May work

Offline

#7 2015-09-13 21:42:05

curmudgeon
Trusted Member
Registered: 2014-12-31
Posts: 19

Re: rsync - trying to moderate system load

Thanks. I'll give it a try (yes it's standard rsync over ssh)

Offline

#8 2015-09-14 01:39:35

willie
Trusted Member
Registered: 2013-05-05
Posts: 423

Re: rsync - trying to moderate system load

Do you know for sure if rsync is using a single ssh connection for all the files, rather than making new ones?  They are rather expensive to set up.  I've generally found it's faster to tar on the origin side and untar on the destination, than do 1000's of small file transfers by any method.  What is in the files?  If they're text files that are changing slightly, another alternative might be to use git.

Offline

#9 2015-09-15 22:42:46

curmudgeon
Trusted Member
Registered: 2014-12-31
Posts: 19

Re: rsync - trying to moderate system load

thansk willie. files are mixed media, so not sure git would help. i'm trying yomero's suggestion to renice the rsync once it's running, so far seems to be an improvement.

Offline

#10 2015-12-24 18:59:24

sleddog
Trusted Member
Registered: 2015-04-29
Posts: 21

Re: rsync - trying to moderate system load

I'm late to the party, sorry.

rsync has an 'rsync-path' switch that lets you control how rsync runs on the remote end, so you can nice/ionice both ends when you do a transfer.

For a minimal-impact rsync...

#!/bin/bash

REMOTE="some.server.com"
NICE="/bin/nice -n 19"
IONICE="/usr/bin/ionice -c3"
RSYNC="/usr/bin/rsync"
IKEY="/root/.ssh/some_key"

$NICE $IONICE $RSYNC -aqz --delete-during --bwlimit=2000 --rsync-path="$NICE $IONICE $RSYNC" --no-inc-recursive -e "ssh -p 1234 -i $IKEY" /local/path/ [email protected]${REMOTE}:/remote/path 

One gotcha is that the path for 'nice' differs between CentOS and Debian so adjust accordingly if you're rsyncing between different distros smile

Last edited by sleddog (2015-12-24 19:17:43)

Offline

#11 2015-12-25 01:19:47

yomero
Trusted Member
Registered: 2014-06-30
Posts: 222

Re: rsync - trying to moderate system load

Thanks sleddog! Nice to see you around

And probably you don't need to specify each path if there is no need to regulate the load on the remote server (not sure if I am totally right).

Offline

#12 2015-12-26 19:03:16

sleddog
Trusted Member
Registered: 2015-04-29
Posts: 21

Re: rsync - trying to moderate system load

Hi yomero! Hope you mexican winter isn't too cold smile

Yes, if you don't want to nice/ionice the remote end, you can omit the -rsync-path="..." altogether.

Offline

#13 2015-12-27 01:40:44

yomero
Trusted Member
Registered: 2014-06-30
Posts: 222

Re: rsync - trying to moderate system load

Hey!

Well, isn't that cold to be honest. Maybe a little bit more than the past year I think. At least at Mexico City, our lowest temps go around 5ºC or something like that.

/offtopic hehe

Offline

Board footer

Powered by FluxBB