I'm trying to use rsync to upload a mailbox archive from my home system to the system that I usually use for handling my mail. I'm already successfully using rsync in the opposite direction. What happens is that rsync gets part of the way through the transfer (it does about 3.9Mb out of 75Mb) and then it just hangs. I've looked at the files it has transferred and the point at which it has stopped and can see no oddities in permissions or anything like that. The command I'm using is:- rsync --rsh='/usr/bin/ssh -l chris -x' --recursive \ --exclude=in/ --exclude=li/ --exclude=mfolder \ /home/chris/MailArchive halkidiki.areti.com: It's normally run by cron in the middle of the night but I'm trying it just now from the command line and the same problem seems to be there. In my mail I'm getting the following error from cron:- Read from remote host halkidiki.areti.com: Connection timed out rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(836) This error arrives 23 minutes (consistent for two nights anyway) after the cron command is run. When run from the command line it looks like it's going to take just as long to fail. Has anyone got any ideas what might be going wrong? -- Chris Green (chris@areti.co.uk) "Never ascribe to malice that which can be explained by incompetence."
On Wed, Jan 11, 2006 at 11:05:45AM +0000, Chris Green wrote:
I'm trying to use rsync to upload a mailbox archive from my home system to the system that I usually use for handling my mail.
I'm already successfully using rsync in the opposite direction.
What happens is that rsync gets part of the way through the transfer (it does about 3.9Mb out of 75Mb) and then it just hangs. I've looked at the files it has transferred and the point at which it has stopped and can see no oddities in permissions or anything like that.
The command I'm using is:-
rsync --rsh='/usr/bin/ssh -l chris -x' --recursive \ --exclude=in/ --exclude=li/ --exclude=mfolder \ /home/chris/MailArchive halkidiki.areti.com:
It's normally run by cron in the middle of the night but I'm trying it just now from the command line and the same problem seems to be there.
In my mail I'm getting the following error from cron:-
Read from remote host halkidiki.areti.com: Connection timed out rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(836)
This error arrives 23 minutes (consistent for two nights anyway) after the cron command is run. When run from the command line it looks like it's going to take just as long to fail.
Has anyone got any ideas what might be going wrong?
Looking at my system now where I ran the above command it seems to have lost the connection to halkidiki.areti.com completely. Although the command still appears to be running there's no rsync process or even an associated shell process. It's as if there's some sort of inactivity timeout that can't see the rsync transfer running and it's pulled the plug. -- Chris Green (chris@areti.co.uk) "Never ascribe to malice that which can be explained by incompetence."
On Wed, Jan 11, 2006 at 11:13:44AM +0000, Chris Green wrote:
Has anyone got any ideas what might be going wrong?
Looking at my system now where I ran the above command it seems to have lost the connection to halkidiki.areti.com completely. Although the command still appears to be running there's no rsync process or even an associated shell process. It's as if there's some sort of inactivity timeout that can't see the rsync transfer running and it's pulled the plug.
It's very peculiar! I added a --progress parameter to the rsync command as that should keep the terminal alive. It gets to around the same place in the copying process and then rapidly slows down and grinds to a halt half way through copying a file. Currently it's stuck at 18% of a largish file. I don't think it's a tiemout as the slowing down and grinding to a halt is pretty rapid, it happens within a minute or so I would guess. Once it's happened I have to kill the terminal window where it's running, nothing else seems to retrieve control. -- Chris Green (chris@areti.co.uk) "Never ascribe to malice that which can be explained by incompetence."
On 11/01/06, Chris Green <chris@areti.co.uk> wrote:
I'm trying to use rsync to upload a mailbox archive from my home system to the system that I usually use for handling my mail.
I'm already successfully using rsync in the opposite direction.
What happens is that rsync gets part of the way through the transfer (it does about 3.9Mb out of 75Mb) and then it just hangs. I've looked at the files it has transferred and the point at which it has stopped and can see no oddities in permissions or anything like that.
Has anyone got any ideas what might be going wrong?
I don't suppose it is hanging on a large compressed file? I had rsync completely hang when it got to a large.iso of 4G or so before. Jen
On Wed, Jan 11, 2006 at 11:23:39AM +0000, Jenny Hopkins wrote:
On 11/01/06, Chris Green <chris@areti.co.uk> wrote:
I'm trying to use rsync to upload a mailbox archive from my home system to the system that I usually use for handling my mail.
I'm already successfully using rsync in the opposite direction.
What happens is that rsync gets part of the way through the transfer (it does about 3.9Mb out of 75Mb) and then it just hangs. I've looked at the files it has transferred and the point at which it has stopped and can see no oddities in permissions or anything like that.
Has anyone got any ideas what might be going wrong?
I don't suppose it is hanging on a large compressed file? I had rsync completely hang when it got to a large.iso of 4G or so before.
None of the files is particularly big, I think the biggest is just one or two megabytes. When run from the command line with the --progress parameter it seems to be getting a little bit further each time. -- Chris Green (chris@areti.co.uk) "Never ascribe to malice that which can be explained by incompetence."
On Wed, Jan 11, 2006 at 11:31:54AM +0000, Chris Green wrote:
On Wed, Jan 11, 2006 at 11:23:39AM +0000, Jenny Hopkins wrote:
On 11/01/06, Chris Green <chris@areti.co.uk> wrote:
I'm trying to use rsync to upload a mailbox archive from my home system to the system that I usually use for handling my mail.
I'm already successfully using rsync in the opposite direction.
What happens is that rsync gets part of the way through the transfer (it does about 3.9Mb out of 75Mb) and then it just hangs. I've looked at the files it has transferred and the point at which it has stopped and can see no oddities in permissions or anything like that.
Has anyone got any ideas what might be going wrong?
I don't suppose it is hanging on a large compressed file? I had rsync completely hang when it got to a large.iso of 4G or so before.
None of the files is particularly big, I think the biggest is just one or two megabytes.
When run from the command line with the --progress parameter it seems to be getting a little bit further each time.
Sorry about this saga! I've tried scp and ftp now as well, they all seem to hang in a similar way when transferring 'large' files, that's files of a few MB. The biggest file I'm trying to copy is about 14MB. Nothing else seems to break, the ADSL continues to function perfectly. The only thing I can think of (for this, not the cron driven one) is an odd interaction between ssh and the file transfer because I'm connecting *into* my system using ssh from a remote system and then running the transfer out from the ssh command prompt. I'll try again this evening when I get home using a local terminal window. -- Chris Green (chris@areti.co.uk) "Never ascribe to malice that which can be explained by incompetence."
Chris Green <chris@areti.co.uk> wrote:
On Wed, Jan 11, 2006 at 11:31:54AM +0000, Chris Green wrote: Sorry about this saga!
I've tried scp and ftp now as well, they all seem to hang in a similar way when transferring 'large' files, that's files of a few MB. The biggest file I'm trying to copy is about 14MB.
Random thought. One thing I've seen before with large files stalling has been where the MTU settings were lower in some part of the network. Usually noticeable on large transfers. Possibly since you're connecting *into* your system maybe your router or the remote system has a lower or higher MTU than your ssh system. Running "ifconfig" will tell you. A value of 1500 is typical for Ethernet/RJ45 networks. If they're all at 1500 then ignore me ;-) Cheers David.
On Thu, Jan 12, 2006 at 08:39:02PM -0000, David Fairey wrote:
Chris Green <chris@areti.co.uk> wrote:
On Wed, Jan 11, 2006 at 11:31:54AM +0000, Chris Green wrote: Sorry about this saga!
I've tried scp and ftp now as well, they all seem to hang in a similar way when transferring 'large' files, that's files of a few MB. The biggest file I'm trying to copy is about 14MB.
Random thought.
One thing I've seen before with large files stalling has been where the MTU settings were lower in some part of the network. Usually noticeable on large transfers.
Possibly since you're connecting *into* your system maybe your router or the remote system has a lower or higher MTU than your ssh system.
Running "ifconfig" will tell you. A value of 1500 is typical for Ethernet/RJ45 networks.
If they're all at 1500 then ignore me ;-)
The mtu on my Linux box is 1500 (just checked). The mtu on the remote system is 1500 (also just checkd) I can't find any reference to MTU in the Zyxel router so I have no idea what that's using. -- Chris Green (chris@areti.co.uk) "Never ascribe to malice that which can be explained by incompetence."
I'm trying to use rsync to upload a mailbox archive from my home system to the system that I usually use for handling my mail.
I'm already successfully using rsync in the opposite direction.
What happens is that rsync gets part of the way through the transfer (it does about 3.9Mb out of 75Mb) and then it just hangs. I've looked at the files it has transferred and the point at which it has stopped and can see no oddities in permissions or anything like that.
The command I'm using is:-
rsync --rsh='/usr/bin/ssh -l chris -x' --recursive \ --exclude=in/ --exclude=li/ --exclude=mfolder \ /home/chris/MailArchive halkidiki.areti.com:
It's normally run by cron in the middle of the night but I'm trying it just now from the command line and the same problem seems to be there.
In my mail I'm getting the following error from cron:-
Read from remote host halkidiki.areti.com: Connection timed out rsync: writefd_unbuffered failed to write 4 bytes: phase "unknown": Broken pipe rsync error: error in rsync protocol data stream (code 12) at io.c(836)
This error arrives 23 minutes (consistent for two nights anyway) after the cron command is run. When run from the command line it looks like it's going to take just as long to fail.
Has anyone got any ideas what might be going wrong?
-- Chris Green (chris@areti.co.uk)
"Never ascribe to malice that which can be explained by incompetence."
_______________________________________________ main@lists.alug.org.uk http://www.alug.org.uk/ http://lists.alug.org.uk/mailman/listinfo/main Unsubscribe? See message headers or the web site above!
Sound to me like an ssh timeout problem not a rsync problem. You have 2 options 1) change the rsync command to rsync --progress -v, that will get some characters flowing over the link and keep it open 2) Edit /etc/sshd/ssh_config to use TCPKeepAlives Hope this helps Stuart
participants (4)
-
Chris Green -
David Fairey -
Jenny Hopkins -
stuart＠dontuse.ms