cancel
Showing results for 
Search instead for 
Did you mean: 

A Linux / Unix question for you...

mcgurka
Grafter
Posts: 764
Registered: ‎09-10-2007

A Linux / Unix question for you...

Right, OK, it goes like this!
In Dundee, there are a number of core machines which function as routers within the network, and they have graphing capabilities on them. Access to the graphs is UN/PW protected, but you can access the files by SSH into the box.
What I would like to do is periodically ( 5 mins or so ) grab the files, and transfer them (possibly by FTP?) to a central machine for processing..
I know I would use cron jobs to schedule the transfers.. but as to how to do the actually push of the files Im not so sure!
Hopefully there is a linux guru out there who can help Smiley
16 REPLIES
Community Veteran
Posts: 1,264
Registered: ‎04-04-2007

Re: A Linux / Unix question for you...

Last time I needed something to push something on a schedule via FTP I just used a Perl Script and the Net::FTP module... but I'm not 100% sure that's what you're trying to do.
Plusnet Alumni (retired) orbrey
Plusnet Alumni (retired)
Posts: 10,540
Registered: ‎18-07-2007

Re: A Linux / Unix question for you...

Surely you could add a line to crontab (on either your own machine or the server) that will scp the relevant files to you after your given interval?
Community Veteran
Posts: 4,729
Registered: ‎04-04-2007

Re: A Linux / Unix question for you...

I believe that you can direct a text file into ftp with the command to connect and transfer the files.
ftp < ftpcommandfile

[quote=ftpcommandfile]
USER
username
PASS
passwordinplaintext
ls
etc
mcgurka
Grafter
Posts: 764
Registered: ‎09-10-2007

Re: A Linux / Unix question for you...

Right, OK, i will clarify Smiley
It need not be only FTP, im open to any file transfer options.
Pretty much, the setup is like this...
Gardner-St.Core - 192.168.0.1 (servs GS)
Dock-St-East.core - 192.168.10.1 (servs DSE)
Dock-St-West.Core - 192.168.20.1 (servs DSW)
There are more, but that should suffice for now
There is a monitoring box which sits in the managed lan, 192.168.254.27 I belive it is.
All of the core nodes, generate graphs. One such example is the data file "/var/dscn/logging/stat-wan-qos-queues.rrd" or "/var/dscn/logging/stat-global-traffic-load.rrd". What im am after, is to push these files out to 192.168.254.27 at, say 5 min intervals or such so they can be processed and displayed centrally.
Hope that clears up what Im trying to do Smiley
Community Veteran
Posts: 3,789
Registered: ‎08-06-2007

Re: A Linux / Unix question for you...

Investigate scp.

scp allows you to copy files from remote servers using an ssh tunnel
The most basic usage is something like:
scp root@radagast:/var/log/messages ./messages

Where root is the remote user, and radagast is the name of the server (or IP address)
This uses an interactive prompt for the password.  For security, you cannot pass the password on the command line.
There is a way to prevent the remote server requesting a password, which you can do by doing the following:

  • Decide which user on the local machine will be using scp later on. Of course, root gives you the most power, and that's how I personally have done it. I'm not going to give you a lecture here on the dangers of root, so if you don't understand them, choose a different user. Whatever you choose, log in as that user now and stay there for the rest of the procedure. Log in as this same user when you use scp later on.


  • Generate a public/private key pair on the local machine. Say what? If you're not familiar with public key cryptography, here's the 15-second explanation. In public key cryptography, you generate a pair of mathematically related keys, one public and one private. You then give your public key to anyone and everyone in the world, but you never ever give out your private key. The magic is in the mathematical makeup of the keys; anyone with your public key can use it to encrypt a message, but only you can decrypt it with your private key. Anyway, the syntax to create the key pair is:
    ssh-keygen -t rsa


  • In response, you should see:
          Generating public/private rsa key pair
          Enter file in which to save the key ...
    Press Enter to accept this.


  • In response, you should see:
          Enter passphrase (empty for no passphrase):
    You don't need a passphrase, so press Enter twice.


  • In response, you should see:
          Your identification has been saved in ...
          Your public key has been saved in ...
    Note the name and location of the public key just generated. It always ends in .pub.

  • Copy the public key just generated to all of your remote Linux boxes. You can use scp or FTP or whatever to make the copy. Assuming you're using root--again, see my warning in step 1--the key must be contained in the file /root/.ssh/authorized_keys. Or, if you are logging in as a user, for example, clyde, it would be in /home/clyde/authorized_keys. Notice that the authorized_keys file can contain keys from other PCs. So, if the file already exists and contains text, you need to append the contents of your public key file to what already is there.


That should allow you to issue the scp command (and in fact ssh directly) to automatically copy files using a cron job or similar.
A word about security. This local PC just became pretty powerful, as it now has access to all the remote PCs with only the one local password. So that one password better be strong and well guarded.
B.
Community Veteran
Posts: 1,264
Registered: ‎04-04-2007

Re: A Linux / Unix question for you...

Something like:
scp -r /var/dscn/logging/* user@192.168.254.27:/var/dscn/logging/`hostname`/

Would SCP them to a folder named after the hostname of the server on the monitoring server...
Something tells me that RRD files are architecture-dependant though so you may find they don't work...
mcgurka
Grafter
Posts: 764
Registered: ‎09-10-2007

Re: A Linux / Unix question for you...

Dependant as in directory structure, or OS kernel? If its the latter, then they are both running the same OS.
Im also looking at rsync, although the ssh method by barry seems to be pretty straight forward.
I suppose its very much 'suck-it-and-see' as far as wether the graphing works or not!
Community Veteran
Posts: 1,239
Thanks: 4
Registered: ‎30-07-2007

Re: A Linux / Unix question for you...

Can I piggyback on this thread as I am an inveterate penguin-fiddler, but this is the first time I've come across scp ...
A couple of questions for Barry:
Does scp automatically detect the presence of the key-pairs and therefore not require further intervention? (so authentication when using scp becomes transparent to the user "as if" it were a cp on the same box)?
Could  I use  puTTY on an XP box and connect transparently using rsa key-pairs generated with puTTYgen to a Linux box? Or would I have to have a Linux pscp ? (I realise that this may have to be asked of  Simon Tatham, but you may know)
EDIT: Should have tried it before I posted, which I have now done, and the answers are Yes, Yes, Yes and  No
paul
Community Veteran
Posts: 3,789
Registered: ‎08-06-2007

Re: A Linux / Unix question for you...

Quote from: paulh
Does scp automatically detect the presence of the key-pairs and therefore not require further intervention? (so authentication when using scp becomes transparent to the user "as if" it were a cp on the same box)?

Yes is the short answer.
SSH supports Public Key authentication, and scp is effectively opening an ssh connection on your behalf.  It is actually SSH that is establishing the secure connection in the background.  SCP merely tunnels over it.
Quote from: paulh
Could  I use  puTTY on an XP box and connect transparently using rsa key-pairs generated with puTTYgen to a Linux box? Or would I have to have a Linux pscp ? (I realise that this may have to be asked of  Simon Tatham, but you may know)


I use putty extensively, but have never managed to get it to work nicely with OpenSSH on the server.  I'll play a bit more with it to see if I can figure it out.
B.
Community Veteran
Posts: 1,239
Thanks: 4
Registered: ‎30-07-2007

Re: A Linux / Unix question for you...

It turned out ever so simple:
On Linux (in my case Puppy 3.01) run ssh-keygen -t rsa
On XP run puTTYgen, generate a SSH-1 rsa keypair, save the private key as a .ppk file.
Create an authorized_keys file on the Linux machine (in ./.ssh/) with the public key from puttygen pasted into it.
On XP run
putty -i whatever.ppk user@host
and robert's yer father's brother
pscp works equally tidily:
pscp -i whatever.ppk user@host:file file

very nice -- glad I spotted this thread. I've learned something useful.
paul
Community Veteran
Posts: 1,264
Registered: ‎04-04-2007

Re: A Linux / Unix question for you...

You may run in to a problem if your SSH keypair has a passphrase though, as you'd need to type that in -- although there's ssh-agent as well which you can add your keys to, to get round that problem Smiley
mcgurka
Grafter
Posts: 764
Registered: ‎09-10-2007

Re: A Linux / Unix question for you...

Ok, so Ive got my keypair and things between my two servers, scp file copy all ready with the command;
# scp test.rrd usr@80.229.xxx.yyy:/var/rrd/gdst/
Now the problem I have, is that ssh runs on the remote server, for about 5 seconds, before it stops and kicks all clients offline, and only a physical reboot of the server will bring it back again!
Anybody any ideas about this, its a complete new built server!
--Confused as heck--
Community Veteran
Posts: 1,239
Thanks: 4
Registered: ‎30-07-2007

Re: A Linux / Unix question for you...

dunno which Linux you're running  ... but you should be able to start sshd with a debug or high-loglevel option to get some clues.
also post the sshd startup line from rc.d or whatever
have a sniff round the forum for your distro as well, as there's usually someone who's had a similar problem or is willing to help
always good to have a look at the kernel logs and messages file too.
mcgurka
Grafter
Posts: 764
Registered: ‎09-10-2007

Re: A Linux / Unix question for you...

well, it gets more confusing...
according to the service manager, the service is stil running, it hasnt stopped, and has generated not output errors at all!
The dristo im running is Fedora Core 8 running on 2 x 2.66Ghz Dual Core Xeon, 6Gb ram, 5 x 146Gb hdd so the platform should be ok for it.
I will pull that starter lines for it out of the kernel file later on today.
Scott