Does anyone know how I can bypass DNS requests in a general way so that I can do, say: curl http://www.microsoft.com .. but bypassing the DNS lookup so that I can specify the IP I want the http request targetted at? In a simple case I can just add www.microsoft.com to my hosts file, but if I wanted to write (say) a bash or PHP script which did this then it would get messy using hosts that way. What I am looking to do is automate some checks which will run on an Apache server and ensure that all the virtual hosts on the server are working correctly. It is not possible to assume that all site shave the DNS pointed at the server (for example it may be a site which is being migrated to the server at some point in the near future). I may just write my own code to make http(s) requests directly using a socket library or something a bit higher level than that but below curl, but I'd prefer to use a standard tool if possible. -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
On 1 Jun 2009, at 14:14, Mark Rogers wrote:
Does anyone know how I can bypass DNS requests in a general way so that I can do, say: curl http://www.microsoft.com .. but bypassing the DNS lookup so that I can specify the IP I want the http request targetted at?
In a simple case I can just add www.microsoft.com to my hosts file, but if I wanted to write (say) a bash or PHP script which did this then it would get messy using hosts that way.
What I am looking to do is automate some checks which will run on an Apache server and ensure that all the virtual hosts on the server are working correctly. It is not possible to assume that all site shave the DNS pointed at the server (for example it may be a site which is being migrated to the server at some point in the near future).
I may just write my own code to make http(s) requests directly using a socket library or something a bit higher level than that but below curl, but I'd prefer to use a standard tool if possible.
If I understand what you're saying correctly (and it took me a couple of reads to get to the point I'm at now, so it's quite possible I don't..), I think you might want to script something like telnet server.ip.address 80 GET / HTTP/1.1 Host: www.the-virtualhost-you-want-to-test.com That way you are talking to the server you think the site is on, regardless of the DNS. You ought to be able to script the above with some sort of socket library, as you suggested Cheers, Dave -- David Reynolds david@reynoldsfamily.org.uk
David Reynolds wrote:
If I understand what you're saying correctly (and it took me a couple of reads to get to the point I'm at now, so it's quite possible I don't..), I think you might want to script something like
telnet server.ip.address 80 GET / HTTP/1.1 Host: www.the-virtualhost-you-want-to-test.com
That way you are talking to the server you think the site is on, regardless of the DNS.
You're spot on with what I'm trying to do. And yes, the above should work. However, using something like curl would allow me to handle stuff like sessions and https without jumping through too many hoops, if I decide to do that (I don't really need to right now, though). Thanks for clarifying what I am trying to do, though, since I did a bad job of explaining it! -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
On 01 Jun 15:52, Mark Rogers wrote:
David Reynolds wrote:
If I understand what you're saying correctly (and it took me a couple of reads to get to the point I'm at now, so it's quite possible I don't..), I think you might want to script something like
telnet server.ip.address 80 GET / HTTP/1.1 Host: www.the-virtualhost-you-want-to-test.com
That way you are talking to the server you think the site is on, regardless of the DNS.
You're spot on with what I'm trying to do. And yes, the above should work.
However, using something like curl would allow me to handle stuff like sessions and https without jumping through too many hoops, if I decide to do that (I don't really need to right now, though).
Thanks for clarifying what I am trying to do, though, since I did a bad job of explaining it!
wget will let you override headers... so... wget --header="Host: foo.bar" http://the.ip.add.ress/ Would be very similar to what you want. Hope that 'elps. Cheers, -- Brett Parker
Martijn Koster wrote:
curl -H "Host: www.google.co.uk" http://209.85.229.147/
.. and Brett Parker wrote:
wget will let you override headers... so...
wget --header="Host: foo.bar" http://the.ip.add.ress/
Would be very similar to what you want.
Those two look perfect, thanks! -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Mark Rogers wrote:
Those two look perfect, thanks!
In case anyone cares, the PHP version would be (using the curl library): <?php $ch = curl_init('http://209.85.229.147/search?q=test'); curl_setopt ($ch, CURLOPT_HTTPHEADER, array('Host: www.google.co.uk')); curl_setopt ($ch, CURLOPT_RETURNTRANSFER, TRUE); $html = curl_exec($ch); echo $html; ?> -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
On 01 Jun 14:14, Mark Rogers wrote:
Does anyone know how I can bypass DNS requests in a general way so that I can do, say: curl http://www.microsoft.com .. but bypassing the DNS lookup so that I can specify the IP I want the http request targetted at?
In a simple case I can just add www.microsoft.com to my hosts file, but if I wanted to write (say) a bash or PHP script which did this then it would get messy using hosts that way.
What I am looking to do is automate some checks which will run on an Apache server and ensure that all the virtual hosts on the server are working correctly. It is not possible to assume that all site shave the DNS pointed at the server (for example it may be a site which is being migrated to the server at some point in the near future).
I may just write my own code to make http(s) requests directly using a socket library or something a bit higher level than that but below curl, but I'd prefer to use a standard tool if possible.
Alternatively, this is really simple in python... --- Begin Snippet --- import urllib2 req = urllib2.Request("http://ip.add.of.server/path/to/thing/") req.add_header("Host", "the hostname you really want to present") resp = urllib2.urlopen(req) data = resp.read() --- End Snippet --- Now the content will be in data (unless it hit a 404, in which case it'll have thrown an Exception and you'll be outside of the scope of the snippet ;) Cheers, -- Brett Parker
Brett Parker wrote:
--- Begin Snippet ---
import urllib2
req = urllib2.Request("http://ip.add.of.server/path/to/thing/") req.add_header("Host", "the hostname you really want to present") resp = urllib2.urlopen(req) data = resp.read()
--- End Snippet ---
Note to self: really must learn python! That said I think I can replicate the above in PHP. Thanks for that, I should have thought about it that way round myself. (I'd been thinking about how to connect by hostname but override the IP, rather than connect by IP but then override the host.) -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
On 01 Jun 15:54, Mark Rogers wrote:
Brett Parker wrote:
--- Begin Snippet ---
import urllib2
req = urllib2.Request("http://ip.add.of.server/path/to/thing/") req.add_header("Host", "the hostname you really want to present") resp = urllib2.urlopen(req) data = resp.read()
--- End Snippet ---
Note to self: really must learn python! That said I think I can replicate the above in PHP.
Thanks for that, I should have thought about it that way round myself. (I'd been thinking about how to connect by hostname but override the IP, rather than connect by IP but then override the host.)
Well, the hostname is part of the HTTP/1.1 spec, it doesn't exist in HTTP/1.0 and so it's the right way round to think of it ;) Back in the HTTP/1.0 days there wasn't a way to have multiple virtual hosts on the same IP (well, other than different port numbers). I tend to think things through in terms of the specs, which is quite handy :) But, anyways - see the other post in this thread for how to do it with wget - there's probably a way of doing that with curl but I don't know how at the moment... oh, apparently... curl -H "Host: the.host.name" http://the.ip.add.ress/ Should do what you're after too :) Cheers, -- Brett Parker
Brett Parker wrote:
Well, the hostname is part of the HTTP/1.1 spec, it doesn't exist in HTTP/1.0 and so it's the right way round to think of it ;)
Fair point! My normal (non-scripted) workaround would be to add an entry to my hosts file, which is why I came at it from that angle. Very easy not to stand back from something and see the whole picture! -- Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555 Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG
Mark Rogers wrote:
Does anyone know how I can bypass DNS requests in a general way so that I can do, say: curl http://www.microsoft.com .. but bypassing the DNS lookup so that I can specify the IP I want the http request targetted at? curl -H "Host: www.google.co.uk" http://209.85.229.147/
-- Martijn
participants (4)
-
Brett Parker -
David Reynolds -
Mark Rogers -
Martijn Koster