A question for the Python Programmers

[#1]

Has anyone done any socket programming with Python on Windows?

I have written code that works perfectly well on my iMac but blocks when run on Windows 7 and 10!

The code blocks on:

iSocket.recvfrom(512)

So to prevent it from locking the application up as the method never returns I have to set the socket to non blocking mode:

iSocket.setblocking(0)
ready = select.select([iSocket], [], [], 5)
    if (ready[0]):
         bytes, addr = iSocket.recvfrom(512)

But as a consequence I get no data (bytes) returned, regardless of the timeout, but on the iMac the response from either method is near instant.

Because this uses raw sockets the code has to be run as root on the Mac, or administrator on Windows but this is not the issue here. If anyone has any suggestions other than searching the internet for something I can’t find I’d really appreciate the insight.

[#2]

If its blocking, its likely that the Windows PC isn't connecting to the other end and the outbound connection is getting stopped. It could be that an Anit Virus product or the built in Windows security is blocking the connection. Windows has a built in Firewall that blocks by default, you would need to create a rule that allows Python.exe to establish connections.

see: https://stackoverflow.com/questions/36646093/allowing-a-program-through-windows-firewall

The stackoverflow post is for inbound, but you can equally create an outbound rule. Some Anti Virus / Security Products replace the Windows Firewall with their own firewalls that work in a similar way. If you are using one of these you can check the manufacturers website to see how to create a rule.

A quick and dirty test may be to temporarily disable the Windows Firewall  / AV suite and see if the code works. Although I wouldn't recommend leaving it that way for any length of time, but at least you would prove what the issue was.

[#3]

Thanks for the reply @mikelahey but the issue isn’t down to AV or Firewall as I have other Python code that works as expected in the same application.

The code in question is in fact my attempt at implementing a traceroute in Python, my ping implementation works as expected which makes this even more confusing, so it looks like I’ll need to get the sleeves rolled up on this one.

[#4]

I don't do python but I do object pascal using indy sockets which are blocking so i'm poking at shadows with similar experience.

The only thing I can think of is that there isn't any data actually being sent to your app. Blocking receive calls can sit there forever waiting if there is no data available to them. If you're also not getting the expected data when using non-blocking then this would (IMO) point at the lack of available data again.

Where is the expected data being sent from and are you sure it's actually being sent? - Can you confirm that there is definitely a connection between the two points?

[#5]

I've had the opposite problem, which may be related - probably I'm wrong knowing me.

I wrote a Mac XCode app just to get content from a site. Worked for some sites, not others. I can't remember the exact error message but would just jump into the exception handler. So I couldn't understand why it is for some and not others.

Turned out the API would only fetch content from a HTTPS connection and deny you HTTP by default. There was some parameter you had to pass in.

It was doing my head in and got it fixed - so I remember reading BBC News from the HTML source once it did work from the XCode debug window 

One original way of reading the news I suppose - but I was relived in the end. 

[#6]

Thanks for the input guys. @7up I quite agree with your comment regarding the lack of data but I have just found this on the net:

https://gist.github.com/jcjones/0f3f11a785a833e0a216 (A Python TraceRoute!)

But it doesn’t work for me and blocks on the receive! Is anyone able to test this for me please?

I’m Using Python 3.5 On Windows 10 for this.

[#7]

I couldn't get the example you linked to to work on Python 3.6 on Windows, it gave me syntax errors when I tried to run it.

I did try this: https://github.com/wsimmerson/traceroute/blob/master/traceroute.py

But every response timed out after one second, even when running as admin with the firewall turned off.

The Python docs for sockets have an ominous postscript at the bottom:

"Portability alert: On Unix, select works both with the sockets and files. Don’t try this on Windows. On Windows, select works with sockets only. Also note that in C, many of the more advanced socket options are done differently on Windows. In fact, on Windows I usually use threads (which work very, very well) with my sockets. Face it, if you want any kind of performance, your code will look very different on Windows than on Unix."

[#8]

I've a suspicion, yet unproven, that the Python socket is not getting to the physical network interface. I need to wireshark it to see what's happening, but I haven't got the time to look into it further at the moment.

Even traceroute to localhost fails on Windows.

[#9]

The syntax errors can be removed by changing the following, line 38 should be:

... .sendto(b’’, (dest_name, port))

and line 52 should be :

… socket.error as e:

The line numbers are based on the line numbers on gitgub.

On running the example posted by @mikelahey, I got the same result with the recvfrom timing out after the one second timeout interval.

Odd, very odd.

[#10]

OK, I've made the mods to the code you linked to get rid of the erros and run it while wireshark is running. The send side seems to be working, the issue is on the receive side, the Python Code is not binding to host network interface for ICMP Packets so they are getting thrown out by windows.

Error is : "Destination unreachable (Port Unreachable)"

[#11]

I've been doing some more thinking about this and it may be a timing issue in that the response is coming back before the receiving socket is established by Python. I also read somewhere that RAW Sockets aren't properly implemented on Windows after Windows XP SP1. 

There are numerous unanswered posts on Stackoverflow from people who have tried to get this to work on Windows and failed, so we're not alone in that.

I think that the way to get this to work may be to have the receiver and sender running on separate threads as eluded to in the Python docs, however multi threaded Python apps are beyond my ability in Python.

[#12]

This really smells of Windowsitis to me, so I’m going to change my code to send / receive an ICMP Echo request / response to see if that works. This is at the moment a pet project for me and I have other things I need to do today so I’m going to have to come back to it later, but I will post an update later in the week.

If the aforementioned changes fail then I’ll do a version in C++ and see if the level of success changes.

In the meantime thanks for the input, and if anything else occurs to you please share it.

While I was composing this a new reply arrived regarding a possible solution could be threading the calls so I’ve threaded the calls and it still doesn’t work!

def traceRoute(self):
        ttl = 1
        max_hops = 30
        quit = None
        def sendCallback(ttl):
            print("Send Thread...")
            try:
                oSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM,  socket.getprotobyname('udp'))
                oSocket.setsockopt(socket.SOL_IP, socket.IP_TTL, struct.pack('b', ttl))
                oSocket.sendto(b'', (socket.gethostbyname("NAS"), self.port))
            except socket.error as se:
                print ("Send %s" % se)
            print("Send Thread. Done. TTL : %d" % ttl)
            
        def recvCallback():
            print("Recieve Thread...")
            try:
                rSocket = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.getprotobyname('icmp'))
                #Remove the comment marker from the line below in set the timeout to 5 seconds
                #rSocket.setsockopt(socket.SOL_SOCKET, socket.SO_RCVTIMEO, struct.pack("ll", 5, 0))
                rSocket.bind(('', 33434))
                tries = 3
                finished = False
                while not finished and tries > 0:
                    try:
                        bytes, addr = rSocket.recvfrom(512)
                        finished = True
                        if (addr != None):
                            addr = addr[0]
                            try:
                                name = socket.gethostbyaddr(addr)[0]
                                print("Name : %s " % name)
                            except socket.error as se:
                                name = addr
                    except socket.error as se:
                        tries -= 1
            except socket.error as se:
                print ("Recv %s" % se)
            print("Recieve Thread. Done")
            
        while 1:
            try:
                r = Thread(target=recvCallback)
                s = Thread(target=sendCallback, kwargs={'ttl': ttl})
                r.start()
                s.start()
                r.join()
                s.join()

                ttl += 1
                if (ttl > max_hops):
                    break
            except Exception as e:
                print("Exception : %s in TraceRoute" % e)
                quit = True
            if (quit != None): break
        return None