I started learning Python last night. I will be lying if I say “it’s good”. I’ll tell you what, “it’s f**king good”. I am a polyglot programmer and diving in Python was a cinch. It felt as if I knew it already, because “it makes sense”. I’ve been playing around with Python 3.4 and here’s an experiment.
I wanted to explore some modules to take my learning to the next level. In a night, I’ve caught up on a lot of Pythonic ways, ‘pip’, ‘modules’, ‘syntax’ and more. Python is the “size 0” of the programming world. Don’t be alarmed, it just means “light-weight”.
To be honest, I’ve been hacking Python and reading Pythonic code for a long time, but it’s only recently that I started getting serious about formally learning the language. I am working on a startup and decided to use Python for building the machine learning layer.
Anyway, here’s the little experiment. I tried to test out Requests, a nice HTTP library “for humans”, but I found Async HTTP Requests to be more efficient and interesting. The best library, for the task, is “requests-futures“. It’s quite well maintained.
“Requests-futures” makes use of Python 3’s concurrent.futures module. This module provides a high level interface for asynchronously executing callables. It ships with all Python versions 3.2 and beyond. Async I/O magic. Love the sound of that!
Installing Requests-Futures
To install requests-futures, you must have Python 3.2+ and PIP. On a Mac, this would be:
#install requests
pip3 install requests
#install requests-futures
pip3 install requests-futures
#check modules
python3
help('modules')
#ensure requests and requests-futures
#are listed in the modules section
Okay, so looks like you have successfully got these two libraries up and running. Let’s tap into their powers and perform some asynchronous tasks, like hitting up URLs in parallel and getting their response codes.
# Asynchronous HTTP Requests in Python 3.4
from requests_futures.sessions import FuturesSession
# max workers set to 10, default is 2
session = FuturesSession(max_workers=10)
# requests are started in background
future_one = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_two = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_three = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_four = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_five = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
# just a line break for readibility
future_six = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_seven = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_eight = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_nine = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
future_ten = session.get('http://cdn.api.twitter.com/1/urls/count.json?url=http://www.aligajani.com')
# wait for the requests to complete, if they haven't already
response_one = future_one.result()
print('response one status: {0}'.format(response_one.status_code))
response_two = future_two.result()
print('response two status: {0}'.format(response_two.status_code))
response_three = future_three.result()
print('response two status: {0}'.format(response_three.status_code))
response_four = future_four.result()
print('response two status: {0}'.format(response_four.status_code))
response_five = future_five.result()
print('response two status: {0}'.format(response_five.status_code))
response_six = future_six.result()
print('response two status: {0}'.format(response_six.status_code))
response_seven = future_seven.result()
print('response two status: {0}'.format(response_seven.status_code))
response_eight = future_eight.result()
print('response two status: {0}'.format(response_eight.status_code))
response_nine = future_nine.result()
print('response two status: {0}'.format(response_nine.status_code))
response_ten = future_ten.result()
print('response two status: {0}'.format(response_ten.status_code))
GIF Animation
I have recorded a short GIF of how changing the “max_workers” number in the FutureSession constructor impacts the parallelism. Quite obvious, but quite visibly true. See the terminal below. With 10 URLs and 10 concurrent workers, it goes like “boom”. 200 goodness all the way.
Conclusion
Hope you liked this little nugget of an idea. Subscribe, like and keep in touch for more. Also, let me know if you are aware of an elegant way of doing this very thing. I am sure this isn’t scalable if I am pulling URLs from a database, but well, you get the idea. Adios.
About Ali Gajani
Hi. I am Ali Gajani. I started Mr. Geek in early 2012 as a result of my growing enthusiasm and passion for technology. I love sharing my knowledge and helping out the community by creating useful, engaging and compelling content. If you want to write for Mr. Geek, just PM me on my Facebook profile.
