Posted on March 15, 2010 - by jono
Downloading Large Files Async With GIO
Slightly technical question for my friends on Planet GNOME. I have been hunting around for some help online with no luck, so I figured I would post here and hopefully this blog entry can be a solution for those who have similar questions.
I am in the process of porting App Of Jaq to to async. To do this I am using gio and have the code that downloads XML feeds up and running pretty well, and the app feels much more responsive. Now I need to have the application download an Ogg asynchronously without freezing the GUI.
My code currently looks like this:
def download_latest_shot(self):
audiourl = "http://....the url to the Ogg file...."
self.shot_stream = gio.File(audiourl)
self.shot_stream.read_async(self.download_latest_shot_complete)
It then calls this callback:
def download_latest_shot_complete(self, gdaemonfile, result):
f = self.shot_stream.read_finish(result).read()
outputfile = open("/home/jono/Desktop/shot.ogg","w")
outputfile.writelines(f)
Now, I am still pretty new to this, and while this code does work, it freezes the GUI pretty hard. Can anyone recommend some next steps for how I can download the file without it freezing? Some example code would be great. 
Thanks in advance for anyone who can help.
16 Comments
We'd love to hear yours!
Leave a Reply
Here's your chance to speak.
-
Flickr Photos
Visit My Website
March 15, 2010
Permalink
The async_read on the GFile is just to open the file for reading. The read_finish will then result in an InputStream. In your little snippet you then call the sync read method on the stream, which just blocks everything again. Instead use the async methods of the gio input stream.
Visit My Website
March 15, 2010
Permalink
Thanks for the response, Jesse! Do you know which async methods I should be using?
Visit My Website
March 15, 2010
Permalink
Jesse, one other thing: I am using http://faq.pygtk.org/index.py?req=show&file=faq20.021.htp as my point of reference on this which is why I use read_finish(). Thanks!
Visit My Website
March 15, 2010
Permalink
The methods for async loading the contents of a file are load_contents_async and load_contents_finish
Visit My Website
March 15, 2010
Permalink
I think that you need to call read_finish() on the read_finish(result) result.
After looking at the PyGTK docs I can understand why this is not immediately obvious.
Visit My Website
March 15, 2010
Permalink
Forget my last comment, I was confused.
self.shot_stream.read_finish(result) returns a gio.FileInputStream, and you need to call the async_read() method on that object just as with gio.File. Except this time you’re not just opening the file but actually reading the file data. Use read_finish(result) on the gio.FileInputStream to get the result. In the callback for the gio.FileInputStream read_async call, you need to call read_async again to continue reading; keep doing this until read_async returns 0 which signifies EOF.
Visit My Website
March 15, 2010
Permalink
Hi, I’m not an expert but I think at least the write isn’t async here (this alone can freeze, esp. on slow disks). And I think (not sure) the read callback will be triggered as soon as the fd opened by read_async() is available (if so read_finish(result).read() will block).
There’s a complete and working example (using gio’s copy_async()) in Rhythmbox source code : http://osdir.com/ml/general/2010-02/msg24291.html http://git.gnome.org/browse/rhythmbox/tree/plugins/magnatune/magnatune/MagnatuneSource.py
The faq.pygtk example isn’t likely to block because it doesn’t write() as your code snippet do, and only display 100 bytes (less than a single tcp packet, should be fully available in one shot when the read callback is triggered or soon after).
Visit My Website
March 15, 2010
Permalink
Already considered connecting both streams with gio.OutputStream.splice_async()?
Visit My Website
March 15, 2010
Permalink
If you want the whole file contents then you should use load_contents_async/load_contents_finish as Jan said. read_async() just returns an opened stream you can read from.
However, all this is void unless you also enable threads by calling g_thread_init() (or whatever this is in python) early in your app. Its not possible to implement async local file i/o without threads, so when its not enabled gio “emulates” it by reading from idle handlers.
Additionally, this part of your code: outputfile = open(“/home/jono/Desktop/shot.ogg”,”w”) outputfile.writelines(f)
Is not async and will block your app.
Visit My Website
March 15, 2010
Permalink
Hi Jono,
I have a simple Python app for the N900 and I make a few async calls in it.
The concept is really simple, I have an Async worker which is a thread and has a queue of items to be dealt with. An item receives a target method to execute and a callback to be called when it is done, each with its respective arguments.
Then the items are put in the queue and the async worker just goes through running them.
Here you have the code for the AsyncWorker and Items:
http://gitorious.org/seriesfinale/seriesfinale/blobs/master/src/SeriesFinale/asyncworker.py
Take a look here for an example of how it’s used:
http://gitorious.org/seriesfinale/seriesfinale/blobs/master/src/SeriesFinale/series.py
Hope it helps you,
Visit My Website
March 15, 2010
Permalink
A mostly async example is available here:
http://git.gnome.org/browse/pygobject/tree/examples/gio/downloader.py
Visit My Website
March 15, 2010
Permalink
Wouldn’t it be easier to use subprocess module to execute a wget commend. command = ['wget',path_name] wget_process = subprocess.Popen(command)
You can then check on when it is complete with the wget_process.returncode == None if it is still running.
Visit My Website
March 15, 2010
Permalink
Why don’t you just copy the file using http://library.gnome.org/devel/gio/stable/GFile.html#g-file-copy-async – or its pygtk equivalent?
Visit My Website
March 15, 2010
Permalink
If you just want to download the file you probably should use copy_async().