Downloading Large Files Async With GIO

Slightly technical question for my friends on Planet GNOME. I have been hunting around for some help online with no luck, so I figured I would post here and hopefully this blog entry can be a solution for those who have similar questions.

I am in the process of porting App Of Jaq to to async. To do this I am using gio and have the code that downloads XML feeds up and running pretty well, and the app feels much more responsive. Now I need to have the application download an Ogg asynchronously without freezing the GUI.

My code currently looks like this:

def download_latest_shot(self):
    audiourl = "http://....the url to the Ogg file...."

    self.shot_stream = gio.File(audiourl)
    self.shot_stream.read_async(self.download_latest_shot_complete)

It then calls this callback:

def download_latest_shot_complete(self, gdaemonfile, result):
    f = self.shot_stream.read_finish(result).read()

    outputfile = open("/home/jono/Desktop/shot.ogg","w")
    outputfile.writelines(f)

Now, I am still pretty new to this, and while this code does work, it freezes the GUI pretty hard. Can anyone recommend some next steps for how I can download the file without it freezing? Some example code would be great. :-)

Thanks in advance for anyone who can help. :-)

  • http://blogs.gnome.org/jessevdk Jesse van den Kieboom

    The async_read on the GFile is just to open the file for reading. The read_finish will then result in an InputStream. In your little snippet you then call the sync read method on the stream, which just blocks everything again. Instead use the async methods of the gio input stream.

  • jono

    Thanks for the response, Jesse! Do you know which async methods I should be using?

  • jono

    Jesse, one other thing: I am using http://faq.pygtk.org/index.py?req=show&file=faq20.021.htp as my point of reference on this which is why I use read_finish(). Thanks!

  • Hongli Lai

    I think that you need to call read_finish() on the read_finish(result) result.

    After looking at the PyGTK docs I can understand why this is not immediately obvious.

  • http://blog.jpetersen.org Jan Arne Petersen

    The methods for async loading the contents of a file are load_contents_async and load_contents_finish

  • Hongli Lai

    Forget my last comment, I was confused. :)

    self.shot_stream.read_finish(result) returns a gio.FileInputStream, and you need to call the async_read() method on that object just as with gio.File. Except this time you’re not just opening the file but actually reading the file data. Use read_finish(result) on the gio.FileInputStream to get the result. In the callback for the gio.FileInputStream read_async call, you need to call read_async again to continue reading; keep doing this until read_async returns 0 which signifies EOF.

  • herodiade

    Hi, I’m not an expert but I think at least the write isn’t async here (this alone can freeze, esp. on slow disks). And I think (not sure) the read callback will be triggered as soon as the fd opened by read_async() is available (if so read_finish(result).read() will block).

    There’s a complete and working example (using gio’s copy_async()) in Rhythmbox source code : http://osdir.com/ml/general/2010-02/msg24291.html http://git.gnome.org/browse/rhythmbox/tree/plugins/magnatune/magnatune/MagnatuneSource.py

    The faq.pygtk example isn’t likely to block because it doesn’t write() as your code snippet do, and only display 100 bytes (less than a single tcp packet, should be fully available in one shot when the read callback is triggered or soon after).

  • Tomeu Vizoso

    Already considered connecting both streams with gio.OutputStream.splice_async()?

  • Alexander Larsson

    If you want the whole file contents then you should use load_contents_async/load_contents_finish as Jan said. read_async() just returns an opened stream you can read from.

    However, all this is void unless you also enable threads by calling g_thread_init() (or whatever this is in python) early in your app. Its not possible to implement async local file i/o without threads, so when its not enabled gio “emulates” it by reading from idle handlers.

    Additionally, this part of your code: outputfile = open(“/home/jono/Desktop/shot.ogg”,”w”) outputfile.writelines(f)

    Is not async and will block your app.

  • http://www.joaquimrocha.com Joaquim Rocha

    Hi Jono,

    I have a simple Python app for the N900 and I make a few async calls in it.

    The concept is really simple, I have an Async worker which is a thread and has a queue of items to be dealt with. An item receives a target method to execute and a callback to be called when it is done, each with its respective arguments.

    Then the items are put in the queue and the async worker just goes through running them.

    Here you have the code for the AsyncWorker and Items:

    http://gitorious.org/seriesfinale/seriesfinale/blobs/master/src/SeriesFinale/asyncworker.py

    Take a look here for an example of how it’s used:

    http://gitorious.org/seriesfinale/seriesfinale/blobs/master/src/SeriesFinale/series.py

    Hope it helps you,

  • http://punchingbagset.info/punching_bag_stands/punch-bag-stand/ punch bag stand | punching bag sets

    [...] Downloading Large Files Async With GIO | jonobacon@home [...]

  • Johan Dahlin
  • Stuart Ward

    Wouldn’t it be easier to use subprocess module to execute a wget commend. command = ['wget',path_name] wget_process = subprocess.Popen(command)

    You can then check on when it is complete with the wget_process.returncode == None if it is still running.

  • Benjamin Otte

    Why don’t you just copy the file using http://library.gnome.org/devel/gio/stable/GFile.html#g-file-copy-async – or its pygtk equivalent?

  • http://ch.soup.io chris

    If you just want to download the file you probably should use copy_async().

  • http://www.jonobacon.org/2010/03/15/download-files-async-with-gio-and-python/ Download Files Async With Gio And Python | jonobacon@home

    [...] I asked for some help on how to download a file without blocking the GUI. Thanks to everyone who contributed their expertise in the post comments: I now have my program [...]