Debugging Jokosher Guide

Hello everyone. I like Jokosher. I think its a cool piece of software, with the right direction and a great bunch of people working on. Sure, I am biased, but nonetheless, Jokosher is an exciting project that has the opportunity to really redefine how we record music on a free software platform. But we have a problem…

…we need more testers. We need more people testing Jokosher, finding problems and reporting those problems. Now, some people are doing this and reporting bugs and reporting things on our excellent forums, but what would be really useful would be if some people could have a go at debugging Jokosher, and this in turns means being able to debug a GStreamer application.

We had a meeting about the new Jokosher QA Team yesterday and I agreed to write up a quick guide to getting started doing this kind of debugging. Hopefully this will help some of you get started in this incredibly important aspect of Jokosher. If you read this guide, it should give you a good solid grounding in how to debug Jokosher and contribute essential debugging information to our developers who can fix the problems. :)

A bit of preparation

Before you begin, I recommend you get a quick overview of how GStreamer works. You basically need to know that you string together elements (plugins that do something specific such as reading in a file, decoding music or applying effects) and those elements are linked and have their pads connected together. I wrote a few little guides that explain how GStreamer and Gnonlin work. Although these guides cover how to write code, read them and understand the theory in them and it should give you a firm grounding in GStreamer. They are at:

You should also take a look at the diagram of the Jokosher pipeline:

This pipeline shows all of the elements and how they fit together in Jokosher.

In terms of software, it is recommended you are running Jokosher from SVN trunk and running a CVS copy of GStreamer. Instructions for this can be found here.

Getting started

The first thing you should do is initialise your CVS GStreamer setup. If you are using the guide linked above, you run a script that initialised CVS GStreamer for you. For me I run:

jono@forge:~$ bin/gst-head

The next step is to find a bug to work on. Take a look at our bug tracker and find a bug to do with playback or recording. This bug should preferably include some steps to reproduce the bug.

The next step is to run Jokosher in debug mode, reproduce a bug and then explore the log file. To do this, move to the directory where the main Jokosher code lives (namely where the JokosherApp.py file is present) and run the following command:

jono@forge:~/jokosher/trunk/Jokosher$ GST_DEBUG=3,python:5,gnl*:5 ./Jokosher -g > log 2>&1

This command does a number of things. Lets break it down:

  • GST_DEBUG=3, – set the GStreamer debug detail level to 3 (1 is least detailed, 5 is insane uber-detail).
  • python:5, – set the python debug level to 5.
  • gnl*:5 – set the Gnonlin debug level to 5.
  • ./Jokosher -g – run Jokosher and pass it the -g option which directs all debug output to the log file.
  • > log 2>&1 – send all debug output to a file called log.

Run the command and reproduce the bug by performing actions in Jokosher until the bug occurs. Try to remember the steps you follow – you will need to match your actions to different parts of the log file. If you get a GStreamer error message (one of the errors that says Arggh! in Jokosher) that will also be logged to the log file.

Looking at the log file

When you run the last command you set GStreamer debugging into its default mode, which rather handily adds ANSI colours to the log file to distinguish between different types of information. To understand these colours, you need to run less to look at the log file. Run this command:

jono@forge:~/jokosher/trunk/Jokosher$ less -R log

It will then say "log" may be a binary file. See it anyway?. Type y to continue. The log file will appear and you will be presented with thousands of lines of debug code. Don’t be put off though, you will be skipping through large chunks of this code.

The debug log consists of a bunch of different types of information. A typical line looks like this:

0:00:06.855443000 15857 0x8249040 INFO   GST_ELEMENT_FACTORY gstelementfactory.c:377:gst_element_factory_create: 

creating element “bin” named “Volume_fades_bin”

If you look at the above line, there are two key bits of information you need to be aware of. The INFO bit tells you the type of debug message this line is and the content to the right of it is the message itself. There are a few different types of message:

  • INFO – general information about what GStreamer is doing.
  • DEBUG – debugging information from the application itself. More on this below.
  • LOG – more logging information, typically used by Gnonlin.
  • WARN – GStreamer warnings.

…and I believe there is an ERROR type as well, which are GStreamer errors. I seem to remember seeing this, but I am not 100% sure.

To the right of the type of message is the actual log message, which in the above example is GST_ELEMENT_FACTORY gstelementfactory.c:377:gst_element_factory_create: creating element "bin" named "Volume_fades_bin". In this message a bin called Volume_fades_bin is being created.

You job now is to read through the lines in the debug log, matching the different parts to the elements and connections in the pipeline map, and try to figure out where the bug is occurring. Reading the log will be rather confusing at first, but after a few times you will get the hang of it, and after asking some questions about what debug messages mean, you will get the hang of how to understand the log file and see what is going on.

The log file is chronologically organised – the content is appended to the file as Jokosher runs. When Jokosher initialises, the pipeline is constructed, and as different things happen, the pipeline is updated and adjusted as Jokosher runs. So, as an example, when you add an instrument, a sub-pipeline (a small bunch of elements that are part of the wider pipeline) is added to the main pipeline. When this happens, some elements from the main pipeline need to be un-linked, the new sub-pipeline created and then hooked into the main pipeline. You can see how this works by reading the log file and tracking how the pipeline map is created.

Aside from GStreamer debug messages, there are also a number of debug messages written into Jokosher by our coders. The -g switch that you pass to Jokosher switches on these debug messages so they are saved in the log file. If you search the log file for python you can see these messages. You can then search the Jokosher code for Globals.debug to see where the debug messages are added. So, as an example, imagine this line in a Jokosher source code file:

Globals.debug("foo bar")

This line would generate the following debug log message (as an example):

0:00:06.896163000 15857 0x8249040 DEBUG       python Globals.py:147:debug: foo bar

So you can match the contents of the log file to different bits of code.

Navigating the log file with the less command uses a few keyboard commands which you can learn by running man less and reading the information or by reading about it here. The key skill is typing a slash and a search term will highlight the search term throughout the file (e.g. type /myterm will highlight all instances of myterm in the log file).

Debugging is like forensics in CSI – you piece together a picture of the crime from the evidence you have, the main evidence being the pipeline map (which shows the GStreamer structure inside Jokosher), the debug log and the actions in Jokosher that were performed to trigger the bug. You use this evidence to narrow in on the problem. It is a fun, challenging and rewarding process when you explore the evidence and then discover what the bug is. They key thing then is sharing your findings with the rest of the team so that the bug can be fixed.

You should share any findings on the bug report of the bug you are investigating. If a bug report does not exist yet, create one and add your findings.

Getting help

Debugging is a science that cannot be learned in a single evening, and you will need to ask plenty of questions as you learn to debug. Do hang in there though! Debugging is an incredibly useful thing for you to do to help the Jokosher project. You are welcome to ask questions in #jokosher on Freenode, post to our mailing list or ask in the Jokosher forums.

Good luck, and thanks for getting involved in this fun, exciting and important project. Feel free to ask questions or clarify points in the comments on this blog entry! :)

  • FXB

    Just out of interest, what ever happened to Jokosher mini ? Pretty sure There were a lot of nokia 770 users looking forward to it, and it’d be a real boon to the n800 too.

  • jj

    Is there no debugger for python? Is printing log messages really the only way to debug jokosher, or is it just your preferred method?

  • Laszlo

    Yes there a debugger for python, but the problem is not with python. Most of the hard to solve errors in Jokosher occur from Gstreamer which is written C. And you would have to be a pro with the C debugger to be able to handle all of Gstreamer + the entire python interpreter running through it. For the most part, bugs caused entirely by the Jokosher python code can be detected and fixed easily, but for the Gstreamer bugs using the very extensive debug logs are probably your best bet.

    On a different note, for those of you who don’t want to use less on the command line to view the debug files try the KDE editor KWrite. Turn off “Dynamic Word Wrap” in the View menu so that all timestamps and message type names line up, and turn on C syntax highlighting by going Tools->Highlighting->Sources->C. By coincidence the Gstreamer debug logs look very nice using the C code syntax highlighting in KWrite. Much easier than using the command line if you ask me.

  • Félim Whiteley

    The minute I figure out how to get audio on this Mac G4 I just built consider me in for testing… looks like might take a while though seems to be a Kernel problem… sigh (Whole reason I built the damn thing.. oh well!)

  • Félim Whiteley

    Eh that’s a Fiesty G4 just in case you think I’m on crack.. :-D

  • http://jamalfanaian.com Jamal Fanaian

    I know this is old, but I just wanted to give my thanks as I recently found this article very useful. I’m not working on Jokosher but the gstreamer debugging tips helped me a lot. Thanks!