Watch behind my shoulder: Debugging

Warning: long-winded tech post. If you don’t want to improve your debugging skills or don’t have lots of free time, don’t read this.

I decided to do some pro-bono debugging, and as I was diving in I thought that some of you might be interested in a glimpse of how I work.

I find that when debugging complex issues, keeping a “lab manual” and documenting every single action you take before you take it, although tedious and boring, gets the problem solved much faster. This time I’ll open my manual.

I will document what I think and what I do. I won’t document general debugging theory and techniques, as that would interrupt my flow too much (and probably make this into a book). I’ll just say this:

When debugging, you should throw all trust you have in yourself out the window. You start knowing that you’re wrong  (there is a bug) despite being confident you’ve done every single thing right, so thinking “I can see the problem is not here” without proving it by experiment is almost thought-crime and will waste you hours and sanity.


I’m considering submitting a talk proposal for PyCon 2014. They have a page with proposal resources. Unfortunately, I cannot click any links on the page. Normally I’d just go do something else, preferring not to waste time on websites that hurt so much to use. Since this is an open source web site for a project that has a lot of value for me, I figured why not exercise those debugging muscles and help them out with the issue.


I click a few times, trying different things and observing the outcomes. Middle click – my habit – fails. Ctrl+left click – another way to “open in new tab” in Firefox – fails. Left click succeeds, but this doesn’t solve the issue to my satisfaction. I like new pages to open in new tabs. I see the same behaviour for different links – HTTP and HTTPS links, same-site and other-site links. The browser seems to register my clicks, in the sense that the link is marked as focused after my mouse leaves it.

Is the issue with the site or my browser? I click the FAQ link on a Paul Graham essay that I have open. It opens the link in a new tab. I click again in – still nothing. The issue is with the site.

Is the issue with the page or the site? I left-click to another page on the menu (turns out middle clicking a menu item closes the menu but doesn’t open the link) and try links there – still nothing. The issue is with the entire site.

Is the issue with my browser or all browsers? I initially think I don’t care enough to install Chrome, but having to write it in my manual it smells wrong. Many scenarios where knowing the answer will narrow my search pop into my mind, and I’d rather spend a few minutes solving some issues with my Arch Linux installations so I can install Chrome than spending a few hours mulling over the PyCon issue.

I’ll have to update my OS first, so in the meantime lets continue. I’ll add TODO: Chrome to the end of this document, keep it after the cursor and remove it when I check, so I don’t forget this.

Is it an HTML issue? A quick right-click-inspect-element shows perfectly normal HTML, so probably not:

<a href="/2014/dashboard/">account dashboard!</a>
<a href="">Proposal Advice</a>

What does the site Javascript look like? I open Firebug “script” tab. There’s a little but of inline JS, an that probably cobbles together all  page-specific JS, jquery.min.js, bootstrap.min.js, and theme.js. jQuery is 1.8.3, bootstrap is 2.3.1.

theme.js is short and has a call to jQuery ajaxSend that looks very suspect, a quick google explains that it sets a hook that runs every time an AJAX request is sent, in this case it seems like the hook adds a CSRF token. Is it responsible? Easy to check: Add a breakpoint in the hook, click a link. The breakpoint is not invoked. Nothing to see here, moving on.

What happens when I middle click a link

In Firebug’s “Script” tab, I click the “pause play” icon – the great “break on next” feature. Nothing happens. Good, no periodic JS to confuse me. Then I click a link. It pauses in minified jQuery, in line 4 of:

}, function(e, t) {
var n = 0,
r = function(e) {
v.event.simulate(t,, v.event.fix(e), !0)
view raw gistfile1.js hosted with ❤ by GitHub

I try again, this time stopping in line 5 of:

v.event = {
add: function(e, n, r, i, s) {
var o, u, a, f, l, c, h, p, d, m, g;
if (e.nodeType === 3 || e.nodeType === 8 || !n || !r || !(o = v._data(e))) return;
r.handler && (d = r, r = d.handler, s = d.selector), r.guid || (r.guid = v.guid++), a =, a || ( = a = {}), u = o.handle, u || (o.handle = u = function(e) {
return typeof v == "undefined" || !! e && v.event.triggered === e.type ? t : v.event.dispatch.apply(u.elem, arguments)
}, u.elem = e), n = v.trim(Z(n)).split(" ");
for (f = 0; f < n.length; f++) {
l = J.exec(n[f]) || [], c = l[1], h = (l[2] || "").split(".").sort(), g = v.event.special[c] || {}, c = (s ? g.delegateType : g.bindType) || c, g = v.event.special[c] || {}, p = v.extend({
type: c,
origType: l[1],
data: i,
handler: r,
guid: r.guid,
selector: s,
needsContext: s && v.expr.match.needsContext.test(s),
namespace: h.join(".")
}, d), m = a[c];
if (!m) {
m = a[c] = [], m.delegateCount = 0;
if (!g.setup ||, i, h, u) === !1) e.addEventListener ? e.addEventListener(c, u, !1) : e.attachEvent && e.attachEvent("on" + c, u)
g.add && (, p), p.handler.guid || (p.handler.guid = r.guid)), s ? m.splice(m.delegateCount++, 0, p) : m.push(p),[c] = !0
e = null
view raw gistfile1.js hosted with ❤ by GitHub

I try a few more times, getting the same two lines.

Actually, at first it always stopped in the one and only line of jquery.min.js. I googled and found Javascript Deminifier, a Firefox extention that prettified the source for me (variables still have one-letter names, but I’ll figure that out later if needed.)

I suddenly remember that I have NoScript installed (although disabled for the site), which tends to cause strange JS problems. Is it responsible for the bug? In a different browser profile with no NoScript, still no middle click.

OK, this would be a good time to stop what I’m doing and search for existing bug reports in jQuery. Who knows, I might be trying to solve a solved problem. I find references to problems with jQuery’s live() function, later deprecated in favour of deleteage() and on(), but I can only find a few uses of delegate() and breakpoints on their hooks don’t trigger when I middle click.

I find a page indicating that I should probably look for return False, .preventDefault() or .stopPropagation() as culprits. I search the code for them, ignoring results in jQuery.min.js and resisting the urge to ignore results in bootstrap.min.js (despite my instinct that well-tested frameworks are bug-free, it might be a Bootstrap behaviour. Note to self: try other Bootstrap sites with the same version).

I find a few uses of said functions, set breakpoints and middle click a link. Break! This might be it. The debugger stopped in bootstrap.min.js, in a piece of code that looks very unashamed:

}).on("click.dropdown-menu", function(e) {
view raw gistfile1.js hosted with ❤ by GitHub

Is this intended Bootstrap behaviour? Googling for bootstrap “on(click.dropdown-menu)” turns out this page that tells me that what I’m looking for is bug 7118 in Bootstrap versions 2.3.1 and 2.3.2 (bingo! That’s the version the site uses!).

So how should they fix it? Upgrade Bootstrap? According to the drupal bug, 2.3.2 – the last release before 3.0 – suffers from the issue. According to the Bootstrap issue tracker, it seems that it was fixed:

Skud commented June 11, 2013

Has this actually been rolled into a bootstrap release yet?

@Skud: #7614 was declined. b9bd82b was rolled into v2.3.2 to fix #7118. Unfortunately, that fix introduced another bug: #7968.

According to my own testing, it doesn’t work in 2.3.1 and does in 2.3.2.

So PyCon webmaster, please upgrade to Bootstrap 2.3.2!


Well, this was a bit anticlimatic. I was hoping for an interesting Javascript bug and found a reported Bootstrap issue. However, this is still a good example of how I debug complex issues in code I don’t understand well (in this example, someone else’s code).

I found the problem and isolated it to a specific line without understanding what the huge codebase does. I didn’t click “step into” even once. I documented everything I did and it made me follow good paths that I wouldn’t have followed otherwise. I resisted my self-confidence many times and one of them proved critical. All in all, I consider this a good learning resource on debugging.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s