Trying to fix a weird bug in my Python script took me 4 hours yesterday

I was automating a simple data pull from a website, and the script kept crashing on one specific page for no clear reason. Turns out there was a hidden HTML comment with a weird character that my parser couldn't handle, and I had to write a custom filter. What's the most random bug you've spent way too long solving?

3 comments

3 Comments

evah402mo ago

Ugh. @joel_adams79, what encoding were you even using?

morgan_lewis181mo ago

Yeah, those single smart quotes are the absolute worst. I had a CSV file once that had them hidden in a field and my import script just silently dropped half the rows. Ended up having to run a regex replace on the whole file before it would even parse right. Did you end up finding a tool that handles this kind of stuff automatically or do you just check everything manually now?

joel_adams792mo ago

That hidden HTML comment sounds like a nightmare. I once lost a whole afternoon because a JSON file had a single smart quote instead of a regular apostrophe. My script just kept throwing a generic encoding error. Did your custom filter just strip out the weird characters, or did you have to change the whole way your parser read the page?