Mac OS X, Python, PyCharm, and Pain
Apr. 7th, 2013 10:25 pm![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
geekchick77So, this weekend has been frustrating. On Friday, I got Django unit tests for my work codebase running in PyCharm. I was very pleased, because visual debugging and in-editor code coverage are fantastic!
So, I get home, update my code, and try to run unit tests... and an import fails with:
AttributeError: 'module' object has no attribute 'Products'
Not a regular import, but a call to __import__() buried deep in the test runner code.
I investigate and discover that, mysteriously enough, the module from which it is trying to import is missing most of its subdirectories. In fact, it has ONLY the directories that lead to the test being run. This is odd... for one thing, it worked at work, and for another, those packages load fine in every other context. I can run the unit tests from the command line, I can run the app server from within PyCharm, etc.
I spent a good long time comparing, only to conclude the environments are nearly identical. Same version of OS X, same version of PyCharm, virtualenv created from the same requirements files, everything. And yet, somehow, at work the module has the right subdirectories, and at home it does not.
Finally, I had a brain flash. Why not run the tests through the debugger, stepping through setup with a watch on the module, to see when it loses its subdirectories? It took quite a while, because I'd find the place where they disappeared, rerun stepping in instead of over, rinse & repeat. I was actually simultaneously debugging the same code at home and work (via VPN) so I could compare. Finally, I narrowed it down to the nose importer. At a particular point, it compares two directories. The comparison succeeded at work and failed at home. At first, I couldn't figure it out. They looked the same! It was actually pasting it somewhere that didn't have a fixed-width font, and the lines were different lengths! And then, I saw it. /Users/Jessamyn vs /Users/jessamyn.
But, how very mysterious! My home directory on my home computer is very clearly /Users/jessamyn with a small 'j'. Where was this capitalized version coming from? As it turns out, from Python's os.getcwd(). For some reason, this was always returning a capitalized version, which the nose importer stores as it is setting up. Later on, it tries to compare this to the contents of sys.modules, which for some reason had the correct case, so the comparison failed. To make matters even weirder, this was ONLY true inside PyCharm. When running from the command line, everything was capitalized.
I was able to verify that changing the comparison to use .lower() calls fixed the problem, but this would of course create issues on case-sensitive filesystems. In the end, I created a second administrative user so I could log in to that account, rename my home directory to be /Users/Jessamyn, and get on with life. There is still a slight issue, in that ~ still refers to /Users/jessamyn, but so far that hasn't caused any problems.
TL;DR
* OS X is sort-of case-insensitive
* The results of os.getcwd() and the contents of sys.modules are unpredictable with respect to capitalization
* The nose test runner contains some case-sensitive code
So, I get home, update my code, and try to run unit tests... and an import fails with:
AttributeError: 'module' object has no attribute 'Products'
Not a regular import, but a call to __import__() buried deep in the test runner code.
I investigate and discover that, mysteriously enough, the module from which it is trying to import is missing most of its subdirectories. In fact, it has ONLY the directories that lead to the test being run. This is odd... for one thing, it worked at work, and for another, those packages load fine in every other context. I can run the unit tests from the command line, I can run the app server from within PyCharm, etc.
I spent a good long time comparing, only to conclude the environments are nearly identical. Same version of OS X, same version of PyCharm, virtualenv created from the same requirements files, everything. And yet, somehow, at work the module has the right subdirectories, and at home it does not.
Finally, I had a brain flash. Why not run the tests through the debugger, stepping through setup with a watch on the module, to see when it loses its subdirectories? It took quite a while, because I'd find the place where they disappeared, rerun stepping in instead of over, rinse & repeat. I was actually simultaneously debugging the same code at home and work (via VPN) so I could compare. Finally, I narrowed it down to the nose importer. At a particular point, it compares two directories. The comparison succeeded at work and failed at home. At first, I couldn't figure it out. They looked the same! It was actually pasting it somewhere that didn't have a fixed-width font, and the lines were different lengths! And then, I saw it. /Users/Jessamyn vs /Users/jessamyn.
But, how very mysterious! My home directory on my home computer is very clearly /Users/jessamyn with a small 'j'. Where was this capitalized version coming from? As it turns out, from Python's os.getcwd(). For some reason, this was always returning a capitalized version, which the nose importer stores as it is setting up. Later on, it tries to compare this to the contents of sys.modules, which for some reason had the correct case, so the comparison failed. To make matters even weirder, this was ONLY true inside PyCharm. When running from the command line, everything was capitalized.
I was able to verify that changing the comparison to use .lower() calls fixed the problem, but this would of course create issues on case-sensitive filesystems. In the end, I created a second administrative user so I could log in to that account, rename my home directory to be /Users/Jessamyn, and get on with life. There is still a slight issue, in that ~ still refers to /Users/jessamyn, but so far that hasn't caused any problems.
TL;DR
* OS X is sort-of case-insensitive
* The results of os.getcwd() and the contents of sys.modules are unpredictable with respect to capitalization
* The nose test runner contains some case-sensitive code
