Lately I dove into packaging python code. And I wrote about it here. However, I concentrated on making a minimal package, which is not really ready for publishing. To bring it closer to something I wouldn’t be ashamed of, I need to do some testing. As a minimum, I need to write some unit tests for it. Or perhaps even follow PyPA’s guidelines for proper Python Package tests.
The tests aren’t a package
First of all, lets start with the structure of a Python Package. When you look at the PyPA’s packaging tutorial, you can see they tell you to use this structure:
packaging_tutorial ├── LICENSE ├── README.md ├── example_pkg │ └── __init__.py ├── setup.py └── tests
Clearly, all the testing happens in the folder tests. Also, you can see there is no __init__.py file in that directory. Basically, this makes the files in it just some simple modules placed outside of the future python package.
But who cares about importing tests! All we need is to execute them and see our library is running. After all that’s the value of testing.
You can’t import the production code
So I started writing my testcases. I imported unittest and went on to write a simple case:
import unittest
from example_pkg import storage as st
import pathlib
class TestStorage(unittest.TestCase):
"""Unit tests for Storage"""
def test_init_sets_db_path(self):
"""after initiating, self.db_path should be a pathlib.Path object
with expanded home directory"""
sample_path = pathlib.Path('~/test/dir').expanduser()
storage = st.Storage('~/test/dir')
self.assertEquals(storage.db_path, sample_path)
if __name__ == '__main__':
unittest.main()
What it does isn’t really important here, rather how it fails. I tried running it in the console using the following command:
python test_storage.py
I was expecting something to fail, since that’s what programming is after all. But I was really surprised by the error I received in the console:
Traceback (most recent call last):
File "~/packaging_tutorial/example_pkg/tests/test_storage.py", line 3, in <module>
from example_pkg import storage as st
ModuleNotFoundError: No module named 'example_pkg'
At that point I’m pretty sure there is such package and I can see it waiting there in my project. Perhaps, I thought, I need to use relative import, so the interpreter would see it too. I changed the import statement to this:
from ..example_pkg import storage as st
The tests have no parent package
And then I got the next error. This one was even stranger. It seems finding the package isn’t a problem anymore. The problem is much earlier now:
Traceback (most recent call last):
File "~/packaging_tutorial/example_pkg/tests/test_storage.py", line 3, in <module>
from ..example_pkg import storage as st
ImportError: attempted relative import with no known parent package
Since the tests directory isn’t in a source package (there is no __init__.py in its parent directory), the interpreter is very unhappy.
I can, of course, just add some __init__.py files around, but then my package structure would be quite different from the tutorial. So I was in a dead end…
I wondered if the tutorial is already outdated and therefore showed some structure that didn’t fit. But since it is the official tutorial of the Python Packaging Authority, perhaps it is up to date. There is something, I am missing and I have no idea what it is.
How does Flask deal with it?
Finally I decided I need to check how other people are dealing with it. Therefore I chose a very popular Python Package and looked into their documentation. As you can probably guess by now, I went to the Github repository of Flask.
Following Flask’s steps into testing
There, I looked at their file structure. What I found was pretty much in line with the tutorial I was following. So far so good! But what’s my problem then?
Then I looked at their contributors instructions. This is the document, explaining to people what they need to do to make their contribution ready for submission. At first it sounds this has nothing to do with my tests failing with nonsense errors, but the thing is, that you need to test everything you want to send to the maintainers and you need to run all the tests of the framework.
This means they have a system put in place to automate this testing. Further their structure is similar to mine. That said, I can probably apply their approach to my code with little effort. In a way, this is a learning approach. Monkey see, monkey do, right?
Well, it seems they are using some kind of test framework – pytest. Now, I’m not really fond of starting off my new project with a framework, but it’s all about the trade-off between being stuck and actually finishing something today. So I went on and tried to install it.
I did have some weird troubles installing it in the virtual environment of the package. But the solution was simply wiping out the environment and creating it again. Crude, but it worked.
Running tests using Pytest
Once I had Pytest installed into the virtual environment I tried running a test by calling
pytest tests/test_storage.py
Of course Pytest politely informed me I should read a bit more about how it is used. It did that by saying the following:
ImportError while importing test module 'packaging_tutorial/example_pkg/tests/test_storage.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.9/importlib/__init__.py:127: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
tests/test_storage.py:2: in <module>
from example_pkg import storage
E ModuleNotFoundError: No module named 'example_pkg'
So if that happens to you too, don’t worry. Just run the right command:
python -m pytest
And this solved my problems with structuring.
Some debugging necessary
Now that said, I needed also to put my tests into appropriate form for the new framework I introduced. Suddenly unittest module styled tests were a problem. I needed to get rid of all the unittest specifics and introduce other asserts and stuff.
The tests don’t need parent package
After the odyssey I described above, however, I have no more errors saying there is no parent package. This is not really because of the magic of the framework, rather because the tests don’t need relative imports anymore.
If however, you decide you need to reuse test code, you’d be in trouble again. That said, I believe if there is a need for epic test infrastructure (symptoms of which are relative imports), then you probably have to think hard about the architecture of the application, rather than hack your way away.
You can import production code
Importing production code is now possible too! Even though when executing the test directly there’s a bunch of errors in the console saying the interpreter can’t find a package, be brave and continue doing so! The error message is lying.
Running python -m pytest is powerful enough to resolve the imports and execute the tests. Any module in the example_pkg package is importable in your tests and can be used.
The tests aren’t a package
The Python Packaging Authority has a reason behind recommending this structure. It puts your tests in a directory outside of any source package.
And this is great! Because if you don’t put the tests into a package, then they will not be bundled and distributed. After all, why would anyone need to use the tests of the library? All people need is the functionality, right?
Testing a Python Package isn’t that weird after all
After all these problems finding out how to test my package, I found out that even weird at first, there is a reason behind every little quirk in this approach.
If you have a decent architecture of your production code, and you use a CI/CD service, to run your tests automatically, every little thing snaps into its place and makes the quirks look like deliberate sensible decisions, made to put your package on the road to success.
Now go and test your code
Well, there it is. Now you are able to test your package and ensure that when people use it, it doesn’t fail. To make it perform what you it intended to do and deliver happiness among its users.
So what are you waiting for? Go and make the world better!
Or maybe look at another article I wrote about testing: Always Test Your Code