How do you include several subdirectories with sphinx-apidoc?

Question

Problem

I'm trying to build documentation for a Python project that contains a number of subdirectories. I'm trying to use sphinx-apidoc to make the process seamless. However, despite my best efforts, I can't deal with code that is in subdirectories. I followed a great short tutorial on YouTube here and it worked well. To mock the issue that I've been having, I put one of the files in a can_it_handle_folders directory, as below.

spamfilter-py
│   __init__.py  (**)
│   readme.md
│   sample.py
│   spamfilter.py
│   token.py
├───can_it_handle_folders
│       __init__.py
│       test_spamfilter.py
├───docs
│   ├───build
│   │   └───html
│   └───source
│       ├───conf.py
│       ├───index.rst
│       ├───modules.rst
│       ├───sample.rst
│       ├───spamfilter.rst
│       ├───token.rst
│       ├───spamfilter-py.test_spamfilter.rst
│       └───html
...

I go to the docs directory and run sphinx-apidoc -o . .. to generate .rst files based on the root spamfilter directory. I've added the following line to the conf.py:

sys.path.insert(0, os.path.abspath('..'))

and my generated modules.rst looks like:

spamfilter-py
=============

.. toctree::
   :maxdepth: 4

   sample
   spamfilter
   token

The html for the anything in the can_it_handle_folders directory does not generate. If I try to add can_it_handle_folders/test_spamfilter to the toctree, I get a toctree contains reference to nonexisting document 'can_it_handle_folders/test_spamfilter' error.

Question

I want the test_spamfilter module to show up in the generated html. How do I do this? What is the best way to set this up?

UPDATE: it is the init file

I've isolated the issue, which seems to be the __init__.py in the root directory (marked with ** above). I don't know how to deal with this. If I remove it, sphinx works exactly how I want it to. If I keep it, the issue becomes this one here. It seems that this must be an issue with the sys.path. This is the error I get, for each of the files:

WARNING: invalid signature for automodule ('spamfilter-py.can_it_handle_folders')
WARNING: don't know which module to import for autodocumenting 'spamfilter-py.can_it_handle_folders' (try placing a "module" or "currentmodule" directive in the document, or giving an explicit module name)

I have tried the following in the conf.py file:

• sys.path.insert(0, os.path.abspath(os.path.join('..')))
• sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))
• sys.path.insert(0, os.path.abspath(os.path.join('..', '..', '..')))

All have resulted in the same error message above. I have also tried running sphinx-apidoc -o ./source ../.. in addition to sphinx-apidoc -o ./source .. for each of the paths above. What could be the issue? I know it must be some small configuration thing.

Context

I am an absolute beginner to Sphinx. I have tried to read the documentation, but it has not addressed this issue. I've tried a number of things that I guess are clearly incorrect. If there is no easy answer, I'll add my attempts here. I know that there are other stack overflow questions on this, but they are not recent and unhelpful.

• Windows 10 computer using PowerShell for commands

Included files:

• index.rst

Welcome to j's documentation!
=============================

.. toctree::
   :maxdepth: 2
   :caption: Contents:

   modules

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

• conf.py

import os
import sys
sys.path.insert(0, os.path.abspath(os.path.join('..', '..')))


# -- Project information -----------------------------------------------------

project = 'j'
copyright = '2020, k'
author = 'k'


# -- General configuration ---------------------------------------------------

# Add any Sphinx extension module names here, as strings. They can be
# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
# ones.
extensions = ['sphinx.ext.autodoc'
]

# Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates']

# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
# This pattern also affects html_static_path and html_extra_path.
exclude_patterns = []


# -- Options for HTML output -------------------------------------------------

# The theme to use for HTML and HTML Help pages.  See the documentation for
# a list of builtin themes.
#
html_theme = 'alabaster'

# Add any paths that contain custom static files (such as style sheets) here,
# relative to this directory. They are copied after the builtin static files,
# so a file named "default.css" will overwrite the builtin "default.css".
html_static_path = ['_static']

• one rst that is not working, spamfilter-py.rst:

spamfilter\-py package
======================

Subpackages
-----------

.. toctree::
   :maxdepth: 4

   spamfilter-py.can_it_handle_folders

Submodules
----------

spamfilter\-py.sample module
----------------------------

.. automodule:: spamfilter-py.sample
   :members:
   :undoc-members:
   :show-inheritance:

spamfilter\-py.spamfilter module
--------------------------------

.. automodule:: spamfilter-py.spamfilter
   :members:
   :undoc-members:
   :show-inheritance:

spamfilter\-py.token module
---------------------------

.. automodule:: spamfilter-py.token
   :members:
   :undoc-members:
   :show-inheritance:

Module contents
---------------

.. automodule:: spamfilter-py
   :members:
   :undoc-members:
   :show-inheritance:

Here is what the documentation looks like: