From https://travis-ci.org/nltk/nltk/builds/529221349, we see several builds and all other builds have the logs with the passed/failed/skip colored when running tox, e.g.

enter image description here

using tox.ini configuration for the py-travis environment:

[tox]
envlist =
    py{27,35,36,37}
    pypy
    py{27,35,36}-nodeps
    py{27,35,36}-jenkins
    py-cythonized
    py-travis

[testenv]
; simplify numpy installation
setenv =
    LAPACK=
    ATLAS=None
    PYTHONWARNINGS=ignore

; Copy all environment variables to the tox test environment
passenv = *

deps =
    numpy
    nose >= 1.2.1
    coverage
    text-unidecode
    twython
    pyparsing
    python-crfsuite
    rednose

changedir = nltk/test
commands =
    ; scipy and scikit-learn requires numpy even to run setup.py so
    ; they can't be installed in one command
    pip install scipy scikit-learn

    ; python runtests.py --with-coverage --cover-inclusive --cover-package=nltk --cover-html --cover-html-dir={envdir}/docs []
    python runtests.py []

commands =
    python runtests.py []


[testenv:py-travis]
extras = all
setenv =
    NLTK_DATA = {homedir}/nltk_data/
commands = {toxinidir}/tools/travis/coverage-pylint.sh

But when the cythonization from the setup.py kicks in, the tox and travis configuration seems the same,

[testenv:py-cythonized]
deps =
    Cython >= 0.28.5
setenv =
    CYTHONIZE_NLTK = true
    NLTK_DATA = {homedir}/nltk_data/
extras = all
commands = {toxinidir}/tools/travis/coverage-pylint.sh

when running the build, the colors disappears:

enter image description here

The installation setup.py is exactly the same for both py-travis and py-cynthonized builds:

# Work around mbcs bug in distutils.
# http://bugs.python.org/issue10945
import codecs
try:
    codecs.lookup('mbcs')
except LookupError:
    ascii = codecs.lookup('ascii')
    func = lambda name, enc=ascii: {True: enc}.get(name == 'mbcs')
    codecs.register(func)

import os

# Use the VERSION file to get NLTK version
version_file = os.path.join(os.path.dirname(__file__), 'nltk', 'VERSION')
with open(version_file) as fh:
    nltk_version = fh.read().strip()

# setuptools
from setuptools import setup, find_packages

# Specify groups of optional dependencies
extras_require = {
    'machine_learning': ['gensim', 'numpy', 'python-crfsuite', 'scikit-learn', 'scipy'],
    'plot': ['matplotlib'],
    'tgrep': ['pyparsing'],
    'twitter': ['twython'],
    'corenlp': ['requests'],
}

# Add a group made up of all optional dependencies
extras_require['all'] = set(
    package for group in extras_require.values() for package in group
)

MODULES_TO_COMPILE = [
    'nltk.grammar',
    'nltk.parse.chart',
    'nltk.tokenize.*',
    'nltk.probability',
    'nltk.util',
    'nltk.stem.*',
    'nltk.lm.*',
    'nltk.translate.*',
    'nltk.tbl.*',
    'nltk.sentiment.*',
    'nltk.cluster.*',
    'nltk.classify.*',
    'nltk.metrics.*',
    'nltk.chunk.*',
    'nltk.sem.*',

]


def compile_modules(modules):
    """
    Compile the named modules using Cython, using the clearer Python 3 semantics.
    """
    import Cython
    from Cython.Build import cythonize
    files = [name.replace('.', os.path.sep) + '.py' for name in modules]
    print("Compiling %d modules using Cython %s" % (len(modules), Cython.__version__))
    return cythonize(files, language_level=3)


if os.getenv('CYTHONIZE_NLTK') == 'true':
    ext_modules = compile_modules(MODULES_TO_COMPILE)
else:
    ext_modules = None

setup(
    name="nltk",
    description="Natural Language Toolkit",
    version=nltk_version,
    url="http://nltk.org/",
    long_description="""\
The Natural Language Toolkit (NLTK) is a Python package for
natural language processing.  NLTK requires Python 2.7, 3.5, 3.6, or 3.7.""",
    license="Apache License, Version 2.0",
    keywords=[
        'NLP',
        'CL',
        'natural language processing',
        'computational linguistics',
        'parsing',
        'tagging',
        'tokenizing',
        'syntax',
        'linguistics',
        'language',
        'natural language',
        'text analytics',
    ],
    maintainer="Steven Bird",
    maintainer_email="[email protected]",
    author="Steven Bird",
    author_email="[email protected]",
    classifiers=[
        'Development Status :: 5 - Production/Stable',
        'Intended Audience :: Developers',
        'Intended Audience :: Education',
        'Intended Audience :: Information Technology',
        'Intended Audience :: Science/Research',
        'License :: OSI Approved :: Apache Software License',
        'Operating System :: OS Independent',
        'Programming Language :: Python :: 2.7',
        'Programming Language :: Python :: 3.5',
        'Programming Language :: Python :: 3.6',
        'Programming Language :: Python :: 3.7',
        'Topic :: Scientific/Engineering',
        'Topic :: Scientific/Engineering :: Artificial Intelligence',
        'Topic :: Scientific/Engineering :: Human Machine Interfaces',
        'Topic :: Scientific/Engineering :: Information Analysis',
        'Topic :: Text Processing',
        'Topic :: Text Processing :: Filters',
        'Topic :: Text Processing :: General',
        'Topic :: Text Processing :: Indexing',
        'Topic :: Text Processing :: Linguistic',
    ],
    package_data={'nltk': ['test/*.doctest', 'VERSION']},
    install_requires=[
        'six',
        'singledispatch; python_version < "3.4"'
    ],
    extras_require=extras_require,
    packages=find_packages(),
    ext_modules=ext_modules,
    zip_safe=False,  # since normal files will be present too?
)

Why does the color disappear for the Cythonized builds?

How to enable the color for the Cythonized builds?


For some background the code comes from the nltk library and the full branch for the cynthonization tests/builds are on https://github.com/alvations/nltk/tree/cythonize

1 Answers

1
hoefling On Best Solutions

Why does the color disappear for the Cythonized builds?

Because py-cythonized environment doesn't install nose (this is why the tests are being run with stdlib's unittest in first place) and rednose (nose output coloring).

This happens because the env has deps overridden. py-travis doesn't declare any own deps, so it inherits the deps setting from the global testenv configuration. py-cythonized needs Cython, so it redefines deps list, losing all the packages required for test execution.

How to enable the color for the Cythonized builds?

Copy the dependencies from global testenv to py-cythonized. Proposed patch:

diff --git a/tox.ini b/tox.ini
index a267d9a5a..41740e19b 100644
--- a/tox.ini
+++ b/tox.ini
@@ -133,6 +133,14 @@ commands =
 [testenv:py-cythonized]
 deps =
     Cython >= 0.28.5
+    numpy
+    nose >= 1.2.1
+    coverage
+    text-unidecode
+    twython
+    pyparsing
+    python-crfsuite
+    rednose
 setenv =
     CYTHONIZE_NLTK = true
     NLTK_DATA = {homedir}/nltk_data/

With the patch applied, the test output is nicely colored: example run on travis

Update: using conditional settings

To avoid the duplication of dependencies, you can use conditional dependencies:

[testenv]
deps =
    rednose  # this one is global dependency
    py-cythonized: cython  # this one is specific for testenv:py-cythonized

Proposed patch:

diff --git a/tox.ini b/tox.ini
index a267d9a5a..fa0839b96 100644
--- a/tox.ini
+++ b/tox.ini
@@ -26,6 +26,7 @@ deps =
     pyparsing
     python-crfsuite
     rednose
+    py-cythonized: Cython >= 0.28.5

 changedir = nltk/test
 commands =
@@ -131,8 +132,6 @@ commands =

 # Test Cython compiled installation.
 [testenv:py-cythonized]
-deps =
-    Cython >= 0.28.5
 setenv =
     CYTHONIZE_NLTK = true
     NLTK_DATA = {homedir}/nltk_data/

Source: Factors and factor-conditional settings