Description
Code Sample, a copy-pastable example if possible
python setup.py build_ext --inplace -j 8Problem description
See some discussion in #30862 . I have 8 cores on Windows. When I use -j 8, in the middle of the build, I will get this error message:
c:\Code\pandas_dev\pandas\pandas\_libs\tslibs\src\datetime\np_datetime.c : fatal error C1083: Cannot open compiler generated file: 'c:\Code\pandas_dev\pandas\build\temp.win-amd64-3.7\Release\pandas\_libs\tslibs\src\datetime\np_datetime.obj': Permission denied
What I think is happening is that in setup.py, _libs.tslibs.conversion, libs.tslibs.np_datetime and _libs.tslibs.period all have to compile pandas/_libs/tslibs/src/datetime/np_datetime.c . If the parallel build is timed in such a way that two of those extensions are being built at the same time, then they can conflict in writing np_datetime.obj. A similar thing can happen with _libs.lib, _libs.parsers, and _libs.tslibs.parsing all having to compile pandas/_libs/src/parser/tokenizer.c .
If I do two python setup.py build_ext --inplace -j 8 steps in a row, then everything is fine. Also things are fine with -j 4. But I can imagine that even with -j 4, you could end up with the same kind of issue, even in CI.
I don't know enough about how extensions are built, distutils and setuptools in order to avoid this potential collision during the build process.
Output of pd.show_versions()
c:\Code\pandas_dev\pandas\pandas\core\index.py:29: FutureWarning: pandas.core.index is deprecated and will be removed in a future version. The public classes are available in the top-level namespace.
FutureWarning,
INSTALLED VERSIONS
commit : 13858f6
python : 3.7.6.final.0
python-bits : 64
OS : Windows
OS-release : 10
machine : AMD64
processor : Intel64 Family 6 Model 158 Stepping 13, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : None.None
pandas : 0.26.0.dev0+1734.g13858f6e1
numpy : 1.17.3
pytz : 2019.3
dateutil : 2.8.1
pip : 19.3.1
setuptools : 42.0.2.post20191201
Cython : 0.29.14
pytest : 5.3.2
hypothesis : 4.57.1
sphinx : 2.3.1
blosc : None
feather : None
xlsxwriter : 1.2.7
lxml.etree : 4.4.2
html5lib : 1.0.1
pymysql : None
psycopg2 : None
jinja2 : 2.10.3
IPython : 7.10.1
pandas_datareader: None
bs4 : 4.8.2
bottleneck : 1.3.1
fastparquet : 0.3.2
gcsfs : None
lxml.etree : 4.4.2
matplotlib : 3.1.2
numexpr : 2.7.0
odfpy : None
openpyxl : 3.0.1
pandas_gbq : None
pyarrow : 0.15.1
pytables : None
pytest : 5.3.2
s3fs : 0.4.0
scipy : 1.3.1
sqlalchemy : 1.3.12
tables : 3.6.1
tabulate : 0.8.6
xarray : 0.14.1
xlrd : 1.2.0
xlwt : 1.3.0
xlsxwriter : 1.2.7
numba : 0.46.0