Skip to content

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

BUG: pandas styling doesn't display for all rows in large dataframes #40913

Copy link
Copy link
Open
@trenton3983

Description

@trenton3983
Issue body actions
  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • (optional) I have confirmed this bug exists on the master branch of pandas.


Note: Please read this guide detailing how to provide the necessary information for us to reproduce your bug.

Update 2

  • The issue seems to be Google Chrome and Microsoft Edge
  • Jupyter Lab in Firefox correctly displays all of the styled rows and correctly renders an output HTML file.
  • The question is then, what are the relevant settings in Chrome and Edge, or why doesn't this work in Chrome or Edge?

Original

Code Sample, a copy-pastable example

import pandas as pd
import numpy as np
from faker import Faker  # conda install -c conda-forge faker or pip install Faker

# for fake names
fake = Faker()

# test data
np.random.seed(365)
rows = 11000

# change 36 or 158 to test where the rows stop appearing
vals = {f'val{i}': np.random.randint(1, 11, size=(rows)) for i in range(1, 36)}
data = {'name': np.random.choice([fake.unique.name() for i in range(158)], size=rows),
        'cat': np.random.randint(1, 4, size=(rows))}
data.update(vals)

df = pd.DataFrame(data)

# used to create the mask for the background color
mean = df.groupby('cat').mean().round(2)

# calculate the mean for each name and cat
cat_mean = df.groupby(['name', 'cat']).mean()


def color(x):
    """Function to apply background color"""
    c1 = 'background-color: green'
    c = '' 
    # compare columns
    mask1 = x.gt(mean)
    # DataFrame with same index and columns names as original filled empty strings
    df1 =  pd.DataFrame(c, index=x.index, columns=x.columns)
    # modify values of df1 column by boolean mask
    df1.iloc[mask1] = c1
    display(df1)

    return df1

# displays the notebook in Jupyter
cat_mean.style.apply(color, axis=None)

# In PyCharm saving rendered styler to file
cm = cat_mean.style.apply(color, axis=None).set_precision(3).render()

with open('cm_test.html', 'w') as f:
    f.write(cm)

Problem description

  • Given a large dataframe, in this case 474 rows x 35 columns, the applied styling does not correctly display for all rows in Jupyter or if saving the file to HTML.
    • enter image description here
  • If the number of rows or columns increases beyond this size, then more rows aren't displayed properly
  • We can see from the styling map, that the rows are correctly mapped with a background color, but it isn't displayed.
    • enter image description here
  • If the number of rows or columns is reduced, then all of the rows display the correct styling.
  • jupyterlab v3.0.11 and pandas v1.2.3
  • In PyCharm 2021.1 (Professional Edition) Build #PY-211.6693.115, built on April 6, 2021 saving the redendered styler to a file has the same result, so this isn't just an issue with Jupyter.
  • This issue is reproducible on two different systems that I have tried.
  • If the shape is reduced to 471 rows × 35 columns or 474 rows × 34 columns, then all rows correctly display the highlighting.
  • Associated Stack Overflow question.

Workaround

Split the DataFrame
  • It's dissatisfying, but splitting the DataFrame, and applying the style will work, since if reduces the overall size.
cat_mean.iloc[:237, :].style.apply(color, axis=None)
cat_mean.iloc[237:, :].style.apply(color, axis=None)
Save to Excel
  • All rows are displayed correctly with the highlight color when saving to Excel
test = cat_mean.style.apply(color, axis=None)
test.to_excel('test.xlsx', engine='openpyxl')

Expected Output

  • All rows should display highlighting.

Output1 of pd.show_versions()

INSTALLED VERSIONS

commit : f2c8480
python : 3.8.8.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.2.3
numpy : 1.19.2
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 52.0.0.post20210125
Cython : 0.29.22
pytest : 6.2.3
hypothesis : None
sphinx : 3.5.3
blosc : None
feather : None
xlsxwriter : 1.3.8
lxml.etree : 4.6.3
html5lib : 1.1
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: 0.9.0
bs4 : 4.9.3
bottleneck : 1.3.2
fsspec : 0.9.0
fastparquet : None
gcsfs : None
matplotlib : 3.3.4
numexpr : 2.7.3
odfpy : None
openpyxl : 3.0.7
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : 1.6.2
sqlalchemy : 1.4.5
tables : 3.6.1
tabulate : 0.8.9
xarray : None
xlrd : 2.0.1
xlwt : 1.3.0
numba : 0.53.1

Update 1

  • The following options produced the same results as the original attempts
    • Standard python console creating HTML file
    • PyCharm creating HTML file
    • Jupyter Lab displaying styled dataframe in a cell

Output2 of pd.show_versions()

INSTALLED VERSIONS

commit : 2cb9652
python : 3.9.2.final.0
python-bits : 64
OS : Windows
OS-release : 10
Version : 10.0.19041
machine : AMD64
processor : Intel64 Family 6 Model 60 Stepping 3, GenuineIntel
byteorder : little
LC_ALL : None
LANG : None
LOCALE : English_United States.1252

pandas : 1.2.4
numpy : 1.20.0
pytz : 2021.1
dateutil : 2.8.1
pip : 21.0.1
setuptools : 52.0.0.post20210125
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : None
jinja2 : 2.11.3
IPython : 7.22.0
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugIO HTMLread_html, to_html, Styler.apply, Styler.applymapread_html, to_html, Styler.apply, Styler.applymapStylerconditional formatting using DataFrame.styleconditional formatting using DataFrame.style

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

      Morty Proxy This is a proxified and sanitized view of the page, visit original site.