Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Appearance settings

Latest commit

 

History

History
History
248 lines (202 loc) · 13.8 KB

File metadata and controls

248 lines (202 loc) · 13.8 KB
Copy raw file
Download raw file
Outline
Edit and raw actions

Text (CSV, fixed-width, HTML, and specialized)

The :meth:`~astropy.table.Table.read` and :meth:`~astropy.table.Table.write` methods can be used to read and write text-based table data in a wide variety of supported formats. In addition to common formats like CSV and :ref:`fixed-width <fixed_width_gallery>`, the unified interface also supports specialized formats like LaTeX tables and the AAS :class:`MRT <astropy.io.ascii.Mrt>` format.

Most of the formats are provided by :ref:`table_io_ascii`, which is a flexible and powerful interface for reading and writing text tables. In addition, the interface provides wrappers around select I/O functions in the pandas library for additional flexibility and performance.

Note

For reading large CSV files, the astropy :ref:`PyArrow CSV <table_io_pyarrow_csv>` reader is a good option to consider since it can be up to 15 times faster than other readers.

Supported Formats

Character-delimited Formats

These formats use a character delimiter to separate columns. This is most commonly a comma (CSV) or a whitespace character like space or tab.

Format Write Suffix Description
ascii Yes   ASCII table in most supported formats (uses guessing)
ascii.basic Yes   :class:`~astropy.io.ascii.Basic`: Basic table with custom delimiters
ascii.commented_header Yes   :class:`~astropy.io.ascii.CommentedHeader`: Column names in a commented line
ascii.csv Yes .csv :class:`~astropy.io.ascii.Csv`: Basic table with comma-separated values
ascii.ecsv Yes .ecsv :class:`~astropy.io.ascii.Ecsv`: Basic table with Enhanced CSV (supporting metadata)
ascii.no_header Yes   :class:`~astropy.io.ascii.NoHeader`: Basic table with no headers
ascii.rdb Yes .rdb :class:`~astropy.io.ascii.Rdb`: Tab-separated with a type definition header line
ascii.tab Yes   :class:`~astropy.io.ascii.Tab`: Basic table with tab-separated values
ascii.tdat Yes .tdat :class:`~astropy.io.ascii.Tdat`: Transportable Database Aggregate Table format
pandas.csv Yes   :func:`pandas.read_csv` and :meth:`pandas.DataFrame.to_csv`
pyarrow.csv No   :func:`~astropy.io.misc.pyarrow.csv.read_csv`: Performant CSV reader

Fixed-width Formats

These formats use fixed-width columns, where each column has a fixed width in characters. This can be useful for tables that are intended to also be read by humans.

Format Write Suffix Description
ascii.fixed_width Yes   :class:`~astropy.io.ascii.FixedWidth`: Fixed width
ascii.fixed_width_no_header Yes   :class:`~astropy.io.ascii.FixedWidthNoHeader`: Fixed width with no header
ascii.fixed_width_two_line Yes   :class:`~astropy.io.ascii.FixedWidthTwoLine`: Fixed width with second header line
pandas.fwf No   :func:`pandas.read_fwf` (fixed width format)

HTML and JSON Formats

Be aware that in many cases reading tables from published HTML journal articles will not work for a variety of reasons, including inconsistent mark-ups, CAPTCHAs, changing formats, embedded javascript, or the table actually being an image. If possible you should consider retrieving the table in a standard data format such as CSV or FITS, perhaps from an archive such as CDS/Vizier.

Format Write Suffix Description
ascii.html Yes .html :class:`~astropy.io.ascii.HTML`: HTML table
jsviewer Yes   JavaScript viewer format (write-only)
pandas.html Yes   :func:`pandas.read_html` and :meth:`pandas.DataFrame.to_html`
pandas.json Yes   :func:`pandas.read_json` and :meth:`pandas.DataFrame.to_json`

Specialized Formats

Format Write Suffix Description
ascii.aastex Yes   :class:`~astropy.io.ascii.AASTex`: AASTeX deluxetable used for AAS journals
ascii.cds No   :class:`~astropy.io.ascii.Cds`: CDS format table
ascii.daophot No   :class:`~astropy.io.ascii.Daophot`: IRAF DAOphot format table
ascii.ipac Yes   :class:`~astropy.io.ascii.Ipac`: IPAC format table
ascii.latex Yes .tex :class:`~astropy.io.ascii.Latex`: LaTeX table
ascii.mesa No .data :class:`~astropy.io.ascii.Mesa`: MESA stellar evolution code output format
ascii.mrt Yes   :class:`~astropy.io.ascii.Mrt`: AAS Machine-Readable Table format
ascii.qdp Yes .qdp :class:`~astropy.io.ascii.QDP`: Quick and Dandy Plotter files
ascii.rst Yes .rst :class:`~astropy.io.ascii.RST`: reStructuredText simple format table
ascii.sextractor No   :class:`~astropy.io.ascii.SExtractor`: SExtractor format table

astropy.io.ascii

The :ref:`astropy.io.ascii <io-ascii>` sub-package provides read and write support for :ref:`many different formats <supported_formats>`, including astronomy-specific formats like AAS Machine-Readable Tables (MRT).

We strongly recommend using the unified interface for reading and writing tables via the :ref:`astropy.io.ascii <io-ascii>` sub-package. This is done by prefixing the :ref:`format name <supported_formats>` with the ascii. prefix. For example to read a DAOphot table (in this example we use a file that is installed with astropy):

>>> from astropy.table import Table
>>> from astropy.utils.data import get_pkg_data_filename
>>> photometry_file = get_pkg_data_filename('data/daophot4.dat',
...                                         package='astropy.io.ascii.tests')
>>> t = Table.read(photometry_file, format='ascii.daophot')

Use format='ascii' in order read a table and guess the table format by successively trying most of the available formats in a specific order. This can be slow for large tables and :ref:`might read the table in a different format than expected <io_ascii_should_specify_format>`.

>>> latex_file = get_pkg_data_filename('data/latex1.tex',
...                                     package='astropy.io.ascii.tests')
>>> t = Table.read(latex_file, format='ascii')
>>> print(t)
cola colb colc
---- ---- ----
   a    1    2
   b    3    4

When writing a table with format='ascii' the output is a basic space-delimited file with a single header line containing the column names.

All additional arguments are passed to the astropy.io.ascii :func:`~astropy.io.ascii.read` and :func:`~astropy.io.ascii.write` functions. Further details are available in the sections on :ref:`io_ascii_read_parameters` and :ref:`io_ascii_write_parameters`. For example, to change the column delimiter and the output format for the colc column use:

>>> import sys
>>> t.write(sys.stdout, format='ascii', delimiter='|', formats={'colc': '%0.2f'})
#
#
cola|colb|colc
a|1|2.00
b|3|4.00

Attention!

ECSV is recommended

For writing and reading tables to text in a way that fully reproduces the table data, types, and metadata (i.e., the table will "round-trip"), we highly recommend using the :ref:`ecsv_format` with format="ascii.ecsv". This writes the actual data in a space- or comma-delimited format that most text table readers can parse, but also includes metadata encoded in a comment block that allows full reconstruction of the original columns. This includes support for :ref:`ecsv_format_mixin_columns` (such as ~astropy.coordinates.SkyCoord or ~astropy.time.Time) and :ref:`ecsv_format_masked_columns`.

Pandas

astropy ~astropy.table.Table supports the ability to read or write tables using some of the :ref:`I/O methods <pandas:io>` available within pandas. This interface thus provides convenient wrappers to the following functions / methods:

Format name Data Description Reader Writer
pandas.csv CSV :ref:`read_csv() <pandas:io.read_csv_table>` :ref:`to_csv() <pandas:io.store_in_csv>`
pandas.json JSON :ref:`read_json() <pandas:io.json_reader>` :ref:`to_json() <pandas:io.json_writer>`
pandas.html HTML :ref:`read_html() <pandas:io.read_html>` :ref:`to_html() <pandas:io.html>`
pandas.fwf Fixed Width :ref:`read_fwf() <pandas:io.fwf_reader>`  

Notes:

When reading or writing a table, any keyword arguments apart from the format and file name are passed through to pandas, for instance:

.. doctest-requires:: pandas

  >>> t.write('data.csv', format='pandas.csv', sep=' ', header=False)
  >>> t2 = Table.read('data.csv', format='pandas.csv', sep=' ', names=['a', 'b', 'c'])

.. testcleanup::
  >>> import os
  >>> if os.path.exists('data.csv'):
  ...     os.remove('data.csv')

PyArrow CSV

The pyarrow library provides a highly-performant CSV reader that can be used in Astropy with Table.read(input_file, format="pyarrow.csv", ...). This can by up to 15 times faster and more memory-efficient than the :ref:`astropy.io.ascii <io-ascii>` fast reader or the default pandas.csv reader. The best performance is achieved for files with only numeric data types, but even for files with mixed data types, the performance is still better than the standard :ref:`astropy.io.ascii <io-ascii>` fast CSV reader.

This reader uses the :func:`~astropy.io.misc.pyarrow.csv.read_csv` function, which in turn uses the PyArrow CSV reader and sets the various options to pyarrow.csv.read_csv() appropriately. The interface is designed to be similar to the :ref:`io.ascii read interface <io_ascii_read_parameters>` where possible, but there are differences, most notably:

  • Input can only be a string file name, pathlib.Path, or a binary file-like object.
  • Whitespace in string data fields and header column names is preserved.
  • Use dtypes instead of converters to specify the column data types.
  • Use null_values instead of fill_values to specify the null (missing) values.
  • No guess parameter and no guessing of the table format (e.g., delimiter).
  • No data_end parameter.
  • No exclude_names parameter.
  • Columns consisting of only string values True and False are parsed as boolean data.
  • Columns with ISO 8601 date/time strings are parsed as shown below: - 12:13:14.123456: object[datetime.time] - 2025-01-01: np.datetime64[D] - 2025-01-01T01:02:03: np.datetime64[s] - 2025-01-01T01:02:03.123456: np.datetime64[ns]
  • Timestamp parsing behavior can be customized with the timestamp_parsers parameter.

Using the PyArrow CSV reader directly

The :mod:`astropy.io.misc.pyarrow.csv` module also provides the :func:`~astropy.io.misc.pyarrow.csv.convert_pa_table_to_astropy_table` function to allow converting a pyarrow.Table to an astropy.table.Table. This allows using the PyArrow CSV reader directly with custom options that are not available in the astropy interface.

Morty Proxy This is a proxified and sanitized view of the page, visit original site.