.. data_formats.rst .. This is a port of the original SasView html help file to ReSTructured text .. by S King, ISIS, during SasView CodeCamp-III in Feb 2015. .. WG Bouwman, DUT, added during CodeCamp-V in Oct 2016 the SESANS data format .. WG Bouwman, DUT, updated during CodeCamp-VI in Apr 2017 the SESANS data format .. J Krzywon, P Butler, S King, overhauled during PR Hackathon in Oct 2021 .. _Formats: Data Formats ============ SasView recognizes 1D SAS (*I(Q) vs Q*), 2D SAS(*I(Qx,Qy) vs (Qx,Qy)*) and 1D SESANS (*P(z) vs z*) data in several different file formats. It will also read and analyse other data adhering to the same file formats (e.g. DLS or NR data) but not necessarily recognise what those data represent (e.g. plot axes may be mislabelled). .. note:: From SasView 4.1 onwards (but not versions 5.0.0 or 5.0.1), the :ref:`File_Converter_Tool` allows some legacy formats to be converted into either the canSAS SASXML format or the NeXus NXcanSAS format. These legacy formats include 1D/2D BSL/OTOKO, 1D output from FIT2D and some other SAXS-oriented software, and the ISIS COLETTE (or 'RKH') 2D format. 1D SAS Formats -------------- SasView recognizes 1D data supplied in a number of specific formats, as identified by the file extensions below. It also incorporates a 'generic loader' which is called if all else fails. The generic loader will attempt to read data files of any extension *provided* the file is in ASCII ('text') format (i.e. not binary). So this includes, for example, comma-separated variable (CSV) files from a spreadsheet. The file extensions (which are not case sensitive) with specific meaning are: * .ABS (NIST format) * .ASC (NIST format) * .COR (in canSAS XML v1.0 and v1.1 formats *only*) * .DAT (NIST format) * .H5, .NXS, .HDF, or .HDF5 (in NXcanSAS v1.0 and v1.1 formats *only*) * .PDH (Anton Paar SAXSess format) * .XML (in canSAS XML v1.0 and v1.1 formats *only*) The CanSAS & NXcanSAS standard formats are both output by the `Mantid data reduction framework `_ and the `NIST Igor data reduction routines `_. The ASCII formats can be viewed in any text editor (Notepad, vi, etc) but the HDF formats require a viewer, such as `HDFView `_. The ASCII ('text') files are expected to have 2, 3, or 4 columns of values, separated by whitespaces or commas or semicolons, in the following order: *Q, I(Q), ( dI(Q), dQ(Q) )* where *Q* is assumed to have units of 1/Angstrom, *I(Q)* is assumed to have units of 1/cm, *dI(Q)* is the uncertainty on the intensity value (also as 1/cm), and *dQ(Q)* **is the one-sigma FWHM Gaussian instrumental resolution in** *Q*, **assumed to have arisen from pinhole geometry**. If the data are slit-smeared, see `Slit-Smeared Data`_. There must be a minimum of 5 lines of data in the file, and each line of data **must** contain the same number of entries (i.e. columns of data values). As a general rule, SasView will provide better fits when it is provided with more information (i.e. more columns) about each observation (i.e. data point). If using CSV output, ensure that it is not using commas as delimiters for the thousands. **Examples of these formats can be found in the \\test\\1d_data sub-folder in the SasView installation folder.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ 2D SAS Formats -------------- SasView recognizes 2D data only when supplied in ASCII ('text') files in the NIST 2D format (with the extensions .ASC or .DAT) or HDF files in the NeXus NXcanSAS (HDF5) format (with the extension .H5, .NXS, .HDF, or .HDF5). The file extensions are not case-sensitive. Data in the old ISIS 2D format must be converted using the :ref:`File_Converter_Tool`. The NXcanSAS standard format is output by the `Mantid data reduction framework `_ and the `NIST Igor data reduction routines `_. Most of the header lines in the `NIST 2D format `_ can be removed *except the last line*, and only the first three columns (*Qx, Qy,* and *I(Qx,Qy)*) are actually required. Data values have the same meanings and units as for `1D SAS Formats`_. *2D image data* can be translated into 2D 'pseudo-data' using the :ref:`Image_Viewer_Tool`, but this should only be done with an abundance of caution. **Examples of these formats can be found in the \\test\\2d_data sub-folder in the SasView installation folder.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ 1D SESANS Format ---------------- SasView version 4.1 onwards will read ASCII ('text') files in a prototype SESANS standard format with the extensions .SES or .SESANS (which are not case-sensitive). The file format starts with a list of name-value pairs which detail the general experimental parameters necessary for fitting and analyzing the data. This list should contain all the information necessary for the file to be 'portable' between users. Following the header are up to 8 space-delimited columns of experimental variables of which the first 4 columns are required. In order, these are: - Spin-echo length (z, in Angstroms) - Depolarization (:math:`log(P/P_0)/(lambda^2 * thickness)`, in Angstrom :sup:`-1` cm :sup:`-1`\ ) - Depolarization error (also in in Angstrom :sup:`-1` cm :sup:`-1`\ ) (i.e. the measurement error) - Spin-echo length error (:math:`\Delta`\ z, in Angstroms) (i.e. the experimental resolution) - Neutron wavelength (:math:`\lambda`, in Angstroms) - Neutron wavelength error (:math:`\Delta \lambda`, in Angstroms) - Normalized polarization (:math:`P/P_0`, unitless) - Normalized polarization error (:math:`\Delta(P/P_0)`, unitless) (i.e. the measurement error) **Examples of this format can be found in the \\test\\sesans_data sub-folder in the SasView installation folder.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ Coordinate Formats ------------------ The :ref:`SANS_Calculator_Tool` in SasView recognises ASCII ('text') files containing coordinate data (a grid of 'voxels') with the following extensions (which are not case-sensitive): * .PDB (`Protein Data Bank format `_) * .OMF (`OOMMF micromagnetic simulation format `_) * .SLD (Spin-Lattice Dynamics simulation format) In essence, coordinate formats specify a location and one or more properties of that location (e.g. what it represents, its volume, or magnetisation, etc). The PDB/OMF/SLD formats all use a rectangular grid of voxels. The .STL coordinate format is not currently supported by SasView. **Examples of these formats can be found in the \\test\\coordinate_data sub-folder in the SasView installation folder.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ Slit-Smeared Data ----------------- SasView will only account for slit smearing if the data being processed are recognized as slit-smeared. Currently, only the canSAS \*.XML, NIST \*.ABS and NXcanSAS formats facilitate slit-smeared data. The easiest way to include $\Delta q_v$ in a way recognizable by SasView is to mimic the \*.ABS format. The data must follow the normal rules for general ASCII files **but include 6 columns**, not 4 columns. The SasView general ASCII loader assumes the first four columns are *Q*, *I(Q)*, *dI(Q)*, and *dQ(Q)*. If the data does not contain any *dI(Q)* information, these can be faked by making them ~1% (or less) of the *I(Q)* data. The fourth column **must** then contain the the $\Delta q_v$ value, in |Ang^-1|, but as a **negative number**. Each row of data should have the same value. The 5th column **must** be a duplicate of column 1. **Column 6 can have any value but cannot be empty**. Finally, the line immediately preceding the actual columnar data **must** begin with: "The 6 columns". **For an example of a 6 column file with slit-smeared data, see the example data set 1umSlitSmearSphere.ABS in the \\test\\1d sub-folder in the SasView installation folder.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ Further Information ------------------- ASCII - https://en.wikipedia.org/wiki/ASCII HDF - https://en.wikipedia.org/wiki/Hierarchical_Data_Format NXS - https://en.wikipedia.org/wiki/Nexus_(data_format) - https://www.nexusformat.org/ For a description of the CanSAS SASXML 1D format see: - http://www.cansas.org/formats/canSAS1d/1.1/doc/ For a description of the NXcanSAS format see: - http://cansas-org.github.io/NXcanSAS/classes/contributed_definitions/NXcanSAS.html For descriptions of the NIST 1D & 2D formats see: - https://github.com/sansigormacros/ncnrsansigormacros/wiki For descriptions of the ISIS COLETTE (or 'RKH') 1D & 2D formats see: - https://www.isis.stfc.ac.uk/Pages/colette-ascii-file-format-descriptions.pdf For a description of the BSL/OTOKO format see: - http://www.diamond.ac.uk/Beamlines/Soft-Condensed-Matter/small-angle/SAXS-Software/CCP13/BSL.html .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ .. note:: This help document was last changed by Steve King, 29Oct2021