.. data_formats.rst .. This is a port of the original SasView help file moved to SasData .. by S King, ISIS, during SasView CodeCamp-III in Feb 2015. .. WG Bouwman, DUT, added during CodeCamp-V in Oct 2016 the SESANS data format .. WG Bouwman, DUT, updated during CodeCamp-VI in Apr 2017 the SESANS data format .. J Krzywon, P Butler, S King, overhauled during PR Hackathon in Oct 2021 .. J Krzywon, moved to SasData and updated for recent usage in Sep 2023 .. _Formats: Data Formats ============ SasData recognizes 1D SAS (*I(Q) vs Q*), 2D SAS(*I(Qx,Qy) vs (Qx,Qy)*) and 1D SESANS (*P(z) vs z*) data in several different file formats. It will also load other data adhering to the same file formats (e.g. DLS or NR data) but not necessarily recognise what those data represent (e.g. units may be mislabelled). .. note:: From SasData 0.7 onwards, the :ref:`File_Converter_Tool` allows some legacy formats to be converted into either the canSAS XML format or the NXcanSAS format. These legacy formats include 1D/2D BSL/OTOKO, 1D output from FIT2D and some other SAXS-oriented software, and the ISIS COLETTE (or 'RKH') 2D format. 1D SAS Formats -------------- SasView recognizes 1D data supplied in a number of specific formats, as identified by the file extensions below. It also incorporates a series of 'fall back loaders' which are called if all else fails. The file extensions (which are not case sensitive) with specific meaning are: * .ABS (NIST format) * .ASC (NIST format) * .COR (in canSAS XML v1.0 and v1.1 formats *only*) * .DAT (NIST format) * .H5, .NXS, .HDF, or .HDF5 (in NXcanSAS v1.0 and v1.1 formats *only*) * .PDH (Anton Paar SAXSess format) * .XML (in canSAS XML v1.0 and v1.1 formats *only*) The CanSAS & NXcanSAS standard formats are both output by the `Mantid data reduction framework `_ and the `NIST Igor data reduction routines `_. ASCII formats can be viewed in any text editor (Notepad, vi, etc) but the HDF formats require a binary viewer, such as the downloadable `HDFView `_ or an online tool such as the one created by PaNOSC `H5Web `_. ASCII ('text') files are expected to have 2, 3, or 4 columns of values, separated by whitespaces or commas or semicolons, in the following order: *Q, I(Q), ( dI(Q), dQ(Q) )* where *Q* is assumed to have units of 1/Angstrom, *I(Q)* is assumed to have units of 1/cm, *dI(Q)* is the uncertainty on the intensity value (also as 1/cm), and *dQ(Q)* **is the one-sigma FWHM Gaussian instrumental resolution in** *Q*, **assumed to have arisen from pinhole geometry**. If the data are slit-smeared, see `Slit-Smeared Data`_. There must be a minimum of 5 lines of data in the file, and each line of data **must** contain the same number of entries (i.e. columns of data values). As a general rule, SasView will provide better fits when it is provided with more information (i.e. more columns) about each observation (i.e. data point). If importing CSV data, ensure that it is not using commas for thousands delimiters. **Examples of these formats can be found in the \\example_data\\1d_data sub-folder in the SasData distribution.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ 2D SAS Formats -------------- SasView recognizes 2D data only when supplied in ASCII ('text') files in the NIST 2D format (with the extensions .ASC or .DAT) or HDF files in the NeXus NXcanSAS (HDF5) format (with the extension .H5, .NXS, .HDF, or .HDF5). File extensions are case-insensitive. Data in the old ISIS 2D format must be converted using the :ref:`File_Converter_Tool`. The NXcanSAS standard format is output by the `Mantid data reduction framework `_ and the `NIST Igor data reduction routines `_. Most of the header lines in the `NIST 2D format `_ can be removed *except the last line*, and only the first three columns (*Qx, Qy,* and *I(Qx,Qy)*) are actually required. Data values have the same meanings and units as for `1D SAS Formats`_. *2D image data* can be translated into 2D 'pseudo-data' using the :ref:`Image_Viewer_Tool`, but this should only be done with an abundance of caution. **Examples of these formats can be found in the \\example_data\\2d_data sub-folder in the SasData distribution.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ 1D SESANS Format ---------------- SasData will read ASCII ('text') files in a prototype SESANS standard format with the extensions .SES or .SESANS (case-insensitive). The file format starts with a list of name-value pairs which detail the general experimental parameters necessary for fitting and analyzing the data. This list should contain all the information necessary for the file to be 'portable' between users. Following the header are up to 8 space-delimited columns of experimental variables of which the first 4 columns are required. In order, these are: - Spin-echo length (:math:`\delta`, in Angstroms) - Depolarization (:math:`log(P/P_0)/(\lambda^2 * thickness)`, in Angstrom :sup:`-1` cm :sup:`-1`\ ) - Depolarization error (also in in Angstrom :sup:`-1` cm :sup:`-1`\ ) (i.e. the measurement error) - Spin-echo length error (:math:`\Delta \delta`, in Angstroms) (i.e. the experimental resolution) - Neutron wavelength (:math:`\lambda`, in Angstroms) - Neutron wavelength error (:math:`\Delta \lambda`, in Angstroms) - Normalized polarization (:math:`P/P_0`, unitless) - Normalized polarization error (:math:`\Delta(P/P_0)`, unitless) (i.e. the measurement error) **Examples of this format can be found in the \\example_data\\sesans_data sub-folder in the SasData distribution.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ Coordinate Formats ------------------ The :ref:`SANS_Calculator_Tool` in SasView recognises ASCII ('text') files containing coordinate data (a grid of 'voxels') with the following extensions (which are not case-sensitive): * .PDB (`Protein Data Bank format `_) * .OMF (`OOMMF micromagnetic simulation format `_) * .SLD (Spin-Lattice Dynamics simulation format) In essence, coordinate formats specify a location and one or more properties of that location (e.g. what it represents, its volume, or magnetisation, etc). The PDB/OMF/SLD formats all use a rectangular grid of voxels. The .STL coordinate format is not currently supported by SasView. **Examples of these formats can be found in the \\example_data\\coordinate_data sub-folder in the SasData distribution.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ Slit-Smeared Data ----------------- SasView will only account for slit smearing if the data being processed are recognized as slit-smeared. Currently, only the canSAS \*.XML, NIST \*.ABS and NXcanSAS formats facilitate slit-smeared data. The easiest way to include $\Delta q_v$ in a way recognizable by SasView is to mimic the \*.ABS format. The data must follow the normal rules for general ASCII files **but include 6 columns**, not 4 columns. The SasView general ASCII loader assumes the first four columns are *Q*, *I(Q)*, *dI(Q)*, and *dQ(Q)*. If the data does not contain any *dI(Q)* information, these can be faked by making them ~1% (or less) of the *I(Q)* data. The fourth column **must** then contain the the $\Delta q_v$ value, in |Ang^-1|, but as a **negative number**. Each row of data should have the same value. The 5th column **must** be a duplicate of column 1. **Column 6 can have any value but cannot be empty**. Finally, the line immediately preceding the actual columnar data **must** begin with: "The 6 columns". **For an example of a 6 column file with slit-smeared data, see the example data set 1umSlitSmearSphere.ABS in the \\example_data\\1d_data sub-folder in the SasData distribution folder.** .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ Further Information ------------------- ASCII - https://en.wikipedia.org/wiki/ASCII HDF - https://en.wikipedia.org/wiki/Hierarchical_Data_Format NXS - https://en.wikipedia.org/wiki/Nexus_(data_format) - https://www.nexusformat.org/ For a description of the CanSAS SASXML 1D format see: - http://www.cansas.org/formats/canSAS1d/1.1/doc/ For a description of the NXcanSAS format see: - http://cansas-org.github.io/NXcanSAS/classes/contributed_definitions/NXcanSAS.html For descriptions of the NIST 1D & 2D formats see: - https://github.com/sansigormacros/ncnrsansigormacros/wiki/DataOutputFormats For descriptions of the ISIS COLETTE (or 'RKH') 1D & 2D formats see: - https://www.isis.stfc.ac.uk/Pages/colette-ascii-file-format-descriptions.pdf For a description of the BSL/OTOKO format see: - http://www.diamond.ac.uk/Beamlines/Soft-Condensed-Matter/small-angle/SAXS-Software/CCP13/BSL.html .. ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ .. note:: This help document was last changed by Jeff Krzywon, 28Sep2023