This is gnuastro.info, produced by makeinfo version 7.2 from
gnuastro.texi.

This book documents version 0.23.119-586f of the GNU Astronomy Utilities
(Gnuastro).  Gnuastro provides various programs and libraries for
astronomical data manipulation and analysis.

   Copyright © 2015-2025 Free Software Foundation, Inc.

     Permission is granted to copy, distribute and/or modify this
     document under the terms of the GNU Free Documentation License,
     Version 1.3 or any later version published by the Free Software
     Foundation; with no Invariant Sections, no Front-Cover Texts, and
     no Back-Cover Texts.  A copy of the license is included in the
     section entitled "GNU Free Documentation License".
INFO-DIR-SECTION Astronomy
START-INFO-DIR-ENTRY
* Gnuastro: (gnuastro).       GNU Astronomy Utilities.
* libgnuastro: (gnuastro)Gnuastro library. Full Gnuastro library doc.

* help-gnuastro: (gnuastro)help-gnuastro mailing list. Getting help.

* bug-gnuastro: (gnuastro)Report a bug. How to report bugs

* Arithmetic: (gnuastro)Arithmetic. Arithmetic operations on pixels.
* astarithmetic: (gnuastro)Invoking astarithmetic. Options to Arithmetic.

* BuildProgram: (gnuastro)BuildProgram. Compile and run programs using Gnuastro's library.
* astbuildprog: (gnuastro)Invoking astbuildprog. Options to BuildProgram.

* ConvertType: (gnuastro)ConvertType. Convert different file types.
* astconvertt: (gnuastro)Invoking astconvertt. Options to ConvertType.

* Convolve: (gnuastro)Convolve. Convolve an input file with kernel.
* astconvolve: (gnuastro)Invoking astconvolve. Options to Convolve.

* CosmicCalculator: (gnuastro)CosmicCalculator. For cosmological params.
* astcosmiccal: (gnuastro)Invoking astcosmiccal. Options to CosmicCalculator.

* Crop: (gnuastro)Crop. Crop region(s) from image(s).
* astcrop: (gnuastro)Invoking astcrop. Options to Crop.

* Fits: (gnuastro)Fits. View and manipulate FITS extensions and keywords.
* astfits: (gnuastro)Invoking astfits. Options to Fits.

* MakeCatalog: (gnuastro)MakeCatalog. Make a catalog from labeled image.
* astmkcatalog: (gnuastro)Invoking astmkcatalog. Options to MakeCatalog.

* MakeProfiles: (gnuastro)MakeProfiles. Make mock profiles.
* astmkprof: (gnuastro)Invoking astmkprof. Options to MakeProfiles.

* Match: (gnuastro)Match. Match two separate catalogs.
* astmatch: (gnuastro)Invoking astmatch. Options to Match.

* NoiseChisel: (gnuastro)NoiseChisel. Detect signal in noise.
* astnoisechisel: (gnuastro)Invoking astnoisechisel. Options to NoiseChisel.

* Segment: (gnuastro)Segment. Segment detections based on signal structure.
* astsegment: (gnuastro)Invoking astsegment. Options to Segment.

* Query: (gnuastro)Query. Access remote databases for downloading data.
* astquery: (gnuastro)Invoking astquery. Options to Query.

* Statistics: (gnuastro)Statistics. Get image Statistics.
* aststatistics: (gnuastro)Invoking aststatistics. Options to Statistics.

* Table: (gnuastro)Table. Read and write FITS binary or ASCII tables.
* asttable: (gnuastro)Invoking asttable. Options to Table.

* Warp: (gnuastro)Warp. Warp a dataset to a new grid.
* astwarp: (gnuastro)Invoking astwarp. Options to Warp.

* astscript: (gnuastro)Installed scripts. Gnuastro's installed scripts.
* astscript-ds9-region: (gnuastro)Invoking astscript-ds9-region. Options to this script
* astscript-fits-view: (gnuastro)Invoking astscript-fits-view. Options to this script
* astscript-pointing-simulate: (gnuastro)Invoking astscript-pointing-simulate. Options to this script
* astscript-psf-scale-factor: (gnuastro)Invoking astscript-psf-scale-factor. Options to this script
* astscript-psf-select-stars: (gnuastro)Invoking astscript-psf-select-stars. Options to this script
* astscript-psf-stamp: (gnuastro)Invoking astscript-psf-stamp. Options to this script
* astscript-psf-subtract: (gnuastro)Invoking astscript-psf-subtract. Options to this script
* astscript-psf-unite: (gnuastro)Invoking astscript-psf-unite. Options to this script
* astscript-radial-profile: (gnuastro)Invoking astscript-radial-profile. Options to this script
* astscript-sort-by-night: (gnuastro)Invoking astscript-sort-by-night. Options to this script
* astscript-zeropoint: (gnuastro)Invoking astscript-zeropoint. Options to this script
END-INFO-DIR-ENTRY


File: gnuastro.info,  Node: Makefile extensions,  Next: Library,  Prev: Installed scripts,  Up: Top

11 Makefile extensions (for GNU Make)
*************************************

Make (https://en.wikipedia.org/wiki/Make_(software)) is a build
automation tool.  It can greatly help manage your analysis workflow,
even very complex projects with thousands of files and hundreds of
processing steps.  In this book, we have discussed Make previously in
the context of parallelization (see *note How to run simultaneous
operations::).  For example, Maneage (http://maneage.org) uses Make to
organize complex and reproducible workflows, see Akhlaghi et al.  2021
(https://arxiv.org/abs/2006.03018).

   GNU Make is the most common and powerful implementation of Make, with
many unique additions to the core POSIX standard of Make.  One of those
features is the ability to add extensions using a dynamic library (that
Gnuastro provides).  For the details of this feature from GNU Make's own
manual, see its Loading dynamic objects
(https://www.gnu.org/software/make/manual/html_node/Loading-Objects.html)
section.  Through this feature, Gnuastro provides additional Make
functions that are useful in the context of data analysis.

   To use this feature, Gnuastro has to be built in shared library more.
Gnuastro's Make extensions will not work if you build Gnuastro without
shared libraries (for example, when you configure Gnuastro with
‘--disable-shared’ or ‘--debug’).

* Menu:

* Loading the Gnuastro Make functions::  How to find and load Gnuastro's Make library.
* Makefile functions of Gnuastro::  The available functions.


File: gnuastro.info,  Node: Loading the Gnuastro Make functions,  Next: Makefile functions of Gnuastro,  Prev: Makefile extensions,  Up: Makefile extensions

11.1 Loading the Gnuastro Make functions
========================================

To load Gnuastro's Make functions in your Makefile, you should use the
‘load’ command of GNU Make in your Makefile.  The load command should be
given Gnuastro's ‘libgnuastro_make.so’ dynamic library, which has been
specifically written for being called by GNU Make.  The generic command
looks like this (the ‘/PATH/TO’ part should be changed):

     load /PATH/TO/lib/libgnuastro_make.so

Here are the possible replacements of the ‘/PATH/TO’ component:
‘/usr/local’
     If you installed Gnuastro from source and did not use the
     ‘--prefix’ option at configuration time, you should use this base
     directory.
‘/usr/’
     If you installed Gnuastro through your operating system's package
     manager, it is highly likely that Gnuastro's library is here.
‘~/.local’
     If you installed Gnuastro from source, but used ‘--prefix’ to
     install Gnuastro in your home directory (as described in *note
     Installation directory::).

   If you cannot find ‘libgnuastro_make.so’ in the locations above, the
command below should give you its location.  It assumes that the
libraries are in the same base directory as the programs (which is
usually the case).

     $ which astfits | sed -e's|bin/astfits|lib/libgnuastro_make.so|'

   The problem with the command above is that it is not in the Makefile
and has to be run by the user when running it (adding more complexity
and therefore potential for bugs).  Therefore a more generic way to have
a portable Makefile (that will run independent of where the host's
Gnuastro is installed), you can implement the command above in Make like
below.  For the ‘:=’, see *note Makefile functions of Gnuastro::.

     gnuastro-prefix := $(subst bin/astfits,,$(shell which astfits))
     load $(gnuastro-prefix)/lib/libgnuastro_make.so


File: gnuastro.info,  Node: Makefile functions of Gnuastro,  Prev: Loading the Gnuastro Make functions,  Up: Makefile extensions

11.2 Makefile functions of Gnuastro
===================================

All Gnuastro Make functions start with the ‘ast-’ prefix (similar to the
programs on the command-line, but with a dash).  After you have loaded
Gnuastro's shared library for Makefiles within your Makefile, you can
call these functions just like any Make function.  For instructions on
how to load Gnuastro's Make functions, see *note Loading the Gnuastro
Make functions::.

   There are two types of Make functions in Gnuastro's Make extensions:
1) Basic operations on text which is more general than astronomy or
Gnuastro, see *note Text functions for Makefiles::).  2) Operations that
are directly related to astronomy (mostly FITS files) and Gnuastro, see
*note Astronomy functions for Makefiles::).

*Difference between '‘=’' or '‘:=’' for variable definition* When you
define a variable with '‘=’', its value is expanded only when used, not
when defined.  However, when you use '‘:=’', it is immediately expanded
when defined.  Therefore the location of a '‘:=’' variable in the
Makefile matters: if used before its definition, it will be empty!
Those defined by '‘=’' can be used even before they are defined!  On the
other hand, if your variable invokes functions (like ‘foreach’ or
‘wildcard’), it is better to use '‘:=’'.  Otherwise, each time the value
is used, the function will be expanded (possibly may times) and this
will reduce the speed of your pipeline.  For more, see the The two
flavors of variables
(https://www.gnu.org/software/make/manual/html_node/Flavors.html) in the
GNU Make manual.

* Menu:

* Text functions for Makefiles::  Basic text operations to supplement Make.
* Astronomy functions for Makefiles::  Astronomy/FITS related functions.


File: gnuastro.info,  Node: Text functions for Makefiles,  Next: Astronomy functions for Makefiles,  Prev: Makefile functions of Gnuastro,  Up: Makefile functions of Gnuastro

11.2.1 Text functions for Makefiles
-----------------------------------

The functions described below operate on simple strings (plain text).
They are therefore generic (not limited to astronomy/FITS), but because
they are commonly necessary in astronomical data analysis pipelines and
are not available anywhere else, we have included them in Gnuastro.  The
names of these functions start with ‘ast-text-*’ and each has a fully
working example to demonstrate its usage.

‘$(ast-text-to-upper STRING)’
     Returns the input string but with all characters in UPPER-CASE. For
     example, the following minimal Makefile will print ‘FOOO BAAR UGGH’
     word of the list.

          load /usr/local/lib/libgnuastro_make.so

          list   = fOOo bAar UggH
          ulist := $(ast-text-to-upper $(list))
          all:; echo $(ulist)

‘$(ast-text-to-lower STRING)’
     Returns the input string but with all characters in lower-case.
     For example, the following minimal Makefile will print ‘fooo baar
     uggh’ word of the list.

          load /usr/local/lib/libgnuastro_make.so

          list  = fOOo bAar UggH
          llist := $(ast-text-to-lower $(list))
          all:; echo $(llist)

‘$(ast-text-contains STRING, TEXT)’
     Returns all white-space-separated words in ‘TEXT’ that contain the
     ‘STRING’, removing any words that _do not_ match.  For example, the
     following minimal Makefile will only print the ‘bAaz Aah’ word of
     the list.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-contains Aa, $(list))

     This can be thought of as Make's own ‘filter’ function, but if it
     would accept two patterns in a format like this ‘$(filter
     %Aa%,$(list))’ (for the example above).  In fact, the first
     sentence describing this function is taken from the Make manual's
     first sentence that describes the ‘filter’ function!  However,
     unfortunately Make's ‘filter’ function only accepts a single ‘%’,
     not two!

‘$(ast-text-not-contains STRING, TEXT)’
     Returns all white-space-separated words in ‘TEXT’ that _do not_
     contain the ‘STRING’, removing any words that _do not_ match.  This
     is the inverse of the ‘ast-text-contains’ function.  For example,
     the following minimal Makefile will print ‘fooo baar uggh’ word of
     the list.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-not-contains Aa, $(list))

‘$(ast-text-prev TARGET, LIST)’
     Returns the word in ‘LIST’ that is previous to ‘TARGET’.  If
     ‘TARGET’ is the first word of the list, or is not within it at all,
     this function will return an empty string (nothing).  If any of the
     arguments are an empty string (or only contain space characters
     like '<SPACE>', '<TAB>', new-line and etc), this function will
     return an empty string (having no effect in Make).  The simple
     example below shows a minimal usage scenario where the output will
     be ‘fooo’.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-prev baar, $(list))

     This function is useful in many other scenarios.  For example, one
     scenario when this function can be useful is when you want a list
     of higher-level targets to always be executed in sequence (even
     when Make is run in parallel).  But you want their lower-level
     prerequisites to be executed in parallel.

     The fully working example below shows this in practice: the "final"
     target depends on the sub-components ‘a.fits’, ‘b.fits’, ‘c.fits’
     and ‘d.fits’.  But each one of these has seven dependencies (for
     example ‘a.fits’ depends on the sub-sub-components ‘a-1.fits’,
     ‘a-2.fits’, ‘a-3.fits’, ...).  Without this function, Make will
     first build all the sub-sub-components first, then the
     sub-components and ultimately the final target.

     When the files are small and there aren't too many of them, this is
     not a problem.  But when you hundreds/thousands of
     sub-sub-components, your computer may not have the capacity to hold
     them all in storage or RAM (during processing).  In such cases, you
     want to build the sub-components to built in series, but the
     sub-sub-components of each sub-component to be built in parallel.
     This function allows just this in an easy manner as below: the
     sub-sub-components of each sub-component depend on the previous
     sub-component.

     To see the effect of this function put the example below in a
     ‘Makefile’ and run ‘make -j12’ (to simultaneously execute 12 jobs);
     then comment/remove this function (so there is no prerequisite in
     ‘$(subsubs)’) and re-run ‘make -j12’.

          # Basic settings
          all: final
          .SECONDEXPANSION:
          load /usr/local/lib/libgnuastro_make.so

          # 4 sub-components (alphabetic), each with 7
          # sub-sub-components (numeric).
          subids = a b c d
          subsubids = 1 2 3 4 5 6 7
          subs := $(foreach s, $(subids), $(s).fits)
          subsubs := $(foreach s, $(subids), \
                       $(foreach ss, $(subsubids), \
                         $(s)-$(ss).fits))

          # Build the sub-components:
          $(subsubs): %.fits: $$(ast-text-prev \
                                 $$(word 1, $$(subst -, ,%)).fits, \
                                 $(subs))
                  @echo "$@: $^"

          # Build the final components
          $(subs): %.fits: $$(foreach s, $(subsubids), %-$$(s).fits)
                  @echo "$@: $^"

          # Final
          final: $(subs)
                  @echo "$@: $^"

     As you see, when this function is present, the sub-sub-components
     of each sub-component are executed in parallel, while at each
     moment, only a single sub-component's prerequisites are being made.
     Without this function, make first builds all the
     sub-sub-components, then goes to the sub-components.  There can be
     any level of components between these, allowing this operation to
     be as complex as necessary in your data analysis pipeline.
     Unfortunately the ‘.NOTPARALLEL’ target of GNU Make doesn't allow
     this level of customization.

‘$(ast-text-prev-batch TARGET, NUM, LIST)’
     Returns the previous batch of ‘NUM’ words in ‘LIST’ (in relation to
     the batch containing ‘TARGET’).  ‘NUM’ will be interpreted as an
     unsigned integer and cannot be zero.  If any of the arguments are
     an empty string (or only contain space characters like '<SPACE>',
     '<TAB>', new-line and etc), this function will return an empty
     string (having no effect in Make).  In the special case that
     ‘NUM=1’, this is equivalent to the ‘ast-text-prev’ function that is
     described above.

     Here is one scenario where this function is useful: in astronomy
     datasets are can easily be very large.  Therefore, some Make
     recipes in your pipeline may require a lot of memory; such that
     executing them on all the available threads (for example 12 threads
     with ‘-j12’) will immediately occupy all your RAM, causing a crash
     in your pipeline.  However, let's assume that you have sufficient
     RAM to execute 4 targets of those recipes in parallel.  Therefore
     while you want all the other steps of your pipeline to be using all
     12 threads, you want one rule to only build 4 targets at any time.
     But before starting to use this function, also see
     ‘ast-text-prev-batch-by-ram’.

     The example below demonstrates the usage of this function in a
     minimal working example of the scenario above: we want to build 15
     targets, but in batches of 4 target at a time, irrespective of how
     many threads Make was executed with.

          load /usr/local/lib/libgnuastro_make.so

          .SECONDEXPANSION:

          targets := $(foreach i,$(shell seq 15),a-$(i).fits)

          all: $(targets)

          $(targets): $$(ast-text-prev-batch $$@,4,$(targets))
                  @echo "$@: $^"

     If you place the example above in a plain-text file called
     ‘Makefile’ (correcting for the TAB at the start of the recipe), and
     run Make on 12 threads like below, you will see the following
     output.  The targets in each batch are not ordered (and the order
     may change in different runs) because they have been run in
     parallel.

          $ make -j12
          a-1.fits:
          a-3.fits:
          a-2.fits:
          a-4.fits:
          a-5.fits: a-1.fits a-2.fits a-3.fits a-4.fits
          a-6.fits: a-1.fits a-2.fits a-3.fits a-4.fits
          a-8.fits: a-1.fits a-2.fits a-3.fits a-4.fits
          a-7.fits: a-1.fits a-2.fits a-3.fits a-4.fits
          a-9.fits: a-5.fits a-6.fits a-7.fits a-8.fits
          a-11.fits: a-5.fits a-6.fits a-7.fits a-8.fits
          a-12.fits: a-5.fits a-6.fits a-7.fits a-8.fits
          a-10.fits: a-5.fits a-6.fits a-7.fits a-8.fits
          a-13.fits: a-9.fits a-10.fits a-11.fits a-12.fits
          a-15.fits: a-9.fits a-10.fits a-11.fits a-12.fits
          a-14.fits: a-9.fits a-10.fits a-11.fits a-12.fits

     Any other rule that is later added to this make file (as a
     prerequisite/parent of ‘targets’ or as a child of ‘targets’) will
     be run on 12 threads.

‘$(ast-text-prev-batch-by-ram TARGET, NEEDED_RAM_GB, LIST)’
     Similar to ‘ast-text-prev-batch’, but instead of taking the number
     of words/files in each batch, this function takes the maximum
     amount of RAM that is needed by one instance of the recipe.
     Through the ‘NEEDED_RAM_GB’ argument, you should specify the amount
     of ram that a _single_ instance of the recipe in this rule needs.
     If any of the arguments are an empty string (or only contain space
     characters like '<SPACE>', '<TAB>', new-line and etc), this
     function will return an empty string (having no effect in Make).
     When the needed RAM is larger than the available RAM only one job
     will be done at a time (similar to ‘ast-text-prev’).

     The number of files in each batch is calculated internally by
     reading the available RAM on the system at the moment Make calls
     this function.  Therefore this function is more generalizable to
     different computers (with very different RAM and/or CPU threads).
     But to avoid overlapping with other rules that may consume a lot of
     RAM, it is better to design your Makefile such that other rules are
     only executed once all instances of this rule have been completed.

     For example, assume every instance of one rule in your Makefile
     requires a maximum of 5.2 GB of RAM during its execution, and your
     computer has 32 GB of RAM and 2 threads.  In this case, you do not
     need to manage the targets at all: at the worst moment your
     pipeline will consume 10.4GB of RAM (much smaller than the 32GB of
     RAM that you have).  However, you later run the same pipeline on
     another machine with identical RAM, but 12 threads!  In this case,
     you will need $5.2\times12=62.4$GB of RAM; but the new system
     doesn't have that much RAM, causing your pipeline to crash.  If you
     used ‘ast-text-prev-batch’ function (described above) to manage
     these hardware limitations, you would have to manually change the
     number on every new system; this is inconvenient, can cause many
     bugs, and requires manual intervention (not making your pipeline
     automatic).

     The ‘ast-text-prev-batch-by-ram’ function was designed as a
     solution to the problem above: it will read the amount of available
     RAM at the time that Make starts (before the recipes in your
     pipeline are actually executed).  From the value to
     ‘NEEDED_RAM_GB’, it will then estimate how many instances of that
     recipe can be executed in parallel without breaching the available
     RAM of the system.  Therefore it is important to not run another
     heavy RAM consumer on the system while your pipeline is being
     executed.  Note that this function reads the available RAM, not
     total RAM; it therefore accounts for the background operations of
     the operating system or graphic user environment that are running
     in parallel to your pipeline; and assumes they will remain at the
     same level.

     The fully working example below shows the usage of this function in
     a scenario where we assume the recipe requires 4.2GB of RAM for
     each target.

          load /usr/local/lib/libgnuastro_make.so

          .SECONDEXPANSION:

          targets := $(foreach i,$(shell seq 13),$(i).fits)

          all: $(targets)

          $(targets): $$(ast-text-prev-batch-by-ram $$@,4.2,$(targets))
                  @echo "$@: $^"

     Once the contents above are placed in a ‘Makefile’ and you execute
     the command below in a system with about 27GB of available RAM
     (total RAM is 32GB; the 5GB difference is used by the operating
     system and other background programs), you will get an output like
     below.

          $ make -j12
          1.fits:
          2.fits:
          3.fits:
          4.fits:
          5.fits:
          6.fits:
          7.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
          8.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
          11.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
          10.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
          9.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
          12.fits: 1.fits 2.fits 3.fits 4.fits 5.fits 6.fits
          13.fits: 7.fits 8.fits 9.fits 10.fits 11.fits 12.fits

     Depending on the amount of available RAM on your system, you will
     get a different output.  To see the effect, you can decrease or
     increase the amount of required RAM (‘4.2’ in the example above).

     *What is the maximum RAM required by my command?*  Put a
     '‘/usr/bin/time --format=%M’' prefix behind your full command
     (including any options and arguments).  For example like this for a
     call to Gnuastro's Warp program:

          /usr/bin/time --format=%M astwarp image.fits

     After the regular outputs of the program, you will see a number on
     the last line.  This number is the maximum used RAM (in
     _kilobytes_) during the execution of the program.  Later, you can
     convert this to Gigabytes (to feed into this function) by dividing
     it to $10^6$.

‘$(ast-text-next TARGET, LIST)’

     Returns the word in ‘LIST’ that is next to ‘TARGET’.  Its operation
     is very similar to the ‘ast-text-prev’ function defined above.  The
     output of the example below (that uses this function) is ‘bAaz’.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-next baar, $(list))

‘$(ast-text-next-words TARGET, NUM, LIST)’

     Returns the next ‘NUM’ words in ‘LIST’ (after ‘TARGET’).  The
     output of the example below (that uses this function) is ‘bAaz
     uggh’.

          load /usr/local/lib/libgnuastro_make.so

          list = fooo baar bAaz uggh Aah
          all:
               echo $(ast-text-next baar, 2, $(list))


File: gnuastro.info,  Node: Astronomy functions for Makefiles,  Prev: Text functions for Makefiles,  Up: Makefile functions of Gnuastro

11.2.2 Astronomy functions for Makefiles
----------------------------------------

FITS files (the standard data format in astronomy) have unique features
(header keywords and HDUs) that can greatly help designing workflows in
Makefiles.  The Makefile extension functions of this section allow you
to optimally use those features within your pipelines.  Besides FITS,
when designing your workflow/pipeline with Gnuastro, there are also
special features like version checking that simplify your design.

‘$(ast-version-is STRING)’
     Returns ‘1’ if the version of the used Gnuastro is equal to
     ‘STRING’, and ‘0’ otherwise.  This is useful/critical for obtaining
     reproducible results on different systems.  It can be used in
     combination with Conditionals in Make
     (https://www.gnu.org/software/make/manual/html_node/Conditionals.html)
     to ensure the required version of Gnuastro is going to be used in
     your workflow.

     For example, in the minimal working Makefile below, we are using it
     to specify if the default (first) target (‘all’) should have any
     prerequisites (and let the workflow start), or if it should simply
     print a message (that the required version of Gnuastro isn't
     installed) and abort (without any prerequisites).

          load /usr/local/lib/libgnuastro_make.so

          gnuastro-version = 0.19
          ifeq ($(ast-version-is $(gnuastro-version)),1)
          all: paper.pdf
          else
          all:; @echo "Please use Gnuastro $(gnuastro-version)"
          endif

          result.fits: input.fits
                  astnoisechisel $< --output=$@

          paper.pdf: result.fits
                  pdflatex --halt-on-error paper.tex

‘$(ast-fits-with-keyvalue KEYNAME, KEYVALUES, HDU, FITS_FILES)’
     Will select only the FITS files (from a list of many in
     ‘FITS_FILES’, non-FITS files are ignored), where the ‘KEYNAME’
     keyword has the value(s) given in ‘KEYVALUES’.  Only the HDU given
     in the ‘HDU’ argument will be checked.  According to the FITS
     standard, the keyword name is not case sensitive, but the keyword
     value is.

     For example, if you have many FITS files in the ‘/datasets/images’
     directory, the minimal Makefile below will put those with a value
     of ‘BAR’ or ‘BAZ’ for the ‘FOO’ keyword in HDU number ‘1’ in the
     ‘selected’ Make variable.  Notice how there is no comma between
     ‘BAR’ and ‘BAZ’: you can specify any series of values.

     load /usr/local/lib/libgnuastro_make.so
     
     files := $(wildcard /datasets/images/*.fits)
     selected := $(ast-fits-with-keyvalue FOO, BAR BAZ, 1, $(files))
     
     all:
     	echo "Full:     $(words $(files)) files";
     	echo "Selected: $(words $(selected)) files"

‘$(ast-fits-unique-keyvalues KEYNAME, HDU, FITS_FILES)’
     Will return the unique values given to the given FITS keyword
     (‘KEYNAME’) in the given HDU of all the input FITS files (non-FITS
     files are ignored).  For example, after the commands below, the
     ‘keyvalues’ variable will contain the unique values given to the
     ‘FOO’ keyword in HDU number 1 of all the FITS files in
     ‘/datasets/images/*.fits’.

          files := $(wildcard /datasets/images/*.fits)
          keyvalues := $(ast-fits-unique-keyvalues FOO, 1, $(files))

     This is useful when you do not know the full range of values
     a-priori.  For example, let's assume that you are looking at a
     night's observations with a telescope and the purpose of the FITS
     image is written in the ‘OBJECT’ keyword of the image (which we can
     assume is in HDU number 1).  This keyword can have the name of the
     various science targets (for example, ‘NGC123’ and ‘M31’) and
     calibration targets (for example, ‘BIAS’ and ‘FLAT’).  The list of
     science targets is different from project to project, such that in
     one night, you can observe multiple projects.  But the calibration
     frames have unique names.  Knowing the calibration keyword values,
     you can extract the science keyword values of the night with the
     command below (feeding the output of this function to Make's
     ‘filter-out’ function).

          calib = BIAS FLAT
          files := $(wildcard /datasets/images/*.fits)
          science := $(filter-out $(calib), \
                       $(ast-fits-unique-keyvalues OBJECT, 1, $(files)))

     The ‘science’ variable will now contain the unique science targets
     that were observed in your selected FITS images.  You can use it to
     group the various exposures together in the next stages to make
     separate coadds of deep images for each science target (you can
     select FITS files based on their keyword values using the
     ‘ast-fits-with-keyvalue’ function, which is described separately in
     this section).


File: gnuastro.info,  Node: Library,  Next: Developing,  Prev: Makefile extensions,  Up: Top

12 Library
**********

Each program in Gnuastro that was discussed in the prior chapters (or
any program in general) is a collection of functions that is compiled
into one executable file which can communicate directly with the outside
world.  The outside world in this context is the operating system.  By
communication, we mean that control is directly passed to a program from
the operating system with a (possible) set of inputs and after it is
finished, the program will pass control back to the operating system.
For programs written in C and C++, the unique ‘main’ function is in
charge of this communication.

   Similar to a program, a library is also a collection of functions
that is compiled into one executable file.  However, unlike programs,
libraries do not have a ‘main’ function.  Therefore they cannot
communicate directly with the outside world.  This gives you the chance
to write your own ‘main’ function and call library functions from within
it.  After compiling your program into a binary executable, you just
have to _link_ it to the library and you are ready to run (execute) your
program.  In this way, you can use Gnuastro at a much lower-level, and
in combination with other libraries on your system, you can
significantly boost your creativity.

   This chapter starts with a basic introduction to libraries and how
you can use them in *note Review of library fundamentals::.  The
separate functions in the Gnuastro library are then introduced
(classified by context) in *note Gnuastro library::.  If you end up
routinely using a fixed set of library functions, with a well-defined
input and output, it will be much more beneficial if you define a
program for the job.  Therefore, in its *note Version controlled
source::, Gnuastro comes with the *note The TEMPLATE program:: to easily
define your own programs(s).

* Menu:

* Review of library fundamentals::  Guide on libraries and linking.
* BuildProgram::                Link and run source files with this library.
* Gnuastro library::            Description of all library functions.
* Library demo programs::       Demonstration for using libraries.


File: gnuastro.info,  Node: Review of library fundamentals,  Next: BuildProgram,  Prev: Library,  Up: Library

12.1 Review of library fundamentals
===================================

Gnuastro's libraries are written in the C programming language.  In
*note Why C::, we have thoroughly discussed the reasons behind this
choice.  C was actually created to write Unix, thus understanding the
way C works can greatly help in effectively using programs and libraries
in all Unix-like operating systems.  Therefore, in the following
subsections some important aspects of C, as it relates to libraries (and
thus programs that depend on them) on Unix are reviewed.  First we will
discuss header files in *note Headers:: and then go onto *note
Linking::.  This section finishes with *note Summary and example on
libraries::.  If you are already familiar with these concepts, please
skip this section and go directly to *note Gnuastro library::.

   In theory, a full operating system (or any software) can be written
as one function.  Such a software would not need any headers or linking
(that are discussed in the subsections below).  However, writing that
single function and maintaining it (adding new features, fixing bugs,
documentation, etc.)  would be a programmer or scientist's worst
nightmare!  Furthermore, all the hard work that went into creating it
cannot be reused in other software: every other programmer or scientist
would have to re-invent the wheel.  The ultimate purpose behind
libraries (which come with headers and have to be linked) is to address
this problem and increase modularity: "the degree to which a system's
components may be separated and recombined" (from Wikipedia).  The more
modular the source code of a program or library, the easier maintaining
it will be, and all the hard work that went into creating it can be
reused for a wider range of problems.

* Menu:

* Headers::                     Header files included in source.
* Linking::                     Linking the compiled source files into one.
* Summary and example on libraries::  A summary and example on using libraries.


File: gnuastro.info,  Node: Headers,  Next: Linking,  Prev: Review of library fundamentals,  Up: Review of library fundamentals

12.1.1 Headers
--------------

C source code is read from top to bottom in the source file, therefore
program components (for example, variables, data structures and
functions) should all be _defined_ or _declared_ closer to the top of
the source file: before they are used.  _Defining_ something in C or C++
is jargon for providing its full details.  _Declaring_ it, on the
other-hand, is jargon for only providing the minimum information needed
for the compiler to pass it temporarily and fill in the detailed
definition later.

   For a function, the _declaration_ only contains the inputs and their
data-types along with the output's type(1).  The _definition_ adds to
the declaration by including the exact details of what operations are
done to the inputs to generate the output.  As an example, take this
simple summation function:

     double
     sum(double a, double b)
     {
       return a + b;
     }
What you see above is the _definition_ of this function: it shows you
(and the compiler) exactly what it does to the two ‘double’ type inputs
and that the output also has a ‘double’ type.  Note that a function's
internal operations are rarely so simple and short, it can be
arbitrarily long and complicated.  This unreasonably short and simple
function was chosen here for ease of reading.  The declaration for this
function is:

     double
     sum(double a, double b);

You can think of a function's declaration as a building's address in the
city, and the definition as the building's complete blueprints.  When
the compiler confronts a call to a function during its processing, it
does not need to know anything about how the inputs are processed to
generate the output.  Just as the postman does not need to know the
inner structure of a building when delivering the mail.  The declaration
(address) is enough.  Therefore by _declaring_ the functions once at the
start of the source files, we do not have to worry about _defining_ them
after they are used.

   Even for a simple real-world operation (not a simple summation like
above!), you will soon need many functions (for example, some for
reading/preparing the inputs, some for the processing, and some for
preparing the output).  Although it is technically possible, managing
all the necessary functions in one file is not easy and is contrary to
the modularity principle (see *note Review of library fundamentals::),
for example, the functions for preparing the input can be usable in your
other projects with a different processing.  Therefore, as we will see
later (in *note Linking::), the functions do not necessarily need to be
defined in the source file where they are used.  As long as their
definitions are ultimately linked to the final executable, everything
will be fine.  For now, it is just important to remember that the
functions that are called within one source file must be declared within
the source file (declarations are mandatory), but not necessarily
defined there.

   In the spirit of modularity, it is common to define contextually
similar functions in one source file.  For example, in Gnuastro,
functions that calculate the median, mean and other statistical
functions are defined in ‘lib/statistics.c’, while functions that deal
directly with FITS files are defined in ‘lib/fits.c’.

   Keeping the definition of similar functions in a separate file
greatly helps their management and modularity, but this fact alone does
not make things much easier for the caller's source code: recall that
while definitions are optional, declarations are mandatory.  So if this
was all, the caller would have to manually copy and paste (_include_)
all the declarations from the various source files into the file they
are working on now.  To address this problem, programmers have adopted
the header file convention: the header file of a source code contains
all the declarations that a caller would need to be able to use any of
its functions.  For example, in Gnuastro, ‘lib/statistics.c’ (file
containing function definitions) comes with ‘lib/gnuastro/statistics.h’
(only containing function declarations).

   The discussion above was mainly focused on functions, however, there
are many more programming constructs such as preprocessor macros and
data structures.  Like functions, they also need to be known to the
compiler when it confronts a call to them.  So the header file also
contains their definitions or declarations when they are necessary for
the functions.

   Preprocessor macros (or macros for short) are replaced with their
defined value by the preprocessor before compilation.  Conventionally
they are written only in capital letters to be easily recognized.  It is
just important to understand that the compiler does not see the macros,
it sees their fixed values.  So when a header specifies macros you can
do your programming without worrying about the actual values.  The
standard C types (for example, ‘int’, or ‘float’) are very low-level and
basic.  We can collect multiple C types into a _structure_ for a
higher-level way to keep and pass-along data.  See *note Generic data
container:: for some examples of macros and data structures.

   The contents in the header need to be _include_d into the caller's
source code with a special preprocessor command: ‘#include
<path/to/header.h>’.  As the name suggests, the _preprocessor_ goes
through the source code prior to the processor (or compiler).  One of
its jobs is to include, or merge, the contents of files that are
mentioned with this directive in the source code.  Therefore the
compiler sees a single entity containing the contents of the main file
and all the included files.  This allows you to include many (sometimes
thousands of) declarations into your code with only one line.  Since the
headers are also installed with the library into your system, you do not
even need to keep a copy of them for each separate program, making
things even more convenient.

   Try opening some of the ‘.c’ files in Gnuastro's ‘lib/’ directory
with a text editor to check out the include directives at the start of
the file (after the copyright notice).  Let's take ‘lib/fits.c’ as an
example.  You will notice that Gnuastro's header files (like
‘gnuastro/fits.h’) are indeed within this directory (the ‘fits.h’ file
is in the ‘gnuastro/’ directory).  You will notice that files like
‘stdio.h’, or ‘string.h’ are not in this directory (or anywhere within
Gnuastro).

   On most systems the basic C header files (like ‘stdio.h’ and
‘string.h’ mentioned above) are located in ‘/usr/include/’(2).  Your
compiler is configured to automatically search that directory (and
possibly others), so you do not have to explicitly mention these
directories.  Go ahead, look into the ‘/usr/include’ directory and find
‘stdio.h’ for example.  When the necessary header files are not in those
specific libraries, the preprocessor can also search in places other
than the current directory.  You can specify those directories with this
preprocessor option(3):

‘-I DIR’
     "Add the directory ‘DIR’ to the list of directories to be searched
     for header files.  Directories named by '-I' are searched before
     the standard system include directories.  If the directory ‘DIR’ is
     a standard system include directory, the option is ignored to
     ensure that the default search order for system directories and the
     special treatment of system headers are not defeated..."  (quoted
     from the GNU Compiler Collection manual).  Note that the space
     between <I> and the directory is optional and commonly not used.

   If the preprocessor cannot find the included files, it will abort
with an error.  In fact a common error when building programs that
depend on a library is that the compiler does not know where a library's
header is (see *note Known issues::).  So you have to manually tell the
compiler where to look for the library's headers with the ‘-I’ option.
For a small software with one or two source files, this can be done
manually (see *note Summary and example on libraries::).  However, to
enhance modularity, Gnuastro (and most other bin/libraries) contain many
source files, so the compiler is invoked many times(4).  This makes
manual addition or modification of this option practically impossible.

   To solve this problem, in the GNU build system, there are
conventional environment variables for the various kinds of compiler
options (or flags).  These environment variables are used in every call
to the compiler (they can be empty).  The environment variable used for
the C preprocessor (or CPP) is ‘CPPFLAGS’.  By giving ‘CPPFLAGS’ a value
once, you can be sure that each call to the compiler will be affected.
See *note Known issues:: for an example of how to set this variable at
configure time.

   As described in *note Installation directory::, you can select the
top installation directory of a software using the GNU build system,
when you ‘./configure’ it.  All the separate components will be put in
their separate sub-directory under that, for example, the programs,
compiled libraries and library headers will go into ‘$prefix/bin’
(replace ‘$prefix’ with a directory), ‘$prefix/lib’, and
‘$prefix/include’ respectively.  For enhanced modularity, libraries that
contain diverse collections of functions (like GSL, WCSLIB, and
Gnuastro), put their header files in a sub-directory unique to
themselves.  For example, all Gnuastro's header files are installed in
‘$prefix/include/gnuastro’.  In your source code, you need to keep the
library's sub-directory when including the headers from such libraries,
for example, ‘#include <gnuastro/fits.h>’(5).  Not all libraries need to
follow this convention, for example, CFITSIO only has one header
(‘fitsio.h’) which is directly installed in ‘$prefix/include’.

   ---------- Footnotes ----------

   (1) Recall that in C, functions only have one output.

   (2) The ‘include/’ directory name is taken from the pre-processor's
‘#include’ directive, which is also the motivation behind the 'I' in the
‘-I’ option to the pre-processor.

   (3) Try running Gnuastro's ‘make’ and find the directories given to
the compiler with the ‘-I’ option.

   (4) Nearly every command you see being executed after running ‘make’
is one call to the compiler.

   (5) the top ‘$prefix/include’ directory is usually known to the
compiler


File: gnuastro.info,  Node: Linking,  Next: Summary and example on libraries,  Prev: Headers,  Up: Review of library fundamentals

12.1.2 Linking
--------------

To enhance modularity, similar functions are defined in one source file
(with a ‘.c’ suffix, see *note Headers:: for more).  After running
‘make’, each human-readable, ‘.c’ file is translated (or compiled) into
a computer-readable "object" file (ending with ‘.o’).  Note that object
files are also created when building programs, they are not particular
to libraries.  Try opening Gnuastro's ‘lib/’ and ‘bin/progname/’
directories after running ‘make’ to see these object files(1).
Afterwards, the object files are _linked_ together to create an
executable program or a library.

   The object files contain the full definition of the functions in the
respective ‘.c’ file along with a list of any other function (or
generally "symbol") that is referenced there.  To get a list of those
functions you can use the ‘nm’ program which is part of GNU Binutils.
For example, from the top Gnuastro directory, run:

     $ nm bin/arithmetic/arithmetic.o

This will print a list of all the functions (more generally, 'symbols')
that were called within ‘bin/arithmetic/arithmetic.c’ along with some
further information (for example, a ‘T’ in the second column shows that
this function is actually defined here, ‘U’ says that it is undefined
here).  Try opening the ‘.c’ file to check some of these functions for
yourself.  Run ‘info nm’ for more information.

   To recap, the _compiler_ created the separate object files mentioned
above for each ‘.c’ file.  The _linker_ will then combine all the
symbols of the various object files (and libraries) into one program or
library.  In the case of Arithmetic (a program) the contents of the
object files in ‘bin/arithmetic/’ are copied (and re-ordered) into one
final executable file which we can run from the operating system.

   There are two ways to _link_ all the necessary symbols: static and
dynamic/shared.  When the symbols (computer-readable function
definitions in most cases) are copied into the output, it is called
_static_ linking.  When the symbols are kept in their original file and
only a reference to them is kept in the executable, it is called
_dynamic_, or _shared_ linking.

   Let's have a closer look at the executable to understand this better:
we will assume you have built Gnuastro without any customization and
installed Gnuastro into the default ‘/usr/local/’ directory (see *note
Installation directory::).  If you tried the ‘nm’ command on one of
Arithmetic's object files above, then with the command below you can
confirm that all the functions that were defined in the object file
above (had a ‘T’ in the second column) are also defined in the
‘astarithmetic’ executable:

     $ nm /usr/local/bin/astarithmetic

These symbols/function have been statically linked (copied) in the final
executable.  But you will notice that there are still many undefined
symbols in the executable (those with a ‘U’ in the second column).  One
class of such functions are Gnuastro's own library functions that start
with '‘gal_’':

     $ nm /usr/local/bin/astarithmetic | grep gal_

   These undefined symbols (functions) are present in another file and
will be linked to the Arithmetic program every time you run it.
Therefore they are known as dynamically _linked_ libraries (2).  As we
saw above, static linking is done when the executable is being built.
However, when a program is dynamically linked to a library, at
build-time, the library's symbols are only checked with the available
libraries: they are not actually copied into the program's executable.
Every time you run the program, the (dynamic) linker will be activated
and will try to link the program to the installed library before the
program starts.

   If you want all the libraries to be statically linked to the
executables, you have to tell Libtool (which Gnuastro uses for the
linking) to disable shared libraries at configure time(3):

     $ configure --disable-shared

Try configuring Gnuastro with the command above, then build and install
it (as described in *note Quick start::).  Afterwards, check the ‘gal_’
symbols in the installed Arithmetic executable like before.  You will
see that they are actually copied this time (have a ‘T’ in the second
column).  If the second column does not convince you, look at the
executable file size with the following command:

     $ ls -lh /usr/local/bin/astarithmetic

It should be around 4.2 Megabytes with this static linking.  If you
configure and build Gnuastro again with shared libraries enabled (which
is the default), you will notice that it is roughly 100 Kilobytes!

   This huge difference would have been very significant in the old
days, but with the roughly Terabyte storage drives commonly in use
today, it is negligible.  Fortunately, output file size is not the only
benefit of dynamic linking: since it links to the libraries at run-time
(rather than build-time), you do not have to rebuild a higher-level
program or library when an update comes for one of the lower-level
libraries it depends on.  You just install the new low-level library and
it will automatically be used/linked next time in the programs that use
it.  To be fair, this also creates a few complications(4):

   • Reproducibility: Even though your high-level tool has the same
     version as before, with the updated library, you might not get the
     same results.
   • Broken links: if some functions have been changed or removed in the
     updated library, then the linker will abort with an error at
     run-time.  Therefore you need to rebuild your higher-level program
     or library.

   To see a list of all the shared libraries that are needed for a
program or a shared library to run, you can use GNU C library's ‘ldd’(5)
program, for example:

     $ ldd /usr/local/bin/astarithmetic

   Library file names (in their installation directory) start with a
‘lib’ and their ending (suffix) shows if they are static (‘.a’) or
dynamic (‘.so’), as described below.  The name of the library is in the
middle of these two, for example, ‘libgsl.a’ or ‘libgnuastro.a’ (GSL and
Gnuastro's static libraries), and ‘libgsl.so.23.0.0’ or
‘libgnuastro.so.4.0.0’ (GSL and Gnuastro's shared library, the numbers
may be different).

   • A static library is known as an archive file and has the ‘.a’
     suffix.  A static library is not an executable file.

   • A shared library ends with the ‘.so.X.Y.Z’ suffix and is
     executable.  The three numbers in the suffix, describe the version
     of the shared library.  Shared library versions are defined to
     allow multiple versions of a shared library simultaneously on a
     system and to help detect possible updates in the library and
     programs that depend on it by the linker.

     It is very important to mention that this version number is
     different from the software version number (see *note Version
     numbering::), so do not confuse the two.  See the "Library
     interface versions" chapter of GNU Libtool for more.

     For each shared library, we also have two symbolic links ending
     with ‘.so.X’ and ‘.so’.  They are automatically set by the
     installer, but you can change them (point them to another version
     of the library) when you have multiple versions of a library on
     your system.

   Libraries that are built with GNU Libtool (including Gnuastro and its
dependencies), build both static and dynamic libraries by default and
install them in ‘prefix/lib/’ directory (for more on ‘prefix’, see *note
Installation directory::).  In this way, programs depending on the
libraries can link with them however they prefer.  See the contents of
‘/usr/local/lib’ with the command below to see both the static and
shared libraries available there, along with their executable nature and
the symbolic links:

     $ ls -l /usr/local/lib/

   To link with a library, the linker needs to know where to find the
library.  _At compilation time_, these locations can be passed to the
linker with two separate options (see *note Summary and example on
libraries:: for an example) as described below.  You can see these
options and their usage in practice while building Gnuastro (after
running ‘make’):

‘-L DIR’
     Will tell the linker to look into ‘DIR’ for the libraries.  For
     example, ‘-L/usr/local/lib’, or ‘-L/home/yourname/.local/lib’.  You
     can make multiple calls to this option, so the linker looks into
     several directories at compilation time.  Note that the space
     between <L> and the directory is optional and commonly ignored
     (written as ‘-LDIR’).

‘-lLIBRARY’
     Specify the unique library identifier/name (not containing
     directory or shared/dynamic nature) to be linked with the
     executable.  As discussed above, library file names have fixed
     parts which must not be given to this option.  So ‘-lgsl’ will
     guide the linker to either look for ‘libgsl.a’ or ‘libgsl.so’
     (depending on the type of linking it is suppose to do).  You can
     link many libraries by repeated calls to this option.

     *Very important: * The place of this option on the compiler's
     command matters.  This is often a source of confusion for
     beginners, so let's assume you have asked the linker to link with
     library A using this option.  As soon as the linker confronts this
     option, it looks into the list of the undefined symbols it has
     found until that point and does a search in library A for any of
     those symbols.  If any pending undefined symbol is found in library
     A, it is used.  After the search in undefined symbols is complete,
     the contents of library A are completely discarded from the
     linker's memory.  Therefore, if a later object file or library uses
     an unlinked symbol in library A, the linker will abort after it has
     finished its search in all the input libraries or object files.

     As an example, Gnuastro's ‘gal_fits_img_read’ function depends on
     the ‘fits_read_pix’ function of CFITSIO (specified with
     ‘-lcfitsio’, which in turn depends on the cURL library, called with
     ‘-lcurl’).  So the proper way to link something that uses this
     function is ‘-lgnuastro -lcfitsio -lcurl’.  If instead, you give:
     ‘-lcfitsio -lgnuastro’ the linker will complain and abort.  To
     avoid such linking complexities when using Gnuastro's library, we
     recommend using *note BuildProgram::.

   If you have compiled and linked your program with a dynamic library,
then the dynamic linker also needs to know the location of the libraries
after building the program: _every time_ the program is run afterwards.
Therefore, it may happen that you do not get any errors when
compiling/linking a program, but are unable to run your program because
of a failure to find a library.  This happens because the dynamic linker
has not found the dynamic library _at run time_.

   To find the dynamic libraries at run-time, the linker looks into the
paths, or directories, in the ‘LD_LIBRARY_PATH’ environment variable.
For a discussion on environment variables, especially search paths like
‘LD_LIBRARY_PATH’, and how you can add new directories to them, see
*note Installation directory::.

   ---------- Footnotes ----------

   (1) Gnuastro uses GNU Libtool for portable library creation.  Libtool
will also make a ‘.lo’ file for each ‘.c’ file when building libraries
(‘.lo’ files are human-readable).

   (2) Do not confuse dynamically _linked_ libraries with dynamically
_loaded_ libraries.  The former (that is discussed here) are only loaded
once at the program startup.  However, the latter can be loaded anytime
during the program's execution, they are also known as plugins.

   (3) Libtool is very common and is commonly used.  Therefore, you can
use this option to configure on most programs using the GNU build system
if you want static linking.

   (4) Both of these can be avoided by joining the mailing lists of the
lower-level libraries and checking the changes in newer versions before
installing them.  Updates that result in such behaviors are generally
heavily emphasized in the release notes.

   (5) If your operating system is not using the GNU C library, you
might need another tool.


File: gnuastro.info,  Node: Summary and example on libraries,  Prev: Linking,  Up: Review of library fundamentals

12.1.3 Summary and example on libraries
---------------------------------------

After the mostly abstract discussions of *note Headers:: and *note
Linking::, we will give a small tutorial here.  But before that, let's
recall the general steps of how your source code is prepared, compiled
and linked to the libraries it depends on so you can run it:

  1. The *preprocessor* includes the header (‘.h’) files into the
     function definition (‘.c’) files, expands preprocessor macros.
     Generally the preprocessor prepares the human-readable source for
     compilation (reviewed in *note Headers::).

  2. The *compiler* will translate (compile) the human-readable contents
     of each source (merged ‘.c’ and the ‘.h’ files, or generally the
     output of the preprocessor) into the computer-readable code of ‘.o’
     files.

  3. The *linker* will link the called function definitions from various
     compiled files to create one unified object.  When the unified
     product has a ‘main’ function, this function is the product's only
     entry point, enabling the operating system or user to directly
     interact with it, so the product is a program.  When the product
     does not have a ‘main’ function, the linker's product is a library
     and it is exported functions can be linked to other executables (it
     has many entry points).

   The GNU Compiler Collection (or GCC for short) will do all three
steps.  So as a first example, from Gnuastro's source, go to
‘tests/lib/’.  This directory contains the library tests, you can use
these as some simple tutorials.  For this demonstration, we will compile
and run the ‘arraymanip.c’.  This small program will call Gnuastro
library for some simple operations on an array (open it and have a
look).  To compile this program, run this command inside the directory
containing it.

     $ gcc arraymanip.c -lgnuastro -lm -o arraymanip

The two ‘-lgnuastro’ and ‘-lm’ options (in this order) tell GCC to first
link with the Gnuastro library and then with C's math library.  The ‘-o’
option is used to specify the name of the output executable, without it
the output file name will be ‘a.out’ (on most OSs), independent of your
input file name(s).

   If your top Gnuastro installation directory (let's call it ‘$prefix’,
see *note Installation directory::) is not recognized by GCC, you will
get preprocessor errors for unknown header files.  Once you fix it, you
will get linker errors for undefined functions.  To fix both, you should
run GCC as follows: additionally telling it which directories it can
find Gnuastro's headers and compiled library (see *note Headers:: and
*note Linking::):

     $ gcc -I$prefix/include -L$prefix/lib arraymanip.c -lgnuastro -lm     \
           -o arraymanip

This single command has done all the preprocessor, compilation and
linker operations.  Therefore no intermediate files (object files in
particular) were created, only a single output executable was created.
You are now ready to run the program with:

     $ ./arraymanip

   The Gnuastro functions called by this program only needed to be
linked with the C math library.  But if your program needs WCS
coordinate transformations, needs to read a FITS file, needs special
math operations (which include its linear algebra operations), or you
want it to run on multiple CPU threads, you also need to add these
libraries in the call to GCC: ‘-lgnuastro -lwcs -lcfitsio -lgsl
-lgslcblas -pthread -lm’.  In *note Gnuastro library::, where each
function is documented, it is mentioned which libraries (if any) must
also be linked when you call a function.  If you feel all these linkings
can be confusing, please consider Gnuastro's *note BuildProgram::
program.


File: gnuastro.info,  Node: BuildProgram,  Next: Gnuastro library,  Prev: Review of library fundamentals,  Up: Library

12.2 BuildProgram
=================

The number and order of libraries that are necessary for linking a
program with Gnuastro library might be too confusing when you need to
compile a small program for one particular job (with one source file).
BuildProgram will use the information gathered during configuring
Gnuastro and link with all the appropriate libraries on your system.
This will allow you to easily compile, link and run programs that use
Gnuastro's library with one simple command and not worry about which
libraries to link to, or the linking order.

   BuildProgram uses GNU Libtool to find the necessary libraries to link
against (GNU Libtool is the same program that builds all of Gnuastro's
libraries and programs when you run ‘make’).  So in the future, if
Gnuastro's prerequisite libraries change or other libraries are added,
you do not have to worry, you can just run BuildProgram and internal
linking will be done correctly.

*BuildProgram requires GNU Libtool:* BuildProgram depends on GNU
Libtool, other implementations do not have some necessary features.  If
GNU Libtool is not available at Gnuastro's configure time, you will get
a notice at the end of the configuration step and BuildProgram will not
be built or installed.  Please see *note Optional dependencies:: for
more information.

* Menu:

* Invoking astbuildprog::       Options and examples for using this program.


File: gnuastro.info,  Node: Invoking astbuildprog,  Prev: BuildProgram,  Up: BuildProgram

12.2.1 Invoking BuildProgram
----------------------------

BuildProgram will compile and link a C source program with Gnuastro's
library and all its dependencies, greatly facilitating the compilation
and running of small programs that use Gnuastro's library.  The
executable name is ‘astbuildprog’ with the following general template:

     $ astbuildprog [OPTION...] C_SOURCE_FILE

One line examples:

     ## Compile, link and run `myprogram.c':
     $ astbuildprog myprogram.c

     ## Similar to previous, but with optimization and compiler warnings:
     $ astbuildprog -Wall -O2 myprogram.c

     ## Compile and link `myprogram.c', then run it with `image.fits'
     ## as its argument:
     $ astbuildprog myprogram.c image.fits

     ## Also look in other directories for headers and linking:
     $ astbuildprog -Lother -Iother/dir myprogram.c

     ## Just build (compile and link) `myprogram.c', do not run it:
     $ astbuildprog --onlybuild myprogram.c

   If BuildProgram is to run, it needs a C programming language source
file as input.  By default it will compile and link the given source
into a final executable program and run it.  The built executable name
can be set with the optional ‘--output’ option.  When no output name is
set, BuildProgram will use Gnuastro's *note Automatic output:: system to
remove the suffix of the input source file (usually ‘.c’) and use the
resulting name as the built program name.

   For the full list of options that BuildProgram shares with other
Gnuastro programs, see *note Common options::.  You may also use
Gnuastro's *note Configuration files:: to specify other
libraries/headers to use for special directories and not have to type
them in every time.

   The C compiler can be chosen with the ‘--cc’ option, or environment
variables, please see the description of ‘--cc’ for more.  The two
common ‘LDFLAGS’ and ‘CPPFLAGS’ environment variables are also checked
and used in the build by default.  Note that they are placed after the
values to the corresponding options ‘--includedir’ and ‘--linkdir’.
Therefore BuildProgram's own options take precedence.  Using environment
variables can be disabled with the ‘--noenv’ option.  Just note that
BuildProgram also keeps the important flags in these environment
variables in its configuration file.  Therefore, in many cases, even
though you may needed them to build Gnuastro, you will not need them in
BuildProgram.

   The first argument is considered to be the C source file that must be
compiled and linked.  Any other arguments (non-option tokens on the
command-line) will be passed onto the program when BuildProgram wants to
run it.  Recall that by default BuildProgram will run the program after
building it.  This behavior can be disabled with the ‘--onlybuild’
option.

   When the ‘--quiet’ option (see *note Operating mode options::) is not
called, BuildPrograms will print the compilation and running commands.
Once your program grows and you break it up into multiple files (which
are much more easily managed with Make), you can use the linking flags
of the non-quiet output in your ‘Makefile’.

‘-c STR’
‘--cc=STR’
     C compiler to use for the compilation, if not given environment
     variables will be used as described in the next paragraph.  If the
     compiler is in your system's search path, you can simply give its
     name, for example, ‘--cc=gcc’.  If it is not in your system's
     search path, you can give its full path, for example,
     ‘--cc=/path/to/your/custom/cc’.

     If this option has no value after parsing the command-line and all
     configuration files (see *note Configuration file precedence::),
     then BuildProgram will look into the following environment
     variables in the given order ‘CC’ and ‘GCC’.  If they are also not
     defined, BuildProgram will ultimately default to the ‘gcc’ command
     which is present in many systems (sometimes as a link to other
     compilers).

‘-I STR’
‘--includedir=STR’
     Directory to search for files that you ‘#include’ in your C
     program.  Note that headers relating to Gnuastro and its
     dependencies do not need this option.  This is only necessary if
     you want to use other headers.  It may be called multiple times and
     order matters.  This directory will be searched before those of
     Gnuastro's build and also the system search directories.  See *note
     Headers:: for a thorough introduction.

     From the GNU C preprocessor manual: "Add the directory ‘STR’ to the
     list of directories to be searched for header files.  Directories
     named by ‘-I’ are searched before the standard system include
     directories.  If the directory ‘STR’ is a standard system include
     directory, the option is ignored to ensure that the default search
     order for system directories and the special treatment of system
     headers are not defeated".

‘-L STR’
‘--linkdir=STR’
     Directory to search for compiled libraries to link the program
     with.  Note that all the directories that Gnuastro was built with
     will already be used by BuildProgram (GNU Libtool).  This option is
     only necessary if your libraries are in other directories.
     Multiple calls to this option are possible and order matters.  This
     directory will be searched before those of Gnuastro's build and
     also the system search directories.  See *note Linking:: for a
     thorough introduction.

‘-l STR’
‘--linklib=STR’
     Library to link with your program.  Note that all the libraries
     that Gnuastro was built with will already be linked by BuildProgram
     (GNU Libtool).  This option is only necessary if you want to link
     with other directories.  Multiple calls to this option are possible
     and order matters.  This library will be linked before Gnuastro's
     library or its dependencies.  See *note Linking:: for a thorough
     introduction.

‘-O INT/STR’
‘--optimize=INT/STR’
     Compiler optimization level: 0 (for no optimization, good
     debugging), 1, 2, 3 (for the highest level of optimizations).  From
     the GNU Compiler Collection (GCC) manual: "Without any optimization
     option, the compiler's goal is to reduce the cost of compilation
     and to make debugging produce the expected results.  Statements are
     independent: if you stop the program with a break point between
     statements, you can then assign a new value to any variable or
     change the program counter to any other statement in the function
     and get exactly the results you expect from the source code.
     Turning on optimization flags makes the compiler attempt to improve
     the performance and/or code size at the expense of compilation time
     and possibly the ability to debug the program."  Please see your
     compiler's manual for the full list of acceptable values to this
     option.

‘-g’
‘--debug’
     Emit extra information in the compiled binary for use by a
     debugger.  When calling this option, it is best to explicitly
     disable optimization with ‘-O0’.  To combine both options you can
     run ‘-gO0’ (see *note Options:: for how short options can be merged
     into one).

‘-W STR’
‘--warning=STR’
     Print compiler warnings on command-line during compilation.
     "Warnings are diagnostic messages that report constructions that
     are not inherently erroneous but that are risky or suggest there
     may have been an error."  (from the GCC manual).  It is always
     recommended to compile your programs with warnings enabled.

     All compiler warning options that start with ‘W’ are usable by this
     option in BuildProgram also, see your compiler's manual for the
     full list.  Some of the most common values to this option are:
     ‘pedantic’ (Warnings related to standard C) and ‘all’ (all issues
     the compiler confronts).

‘-t’
‘--tag=STR’
     The language configuration information.  Libtool can build objects
     and libraries in many languages.  In many cases, it can identify
     the language automatically, but when it does not you can use this
     option to explicitly notify Libtool of the language.  The
     acceptable values are: ‘CC’ for C, ‘CXX’ for C++, ‘GCJ’ for Java,
     ‘F77’ for Fortran 77, ‘FC’ for Fortran, ‘GO’ for Go and ‘RC’ for
     Windows Resource.  Note that the Gnuastro library is not yet fully
     compatible with all these languages.

‘-b’
‘--onlybuild’
     Only build the program, do not run it.  By default, the built
     program is immediately run afterwards.

‘-d’
‘--deletecompiled’
     Delete the compiled binary file after running it.  This option is
     only relevant when the compiled program is run after being built.
     In other words, it is only relevant when ‘--onlybuild’ is not
     called.  It can be useful when you are busy testing a program or
     just want a fast result and the actual binary/compiled file is not
     of later use.

‘-a STR’
‘--la=STR’
     Use the given ‘.la’ file (Libtool control file) instead of the one
     that was produced from Gnuastro's configuration results.  The
     Libtool control file keeps all the necessary information for
     building and linking a program with a library built by Libtool.
     The default ‘prefix/lib/libgnuastro.la’ keeps all the information
     necessary to build a program using the Gnuastro library gathered
     during configure time (see *note Installation directory:: for
     prefix).  This option is useful when you prefer to use another
     Libtool control file.

‘-e’
‘--noenv’
     Do not use environment variables in the build, just use the values
     given to the options.  As described above, environment variables
     like ‘CC’, ‘GCC’, ‘LDFLAGS’, ‘CPPFLAGS’ will be read by default and
     used in the build if they have been defined.


File: gnuastro.info,  Node: Gnuastro library,  Next: Library demo programs,  Prev: BuildProgram,  Up: Library

12.3 Gnuastro library
=====================

Gnuastro library's programming constructs (function declarations,
macros, data structures, or global variables) are classified by context
into multiple header files (see *note Headers::)(1).  In this section,
the functions in each header will be discussed under a separate
sub-section, which includes the name of the header.  Assuming a function
declaration is in ‘headername.h’, you can include its declaration in
your source code with:

     # include <gnuastro/headername.h>

The names of all constructs in ‘headername.h’ are prefixed with
‘gal_headername_’ (or ‘GAL_HEADERNAME_’ for macros).  The ‘gal_’ prefix
stands for _G_NU _A_stronomy _L_ibrary.

   Gnuastro library functions are compiled into a single file which can
be linked on the command-line with the ‘-lgnuastro’ option.  See *note
Linking:: and *note Summary and example on libraries:: for an
introduction on linking and some fully working examples of the
libraries.

   Gnuastro's library is a high-level library which depends on lower
level libraries for some operations (see *note Dependencies::).
Therefore if at least one of Gnuastro's functions in your program use
functions from the dependencies, you will also need to link those
dependencies after linking with Gnuastro.  See *note BuildProgram:: for
a convenient way to deal with the dependencies.  BuildProgram will take
care of the libraries to link with your program (which uses the Gnuastro
library), and can even run the built program afterwards.  Therefore it
allows you to conveniently focus on your exciting science/research when
using Gnuastro's libraries.

*Libraries are still under heavy development: * Gnuastro was initially
created to be a collection of command-line programs.  However, as the
programs and their the shared functions grew, internal (not installed)
libraries were added.  Since the 0.2 release, the libraries are
install-able.  Hence the libraries are currently under heavy development
and will significantly evolve between releases and will become more
mature and stable in due time.  It will stabilize with the removal of
this notice.  Check the ‘NEWS’ file for interface changes.  If you use
the Info version of this manual (see *note Info::), you do not have to
worry: the documentation will correspond to your installed version.

* Menu:

* Configuration information::   General information about library config.
* Multithreaded programming::   Tools for easy multi-threaded operations.
* Library data types::          Definitions and functions for types.
* Pointers::                    Wrappers for easy working with pointers.**
* Library blank values::        Blank values and functions to deal with them.
* Library data container::      General data container in Gnuastro.
* Dimensions::                  Dealing with coordinates and dimensions.
* Linked lists::                Various types of linked lists.
* Array input output::          Reading and writing images or cubes.
* Table input output::          Reading and writing table columns.
* FITS files::                  Working with FITS data.
* File input output::           Reading and writing to various file formats.
* World Coordinate System::     Dealing with the world coordinate system.
* Arithmetic on datasets::      Arithmetic operations on a dataset.
* Tessellation library::        Functions for working on tiles.
* Bounding box::                Finding the bounding box.
* Polygons::                    Working with the vertices of a polygon.
* Qsort functions::             Helper functions for Qsort.
* K-d tree::                    Space partitioning in K dimensions.
* Permutations::                Re-order (or permute) the values in a dataset.
* Matching::                    Matching catalogs based on position.
* Statistical operations::      Functions for basic statistics.
* Fitting functions::           Fit independent and measured variables.
* Binary datasets::             Datasets that can only have values of 0 or 1.
* Labeled datasets::            Working with Segmented/labeled datasets.
* Convolution functions::       Library functions to do convolution.
* Pooling functions::           Reduce size of input by statistical methods.
* Interpolation::               Interpolate (over blank values possibly).
* Warp library::                Warp pixel grid to a new one.
* Color functions::             Definitions and operations related to colors.
* Git wrappers::                Wrappers for functions in libgit2.
* Python interface::            Functions to help in writing Python wrappers.
* Unit conversion library::     Converting between recognized units.
* Spectral lines library::      Functions for operating on Spectral lines.
* Cosmology library::           Cosmological calculations.
* SAO DS9 library::             Take inputs from files generated by SAO DS9.

   ---------- Footnotes ----------

   (1) Within Gnuastro's source, all installed ‘.h’ files in
‘lib/gnuastro/’ are accompanied by a ‘.c’ file in ‘/lib/’.


File: gnuastro.info,  Node: Configuration information,  Next: Multithreaded programming,  Prev: Gnuastro library,  Up: Gnuastro library

12.3.1 Configuration information (‘config.h’)
---------------------------------------------

The ‘gnuastro/config.h’ header contains information about the full
Gnuastro installation on your system.  Gnuastro developers should note
that this is the only header that is not available within Gnuastro, it
is only available to a Gnuastro library user _after_ installation.
Within Gnuastro, ‘config.h’ (which is included in every Gnuastro ‘.c’
file, see *note Coding conventions::) has more than enough information
about the overall Gnuastro installation.

 -- Macro: GAL_CONFIG_VERSION
     This macro can be used as a string literal(1) containing the
     version of Gnuastro that is being used.  See *note Version
     numbering:: for the version formats.  For example:

          printf("Gnuastro version: %s\n", GAL_CONFIG_VERSION);

     or

          char *gnuastro_version=GAL_CONFIG_VERSION;

 -- Macro: GAL_CONFIG_HAVE_GSL_INTERP_STEFFEN
     GNU Scientific Library (GSL) is a mandatory dependency of Gnuastro
     (see *note GNU Scientific Library::).  The Steffen interpolation
     function that can be used in Gnuastro was introduced in GSL version
     2.0 (released in October 2015).  This macro will have a value of
     ‘1’ if the host GSL contains this feature at configure time, and
     ‘0’ otherwise.

 -- Macro: GAL_CONFIG_HAVE_FITS_IS_REENTRANT
     This macro will have a value of 1 when the CFITSIO of the host
     system has the ‘fits_is_reentrant’ function (available from CFITSIO
     version 3.30).  This function is used to see if CFITSIO was
     configured to read a FITS file simultaneously on different threads.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_VERSION
     WCSLIB is the reference library for world coordinate system
     transformation (see *note WCSLIB:: and *note World Coordinate
     System::).  However, only more recent versions of WCSLIB also
     provide its version number.  If the WCSLIB that is installed on the
     system provides its version (through the possibly existing
     ‘wcslib_version’ function), this macro will have a value of one,
     otherwise it will have a value of zero.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_DIS_H
     This macro has a value of 1 if the host's WCSLIB has the
     ‘wcslib/dis.h’ header for distortion-related operations.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_MJDREF
     This macro has a value of 1 if the host's WCSLIB reads and stores
     the ‘MJDREF’ FITS header keyword as part of its core ‘wcsprm’
     structure.

 -- Macro: GAL_CONFIG_HAVE_WCSLIB_OBSFIX
     This macro has a value of 1 if the host's WCSLIB supports the
     ‘OBSFIX’ feature (used by ‘wcsfix’ function to parse the input WCS
     for known errors).

 -- Macro: GAL_CONFIG_HAVE_PTHREAD_BARRIER
     The POSIX threads standard define barriers as an optional
     requirement.  Therefore, some operating systems choose to not
     include it.  As one of the ‘./configure’ step checks, Gnuastro we
     check if your system has this POSIX thread barriers.  If so, this
     macro will have a value of ‘1’, otherwise it will have a value of
     ‘0’.  see *note Implementation of pthread_barrier:: for more.

 -- Macro: GAL_CONFIG_SIZEOF_LONG
 -- Macro: GAL_CONFIG_SIZEOF_SIZE_T
     The size of (number of bytes in) the system's ‘long’ and ‘size_t’
     types.  Their values are commonly either 4 or 8 for 32-bit and
     64-bit systems.  You can also get this value with the expression
     '‘sizeof size_t’' for example, without having to include this
     header.

 -- Macro: GAL_CONFIG_HAVE_LIBGIT2
     Libgit2 is an optional dependency of Gnuastro (see *note Optional
     dependencies::).  When it is installed and detected at configure
     time, this macro will have a value of ‘1’ (one).  Otherwise, it
     will have a value of ‘0’ (zero).  Gnuastro also comes with some
     wrappers to make it easier to use libgit2 (see *note Git
     wrappers::).

 -- Macro: GAL_CONFIG_HAVE_PYTHON
     Gnuastro can optionally provide a set of basic functions to
     facilitate wrapper libraries in Python (see *note Python
     interface::).  If a version of Python 3.X was found on the host
     system that has the necessary Numpy headers, this macro will be
     given a value of ‘1’.  Otherwise, it will be given a value of ‘0’
     and the the Python interface functions won't be available in the
     host's Gnuastro library.

 -- Macro: GAL_CONFIG_HAVE_GNUMAKE_H
     Gnuastro provides a set of GNU Make extension functions (see *note
     Makefile extensions::).  In order to use those, the host should
     have ‘gnumake.h’ in its include paths.  This check is done at
     Gnuastro's configuration time.  If it was found, this macro is
     given a value of ‘1’, otherwise, it will have a value of ‘0’.

   ---------- Footnotes ----------

   (1) <https://en.wikipedia.org/wiki/String_literal>


File: gnuastro.info,  Node: Multithreaded programming,  Next: Library data types,  Prev: Configuration information,  Up: Gnuastro library

12.3.2 Multithreaded programming (‘threads.h’)
----------------------------------------------

In recent years, newer CPUs do not have significantly higher frequencies
any more.  However, CPUs are being manufactured with more cores,
enabling more than one operation (thread) at each instant.  This can be
very useful to speed up many aspects of processing and in particular
image processing.

   Most of the programs in Gnuastro utilize multi-threaded programming
for the CPU intensive processing steps.  This can potentially lead to a
significant decrease in the running time of a program, see *note A note
on threads::.  In terms of reading the code, you do not need to know
anything about multi-threaded programming.  You can simply follow the
case where only one thread is to be used.  In these cases, threads are
not used and can be completely ignored.

   When the C language was defined (the K&R's book was written), using
threads was not common, so C's threading capabilities are not introduced
there.  Gnuastro uses POSIX threads for multi-threaded programming,
defined in the ‘pthread.h’ system wide header.  There are various
resources for learning to use POSIX threads.  An excellent tutorial
(https://computing.llnl.gov/tutorials/pthreads/) is provided by the
Lawrence Livermore National Laboratory, with abundant figures to better
understand the concepts, it is a very good start.  The book 'Advanced
programming in the Unix environment'(1), by Richard Stevens and Stephen
Rago, Addison-Wesley, 2013 (Third edition) also has two chapters
explaining the POSIX thread constructs which can be very helpful.

   An alternative to POSIX threads was OpenMP, but POSIX threads are low
level, allowing much more control, while being easier to understand, see
*note Why C::.  All the situations where threads are used in Gnuastro
currently are completely independent with no need of coordination
between the threads.  Such problems are known as "embarrassingly
parallel" problems.  They are some of the simplest problems to solve
with threads and are also the ones that benefit most from them, see the
LLNL introduction(2).

   One very useful POSIX thread concept is ‘pthread_barrier’.
Unfortunately, it is only an optional feature in the POSIX standard, so
some operating systems do not include it.  Therefore in *note
Implementation of pthread_barrier::, we introduce our own
implementation.  This is a rather technical section only necessary for
more technical readers and you can safely ignore it.  Following that, we
describe the helper functions in this header that can greatly simplify
writing a multi-threaded program, see *note Gnuastro's thread related
functions:: for more.

* Menu:

* Implementation of pthread_barrier::  Some systems do not have pthread_barrier
* Gnuastro's thread related functions::  Functions for managing threads.

   ---------- Footnotes ----------

   (1) Do not let the title scare you!  The two chapters on
Multi-threaded programming are very self-sufficient and do not need any
more knowledge than K&R.

   (2) <https://computing.llnl.gov/tutorials/parallel_comp/>


File: gnuastro.info,  Node: Implementation of pthread_barrier,  Next: Gnuastro's thread related functions,  Prev: Multithreaded programming,  Up: Multithreaded programming

12.3.2.1 Implementation of ‘pthread_barrier’
............................................

One optional feature of the POSIX Threads standard is the
‘pthread_barrier’ concept.  It is a very useful high-level construct
that allows for independent threads to "wait" behind a "barrier" for the
rest after they finish.  Barriers can thus greatly simplify the code in
a multi-threaded program, so they are heavily used in Gnuastro.
However, since it is an optional feature in the POSIX standard, some
operating systems do not include it.  So to make Gnuastro portable, we
have written our own implementation of those ‘pthread_barrier’
functions.

   At ‘./configure’ time, Gnuastro will check if ‘pthread_barrier’
constructs are available on your system or not.  If ‘pthread_barrier’ is
not available, our internal implementation will be compiled into the
Gnuastro library and the definitions and declarations below will be
usable in your code with ‘#include <gnuastro/threads.h>’.

 -- Type: pthread_barrierattr_t
     Type to specify the attributes of a POSIX threads barrier.

 -- Type: pthread_barrier_t
     Structure defining the POSIX threads barrier.

 -- Function:
          int
          pthread_barrier_init (pthread_barrier_t *b,
          pthread_barrierattr_t *attr, unsigned int limit)
     Initialize the barrier ‘b’, with the attributes ‘attr’ and total
     ‘limit’ (a number of) threads that must wait behind it.  This
     function must be called before spinning off threads.

 -- Function:
          int
          pthread_barrier_wait (pthread_barrier_t *b)
     This function is called within each thread, just before it is ready
     to return.  Once a thread's function hits this, it will "wait"
     until all the other functions are also finished.

 -- Function:
          int
          pthread_barrier_destroy (pthread_barrier_t *b)
     Destroy all the information in the barrier structure.  This should
     be called by the function that spun-off the threads after all the
     threads have finished.

     *Destroy a barrier before re-using it:* It is very important to
     destroy the barrier before (possibly) reusing it.  This destroy
     function not only destroys the internal structures, it also waits
     (in 1 microsecond intervals, so you will not notice!)  until all
     the threads do not need the barrier structure any more.  If you
     immediately start spinning off new threads with a not-destroyed
     barrier, then the internal structure of the remaining threads will
     get mixed with the new ones and you will get very strange and
     apparently random errors that are extremely hard to debug.


File: gnuastro.info,  Node: Gnuastro's thread related functions,  Prev: Implementation of pthread_barrier,  Up: Multithreaded programming

12.3.2.2 Gnuastro's thread related functions
............................................

The POSIX Threads functions offered in the C library are very low-level
and offer a great range of control over the properties of the threads.
So if you are interested in customizing your tools for complicated
thread applications, it is strongly encouraged to get a nice familiarity
with them.  Some resources were introduced in *note Multithreaded
programming::.

   However, in many cases used in astronomical data analysis, you do not
need communication between threads and each target operation can be done
independently.  Since such operations are very common, Gnuastro provides
the tools below to facilitate the creation and management of jobs
without any particular knowledge of POSIX Threads for such operations.
The most interesting high-level functions of this section are the
‘gal_threads_number’ and ‘gal_threads_spin_off’ that identify the number
of threads on the system and spin-off threads.  You can see a
demonstration of using these functions in *note Library demo -
multi-threaded operation::.

 -- C struct: gal_threads_params
     Structure keeping the parameters of each thread.  When each thread
     is created, a pointer to this structure is passed to it.  The
     ‘params’ element can be the pointer to a structure defined by the
     user which contains all the necessary parameters to pass onto the
     worker function.  The rest of the elements within this structure
     are set internally by ‘gal_threads_spin_off’ and are relevant to
     the worker function.
          struct gal_threads_params
          {
            size_t            id; /* Id of this thread.                  */
            void         *params; /* User-identified pointer.            */
            size_t       *indexs; /* Target indices given to this thread. */
            pthread_barrier_t *b; /* Barrier for all threads.            */
          };

 -- Function:
          size_t
          gal_threads_number ()
     Return the number of threads that the operating system has
     available for your program.  This number is usually fixed for a
     single machine and does not change.  So this function is useful
     when you want to run your program on different machines (with
     different CPUs).

 -- Function:
          void
          gal_threads_spin_off (void *(*worker)(void *), void
          *caller_params, size_t numactions, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Distribute ‘numactions’ jobs between ‘numthreads’ threads and
     spin-off each thread by calling the ‘worker’ function.  The
     ‘caller_params’ pointer will also be passed to ‘worker’ as part of
     the ‘gal_threads_params’ structure.  For a fully working example of
     this function, please see *note Library demo - multi-threaded
     operation::.

     If there are many jobs (millions or billions) to organize, memory
     issues may become important.  With ‘minmapsize’ you can specify the
     minimum byte-size to allocate the necessary space in a
     memory-mapped file or alternatively in RAM. If ‘quietmmap’ is
     non-zero, then a warning will be printed upon creating a
     memory-mapped file.  For more on Gnuastro's memory management, see
     *note Memory management::.

 -- Function:
          void
          gal_threads_attr_barrier_init (pthread_attr_t *attr,
          pthread_barrier_t *b, size_t limit)
     This is a low-level function in case you do not want to use
     ‘gal_threads_spin_off’.  It will initialize the general thread
     attribute ‘attr’ and the barrier ‘b’ with ‘limit’ threads to wait
     behind the barrier.  For maximum efficiency, the threads
     initialized with this function will be detached.  Therefore no
     communication is possible between these threads and in particular
     ‘pthread_join’ will not work on these threads.  You have to use the
     barrier constructs to wait for all threads to finish.

 -- Function:
          char *
          gal_threads_dist_in_threads (size_t numactions, size_t
          numthreads, size_t minmapsize, int quietmmap, size_t **indexs,
          size_t *icols)
     This is a low-level function in case you do not want to use
     ‘gal_threads_spin_off’.  The job of this function is to distribute
     ‘numactions’ jobs/actions in ‘numthreads’ threads.  To do this, it
     will assign each job an ID, ranging from 0 to ‘numactions’-1.  The
     output is the allocated ‘*indexs’ array and the ‘*icols’ number.
     In memory, it is just a simple 1D array that has ‘numthreads’
     $\times$ ‘*icols’ elements.  But you can visualize it as a 2D array
     with ‘numthreads’ rows and ‘*icols’ columns.  For more on the logic
     of the distribution, see below.

     When you have millions/billions of jobs to distribute, ‘indexs’
     will become very large.  For memory management (when to use a
     memory-mapped file, and when to use RAM), you need to specify the
     ‘minmapsize’ and ‘quietmmap’ arguments.  For more on memory
     management, see *note Memory management::.  In general, if your
     distributed jobs will not be on the scale of billions (and you want
     everything to always be written in RAM), just set ‘minmapsize=-1’
     and ‘quietmmap=1’.

     When ‘indexs’ is actually in a memory-mapped file, this function
     will return a string containing the name of the file (that you can
     later give to ‘gal_pointer_mmap_free’ to free/delete).  When
     ‘indexs’ is in RAM, this function will return a ‘NULL’ pointer.  So
     after you are finished with ‘indexs’, you can free it like this:

          char *mmapname;
          int quietmmap=1;
          size_t *indexs, thrdcols;
          size_t numactions=5000, minmapsize=-1;
          size_t numthreads=gal_threads_number();

          /* Distribute the jobs. */
          mmapname=gal_threads_dist_in_threads(numactions, numthreads,
                                               minmapsize, quietmmap,
                                               &indexs, &thrdcols);

          /* Do any processing you want... */

          /* Free the 'indexs' array. */
          if(mmapname) gal_pointer_mmap_free(&mmapname, quietmmap);
          else         free(indexs);

     Here is a brief description of the reasoning behind the ‘indexs’
     array and how the jobs are distributed.  Let's assume you have $A$
     actions (where there is only one function and the input values
     differ for each action) and $T$ threads available to the system
     with $A>T$ (common values for these two would be $A>1000$ and
     $T<10$).  Spinning off a thread is not a cheap job and requires a
     significant number of CPU cycles.  Therefore, creating $A$ threads
     is not the best way to address such a problem.  The most efficient
     way to manage the actions is such that only $T$ threads are
     created, and each thread works on a list of actions identified for
     it in series (one after the other).  This way your CPU will get all
     the actions done with minimal overhead.

     The purpose of this function is to do what we explained above: each
     row in the ‘indexs’ array contains the indices of actions which
     must be done by one thread (so it has ‘numthreads’ rows with
     ‘*icols’ columns).  However, when using ‘indexs’, you do not have
     to know the number of columns.  It is guaranteed that all the rows
     finish with ‘GAL_BLANK_SIZE_T’ (see *note Library blank values::).
     The ‘GAL_BLANK_SIZE_T’ macro plays a role very similar to a
     string's ‘\0’: every row finishes with this macro, so can easily
     stop parsing the indexes in the row as soon as you confront
     ‘GAL_BLANK_SIZE_T’.  For some real examples, please see the example
     program in ‘tests/lib/multithread.c’ for a demonstration.


File: gnuastro.info,  Node: Library data types,  Next: Pointers,  Prev: Multithreaded programming,  Up: Gnuastro library

12.3.3 Library data types (‘type.h’)
------------------------------------

Data in astronomy can have many types, numeric (numbers) and strings
(names, identifiers).  The former can also be divided into integers and
floats, see *note Numeric data types:: for a thorough discussion of the
different numeric data types and which one is useful for different
contexts.

   To deal with the very large diversity of types that are available
(and used in different contexts), in Gnuastro each type is identified
with global integer variable with a fixed name, this variable is then
passed onto functions that can work on any type or is stored in
Gnuastro's *note Generic data container:: as one piece of meta-data.

   The actual values within these integer constants is irrelevant and
you should never rely on them.  When you need to check, explicitly use
the named variable in the table below.  If you want to check with more
than one type, you can use C's ‘switch’ statement.

   Since Gnuastro heavily deals with file input-output, the types it
defines are fixed width types, these types are portable to all systems
and are defined in the standard C header ‘stdint.h’.  You do not need to
include this header, it is included by any Gnuastro header that deals
with the different types.  However, the most commonly used types in a C
(or C++) program (for example, ‘int’ or ‘long’) are not defined by their
exact width (storage size), but by their minimum storage.  So for
example, on some systems, ‘int’ may be 2 bytes (16-bits, the minimum
required by the standard) and on others it may be 4 bytes (32-bits,
common in modern systems).

   With every type, a unique "blank" value (or place-holder showing the
absence of data) can be defined.  Please see *note Library blank
values:: for constants that Gnuastro recognizes as a blank value for
each type.  See *note Numeric data types:: for more explanation on the
limits and particular aspects of each type.

 -- Global integer: GAL_TYPE_INVALID
     This is just a place-holder to specifically mark that no type has
     been set.

 -- Global integer: GAL_TYPE_BIT
     Identifier for a bit-stream.  Currently no program in Gnuastro
     works directly on bits, but features will be added in the future.

 -- Global integer: GAL_TYPE_UINT8
     Identifier for an unsigned, 8-bit integer type: ‘uint8_t’ (from
     ‘stdint.h’), or an ‘unsigned char’ in most modern systems.

 -- Global integer: GAL_TYPE_INT8
     Identifier for a signed, 8-bit integer type: ‘int8_t’ (from
     ‘stdint.h’), or an ‘signed char’ in most modern systems.

 -- Global integer: GAL_TYPE_UINT16
     Identifier for an unsigned, 16-bit integer type: ‘uint16_t’ (from
     ‘stdint.h’), or an ‘unsigned short’ in most modern systems.

 -- Global integer: GAL_TYPE_INT16
     Identifier for a signed, 16-bit integer type: ‘int16_t’ (from
     ‘stdint.h’), or a ‘short’ in most modern systems.

 -- Global integer: GAL_TYPE_UINT32
     Identifier for an unsigned, 32-bit integer type: ‘uint32_t’ (from
     ‘stdint.h’), or an ‘unsigned int’ in most modern systems.

 -- Global integer: GAL_TYPE_INT32
     Identifier for a signed, 32-bit integer type: ‘int32_t’ (from
     ‘stdint.h’), or an ‘int’ in most modern systems.

 -- Global integer: GAL_TYPE_UINT64
     Identifier for an unsigned, 64-bit integer type: ‘uint64_t’ (from
     ‘stdint.h’), or an ‘unsigned long’ in most modern 64-bit systems.

 -- Global integer: GAL_TYPE_INT64
     Identifier for a signed, 64-bit integer type: ‘int64_t’ (from
     ‘stdint.h’), or an ‘long’ in most modern 64-bit systems.

 -- Global integer: GAL_TYPE_INT
     Identifier for a ‘int’ type.  This is just an alias to ‘int16’, or
     ‘int32’ types, depending on the system.

 -- Global integer: GAL_TYPE_UINT
     Identifier for a ‘unsigned int’ type.  This is just an alias to
     ‘uint16’, or ‘uint32’ types, depending on the system.

 -- Global integer: GAL_TYPE_ULONG
     Identifier for a ‘unsigned long’ type.  This is just an alias to
     ‘uint32’, or ‘uint64’ types for 32-bit, or 64-bit systems
     respectively.

 -- Global integer: GAL_TYPE_LONG
     Identifier for a ‘long’ type.  This is just an alias to ‘int32’, or
     ‘int64’ types for 32-bit, or 64-bit systems respectively.

 -- Global integer: GAL_TYPE_SIZE_T
     Identifier for a ‘size_t’ type.  This is just an alias to ‘uint32’,
     or ‘uint64’ types for 32-bit, or 64-bit systems respectively.

 -- Global integer: GAL_TYPE_FLOAT32
     Identifier for a 32-bit single precision floating point type or
     ‘float’ in C.

 -- Global integer: GAL_TYPE_FLOAT64
     Identifier for a 64-bit double precision floating point type or
     ‘double’ in C.

 -- Global integer: GAL_TYPE_COMPLEX32
     Identifier for a complex number composed of two ‘float’ types.
     Note that the complex type is not yet fully implemented in all
     Gnuastro's programs.

 -- Global integer: GAL_TYPE_COMPLEX64
     Identifier for a complex number composed of two ‘double’ types.
     Note that the complex type is not yet fully implemented in all
     Gnuastro's programs.

 -- Global integer: GAL_TYPE_STRING
     Identifier for a string of characters (‘char *’).

 -- Global integer: GAL_TYPE_STRLL
     Identifier for a linked list of string of characters
     (‘gal_list_str_t’, see *note List of strings::).

The functions below are defined to make working with the integer
constants above easier.  In the functions below, the constants above can
be used for the ‘type’ input argument.

 -- Function:
          size_t
          gal_type_sizeof (uint8_t type)
     Return the number of bytes occupied by ‘type’.  Internally, this
     function uses C's ‘sizeof’ operator to measure the size of each
     type.  For strings, this function will return the size of ‘char *’.

 -- Function:
          char *
          gal_type_name (uint8_t type, int long_name)
     Return a string literal that contains the name of ‘type’.  It can
     return both short and long formats of the type names (for example,
     ‘f32’ and ‘float32’).  If ‘long_name’ is non-zero, the long format
     will be returned, otherwise the short name will be returned.  The
     output string is statically allocated, so it should not be freed.
     This function is the inverse of the ‘gal_type_from_name’ function.
     For the full list of names/strings that this function will return,
     see *note Numeric data types::.

 -- Function:
          uint8_t
          gal_type_from_name (char *str)
     Return the Gnuastro integer constant that corresponds to the string
     ‘str’.  This function is the inverse of the ‘gal_type_name’
     function and accepts both the short and long formats of each type.
     For the full list of names/strings that this function will return,
     see *note Numeric data types::.

 -- Function:
          void
          gal_type_min (uint8_t type, void *in)
     Put the minimum possible value of ‘type’ in the space pointed to by
     ‘in’.  Since the value can have any type, this function does not
     return anything, it assumes the space for the given type is
     available to ‘in’ and writes the value there.  Here is one example
          int32_t min;
          gal_type_min(GAL_TYPE_INT32, &min);

     Note: Do not use the minimum value for a blank value of a general
     (initially unknown) type, please use the constants/functions
     provided in *note Library blank values:: for the definition and
     usage of blank values.

 -- Function:
          void
          gal_type_max (uint8_t type, void *in)
     Put the maximum possible value of ‘type’ in the space pointed to by
     ‘in’.  Since the value can have any type, this function does not
     return anything, it assumes the space for the given type is
     available to ‘in’ and writes the value there.  Here is one example
          uint16_t max;
          gal_type_max(GAL_TYPE_INT16, &max);

     Note: Do not use the maximum value for a blank value of a general
     (initially unknown) type, please use the constants/functions
     provided in *note Library blank values:: for the definition and
     usage of blank values.

 -- Function:
          int
          gal_type_is_int (uint8_t type)
     Return 1 if the type is an integer (any width and any sign).

 -- Function:
          int
          gal_type_is_list (uint8_t type)
     Return 1 if the type is a linked list and zero otherwise.

 -- Function:
          int
          gal_type_out (int first_type, int second_type)
     Return the larger of the two given types which can be used for the
     type of the output of an operation involving the two input types.

 -- Function:
          char *
          gal_type_bit_string (void *in, size_t size)
     Return the bit-string in the ‘size’ bytes that ‘in’ points to.  The
     string is dynamically allocated and must be freed afterwards.  You
     can use it to inspect the bits within one region of memory.  Here
     is one short example:

          int32_t a=2017;
          char *bitstr=gal_type_bit_string(&a, 4);
          printf("%d: %s (%X)\n", a, bitstr, a);
          free(bitstr);

     which will produce:
          2017: 11100001000001110000000000000000  (7E1)

     As the example above shows, the bit-string is not the most
     efficient way to inspect bits.  If you are familiar with
     hexadecimal notation, it is much more compact, see
     <https://en.wikipedia.org/wiki/Hexadecimal>.  You can use
     ‘printf’'s ‘%x’ or ‘%X’ to print integers in hexadecimal format.

 -- Function:
          char *
          gal_type_to_string (void *ptr, uint8_t type, int
          quote_if_str_has_space);
     Read the contents of the memory that ‘ptr’ points to (assuming it
     has type ‘type’ and print it into an allocated string which is
     returned.

     If the memory is a string of characters and
     ‘quote_if_str_has_space’ is non-zero, the output string will have
     double-quotes around it if it contains space characters.  Also,
     note that in this case, ‘ptr’ must be a pointer to an array of
     characters (or ‘char **’), as in the example below (which will put
     ‘"sample string"’ into ‘out’):

          char *out, *string="sample string"
          out = gal_type_to_string(&string, GAL_TYPE_STRING, 1);

 -- Function:
          int
          gal_type_from_string (void **out, char *string, uint8_t type)
     Read a string as a given data type and put a pointer to it in
     ‘*out’.  When ‘*out!=NULL’, then it is assumed to be already
     allocated and the value will be simply put there.  If ‘*out==NULL’,
     then space will be allocated for the given type and the string will
     be read into that type.

     Note that when we are dealing with a string type, ‘*out’ should be
     interpreted as ‘char **’ (one element in an array of pointers to
     different strings).  In other words, ‘out’ should be ‘char ***’.

     This function can be used to fill in arrays of numbers from strings
     (in an already allocated data structure), or add nodes to a linked
     list (if the type is a list type).  For an array, you have to pass
     the pointer to the ‘i’th element where you want the value to be
     stored, for example, ‘&(array[i])’.

     If the string was successfully parsed to the requested type, this
     function will return a ‘0’ (zero), otherwise it will return ‘1’
     (one).  This output format will help you check the status of the
     conversion in a code like the example below where we will try
     reading a string as a single precision floating point number.

          float out;
          void *outptr=&out;
          if( gal_type_from_string(&outptr, string, GAL_TYPE_FLOAT32) )
            {
              fprintf(stderr, "%s could not be read as float32\n", string);
              exit(EXIT_FAILURE);
            }

     When you need to read many numbers into an array, ‘out’ would be an
     array, and you can simply increment ‘outptr=out+i’ (where you
     increment ‘i’).

 -- Function:
          void *
          gal_type_string_to_number (char *string, uint8_t *type)
     Read ‘string’ into smallest type that can host the number, the
     allocated space for the number will be returned and the type of the
     number will be put into the memory that ‘type’ points to.  If
     ‘string’ could not be read as a number, this function will return
     ‘NULL’.

     This function first calls the C library's ‘strtod’ function to read
     ‘string’ as a double-precision floating point number.  When
     successful, it will check the value to put it in the smallest
     numerical data type that can handle it; for example, ‘120’ and
     ‘50000’ will be read as a signed 8-bit integer and unsigned 16-bit
     integer types.  When reading as an integer, the C library's
     ‘strtol’ function is used (in base-10) to parse the string again.
     This re-parsing as an integer is necessary because integers with
     many digits (for example, the Unix epoch seconds) will not be
     accurately stored as a floating point and we cannot use the result
     of ‘strtod’.

     When ‘string’ is successfully parsed as a number _and_ there is ‘.’
     in ‘string’, it will force the number into floating point types.
     For example, ‘"5"’ is read as an integer, while ‘"5."’ or ‘"5.0"’,
     or ‘"5.00"’ will be read as a floating point (single-precision).

     For floating point types, this function will count the number of
     significant digits and determine if the given string is single or
     double precision as described in *note Numeric data types::.

     For integers, negative numbers will always be placed in signed
     types (as expected).  If a positive integer falls below the maximum
     of a signed type of a certain width, it will be signed (for
     example, ‘10’ and ‘150’ will be defined as a signed and unsigned
     8-bit integer respectively).  In other words, even though ‘10’ can
     be unsigned, it will be read as a signed 8-bit integer.  This is
     done to respect the C implicit type conversion in binary operators,
     where signed integers will be interpreted as unsigned, when the
     other operand is an unsigned integer of the same width.

     For example, see the short program below.  It will print ‘-50 is
     larger than 100000’ (which is wrong!).  This happens because when a
     negative number is parsed as an unsigned, the value is effectively
     subtracted from the maximum and $4294967295-50$ is indeed larger
     than 100000 (recall that $4294967295$ is the largest unsigned
     32-bit integer, see *note Numeric data types::).

          #include <stdio.h>
          #include <stdlib.h>
          #include <stdint.h>

          int
          main(void)
          {
            int32_t  a=-50;
            uint32_t b=100000;
            printf("%d is %s than %d\n", a,
                   a>b ? "larger" : "less or equal", b);
            return 0;
          }

     However, if we read 100000 as a signed 32-bit integer, there will
     not be any problem and the printed sentence will be logically
     correct (for someone who does not know anything about numeric data
     types: users of your programs).  For the advantages of integers,
     see *note Integer benefits and pitfalls::.


File: gnuastro.info,  Node: Pointers,  Next: Library blank values,  Prev: Library data types,  Up: Gnuastro library

12.3.4 Pointers (‘pointer.h’)
-----------------------------

Pointers play an important role in the C programming language.  As the
name suggests, they _point_ to a byte in memory (like an address in a
city).  The C programming language gives you complete freedom in how to
use the byte (and the bytes that follow it).  Pointers are thus a very
powerful feature of C. However, as the saying goes: "With great power
comes great responsibility", so they must be approached with care.  The
functions in this header are not very complex, they are just wrappers
over some basic pointer functionality regarding pointer arithmetic and
allocation (in memory or HDD/SSD).

 -- Function:
          void *
          gal_pointer_increment (void *pointer, size_t increment,
          uint8_t type)
     Return a pointer to an element that is ‘increment’ elements ahead
     of ‘pointer’, assuming each element has type of ‘type’.  For the
     type codes, see *note Library data types::.

     When working with the ‘array’ elements of ‘gal_data_t’, we are
     actually dealing with ‘void *’ pointers.  However, pointer
     arithmetic does not apply to ‘void *’, because the system does not
     know how many bytes there are in each element to increment the
     pointer respectively.  This function will use the given ‘type’ to
     calculate where the incremented element is located in memory.

 -- Function:
          size_t
          gal_pointer_num_between (void *earlier, void *later, uint8_t
          type)
     Return the number of elements (in the given ‘type’) between
     ‘earlier’ and ‘later’.  For the type codes, see *note Library data
     types::).

 -- Function:
          void *
          gal_pointer_allocate (uint8_t type, size_t size, int clear,
          const char *funcname, const char *varname)
     Allocate an array of type ‘type’ with ‘size’ elements in RAM (for
     the type codes, see *note Library data types::).  If ‘clear!=0’,
     then the allocated space is set to zero (cleared).

     This is effectively just a wrapper around C's ‘malloc’ or ‘calloc’
     functions but takes Gnuastro's integer type codes and will also
     abort with a clear error if there the allocation was not
     successful.  The number of allocated bytes is the value given to
     ‘size’ that is multiplied by the returned value of
     ‘gal_type_sizeof’ for the given type.  So if you want to allocate
     space for an array of strings you should pass the type
     ‘GAL_TYPE_STRING’.  Otherwise, if you just want space for one
     string (for example, 6 bytes for ‘hello’, including the
     string-termination character), you should set the type
     ‘GAL_TYPE_UINT8’.

     When space cannot be allocated, this function will abort the
     program with a message containing the reason for the failure.
     ‘funcname’ (name of the function calling this function) and
     ‘varname’ (name of variable that needs this space) will be used in
     this error message if they are not ‘NULL’.  In most modern
     compilers, you can use the generic ‘__func__’ variable for
     ‘funcname’.  In this way, you do not have to manually copy and
     paste the function name or worry about it changing later
     (‘__func__’ was standardized in C99).  To use this function
     effectively and avoid memory leaks, make sure to free the allocated
     array after you are done with it.  Also, be mindful of any
     functions that make use of this function as they should also free
     any allocated arrays to maintain memory management and prevent
     issues with the system.

 -- Function:
          void *
          gal_pointer_allocate_ram_or_mmap (uint8_t type, size_t size,
          int clear, size_t minmapsize, char **mmapname, int quietmmap,
          const char *funcname, const char *varname)
     Allocate the given space either in RAM or in a memory-mapped file.
     This function is just a high-level wrapper to
     ‘gal_pointer_allocate’ (to allocate in RAM) or
     ‘gal_pointer_mmap_allocate’ (to use a memory-mapped file).  For
     more on memory management in Gnuastro, please see *note Memory
     management::.  The various arguments are more fully explained in
     the two functions above.

 -- Function:
          void *
          gal_pointer_mmap_allocate (size_t size, uint8_t type, int
          clear, char **mmapname, int allocfailed)
     Allocate the necessary space to keep ‘size’ elements of type ‘type’
     in HDD/SSD (a file, not in RAM). For the type codes, see *note
     Library data types::.  If ‘clear!=0’, then the allocated space will
     also be cleared.  The allocation is done using C's ‘mmap’ function.
     The name of the file containing the allocated space is an allocated
     string that will be put in ‘*mmapname’.

     Note that the kernel does not allow an infinite number of memory
     mappings to files.  So it is not recommended to use this function
     with every allocation.  The best-case scenario to use this function
     is for arrays that are very large and can fill up the RAM. Keep the
     smaller arrays in RAM, which is faster and can have a
     (theoretically) unlimited number of allocations.

     When you are done with the dataset and do not need it anymore, do
     not use ‘free’ (the dataset is not in RAM). Just delete the file
     (and the allocated space for the filename) with the commands below,
     or simply use ‘gal_pointer_mmap_free’.

          remove(mmapname);
          free(mmapname);

     If ‘allocfailed!=0’ and the memory mapping attempt fails, the
     warning message will say something like this (assuming you have
     tried something like ‘malloc’ before calling this function): even
     though there was enough space in RAM, the previous attempts at
     allocation in RAM failed, so we tried memory mapping, but that also
     failed.

 -- Function:
          void
          gal_pointer_mmap_free (char **mmapname, int quietmmap)
     "Free" (actually delete) the memory-mapped file that is named
     ‘*mmapname’, then free the string.  If ‘quietmmap’ is non-zero,
     then a warning will be printed for the user to know that the given
     file has been deleted.


File: gnuastro.info,  Node: Library blank values,  Next: Library data container,  Prev: Pointers,  Up: Gnuastro library

12.3.5 Library blank values (‘blank.h’)
---------------------------------------

When the position of an element in a dataset is important (for example,
a pixel in an image), a place-holder is necessary for the element if we
do not have a value to fill it with (for example, the CCD cannot read
those pixels).  We cannot simply shift all the other pixels to fill in
the one we have no value for.  In other cases, it often occurs that the
field of sky that you are studying is not a clean rectangle to nicely
fit into the boundaries of an image.  You need a way to separate the
pixels outside your scientific field from those inside it.  Blank values
act as these place holders in a dataset.  They have no usable value but
they have a position.

   Every type needs a corresponding blank value (see *note Numeric data
types:: and *note Library data types::).  Floating point types have a
unique value identified by IEEE known as Not-a-Number (or NaN) which is
a unique value that is recognized by the compiler.  However, integer and
string types do not have any standard value.  For integers, in Gnuastro
we take an extremum of the given type: for signed types (that allow
negatives), the minimum possible value is used as blank and for unsigned
types (that only accept positives), the maximum possible value is used.
To be generic and easy to read/write we define a macro for these blank
values and strongly encourage you only use these, and never make any
assumption on the value of a type's blank value.

   The IEEE NaN blank value type is defined to fail on any comparison,
so if you are dealing with floating point types, you cannot use equality
(a NaN will _not_ be equal to a NaN). If you know your dataset is
floating point, you can use the ‘isnan’ function in C's ‘math.h’ header.
For a description of numeric data types see *note Numeric data types::.
For the constants identifying integers, please see *note Library data
types::.

 -- Global integer: GAL_BLANK_UINT8
     Blank value for an unsigned, 8-bit integer.

 -- Global integer: GAL_BLANK_INT8
     Blank value for a signed, 8-bit integer.

 -- Global integer: GAL_BLANK_UINT16
     Blank value for an unsigned, 16-bit integer.

 -- Global integer: GAL_BLANK_INT16
     Blank value for a signed, 16-bit integer.

 -- Global integer: GAL_BLANK_UINT32
     Blank value for an unsigned, 32-bit integer.

 -- Global integer: GAL_BLANK_INT32
     Blank value for a signed, 32-bit integer.

 -- Global integer: GAL_BLANK_UINT64
     Blank value for an unsigned, 64-bit integer.

 -- Global integer: GAL_BLANK_INT64
     Blank value for a signed, 64-bit integer.

 -- Global integer: GAL_BLANK_INT
     Blank value for ‘int’ type (‘int16_t’ or ‘int32_t’ depending on the
     system.

 -- Global integer: GAL_BLANK_UINT
     Blank value for ‘int’ type (‘int16_t’ or ‘int32_t’ depending on the
     system.

 -- Global integer: GAL_BLANK_LONG
     Blank value for ‘long’ type (‘int32_t’ or ‘int64_t’ in 32-bit or
     64-bit systems).

 -- Global integer: GAL_BLANK_ULONG
     Blank value for ‘unsigned long’ type (‘uint32_t’ or ‘uint64_t’ in
     32-bit or 64-bit systems).

 -- Global integer: GAL_BLANK_SIZE_T
     Blank value for ‘size_t’ type (‘uint32_t’ or ‘uint64_t’ in 32-bit
     or 64-bit systems).

 -- Global integer: GAL_BLANK_FLOAT32
     Blank value for a single precision, 32-bit floating point type
     (IEEE NaN value).

 -- Global integer: GAL_BLANK_FLOAT64
     Blank value for a double precision, 64-bit floating point type
     (IEEE NaN value).

 -- Global integer: GAL_BLANK_STRING
     Blank value for string types (this is itself a string, it is not
     the ‘NULL’ pointer).

The functions below can be used to work with blank pixels.

 -- Function:
          void
          gal_blank_write (void *pointer, uint8_t type)
     Write the blank value for the given ‘type’ into the space that
     ‘pointer’ points to.  This can be used when the space is already
     allocated (for example, one element in an array or a statically
     allocated variable).

 -- Function:
          void *
          gal_blank_alloc_write (uint8_t type)
     Allocate the space required to keep the blank for the given data
     type ‘type’, write the blank value into it and return the pointer
     to it.

 -- Function:
          void
          gal_blank_initialize (gal_data_t *input)
     Initialize all the elements in the ‘input’ dataset to the blank
     value that corresponds to its type.  If ‘input’ is not a string,
     and is a tile over a larger dataset, only the region that the tile
     covers will be set to blank.  For strings, the full dataset will be
     initialized.

 -- Function:
          void
          gal_blank_initialize_array (void *array, size_t size, uint8_t
          type)
     Initialize all the elements in the ‘array’ to the blank value that
     corresponds to its type (identified with ‘type’), assuming the
     array has ‘size’ elements.

 -- Function:
          char *
          gal_blank_as_string (uint8_t type, int width)
     Write the blank value for the given data type ‘type’ into a string
     and return it.  The space for the string is dynamically allocated
     so it must be freed after you are done with it.  If ‘width!=0’,
     then the final string will be padded with white space characters to
     have the requested width if it is smaller.

 -- Function:
          int
          gal_blank_is (void *pointer, uint8_t type)
     Return 1 if the contents of ‘pointer’ (assuming a type of ‘type’)
     is blank.  Otherwise, return 0.  Note that this function only works
     on one element of the given type.  So if ‘pointer’ is an array,
     only its first element will be checked.  Therefore for strings, the
     type of ‘pointer’ is assumed to be ‘char *’.  To check if an
     array/dataset has blank elements or to find which elements in an
     array are blank, you can use ‘gal_blank_present’ or
     ‘gal_blank_flag’ respectively (described below).

 -- Function:
          int
          gal_blank_present (gal_data_t *input, int updateflag)
     Return 1 if the dataset has a blank value and zero if it does not.
     Before checking the dataset, this function will look at ‘input’'s
     flags.  If the ‘GAL_DATA_FLAG_BLANK_CH’ bit of ‘input->flag’ is on,
     this function will not do any check and will just use the
     information in the flags.  This can greatly speed up processing
     when a dataset needs to be checked multiple times.

     When the dataset's flags were not used and ‘updateflags’ is
     non-zero, this function will set the flags appropriately to avoid
     having to re-check the dataset in future calls.  When
     ‘updateflags==0’, this function has no side-effects on the dataset:
     it will not toggle the flags.

     If you want to re-check a dataset with the blank-value-check flag
     already set (for example, if you have made changes to it), then
     explicitly set the ‘GAL_DATA_FLAG_BLANK_CH’ bit to zero before
     calling this function.  When there are no other flags, you can just
     set the flags to zero (‘input->flag=0’), otherwise you can use this
     expression:

          input->flag &= ~GAL_DATA_FLAG_BLANK_CH;

 -- Function:
          size_t
          gal_blank_number (gal_data_t *input, int updateflag)
     Return the number of blank elements in ‘input’.  If
     ‘updateflag!=0’, then the dataset blank keyword flags will be
     updated.  See the description of ‘gal_blank_present’ (above) for
     more on these flags.  If ‘input==NULL’, then this function will
     return ‘GAL_BLANK_SIZE_T’.

 -- Function:
          gal_data_t *
          gal_blank_flag (gal_data_t *input)
     Return a "flag" dataset with the same size as the input, but with
     an ‘uint8_t’ type that has a value of 1 for data elements that are
     blank and 0 for those that are not.

 -- Function:
          gal_data_t *
          gal_blank_flag_not (gal_data_t *input)
     Return a "flag" dataset with the same size as the input, but with
     an ‘uint8_t’ type that has a value of 1 for data elements that are
     _not_ blank and 0 for those that are blank.

 -- Function:
          size_t *
          gal_blank_not_minmax_coords (gal_data_t *input)
     Find the minimum and maximum coordinates of the non-blank regions
     within the input dataset.  The coordinates are in C order: starting
     from 0, and with the slowest dimension being first.  The output is
     an allocated array (that should be freed later) with $2\times N$
     elements ($N$ is the number of dimensions).  The first two elements
     contain the minimum and maximum of regions containing non-blank
     elements along the 0-th dimension (the slowest), the second two
     elements contain the next dimension's extrema; and so on.

     When all the elements/pixels of the input are blank, the all the
     elements of the output will have a value of 0.

 -- Function:
          gal_data_t *
          gal_blank_trim (gal_data_t *input, int inplace)
     Trim all the outer layers of blank values from the input dataset.
     For example in the 2D image, "layers" would correspond to columns
     or rows that are fully blank and touching the edge of the image.
     For a more complete description, see the description of the ‘trim’
     operator in *note Dimensionality changing operators::.

 -- Function:
          void
          gal_blank_flag_apply (gal_data_t *input, gal_data_t *flag)
     Set all non-zero and non-blank elements of ‘flag’ to blank in
     ‘input’.  ‘flag’ has to have an unsigned 8-bit type and be the same
     size as ‘input’.

 -- Function:
          void
          gal_blank_flag_remove (gal_data_t *input, gal_data_t *flag)
     Remove all elements within ‘input’ that are flagged, convert it to
     a 1D dataset and adjust the size properly (the number of
     non-flagged elements).  In practice this function does not‘realloc’
     the input array (see ‘gal_blank_remove_realloc’ for
     shrinking/re-allocating also), it just shifts the blank elements to
     the end and adjusts the size elements of the ‘gal_data_t’, see
     *note Generic data container::.

     Note that elements that are blank, but not flagged will not be
     removed.  This function will only remove flagged elements.

     If all the elements were flagged, then ‘input->size’ will be zero.
     This is thus a good parameter to check after calling this function
     to see if there actually were any non-flagged elements in the input
     or not and take the appropriate measure.  This check is highly
     recommended because it will avoid strange bugs in later steps.

 -- Function:
          void
          gal_blank_remove (gal_data_t *input, int free_if_all_blank)
     Remove blank elements from a dataset, convert it to a 1D dataset,
     adjust the size properly (the number of non-blank elements), and
     toggle the blank-value-related bit-flags.  In practice this
     function does not‘realloc’ the input array (see
     ‘gal_blank_remove_realloc’ for shrinking/re-allocating also), it
     just shifts the blank elements to the end and adjusts the size
     elements of the ‘gal_data_t’, see *note Generic data container::.

     If all the elements were blank, then ‘input->size’ will be zero.
     In case ‘free_if_all_blank==1’ is also zero, then ‘input->array’
     will also be freed and set to ‘NULL’.  To see if there actually
     were any non-blank elements in the input or not and take the
     appropriate measure, the most generic parameter to check after
     calling this function is therefore ‘input->size’.  This check is
     highly recommended in followup functions because it will avoid
     strange bugs in later steps of your code (reading a possibly freed
     space!).

 -- Function:
          void
          gal_blank_remove_realloc (gal_data_t *input)
     Similar to ‘gal_blank_remove’, but also shrinks/re-allocates the
     dataset's allocated memory.

 -- Function:
          gal_data_t *
          gal_blank_remove_rows (gal_data_t *columns, gal_list_sizet_t
          *column_indexs, int onlydim0)
     Remove (in place) any row that has at least one blank value in any
     of the input columns and return a "flag" dataset (that should be
     freed later).  The input ‘columns’ is a list of ‘gal_data_t’s (see
     *note List of gal_data_t::).  When ‘onlydim0!=0’ the vector columns
     (with 2 dimensions) will not be checked for the presence of blank
     values.

     After this function, all the elements in ‘columns’ will still have
     the same size as each other, but if any of the searched columns has
     blank elements, all their sizes will decrease together.

     The returned flag dataset has the same size as the original input
     dataset, with a type of ‘uint8_t’.  Every row that has been removed
     from the original dataset has a value of 1, and the rest have a
     value of 0.

     When ‘column_indexs!=NULL’, only the columns whose index (counting
     from zero) is in ‘column_indexs’ will be used to check for blank
     values (see *note List of size_t::.  Therefore, if you want to
     check all columns, just set this to ‘NULL’.  In any case (no matter
     which columns are checked for blanks), the selected rows from all
     columns will be removed.


File: gnuastro.info,  Node: Library data container,  Next: Dimensions,  Prev: Library blank values,  Up: Gnuastro library

12.3.6 Data container (‘data.h’)
--------------------------------

Astronomical datasets have various dimensions, for example, 1D spectra
or table columns, 2D images, or 3D Integral field data cubes.  Datasets
can also have various numeric data types, depending on the
operation/purpose, for example, processed images are commonly stored in
floating point format, but their mask images are integers (allowing
bitwise flags to identify certain classes of pixels to keep or mask, see
*note Numeric data types::).  Certain other information about a dataset
are also commonly necessary, for example, the units of the dataset, the
name of the dataset and some comments.  To deal with any generic
dataset, Gnuastro defines the ‘gal_data_t’ as input or output.

* Menu:

* Generic data container::      Definition of Gnuastro's generic container.
* Dataset allocation::          Allocate, initialize and free a dataset.
* Arrays of datasets::          Functions to help with array of datasets.
* Copying datasets::            Functions to copy a dataset to a new one.


File: gnuastro.info,  Node: Generic data container,  Next: Dataset allocation,  Prev: Library data container,  Up: Library data container

12.3.6.1 Generic data container (‘gal_data_t’)
..............................................

To be able to deal with any dataset (various dimensions, numeric data
types, units and higher-level structures), Gnuastro defines the
‘gal_data_t’ type which is the input/output container of choice for many
of Gnuastro library's functions.  It is defined in ‘gnuastro/data.h’.
If you will be using ('‘# include’'ing) those libraries, you do not need
to include this header explicitly, it is already included by any library
header that uses ‘gal_data_t’.

 -- Type (C struct): gal_data_t
     The main container for datasets in Gnuastro.  It can host data of
     any dimensions, with any numeric data type.  It is actually a
     structure, but ‘typedef’'d as a new type to avoid having to write
     the ‘struct’ before any declaration.  The actual structure is shown
     below which is followed by a description of each element.

          typedef struct gal_data_t
          {
            void     *restrict array;  /* Basic array information.   */
            uint8_t             type;
            size_t              ndim;
            size_t            *dsize;
            size_t              size;
            int            quietmmap;
            char           *mmapname;
            size_t        minmapsize;

            int                 nwcs;  /* WCS information.           */
            struct wcsprm       *wcs;

            uint8_t             flag;  /* Content description.       */
            int               status;
            char               *name;
            char               *unit;
            char            *comment;

            int             disp_fmt;  /* For text printing.         */
            int           disp_width;
            int       disp_precision;

            struct gal_data_t  *next;  /* For higher-level datasets. */
            struct gal_data_t *block;
          } gal_data_t;

The list below contains a description for each ‘gal_data_t’ element.

‘void *restrict array’
     This is the pointer to the main array of the dataset containing the
     raw data (values).  All the other elements in this data-structure
     are actually meta-data enabling us to use/understand the series of
     values in this array.  It must allow data of any type (see *note
     Numeric data types::), so it is defined as a ‘void *’ pointer.  A
     ‘void *’ array is not directly usable in C, so you have to cast it
     to proper type before using it, please see *note Library demo -
     reading a FITS image:: for a demonstration.

     The ‘restrict’ keyword was formally introduced in C99 and is used
     to tell the compiler that at any moment only this pointer will
     modify what it points to (a pixel in an image for example)(1).
     This extra piece of information can greatly help in compiler
     optimizations and thus the running time of the program.  But older
     compilers might not have this capability, so at ‘./configure’ time,
     Gnuastro checks this feature and if the user's compiler does not
     support ‘restrict’, it will be removed from this definition.

‘uint8_t type’
     A fixed code (integer) used to identify the type of data in ‘array’
     (see *note Numeric data types::).  For the list of acceptable
     values to this variable, please see *note Library data types::.

‘size_t ndim’
     The dataset's number of dimensions.

‘size_t *dsize’
     The size of the dataset along each dimension.  This is an array
     (with ‘ndim’ elements), of positive integers in row-major order(2)
     (based on C). When a data file is read into memory with Gnuastro's
     libraries, this array is dynamically allocated based on the number
     of dimensions that the dataset has.

     It is important to remember that C's row-major ordering is the
     opposite of the FITS standard which is in column-major order: in
     the FITS standard the fastest dimension's size is specified by
     ‘NAXIS1’, and slower dimensions follow.  The FITS standard was
     defined mainly based on the FORTRAN language which is the opposite
     of C's approach to multi-dimensional arrays (and also starts
     counting from 1 not 0).  Hence if a FITS image has ‘NAXIS1==20’ and
     ‘NAXIS2==50’, the ‘dsize’ array must be filled with ‘dsize[0]==50’
     and ‘dsize[1]==20’.

     The fastest dimension is the one that is contiguous in memory: to
     increment by one along that dimension, just go to the next element
     in the array.  As we go to slower dimensions, the number of memory
     cells we have to skip for an increment along that dimension becomes
     larger.

‘size_t size’
     The total number of elements in the dataset.  This is actually a
     multiplication of all the values in the ‘dsize’ array, so it is not
     an independent parameter.  However, low-level operations with the
     dataset (irrespective of its dimensions) commonly need this number,
     so this element is designed to avoid calculating it every time.

‘int quietmmap’
     When this value is zero, and the dataset must not be allocated in
     RAM (see ‘mmapname’ and ‘minmapsize’ below), a warning will be
     printed to inform the user when the file is created and when it is
     deleted.  The warning includes the filename, the size in bytes, and
     the fact that they can toggle this behavior through ‘--minmapsize’
     option in Gnuastro's programs.

‘char *mmapname’
     Name of file hosting the ‘mmap’'d contents of ‘array’.  If the
     value of this variable is ‘NULL’, then the contents of ‘array’ are
     actually stored in RAM, not in a file on the HDD/SSD. See the
     description of ‘minmapsize’ below for more.

     If a file is used, it will be kept in the ‘gnuastro_mmap’ directory
     of the running directory.  Its name is randomly selected to allow
     multiple arrays at the same time, see description of ‘--minmapsize’
     in *note Processing options::.  When ‘gal_data_free’ is called the
     randomly named file will be deleted.

‘size_t minmapsize’
     The minimum size of an array (in bytes) to store the contents of
     ‘array’ as a file (on the non-volatile HDD/SSD), not in RAM. This
     can be very useful for large datasets which can be very memory
     intensive and the user's RAM might not be sufficient to
     keep/process it.  A random filename is assigned to the array which
     is available in the ‘mmapname’ element of ‘gal_data_t’ (above), see
     there for more.  ‘minmapsize’ is stored in each ‘gal_data_t’, so it
     can be passed on to subsequent/derived datasets.

     See the description of the ‘--minmapsize’ option in *note
     Processing options:: for more on using this value.

‘nwcs’
     The number of WCS coordinate representations (for WCSLIB).

‘struct wcsprm *wcs’
     The main WCSLIB structure keeping all the relevant information
     necessary for WCSLIB to do its processing and convert data-set
     positions into real-world positions.  When it is given a ‘NULL’
     value, all possible WCS calculations/measurements will be ignored.

‘uint8_t flag’
     Bitwise flags to describe general properties of the dataset.  The
     number of bytes available in this flag is stored in the
     ‘GAL_DATA_FLAG_SIZE’ macro.  Note that you should use bitwise
     operators(3) to check these flags.  The currently recognized bits
     are stored in these macros:

     ‘GAL_DATA_FLAG_BLANK_CH’
          Marking that the dataset has been checked for blank values or
          not.  When a dataset does not have any blank values, the
          ‘GAL_DATA_FLAG_HASBLANK’ bit will be zero.  But upon
          initialization, all bits also get a value of zero.  Therefore,
          a checker needs this flag to see if the value in
          ‘GAL_DATA_FLAG_HASBLANK’ is reliable (dataset has actually
          been parsed for a blank value) or not.

          Also, if it is necessary to re-check the presence of flags,
          you just have to set this flag to zero and call
          ‘gal_blank_present’ for example, to parse the dataset and
          check for blank values.  Note that for improved efficiency,
          when this flag is set, ‘gal_blank_present’ will not actually
          parse the dataset, it will just use ‘GAL_DATA_FLAG_HASBLANK’.

     ‘GAL_DATA_FLAG_HASBLANK’
          This bit has a value of ‘1’ when the given dataset has blank
          values.  If this bit is ‘0’ and ‘GAL_DATA_FLAG_BLANK_CH’ is
          ‘1’, then the dataset has been checked and it did not have any
          blank values, so there is no more need for further checks.

     ‘GAL_DATA_FLAG_SORT_CH’
          Marking that the dataset is already checked for being sorted
          or not and thus that the possible ‘0’ values in
          ‘GAL_DATA_FLAG_SORTED_I’ and ‘GAL_DATA_FLAG_SORTED_D’ are
          meaningful.  The logic behind this is similar to that in
          ‘GAL_DATA_FLAG_BLANK_CH’.

     ‘GAL_DATA_FLAG_SORTED_I’
          This bit has a value of ‘1’ when the given dataset is sorted
          in an increasing manner.  If this bit is ‘0’ and
          ‘GAL_DATA_FLAG_SORT_CH’ is ‘1’, then the dataset has been
          checked and was not sorted (increasing), so there is no more
          need for further checks.

     ‘GAL_DATA_FLAG_SORTED_D’
          This bit has a value of ‘1’ when the given dataset is sorted
          in a decreasing manner.  If this bit is ‘0’ and
          ‘GAL_DATA_FLAG_SORT_CH’ is ‘1’, then the dataset has been
          checked and was not sorted (decreasing), so there is no more
          need for further checks.

     The macro ‘GAL_DATA_FLAG_MAXFLAG’ contains the largest internally
     used bit-position.  Higher-level flags can be defined with the
     bitwise shift operators using this macro to define internal flags
     for libraries/programs that depend on Gnuastro without causing any
     possible conflict with the internal flags discussed above or having
     to check the values manually on every release.

‘int status’
     A context-specific status values for this data-structure.  This
     integer will not be set by Gnuastro's libraries.  You can use it
     keep some additional information about the dataset (with integer
     constants) depending on your applications.

‘char *name’
     The name of the dataset.  If the dataset is a multi-dimensional
     array and read/written as a FITS image, this will be the value in
     the ‘EXTNAME’ FITS keyword.  If the dataset is a one-dimensional
     table column, this will be the column name.  If it is set to ‘NULL’
     (by default), it will be ignored.

‘char *unit’
     The units of the dataset (for example, ‘BUNIT’ in the standard FITS
     keywords) that will be read from or written to files/tables along
     with the dataset.  If it is set to ‘NULL’ (by default), it will be
     ignored.

‘char *comment’
     Any further explanation about the dataset which will be written to
     any output file if present.

‘disp_fmt’
     Format to use for printing each element of the dataset to a plain
     text file, the acceptable values to this element are defined in
     *note Table input output::.  Based on C's ‘printf’ standards.

‘disp_width’
     Width of printing each element of the dataset to a plain text file,
     the acceptable values to this element are defined in *note Table
     input output::.  Based on C's ‘printf’ standards.

‘disp_precision’
     Precision of printing each element of the dataset to a plain text
     file, the acceptable values to this element are defined in *note
     Table input output::.  Based on C's ‘printf’ standards.

‘gal_data_t *next’
     Through this pointer, you can link a ‘gal_data_t’ with other
     datasets related datasets, for example, the different columns in a
     dataset each have one ‘gal_data_t’ associate with them and they are
     linked to each other using this element.  There are several
     functions described below to facilitate using ‘gal_data_t’ as a
     linked list.  See *note Linked lists:: for more on these wonderful
     high-level constructs.

‘gal_data_t *block’
     Pointer to the start of the complete allocated block of memory.
     When this pointer is not ‘NULL’, the dataset is not treated as a
     contiguous patch of memory.  Rather, it is seen as covering only a
     portion of the larger patch of memory that ‘block’ points to.  See
     *note Tessellation library:: for a more thorough explanation and
     functions to help work with tiles that are created from this
     pointer.

   ---------- Footnotes ----------

   (1) Also see <https://en.wikipedia.org/wiki/Restrict>.

   (2) Also see
<https://en.wikipedia.org/wiki/Row-_and_column-major_order>.

   (3) See <https://en.wikipedia.org/wiki/Bitwise_operations_in_C>.


File: gnuastro.info,  Node: Dataset allocation,  Next: Arrays of datasets,  Prev: Generic data container,  Up: Library data container

12.3.6.2 Dataset allocation
...........................

Gnuastro's main data container was defined in *note Generic data
container::.  The functions listed in this section describe the most
basic operations on ‘gal_data_t’: those related to allocation and
freeing.  These functions are declared in ‘gnuastro/data.h’ which is
also visible from the function names (see *note Gnuastro library::).

 -- Function:
          gal_data_t *
          gal_data_alloc (void *array, uint8_t type, size_t ndim, size_t
          *dsize, struct wcsprm *wcs, int clear, size_t minmapsize, int
          quietmmap, char *name, char *unit, char *comment)

     Dynamically allocate a ‘gal_data_t’ and initialize it will all the
     given values.  See the description of ‘gal_data_initialize’ and
     *note Generic data container:: for more information.  This function
     will often be the most frequently used because it allocates the
     ‘gal_data_t’ hosting all the values _and_ initializes it.  Once you
     are done with the dataset, be sure to clean up all the allocated
     spaces with ‘gal_data_free’.

 -- Function:
          void
          gal_data_initialize (gal_data_t *data, void *array, uint8_t
          type, size_t ndim, size_t *dsize, struct wcsprm *wcs, int
          clear, size_t minmapsize, int quietmmap, char *name, char
          *unit, char *comment)

     Initialize the given data structure (‘data’) with all the given
     values.  Note that the raw input ‘gal_data_t’ must already have
     been allocated before calling this function.  For a description of
     each variable see *note Generic data container::.  It will set the
     values and do the necessary allocations.  If they are not ‘NULL’,
     all input arrays (‘dsize’, ‘wcs’, ‘name’, ‘unit’, ‘comment’) are
     separately copied (allocated) by this function for usage in ‘data’,
     so you can safely use one value to initialize many datasets or use
     statically allocated variables in this function call.  Once you are
     done with the dataset, you can free all the allocated spaces with
     ‘gal_data_free_contents’.

     If ‘array’ is not ‘NULL’, it will be directly copied into
     ‘data->array’ (based on the total number of elements calculated
     from ‘dsize’) and no new space will be allocated for the array of
     this dataset, this has many low-level advantages and can be used to
     work on regions of a dataset instead of the whole allocated array
     (see the description under ‘block’ in *note Generic data
     container:: for one example).  If the given pointer is not the
     start of an allocated block of memory or it is used in multiple
     datasets, be sure to set it to ‘NULL’ (with ‘data->array=NULL’)
     before cleaning up with ‘gal_data_free_contents’.

     ‘ndim’ may be zero.  In this case no allocation will occur,
     ‘data->array’ and ‘data->dsize’ will be set to ‘NULL’ and
     ‘data->size’ will be zero.  However (when necessary) ‘dsize’ must
     not have any zero values (a dimension of length zero is not
     defined).

 -- Function:
          gal_data_t *
          gal_data_alloc_empty (size_t ndim, size_t minmapsize, int
          quietmmap)
     Allocate an empty dataset with a certain number of dimensions, but
     no 'array' component.  The ‘size’ element will be set to zero and
     the ‘dsize’ array will be properly allocated (based on the number
     of dimensions), but all elements will be zero.  This is useful in
     scenarios where you just need a ‘gal_data_t’ for metadata.

 -- Function:
          void
          gal_data_free_contents (gal_data_t *data)
     Free all the non-‘NULL’ pointers in ‘gal_data_t’ except for ‘next’
     and ‘block’.  All freed arrays are set to ‘NULL’.  If ‘data’ is
     actually a tile (‘data->block!=NULL’, see *note Tessellation
     library::), then ‘data->array’ is not freed.  For a complete
     description of ‘gal_data_t’ and its contents, see *note Generic
     data container::.

 -- Function:
          void
          gal_data_free (gal_data_t *data)
     Free all the non-‘NULL’ pointers in ‘gal_data_t’, then free the
     actual data structure.


File: gnuastro.info,  Node: Arrays of datasets,  Next: Copying datasets,  Prev: Dataset allocation,  Up: Library data container

12.3.6.3 Arrays of datasets
...........................

Gnuastro's generic data container (‘gal_data_t’) is a very versatile
structure that can be used in many higher-level contexts.  One such
higher-level construct is an array of ‘gal_data_t’ structures to
simplify the allocation (and later cleaning) of several ‘gal_data_t’s
that are related.

   For example, each column in a table is usually represented by one
‘gal_data_t’ (so it has its own name, data type, units, etc.).  A table
(with many columns) can be seen as an array of ‘gal_data_t’s (when the
number of columns is known a-priori).  The functions below are defined
to create a cleared array of data structures and to free them when none
are necessary any more.  These functions are declared in
‘gnuastro/data.h’ which is also visible from the function names (see
*note Gnuastro library::).

 -- Function:
          gal_data_t *
          gal_data_array_calloc (size_t size)
     Allocate an array of ‘gal_data_t’ with ‘size’ elements.  This
     function will also initialize all the values (‘NULL’ for pointers
     and 0 for other types).  You can use ‘gal_data_initialize’ to fill
     each element of the array afterwards.  The following code snippet
     is one example of doing this.

          size_t i;
          gal_data_t *dataarr;
          dataarr=gal_data_array_calloc(10);
          for(i=0;i<10;++i) gal_data_initialize(&dataarr[i], ...);
          ...
          gal_data_array_free(dataarr, 10, 1);

 -- Function:
          void
          gal_data_array_free (gal_data_t *dataarr, size_t num, int
          free_array)
     Free all the ‘num’ elements within ‘dataarr’ and the actual
     allocated array.  If ‘free_array’ is not zero, then the ‘array’
     element of all the datasets will also be freed, see *note Generic
     data container::.

 -- Function:
          gal_data_t **
          gal_data_array_ptr_calloc (size_t size)
     Allocate an array of pointers to Gnuastro's generic data structure
     and initialize all pointers to ‘NULL’.  This is useful when you
     want to allocate individual datasets later (for example, with
     ‘gal_data_alloc’).

 -- Function:
          void
          gal_data_array_ptr_free (gal_data_t **dataptr, size_t size,
          int free_array);
     Free all the individual datasets within the elements of ‘dataptr’,
     then free ‘dataptr’ itself (the array of pointers that was probably
     allocated with ‘gal_data_array_ptr_calloc’.


File: gnuastro.info,  Node: Copying datasets,  Prev: Arrays of datasets,  Up: Library data container

12.3.6.4 Copying datasets
.........................

The functions in this section describes Gnuastro's facilities to copy a
given dataset into another.  The new dataset can have a different type
(including a string), it can be already allocated (in which case only
the values will be written into it).  In all these cases, if the input
dataset is a tile or a list, only the data within the given tile, or the
given node in a list, are copied.  If the input is a list, the ‘next’
pointer will also be copied to the output, see *note List of
gal_data_t::.

   In many of the functions here, it is possible to copy the dataset to
a new numeric data type (see *note Numeric data types::.  In such cases,
Gnuastro's library is going to use the native conversion by C. So if you
are converting to a smaller type, it is up to you to make sure that the
values fit into the output type.

 -- Function:
          gal_data_t *
          gal_data_copy (gal_data_t *in)
     Return a new dataset that is a copy of ‘in’, all of ‘in’'s
     meta-data will also copied into the output, except for ‘block’.  If
     the dataset is a tile/list, only the given tile/node will be
     copied, the ‘next’ pointer will also be copied however.

 -- Function:
          gal_data_t *
          gal_data_copy_to_new_type (gal_data_t *in, uint8_t newtype)
     Return a copy of the dataset ‘in’, converted to ‘newtype’, see
     *note Library data types:: for Gnuastro library's type identifiers.
     The returned dataset will have all meta-data except their type and
     ‘block’ equal to the input's metadata.  If the dataset is a
     tile/list, only the given tile/node will be copied, the ‘next’
     pointer will also be copied however.

 -- Function:
          gal_data_t *
          gal_data_copy_to_new_type_free (gal_data_t *in, uint8_t
          newtype)
     Return a copy of the dataset ‘in’ that is converted to ‘newtype’
     and free the input dataset.  See *note Library data types:: for
     Gnuastro library's type identifiers.  The returned dataset will
     have all meta-data, except their type, equal to the input's
     metadata (including ‘next’).  Note that if the input is a tile
     within a larger block, it will not be freed.  This function is
     similar to ‘gal_data_copy_to_new_type’, except that it will free
     the input dataset.

 -- Function:
          void
          gal_data_copy_to_allocated (gal_data_t *in, gal_data_t *out)
     Copy the contents of the array in ‘in’ into the already allocated
     array in ‘out’.  The types of the input and output may be
     different, type conversion will be done internally.  When ‘in->size
     != out->size’ this function will behave as follows:

     ‘out->size < in->size’
          This function will not re-allocate the necessary space, it
          will abort with an error, so please check before calling this
          function.

     ‘out->size > in->size’
          This function will write the values in ‘out->size’ and
          ‘out->dsize’ from the same values of ‘in’.  So if you want to
          use a pre-allocated space/dataset multiple times with varying
          input sizes, be sure to reset ‘out->size’ before every call to
          this function.

 -- Function:
          gal_data_t *
          gal_data_copy_string_to_number (char *string)
     Read ‘string’ into the smallest type that can store the value (see
     *note Numeric data types::).  This function is just a wrapper for
     the ‘gal_type_string_to_number’, but will put the value into a
     single-element dataset.

 -- Function:
          void
          gal_data_append_second_array_to_first_free (gal_data_t *a,
          gal_data_t *b)
     Append the contents of ‘a->array’ and ‘b->array’ into an internal
     array, and replace it with ‘a->array’.  Finally, free ‘b’.  Both
     inputs have to have the same type and have to have a single
     dimension.


File: gnuastro.info,  Node: Dimensions,  Next: Linked lists,  Prev: Library data container,  Up: Gnuastro library

12.3.7 Dimensions (‘dimension.h’)
---------------------------------

An array is a contiguous region of memory.  Hence, at the lowest level,
every element of an array just has one single-valued position: the
number of elements that lie between it and the first element in the
array.  This is also known as the _index_ of the element within the
array.  A dataset's number of dimensions is high-level abstraction
(meta-data) that we project onto that contiguous patch of memory.  When
the array is interpreted as a one-dimensional dataset, this index is
also the _coordinate_ of the element.  But once we associate the patch
of memory with a higher dimension, there must also be one coordinate for
each dimension.

   The functions and macros in this section provide you with the tools
to convert an index into a coordinate and vice-versa along with several
other issues for example, issues with the neighbors of an element in a
multi-dimensional context.

 -- Function:
          size_t
          gal_dimension_total_size (size_t ndim, size_t *dsize)
     Return the total number of elements for a dataset with ‘ndim’
     dimensions that has ‘dsize’ elements along each dimension.

 -- Function:
          int
          gal_dimension_is_different (gal_data_t *first, gal_data_t
          *second)
     Return ‘1’ (one) if the two datasets do not have the same size
     along all dimensions.  This function will also return ‘1’ when the
     number of dimensions of the two datasets are different.

 -- Function:
          size_t *
          gal_dimension_increment (size_t ndim, size_t *dsize)
     Return an allocated array that has the number of elements necessary
     to increment an index along every dimension.  For example, along
     the fastest dimension (last element in the ‘dsize’ and returned
     arrays), the value is ‘1’ (one).

 -- Function:
          size_t
          gal_dimension_num_neighbors (size_t ndim)
     The maximum number of neighbors (any connectivity) that a data
     element can have in ‘ndim’ dimensions.  Effectively, this function
     just returns $3^n-1$ (where $n$ is the number of dimensions).

 -- Function-like macro: GAL_DIMENSION_FLT_TO_INT (FLT)
     Calculate the integer pixel position that the floating point ‘FLT’
     number belongs to.  In the FITS format (and thus in Gnuastro), the
     center of each pixel is allocated on an integer (not it edge), so
     the pixel which hosts a floating point number cannot simply be
     found with internal type conversion.

 -- Function:
          void
          gal_dimension_add_coords (size_t *c1, size_t *c2, size_t *out,
          size_t ndim)
     For every dimension, add the coordinates in ‘c1’ with ‘c2’ and put
     the result into ‘out’.  In other words, for dimension ‘i’ run
     ‘out[i]=c1[i]+c2[i];’.  Hence ‘out’ may be equal to any one of ‘c1’
     or ‘c2’.

 -- Function:
          size_t
          gal_dimension_coord_to_index (size_t ndim, size_t *dsize,
          size_t *coord)
     Return the index (counting from zero) from the coordinates in
     ‘coord’ (counting from zero) assuming the dataset has ‘ndim’
     elements and the size of the dataset along each dimension is in the
     ‘dsize’ array.

 -- Function:
          void
          gal_dimension_index_to_coord (size_t index, size_t ndim,
          size_t *dsize, size_t *coord)
     Fill in the ‘coord’ array with the coordinates that correspond to
     ‘index’ assuming the dataset has ‘ndim’ elements and the size of
     the dataset along each dimension is in the ‘dsize’ array.  Note
     that both ‘index’ and each value in ‘coord’ are assumed to start
     from ‘0’ (zero).  Also that the space which ‘coord’ points to must
     already be allocated before calling this function.

 -- Function:
          size_t
          gal_dimension_dist_manhattan (size_t *a, size_t *b, size_t
          ndim)
     Return the manhattan distance (see Wikipedia
     (https://en.wikipedia.org/wiki/Taxicab_geometry)) between the two
     coordinates ‘a’ and ‘b’ (each an array of ‘ndim’ elements).

 -- Function:
          float
          gal_dimension_dist_radial (size_t *a, size_t *b, size_t ndim)
     Return the radial distance between the two coordinates ‘a’ and ‘b’
     (each an array of ‘ndim’ elements).

 -- Function:
          float
          gal_dimension_dist_elliptical (double *center, double *pa_deg,
          double *q, size_t ndim, double *point)
     Return the elliptical/ellipsoidal distance of the single point
     ‘point’ (containing ‘ndim’ values: coordinates of the point in each
     dimension) from an ellipse that is defined by ‘center’, ‘pa_deg’
     and ‘q’.  ‘center’ is the coordinates of the ellipse center (also
     with ‘ndim’ elements).  ‘pa’ is the position-angle in degrees (the
     angle of the semi-major axis from the first dimension in a 2D
     ellipse) and ‘q’ is the axis ratio.

     In a 2D ellipse, ‘pa’ and ‘q’ are a single-element array.  However,
     in a 3D ellipsoid, ‘pa’ must have three elements, and ‘q’ must have
     2 elements.  For more see *note Defining an ellipse and
     ellipsoid::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sum (gal_data_t *in, size_t c_dim,
          gal_data_t *weight)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by summing all elements in that direction.  If
     ‘weight!=NULL’, it must be a single-dimensional array, with the
     same size as the dimension to be collapsed.  The respective weight
     will be multiplied to each element during the collapse.

     For generality, the returned dataset will have a ‘GAL_TYPE_FLOAT64’
     type.  See *note Copying datasets:: for converting the returned
     dataset to a desired type.  Also, for more on the application of
     this function, see the Arithmetic program's ‘collapse-sum’ operator
     (which uses this function) in *note Arithmetic operators::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mean (gal_data_t *in, size_t c_dim,
          gal_data_t *weight)
     Similar to ‘gal_dimension_collapse_sum’ (above), but the collapse
     will be done by calculating the mean along the requested dimension,
     not summing over it.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_number (gal_data_t *in, size_t c_dim)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by counting how many non-blank elements there are along
     that dimension.

     For generality, the returned dataset will have a ‘GAL_TYPE_INT32’
     type.  See *note Copying datasets:: for converting the returned
     dataset to a desired type.  Also, for more on the application of
     this function, see the Arithmetic program's ‘collapse-number’
     operator (which uses this function) in *note Arithmetic
     operators::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_minmax (gal_data_t *in, size_t c_dim,
          int max1_min0)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by using the largest/smallest non-blank value along
     that dimension.  If ‘max1_min0’ is non-zero, then the collapsed
     dataset will have the maximum value along the given dimension and
     if it is zero, the minimum.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_median (gal_data_t *in, size_t c_dim,
          size_t numthreads, size_t minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median non-blank value along that
     dimension.  Since the median involves sorting, this operator
     benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_std (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the standard deviation of pixels along that
     dimension after sigma-clipping.  Since sigma-clipping involves
     sorting, this operator benefits from many threads (which needs to
     be set with ‘numthreads’).  For more on ‘minmapsize’ and
     ‘quietmmap’ see *note Memory management::.  For more on sigma
     clipping, see *note Sigma clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_fill_std (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_sclip_std’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_mad (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median absolute deviation (MAD) of
     pixels along that dimension after sigma-clipping.  Since
     sigma-clipping involves sorting, this operator benefits from many
     threads (which needs to be set with ‘numthreads’).  For more on
     ‘minmapsize’ and ‘quietmmap’ see *note Memory management::.  For
     more on sigma clipping, see *note Sigma clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_fill_mad (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_sclip_mad’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_mean (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the mean of pixels along that dimension
     after sigma-clipping.  Since sigma-clipping involves sorting, this
     operator benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on sigma clipping, see *note Sigma
     clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_fill_mean (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_sclip_mean’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_median (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median of pixels along that dimension
     after sigma-clipping.  Since sigma-clipping involves sorting, this
     operator benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on sigma clipping, see *note Sigma
     clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_fill_median (gal_data_t *in,
          size_t c_dim, float multip, float param, size_t numthreads,
          size_t minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_sclip_median’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_number (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the number of pixels along that dimension
     after sigma-clipping.  Since sigma-clipping involves sorting, this
     operator benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on sigma clipping, see *note Sigma
     clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_sclip_fill_number (gal_data_t *in,
          size_t c_dim, float multip, float param, size_t numthreads,
          size_t minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_sclip_number’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_std (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the standard deviation of pixels along that
     dimension after median absolute deviation (MAD) clipping.  Since
     MAD-clipping involves sorting, this operator benefits from many
     threads (which needs to be set with ‘numthreads’).  For more on
     ‘minmapsize’ and ‘quietmmap’ see *note Memory management::.  For
     more on MAD-clipping, see *note MAD clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_fill_std (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_mclip_std’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_mad (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median absolute deviation (MAD) of
     pixels along that dimension after median absolute deviation (MAD)
     clipping.  Since MAD-clipping involves sorting, this operator
     benefits from many threads (which needs to be set with
     ‘numthreads’).  For more on ‘minmapsize’ and ‘quietmmap’ see *note
     Memory management::.  For more on MAD-clipping, see *note MAD
     clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_fill_mad (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_mclip_mad’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_mean (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the mean of pixels along that dimension
     after median absolute deviation (MAD) clipping.  Since MAD-clipping
     involves sorting, this operator benefits from many threads (which
     needs to be set with ‘numthreads’).  For more on ‘minmapsize’ and
     ‘quietmmap’ see *note Memory management::.  For more on
     MAD-clipping, see *note MAD clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_fill_mean (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_mclip_mean’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_median (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the median of pixels along that dimension
     after median absolute deviation (MAD) clipping.  Since MAD-clipping
     involves sorting, this operator benefits from many threads (which
     needs to be set with ‘numthreads’).  For more on ‘minmapsize’ and
     ‘quietmmap’ see *note Memory management::.  For more on
     MAD-clipping, see *note MAD clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_fill_median (gal_data_t *in,
          size_t c_dim, float multip, float param, size_t numthreads,
          size_t minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_mclip_median’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_number (gal_data_t *in, size_t
          c_dim, float multip, float param, size_t numthreads, size_t
          minmapsize, int quietmmap)
     Collapse the input dataset (‘in’) along the given dimension
     (‘c_dim’, in C definition: starting from zero, from the slowest
     dimension), by finding the number of pixels along that dimension
     after median absolute deviation (MAD) clipping.  Since MAD-clipping
     involves sorting, this operator benefits from many threads (which
     needs to be set with ‘numthreads’).  For more on ‘minmapsize’ and
     ‘quietmmap’ see *note Memory management::.  For more on
     MAD-clipping, see *note MAD clipping::.

 -- Function:
          gal_data_t *
          gal_dimension_collapse_mclip_fill_number (gal_data_t *in,
          size_t c_dim, float multip, float param, size_t numthreads,
          size_t minmapsize, int quietmmap)
     Similar to ‘gal_dimension_collapse_mclip_number’, but with filled
     re-clipping (see *note Contiguous outliers::).

 -- Function:
          size_t
          gal_dimension_remove_extra (size_t ndim, size_t *dsize, struct
          wcsprm *wcs)
     Remove extra dimensions (those that only have a length of 1) from
     the basic size information of a dataset.  If the total number of
     elements (in all dimensions) is 1, this function will not remove
     anything (because an "extra" is only meaningful when some
     dimensions have more than one element length).

     ‘ndim’ is the number of dimensions and ‘dsize’ is an array with
     ‘ndim’ elements containing the size along each dimension in the C
     dimension order.  When ‘wcs!=NULL’, the respective dimension will
     also be removed from the WCS.

     This function will return the new number of dimensions and the
     ‘dsize’ elements will contain the length along each new dimension.

 -- Function-like macro: GAL_DIMENSION_NEIGHBOR_OP (index, ndim, dsize,
          connectivity, dinc, operation)
     Parse the neighbors of the element located at ‘index’ and do the
     requested operation on them.  This is defined as a macro to allow
     easy definition of any operation on the neighbors of a given
     element without having to use loops within your source code (the
     loops are implemented by this macro).  For an example of using this
     function, please see *note Library demo - inspecting neighbors::.
     The input arguments to this function-like macro are described
     below:

     ‘index’
          Distance of this element from the first element in the array
          on a contiguous patch of memory (starting from 0), see the
          discussion above.
     ‘ndim’
          The number of dimensions associated with the contiguous patch
          of memory.
     ‘dsize’
          The full array size along each dimension.  This must be an
          array and is assumed to have the same number elements as
          ‘ndim’.  See the discussion under the same element in *note
          Generic data container::.
     ‘connectivity’
          Most distant neighbors to consider.  Depending on the number
          of dimensions, different neighbors may be defined for each
          element.  This function-like macro distinguish between these
          different neighbors with this argument.  It has a value
          between ‘1’ (one) and ‘ndim’.  For example, in a 2D dataset,
          4-connected neighbors have a connectivity of ‘1’ and
          8-connected neighbors have a connectivity of ‘2’.  Note that
          this is inclusive, so in this example, a connectivity of ‘2’
          will also include connectivity ‘1’ neighbors.
     ‘dinc’
          An array keeping the length necessary to increment along each
          dimension.  You can make this array with the following
          function.  Just do not forget to free the array after you are
          done with it:

               size_t *dinc=gal_dimension_increment(ndim, dsize);
               free(dinc);

          ‘dinc’ depends on ‘ndim’ and ‘dsize’, but it must be defined
          outside this function-like macro since it involves allocation
          to help in performance.

     ‘operation’
          Any C operation that you would like to do on the neighbor.
          This macro will fill the ‘nind’ variable that can be used as
          the index of the neighbor that is currently being studied.  It
          is defined as '‘size_t nind;’'.  Your given ‘operation’ will
          be repeated the number of times there is a neighbor for this
          element.  See the example in *note Library demo - inspecting
          neighbors:: for a fully working demo.

     This macro works fully within its own ‘{}’ block and except for the
     ‘nind’ variable that shows the neighbor's index, all the variables
     within this macro's block start with ‘gdn_’.


File: gnuastro.info,  Node: Linked lists,  Next: Array input output,  Prev: Dimensions,  Up: Gnuastro library

12.3.8 Linked lists (‘list.h’)
------------------------------

An array is a contiguous region of memory that is very efficient and
easy to use for recording and later accessing any random element as fast
as any other.  This makes array the primary data container when you have
many elements (for example, an image which has millions of pixels).  One
major problem with an array is that the number of elements that go into
it must be known in advance and adding or removing an element will
require a re-set of all the other elements.  For example, if you want to
remove the 3rd element in a 1000 element array, all 997 subsequent
elements have to pulled back by one position, the reverse will happen if
you need to add an element.

   In many contexts such situations never come up, for example, you do
not want to shift all the pixels in an image by one or two pixels from
some random position in the image: their positions have scientific
value.  But in other contexts you will find yourself frequently
adding/removing an a-priori unknown number of elements.  Linked lists
(or _lists_ for short) are the data-container of choice in such
situations.  As in a chain, each _node_ in a list is an independent C
structure, keeping its own data along with pointer(s) to its immediate
neighbor(s).  Below, you can see one simple linked list node structure
along with an ASCII art schematic of how we can use the ‘next’ pointer
to add any number of elements to the list that we want.  By convention,
a list is terminated when ‘next’ is the ‘NULL’ pointer.

     struct list_float          /*     ---------    ---------           */
     {                          /*     | Value |    | Value |           */
       float             value; /*     |  ---  |    |  ---  |           */
       struct list_float *next; /*     |  next-|--> |  next-|--> NULL   */
     }                          /*     ---------    ---------           */

   The schematic shows another great advantage of linked lists: it is
very easy to add or remove/pop a node anywhere in the list.  If you want
to modify the first node, you just have to change one pointer.  If it is
in the middle, you just have to change two.  You initially define a
variable of this type with a ‘NULL’ pointer as shown below:

     struct list_float *list=NULL;

To add or remove/pop a node from the list you can use functions provided
for the respective type in the sections below.

When you add an element to the list, it is conventionally added to the
"top" of the list: the general list pointer will point to the newly
created node, which will point to the previously created node and so on.
So when you "pop" from the top of the list, you are actually retrieving
the last value you put in and changing the list pointer to the next
youngest node.  This is thus known as a "last-in-first-out" list.  This
is the most efficient type of linked list (easier to implement and
faster to process).  Alternatively, you can add each newly created node
at the end of the list.  If you do that, you will get a
"first-in-first-out" list.  But that will force you to go through the
whole list for each new element that is created (this will slow down the
processing)(1).

   The node example above creates the simplest kind of a list.  We can
define each node with two pointers to both the next and previous
neighbors, this is called a "Doubly linked list".  In general, lists are
very powerful and simple constructs that can be very useful.  But going
into more detail would be out of the scope of this short introduction in
this book.  Wikipedia (https://en.wikipedia.org/wiki/Linked_list) has a
nice and more thorough discussion of the various types of lists.  To
appreciate/use the beauty and elegance of these powerful constructs even
further, see Chapter 2 (Information Structures, in volume 1) of Donald
Knuth's "The art of computer programming".

   In this section we will review the functions and structures that are
available in Gnuastro for working on lists.  They differ by the type of
data that each node can keep.  For each linked-list node structure, we
will first introduce the structure, then the functions for working on
the structure.  All these structures and functions are defined and
declared in ‘gnuastro/list.h’.

* Menu:

* List of strings::             Simply linked list of strings.
* List of int32_t::             Simply linked list of int32_ts.
* List of size_t::              Simply linked list of size_ts.
* List of float::               Simply linked list of floats.
* List of double::              Simply linked list of doubles
* List of size_t and double::   Simply linked list of a size_t and double.
* List of two size_ts and a double::  Simply linked list of two size_ts and a double.
* List of void::                Simply linked list of void * pointers.
* Ordered list of size_t::      Simply linked, ordered list of size_t.
* Doubly linked ordered list of size_t::  Definition and functions.
* List of gal_data_t::          Simply linked list Gnuastro's generic datatype.

   ---------- Footnotes ----------

   (1) A better way to get a first-in-first-out is to first keep the
data as last-in-first-out until they are all read.  Afterwards, reverse
the list by popping each node and immediately add it to the new list.
This practically reverses the last-in-first-out list to a
first-in-first-out one.  All the list types discussed in this chapter
have a function with a ‘_reverse’ suffix for this job.


File: gnuastro.info,  Node: List of strings,  Next: List of int32_t,  Prev: Linked lists,  Up: Linked lists

12.3.8.1 List of strings
........................

Probably one of the most common lists you will be using are lists of
strings.  They are the best tools when you are reading the user's
inputs, or when adding comments to the output files.  Below you can see
Gnuastro's string list type and several functions to help in adding,
removing/popping, reversing and freeing the list.

 -- Type (C struct): gal_list_str_t
     A single node in a list containing a string of characters.
          typedef struct gal_list_str_t
          {
            char *v;
            struct gal_list_str_t *next;
          } gal_list_str_t;

 -- Function:
          void
          gal_list_str_add (gal_list_str_t **list, char *value, int
          allocate)
     Add a new node to the list of strings (‘list’) and update it.  The
     new node will contain the string ‘value’.  If ‘allocate’ is not
     zero, space will be allocated specifically for the string of the
     new node and the contents of ‘value’ will be copied into it.  This
     can be useful when your string may be changed later in the program,
     but you want your list to remain.  Here is one short/simple example
     of initializing and adding elements to a string list:

          gal_list_str_t *list=NULL;
          gal_list_str_add(&list, "bottom of list.", 1);
          gal_list_str_add(&list, "second last element of list.", 1);

 -- Function:
          char *
          gal_list_str_pop (gal_list_str_t **list)
     Pop the top element of ‘list’, change ‘list’ to point to the next
     node in the list, and return the string that was in the popped
     node.  If ‘*list==NULL’, then this function will also return a
     ‘NULL’ pointer.

 -- Function:
          size_t
          gal_list_str_number (gal_list_str_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          gal_list_str_t *
          gal_list_str_last (gal_list_str_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_str_print (gal_list_str_t *list)
     Print the strings within each node of ‘*list’ on the standard
     output in the same order that they are stored.  Each string is
     printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly,
     format.  For example, the following code snippet.

          size_t i=0;
          gal_list_str_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("String %zu: %s\n", ++i, tmp->v);

 -- Function:
          void
          gal_list_str_reverse (gal_list_str_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          void
          gal_list_str_free (gal_list_str_t *list, int freevalue)
     Free every node in ‘list’.  If ‘freevalue’ is not zero, also free
     the string within the nodes.

 -- Function:
          gal_list_str_t *
          gal_list_str_extract (char *string)
     Extract space-separated components of the input string.  If any
     space element should be kept (and not considered as a delimiter
     between two tokens), precede it with a backslash (‘\’).  Be aware
     that in C programming, when including a backslash character within
     a string literal, the correct format is indeed to use two
     backslashes ("\\") to represent a single backslash:

          gal_list_str_extract("bottom of\\ list");

 -- Function:
          char *
          gal_list_str_cat (gal_list_str_t *list, char delimiter)
     Concatenate (append) the input list of strings into a single string
     where each node is separated from the next with the given
     ‘delimiter’.  The space for the output string is allocated by this
     function and should be freed when you have finished with it.

     If there is any delimiter characters are present in any of the
     elements, a backslash (‘\’) will be printed before the SPACE
     character.  This is necessary, otherwise, a function like
     ‘gal_list_str_extract’ will not be able to extract the elements
     back into separate elements in a list.


File: gnuastro.info,  Node: List of int32_t,  Next: List of size_t,  Prev: List of strings,  Up: Linked lists

12.3.8.2 List of ‘int32_t’
..........................

Signed integers are the best types when you are dealing with a positive
or negative integers.  The are generally useful in many contexts, for
example when you want to keep the order of a series of states (each
state stored as a given number in an ‘enum’ for example).  On many
modern systems, ‘int32_t’ is just an alias for ‘int’, so you can use
them interchangeably.  To make sure, check the size of ‘int’ on your
system:

 -- Type (C struct): gal_list_i32_t
     A single node in a list containing a 32-bit signed integer (see
     *note Numeric data types::).
          typedef struct gal_list_i32_t
          {
            int32_t v;
            struct gal_list_i32_t *next;
          } gal_list_i32_t;

 -- Function:
          void
          gal_list_i32_add (gal_list_i32_t **list, int32_t value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘int32_t’s (‘uint32_t’ is equal to ‘int’ on many modern systems),
     and update ‘list’.  Here is one short example of initializing and
     adding elements to a string list:

          gal_list_i32_t *list=NULL;
          gal_list_i32_add(&list, 52);
          gal_list_i32_add(&list, -4);

 -- Function:
          int32_t
          gal_list_i32_pop (gal_list_i32_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will also return
     ‘GAL_BLANK_INT32’ (see *note Library blank values::).

 -- Function:
          size_t
          gal_list_i32_number (gal_list_i32_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_i32_last (gal_list_i32_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_i32_print (gal_list_i32_t *list)
     Print the integers within each node of ‘*list’ on the standard
     output in the same order that they are stored.  Each integer is
     printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly
     format.  For example, the following code snippet.  You can also
     modify it to print all values in one line, etc., depending on the
     context of your program.

          size_t i=0;
          gal_list_i32_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("Number %zu: %s\n", ++i, tmp->v);

 -- Function:
          void
          gal_list_i32_reverse (gal_list_i32_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          int32_t *
          gal_list_i32_to_array (gal_list_i32_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     opposite order of elements in ‘list’.  This function can be useful
     after you have finished reading an initially unknown number of
     values and want to put them in an array for easy random access.

 -- Function:
          void
          gal_list_i32_free (gal_list_i32_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: List of size_t,  Next: List of float,  Prev: List of int32_t,  Up: Linked lists

12.3.8.3 List of ‘size_t’
.........................

The ‘size_t’ type is a unique type in C: as the name suggests it is
defined to store sizes, or more accurately, the distances between memory
locations.  Hence it is always positive (an ‘unsigned’ type) and it is
directly related to the address-able spaces on the host system: on
32-bit and 64-bit systems it is an alias for ‘uint32_t’ and ‘uint64_t’,
respectively (see *note Numeric data types::).

   ‘size_t’ is the default compiler type to index an array (recall that
an array index in C is just a pointer increment of a given _size_).
Since it is unsigned, it is a great type for counting (where negative is
not defined), you are always sure it will never exceed the system's
(virtual) memory and since its name has the word "size" inside it, it
provides a good level of documentation(1).  In Gnuastro, we do all
counting and array indexing with this type, so this list is very handy.
As discussed above, ‘size_t’ maps to different types on different
machines, so a portable way to print them with ‘printf’ is to use C99's
‘%zu’ format.

 -- Type (C struct): gal_list_sizet_t
     A single node in a list containing a ‘size_t’ value (which maps to
     ‘uint32_t’ or ‘uint64_t’ on 32-bit and 64-bit systems), see *note
     Numeric data types::.
          typedef struct gal_list_sizet_t
          {
            size_t v;
            struct gal_list_sizet_t *next;
          } gal_list_sizet_t;

 -- Function:
          void
          gal_list_sizet_add (gal_list_sizet_t **list, size_t value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘size_t’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_sizet_t *list=NULL;
          gal_list_sizet_add(&list, 45493);
          gal_list_sizet_add(&list, 930484);

 -- Function:
          sizet_t
          gal_list_sizet_pop (gal_list_sizet_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will also return
     ‘GAL_BLANK_SIZE_T’ (see *note Library blank values::).

 -- Function:
          size_t
          gal_list_sizet_number (gal_list_sizet_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_sizet_last (gal_list_sizet_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_sizet_print (gal_list_sizet_t *list)
     Print the values within each node of ‘*list’ on the standard output
     in the same order that they are stored.  Each integer is printed on
     one line.  This function is mainly good for checking/debugging your
     program.  For program outputs, it is best to make your own
     implementation with a better, more user-friendly format.  For
     example, the following code snippet.  You can also modify it to
     print all values in one line, etc., depending on the context of
     your program.

          size_t i=0;
          gal_list_sizet_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("Number %zu: %zu\n", ++i, tmp->v);

 -- Function:
          void
          gal_list_sizet_reverse (gal_list_sizet_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          size_t *
          gal_list_sizet_to_array (gal_list_sizet_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     inverse of the order of elements in ‘list’.  This function can be
     useful after you have finished reading an initially unknown number
     of values and want to put them in an array for easy random access.

 -- Function:
          void
          gal_list_sizet_free (gal_list_sizet_t *list)
     Free every node in ‘list’.

   ---------- Footnotes ----------

   (1) So you know that a variable of this type is not used to store
some generic state for example.


File: gnuastro.info,  Node: List of float,  Next: List of double,  Prev: List of size_t,  Up: Linked lists

12.3.8.4 List of ‘float’
........................

Single precision floating point numbers can accurately store real number
until 7.2 decimals and only consume 4 bytes (32-bits) of memory, see
*note Numeric data types::.  Since astronomical data rarely reach that
level of precision, single precision floating points are the type of
choice to keep and read data.  However, when processing the data, it is
best to use double precision floating points (since errors propagate).

 -- Type (C struct): gal_list_f32_t
     A single node in a list containing a 32-bit single precision
     ‘float’ value: see *note Numeric data types::.
          typedef struct gal_list_f32_t
          {
            float v;
            struct gal_list_f32_t *next;
          } gal_list_f32_t;

 -- Function:
          void
          gal_list_f32_add (gal_list_f32_t **list, float value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘float’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_f32_t *list=NULL;
          gal_list_f32_add(&list, 3.89);
          gal_list_f32_add(&list, 1.23e-20);

 -- Function:
          float
          gal_list_f32_pop (gal_list_f32_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will return ‘GAL_BLANK_FLOAT32’
     (NaN, see *note Library blank values::).

 -- Function:
          size_t
          gal_list_f32_number (gal_list_f32_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_f32_last (gal_list_f32_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_f32_print (gal_list_f32_t *list)
     Print the values within each node of ‘*list’ on the standard output
     in the same order that they are stored.  Each floating point number
     is printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly
     format.  For example, in the following code snippet.  You can also
     modify it to print all values in one line, etc., depending on the
     context of your program.

          size_t i=0;
          gal_list_f32_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("Number %zu: %f\n", ++i, tmp->v);

 -- Function:
          void
          gal_list_f32_reverse (gal_list_f32_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          float *
          gal_list_f32_to_array (gal_list_f32_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     inverse of the order of elements in ‘list’.  This function can be
     useful after you have finished reading an initially unknown number
     of values and want to put them in an array for easy random access.

 -- Function:
          void
          gal_list_f32_free (gal_list_f32_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: List of double,  Next: List of size_t and double,  Prev: List of float,  Up: Linked lists

12.3.8.5 List of ‘double’
.........................

Double precision floating point numbers can accurately store real number
until 15.9 decimals and consume 8 bytes (64-bits) of memory, see *note
Numeric data types::.  This level of precision makes them very good for
serious processing in the middle of a program's execution: in many
cases, the propagation of errors will still be insignificant compared to
actual observational errors in a data set.  But since they consume 8
bytes and more CPU processing power, they are often not the best choice
for storing and transferring of data.

 -- Type (C struct): gal_list_f64_t
     A single node in a list containing a 64-bit double precision
     ‘double’ value: see *note Numeric data types::.
          typedef struct gal_list_f64_t
          {
            double v;
            struct gal_list_f64_t *next;
          } gal_list_f64_t;

 -- Function:
          void
          gal_list_f64_add (gal_list_f64_t **list, double value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘double’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_f64_t *list=NULL;
          gal_list_f64_add(&list, 3.8129395763193);
          gal_list_f64_add(&list, 1.239378923931e-20);

 -- Function:
          double
          gal_list_f64_pop (gal_list_f64_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will return ‘GAL_BLANK_FLOAT64’
     (NaN, see *note Library blank values::).

 -- Function:
          size_t
          gal_list_f64_number (gal_list_f64_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_f64_last (gal_list_f64_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_f64_print (gal_list_f64_t *list)
     Print the values within each node of ‘*list’ on the standard output
     in the same order that they are stored.  Each floating point number
     is printed on one line.  This function is mainly good for
     checking/debugging your program.  For program outputs, it is best
     to make your own implementation with a better, more user-friendly
     format.  For example, in the following code snippet.  You can also
     modify it to print all values in one line, etc., depending on the
     context of your program.

          size_t i=0;
          gal_list_f64_t *tmp;
          for(tmp=list; tmp!=NULL; tmp=tmp->next)
            printf("Number %zu: %f\n", ++i, tmp->v);

 -- Function:
          void
          gal_list_f64_reverse (gal_list_f64_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          double *
          gal_list_f64_to_array (gal_list_f64_t *list, int reverse,
          size_t *num)
     Dynamically allocate an array and fill it with the values in
     ‘list’.  The function will return a pointer to the allocated array
     and put the number of elements in the array into the ‘num’ pointer.
     If ‘reverse’ has a non-zero value, the array will be filled in the
     inverse of the order of elements in ‘list’.  This function can be
     useful after you have finished reading an initially unknown number
     of values and want to put them in an array for easy random access.

 -- Function:
          gal_data_t *
          gal_list_f64_to_data (gal_list_f64_t *list, uint8_t type,
          size_t minmapsize, int quietmmap)
     Write the values in the given ‘list’ into a ‘gal_data_t’ dataset of
     the requested ‘type’.  The order of the values in the dataset will
     be the same as the order from the top of the list.

 -- Function:
          void
          gal_list_f64_free (gal_list_f64_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: List of size_t and double,  Next: List of two size_ts and a double,  Prev: List of double,  Up: Linked lists

12.3.8.6 List of ‘size_t’ and ‘double’
......................................

The list structure below is useful when you need to keep floating point
values and indexs (most common usage of ‘size_t’ in Gnuastro) together
within a higher-level program.

 -- Type (C struct): gal_list_sizetf64_t
     Each node in this singly-linked list contains a ‘size_t’ value and
     a floating point value with a type of double-precision floating
     point.

     typedef struct gal_list_sizetf64_t
     {
       size_t i;
       double v;
       struct gal_list_sizetf64_t *next;
     } gal_list_sizetf64_t;

 -- Function:
          void
          gal_list_sizetf64_add (gal_list_sizetf64_t **list, size_t
          index, float value)
     Allocate space for a new node in ‘list’, to store the ‘index’ and
     ‘value’ into it.  As a simply-linked list, the added node will be
     placed at the top of the list.

 -- Function:
          size_t
          gal_list_sizetf64_pop (gal_list_sizetf64_t **list, size_t
          *index, float *value)
     Pop a node from the top of ‘list’, return the node's index and
     value in the respective places that ‘index’ and ‘value’ point to.
     This function will also free the allocated space for the popped
     node and after this function, ‘list’ will point to the next node
     (which has a larger ‘tosort’ element).

 -- Function:
          void
          gal_list_sizetf64_free (gal_list_sizetf64_t **list)
     Free any element within the input list.

 -- Type (C struct): gal_list_sizetsizetf64_t
     Each node in this singly-linked list contains a ‘size_t’ value and
     a floating point value with a type of double-precision floating
     point.


File: gnuastro.info,  Node: List of two size_ts and a double,  Next: List of void,  Prev: List of size_t and double,  Up: Linked lists

12.3.8.7 List of two ‘size_t’s and a ‘double’
.............................................

When it is necessary to keep two indexs and a double value, the
structure and respective functions below can be used.

     typedef struct gal_list_sizetsizetf64_t
     {
       size_t i;
       size_t j;
       double v;
       struct gal_list_sizetsizetf64_t *next;
     } gal_list_sizetsizetf64_t;

 -- Function:
          void
          gal_list_sizetsizetf64_add (gal_list_sizetsizetf64_t **list,
          size_t i, size_t i, float v)
     Allocate space for a new node in ‘list’, to store the ‘i’, ‘j’ and
     ‘v’ into it.  As a simply-linked list, the added node will be
     placed at the top of the list.

 -- Function:
          size_t
          gal_list_sizetsizetf64_pop (gal_list_sizetsizetf64_t **list,
          size_t *i, size_t *j, float *v)
     Pop a node from the top of ‘list’, return the node's index and
     value in the respective places that ‘i’, ‘j’ and ‘v’ point to.
     This function will also free the allocated space for the popped
     node and after this function, ‘list’ will point to the next node
     (which has a larger ‘tosort’ element).

 -- Function:
          void
          gal_list_sizetsizetf64_free (gal_list_sizetsizetf64_t **list)
     Free any element within the input list.


File: gnuastro.info,  Node: List of void,  Next: Ordered list of size_t,  Prev: List of two size_ts and a double,  Up: Linked lists

12.3.8.8 List of ‘void *’
.........................

In C, ‘void *’ is the most generic pointer.  Usually pointers are
associated with the type of content they point to.  For example, ‘int *’
means a pointer to an integer.  This ancillary information about the
contents of the memory location is very useful for the compiler,
catching bad errors and also documentation (it helps the reader see what
the address in memory actually contains).  However, ‘void *’ is just a
raw address (pointer), it contains no information on the contents it
points to.

   These properties make the ‘void *’ very useful when you want to treat
the contents of an address in different ways.  You can use the ‘void *’
list defined in this section and its function on any kind of data: for
example, you can use it to keep a list of custom data structures that
you have built for your own separate program.  Each node in the list can
keep anything and this gives you great versatility.  But in using ‘void
*’, please beware that "with great power comes great responsibility".

 -- Type (C struct): gal_list_void_t
     A single node in a list containing a ‘void *’ pointer.
          typedef struct gal_list_void_t
          {
            void *v;
            struct gal_list_void_t *next;
          } gal_list_void_t;

 -- Function:
          void
          gal_list_void_add (gal_list_void_t **list, void *value)
     Add a new node (containing ‘value’) to the top of the ‘list’ of
     ‘void *’s and update ‘list’.  Here is one short example of
     initializing and adding elements to a string list:

          gal_list_void_t *list=NULL;
          gal_list_f64_add(&list, some_pointer);
          gal_list_f64_add(&list, another_pointer);

 -- Function:
          void *
          gal_list_void_pop (gal_list_void_t **list)
     Pop the top element of ‘list’ and return the value.  This function
     will also change ‘list’ to point to the next node in the list.  If
     ‘*list==NULL’, then this function will return ‘NULL’.

 -- Function:
          size_t
          gal_list_void_number (gal_list_void_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          size_t
          gal_list_void_last (gal_list_void_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_void_reverse (gal_list_void_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          void
          gal_list_void_free (gal_list_void_t *list)
     Free every node in ‘list’.


File: gnuastro.info,  Node: Ordered list of size_t,  Next: Doubly linked ordered list of size_t,  Prev: List of void,  Up: Linked lists

12.3.8.9 Ordered list of ‘size_t’
.................................

Positions/sizes in a dataset are conventionally in the ‘size_t’ type
(see *note List of size_t::) and it sometimes occurs that you want to
parse and read the values in a specific order.  For example, you want to
start from one pixel and add pixels to the list based on their distance
to that pixel.  So that ever time you pop an element from the list, you
know it is the nearest that has not yet been studied.  The
‘gal_list_osizet_t’ type and its functions in this section are designed
to facilitate such operations.

 -- Type (C struct): gal_list_osizet_t
     Each node in this singly-linked list contains a ‘size_t’ value and
     a floating point value.  The floating point value is used as a
     reference to add new nodes in a sorted manner.  At any moment, the
     first popped node in this list will have the smallest ‘tosort’
     value, and subsequent nodes will have larger to values.
     typedef struct gal_list_osizet_t
     {
       size_t v;                       /* The actual value. */
       float  s;                       /* The parameter to sort by. */
       struct gal_list_osizet_t *next;
     } gal_list_osizet_t;

 -- Function:
          void
          gal_list_osizet_add (gal_list_osizet_t **list, size_t value,
          float tosort)
     Allocate space for a new node in ‘list’, and store ‘value’ and
     ‘tosort’ into it.  The new node will not necessarily be at the
     "top" of the list.  If ‘*list!=NULL’, then the ‘tosort’ values of
     existing nodes is inspected and the given node is placed in the
     list such that the top element (which is popped with
     ‘gal_list_osizet_pop’) has the smallest ‘tosort’ value.

 -- Function:
          size_t
          gal_list_osizet_pop (gal_list_osizet_t **list, float
          *sortvalue)
     Pop a node from the top of ‘list’, return the node's ‘value’ and
     put its sort value in the space that ‘sortvalue’ points to.  This
     function will also free the allocated space for the popped node and
     after this function, ‘list’ will point to the next node (which has
     a larger ‘tosort’ element).

 -- Function:
          void
          gal_list_osizet_to_sizet_free (gal_list_osizet_t *in,
          gal_list_sizet_t **out)
     Convert the ordered list of ‘size_t’s into an ordinary ‘size_t’
     linked list.  This can be useful when all the elements have been
     added and you just need to pop-out elements and do not care about
     the sorting values any more.  After the conversion is done, this
     function will free the input list.  Note that the ‘out’ list does
     not have to be empty.  If it already contains some nodes, the new
     nodes will be added on top of them.


File: gnuastro.info,  Node: Doubly linked ordered list of size_t,  Next: List of gal_data_t,  Prev: Ordered list of size_t,  Up: Linked lists

12.3.8.10 Doubly linked ordered list of ‘size_t’
................................................

An ordered list of indices is required in many contexts, one example was
discussed at the beginning of *note Ordered list of size_t::.  But the
list that was introduced there only has one point of entry: you can
always only parse the list from smallest to largest.  In this section,
the doubly-linked ‘gal_list_dosizet_t’ node is defined which will allow
us to parse the values in ascending or descending order.

 -- Type (C struct): gal_list_dosizet_t

     Doubly-linked, ordered ‘size_t’ list node structure.  Each node in
     this Doubly-linked list contains a ‘size_t’ value and a floating
     point value.  The floating point value is used as a reference to
     add new nodes in a sorted manner.  In the functions here, this
     linked list can be pointed to by two pointers (largest and
     smallest) with the following format:
                      largest pointer
                      |
             NULL <-- (v0,s0) <--> (v1,s1) <--> ... (vn,sn) --> NULL
                                                    |
                                     smallest pointer
     At any moment, the two pointers will point to the nodes containing
     the "largest" and "smallest" values and the rest of the nodes will
     be sorted.  This is useful when an unknown number of nodes are
     being added continuously and during the operations it is important
     to have the nodes in a sorted format.

          typedef struct gal_list_dosizet_t
          {
            size_t v;                       /* The actual value. */
            float s;                        /* The parameter to sort by. */
            struct gal_list_dosizet_t *prev;
            struct gal_list_dosizet_t *next;
          } gal_list_dosizet_t;

 -- Function:
          void
          gal_list_dosizet_add (gal_list_dosizet_t **largest,
          gal_list_dosizet_t **smallest, size_t value, float tosort)
     Allocate space for a new node in ‘list’, and store ‘value’ and
     ‘tosort’ into it.  If the list is empty, both ‘largest’ and
     ‘smallest’ must be ‘NULL’.

 -- Function:
          size_t
          gal_list_dosizet_pop_smallest (gal_list_dosizet_t **largest,
          gal_list_dosizet_t **smallest, float tosort)
     Pop the value with the smallest reference from the doubly linked
     list and store the reference into the space pointed to by ‘tosort’.
     Note that even though only the smallest pointer will be popped,
     when there was only one node in the list, the ‘largest’ pointer
     also has to change, so we need both.

 -- Function:
          void
          gal_list_dosizet_print (gal_list_dosizet_t *largest,
          gal_list_dosizet_t *smallest)
     Print the largest and smallest values sequentially until the list
     is parsed.

 -- Function:
          void
          gal_list_dosizet_to_sizet (gal_list_dosizet_t *in,
          gal_list_sizet_t **out)
     Convert the doubly linked, ordered ‘size_t’ list into a
     singly-linked list of ‘size_t’.

 -- Function:
          void
          gal_list_dosizet_free (gal_list_dosizet_t *largest)
     Free the doubly linked, ordered ‘sizet_t’ list.


File: gnuastro.info,  Node: List of gal_data_t,  Prev: Doubly linked ordered list of size_t,  Up: Linked lists

12.3.8.11 List of ‘gal_data_t’
..............................

Gnuastro's generic data container has a ‘next’ element which enables it
to be used as a singly-linked list (see *note Generic data container::).
The ability to connect the different data containers offers great
advantages.  For example, each column in a table in an independent
dataset: with its own name, units, numeric data type (see *note Numeric
data types::).  Another application is in Tessellating an input dataset
into separate tiles or only studying particular regions, or tiles, of a
larger dataset (see *note Tessellation:: and *note Tessellation
library::).  Each independent tile over the dataset can be connected to
the others as a linked list and thus any number of tiles can be
represented with one variable.

 -- Function:
          void
          gal_list_data_add (gal_data_t **list, gal_data_t *newnode)
     Add an already allocated dataset (‘newnode’) to top of ‘list’.
     Note that if ‘newnode->next!=NULL’ (‘newnode’ is itself a list),
     then ‘list’ will be added to its end.

     In this example multiple images are linked together as a list:
          int quietmmap=1;
          size_t minmapsize=-1;
          gal_data_t *tmp, *list=NULL;
          tmp = gal_fits_img_read("file1.fits", "1", minmapsize, quietmmap,
                                  NULL);
          gal_list_data_add( &list, tmp );
          tmp = gal_fits_img_read("file2.fits", "1", minmapsize, quietmmap,
                                  NULL);
          gal_list_data_add( &list, tmp );

 -- Function:
          void
          gal_list_data_add_alloc (gal_data_t **list, void *array,
          uint8_t type, size_t ndim, size_t *dsize, struct wcsprm *wcs,
          int clear, size_t minmapsize, int quietmmap, char *name, char
          *unit, char *comment)
     Allocate a new dataset (with ‘gal_data_alloc’ in *note Dataset
     allocation::) and put it as the first element of ‘list’.  Note that
     if this is the first node to be added to the list, ‘list’ must be
     ‘NULL’.

 -- Function:
          gal_data_t *
          gal_list_data_pop (gal_data_t **list)
     Pop the top node from ‘list’ and return it.

 -- Function:
          void
          gal_list_data_remove (gal_data_t **list, gal_data_t *node)
     Remove ‘node’ from the given ‘list’.  After finding the given node,
     this function will just set ‘node->next=NULL’ and correct the
     ‘next’ node of its previous element to its next element (thus
     "removing" it from the list).  If ‘node’ doesn't exist in the list,
     this function won't make any change to list.

 -- Function:
          gal_data_t *
          gal_list_data_select_by_name (gal_data_t *list, char *name)
     Select the dataset within the list, that has a ‘name’ element that
     is identical (case-sensitive) to the given ‘name’.  If not found, a
     ‘NULL’ pointer will be returned.

     Note that this dataset will not be popped from the list, only a
     pointer to it will be returned and if you free it or change its
     ‘next’ element, it may harm your original list.

 -- Function:
          gal_data_t *
          gal_list_data_select_by_id (gal_data_t *table, char *idstr,
          size_t *index)
     Select the dataset within the list that can be identified with the
     string given to ‘idstr’ (which can be a counter, starting from 1,
     or a name).  If not found, a ‘NULL’ pointer will be returned.

     Note that this dataset will not be popped from the list, only a
     pointer to it will be returned and if you free it or change its
     ‘next’ element, it may harm your original list.

 -- Function:
          void
          gal_list_data_reverse (gal_data_t **list)
     Reverse the order of the list such that the top node in the list
     before calling this function becomes the bottom node after it.

 -- Function:
          gal_data_t **
          gal_list_data_to_array_ptr (gal_data_t *list, size_t *num)
     Allocate and return an array of ‘gal_data_t *’ pointers with the
     same number of elements as the nodes in ‘list’.  The pointers will
     be put in the same order that the list is parsed.  Hence the N-th
     element in the array will point to the same dataset that the N-th
     node in the list points to.

 -- Function:
          size_t
          gal_list_data_number (gal_data_t *list)
     Return the number of nodes in ‘list’.

 -- Function:
          gal_data_t *
          gal_list_data_last (gal_data_t *list)
     Return a pointer to the last node in ‘list’.

 -- Function:
          void
          gal_list_data_free (gal_data_t *list)
     Free all the datasets in ‘list’ along with all the allocated spaces
     in each.


File: gnuastro.info,  Node: Array input output,  Next: Table input output,  Prev: Linked lists,  Up: Gnuastro library

12.3.9 Array input output
-------------------------

Getting arrays (commonly images or cubes) from a file into your program
or writing them after the processing into an output file are some of the
most common operations.  The functions in this section are designed for
such operations with the known file types.  The functions here are thus
just wrappers around functions of lower-level file type functions of
this library, for example, *note FITS files:: or *note TIFF files::.  If
the file type of the input/output file is already known, you can use the
functions in those sections respectively.

 -- Function:
          int
          gal_array_name_recognized (char *filename)
     Return 1 if the given file name corresponds to one of the
     recognized file types for reading arrays.

 -- Function:
          int
          gal_array_name_recognized_multiext (char *filename)
     Return 1 if the given file name corresponds to one of the
     recognized file types for reading arrays which may contain multiple
     extensions (for example FITS or TIFF) formats.

 -- Function:
          int
          gal_array_file_recognized (char *filename)
     Similar to ‘gal_array_name_recognized’, but for FITS files, it will
     also check the contents of the file if the recognized file name
     suffix is not found.  See the description of
     ‘gal_fits_file_recognized’ for more (*note FITS macros errors
     filenames::).

 -- Function:
          gal_data_t
          gal_array_read (char *filename, char *extension,
          gal_list_str_t *lines, size_t minmapsize, int quietmmap, char
          *hdu_option_name)
     Read the array within the given extension (‘extension’) of
     ‘filename’, or the ‘lines’ list (see below).  If the array is
     larger than ‘minmapsize’ bytes, then it will not be read into RAM,
     but a file on the HDD/SSD (no difference for the programmer).
     Messages about the memory-mapped file can be disabled with
     ‘quietmmap’.

     ‘extension’ will be ignored for files that do not support them (for
     example JPEG or text).  For FITS files, ‘extension’ can be a number
     or a string (name of the extension), but for TIFF files, it has to
     be number.  In both cases, counting starts from zero.

     For multi-channel formats (like RGB images in JPEG or TIFF), this
     function will return a *note List of gal_data_t::: one data
     structure per channel.  Thus if you just want a single array (and
     want to check if the user has not given a multi-channel input), you
     can check the ‘next’ pointer of the returned ‘gal_data_t’.

     ‘lines’ is a list of strings with each node representing one line
     (including the new-line character), see *note List of strings::.
     It will mostly be the output of ‘gal_txt_stdin_read’, which is used
     to read the program's input as separate lines from the standard
     input (see *note Text files::).  Note that ‘filename’ and ‘lines’
     are mutually exclusive and one of them must be ‘NULL’.

     ‘hdu_option_name’ is used in error messages related to extensions
     (e.g., HDUs in FITS) and is expected to be the command-line option
     name that users of your program can specify to select an input HDU
     for this particular input; for example, if the kernel is used as
     the input, users should determine ‘--khdu’ for this option.  If the
     given ‘extension’ doesn't exist in ‘filename’, a descriptive error
     message is printed instructing the users how to find and fix the
     problem.  This error message also suggests the option to use in
     order to help users of your program to inform them what option they
     should give a HDU to.  If you don't have an option that is
     configured by the users of your program, you can set this to ‘NONE’
     or ‘NULL’.

 -- Function:
          void
          gal_array_read_to_type (char *filename, char *extension,
          gal_list_str_t *lines, uint8_t type, size_t minmapsize, int
          quietmmap, char *hdu_option_name)
     Similar to ‘gal_array_read’, but the output data structure(s) will
     have a numeric data type of ‘type’, see *note Numeric data types::.

 -- Function:
          void
          gal_array_read_one_ch (char *filename, char *extension,
          gal_list_str_t *lines, size_t minmapsize, int quietmmap, char
          *hdu_option_name)
     Read the dataset within ‘filename’ (extension/hdu/dir ‘extension’)
     and make sure it is only a single channel.  This is just a simple
     wrapper around ‘gal_array_read’ that checks if there was more than
     one dataset and aborts with an informative error if there is more
     than one channel in the dataset.

     Formats like JPEG or TIFF support multiple channels per input, but
     it may happen that your program only works on a single dataset.
     This function can be a convenient way to make sure that the data
     that comes into your program is only one channel.

     Regarding ‘*hdu_option_name’, see the description for the same
     argument in the description of ‘gal_array_read’.

 -- Function:
          void
          gal_array_read_one_ch_to_type (char *filename, char
          *extension, gal_list_str_t *lines, uint8_t type, size_t
          minmapsize, int quietmmap, char *hdu_option_name)
     Similar to ‘gal_array_read_one_ch’, but the output data structure
     will has a numeric data type of ‘type’, see *note Numeric data
     types::.


File: gnuastro.info,  Node: Table input output,  Next: FITS files,  Prev: Array input output,  Up: Gnuastro library

12.3.10 Table input output (‘table.h’)
--------------------------------------

Tables are a collection of one dimensional datasets that are packed
together into one file.  They are the single most common format to store
high-level (processed) information, hence they play a very important
role in Gnuastro.  For a more thorough introduction, please see *note
Table::.  Gnuastro's Table program, and all the other programs that can
read from and write into tables, use the functions of this section for
reading and writing their input/output tables.  For a simple
demonstration of using the constructs introduced here, see *note Library
demo - reading and writing table columns::.

   Currently only plain text (see *note Gnuastro text table format::)
and FITS (ASCII and binary) tables are supported by Gnuastro.  However,
the low-level table infra-structure is written such that accommodating
other formats is also possible and in future releases more formats will
hopefully be supported.  Please do not hesitate to suggest your favorite
format so it can be implemented when possible.

 -- Macro: GAL_TABLE_DEF_WIDTH_STR
 -- Macro: GAL_TABLE_DEF_WIDTH_INT
 -- Macro: GAL_TABLE_DEF_WIDTH_LINT
 -- Macro: GAL_TABLE_DEF_WIDTH_FLT
 -- Macro: GAL_TABLE_DEF_WIDTH_DBL
 -- Macro: GAL_TABLE_DEF_PRECISION_INT
 -- Macro: GAL_TABLE_DEF_PRECISION_FLT
 -- Macro: GAL_TABLE_DEF_PRECISION_DBL
     The default width and precision for generic types to use in writing
     numeric types into a text file (plain text and FITS ASCII tables).
     When the dataset does not have any pre-set width and precision (see
     ‘disp_width’ and ‘disp_precision’ in *note Generic data
     container::) these will be directly used in C's ‘printf’ command to
     write the number as a string.

 -- Macro: GAL_TABLE_DISPLAY_FMT_STRING
 -- Macro: GAL_TABLE_DISPLAY_FMT_DECIMAL
 -- Macro: GAL_TABLE_DISPLAY_FMT_UDECIMAL
 -- Macro: GAL_TABLE_DISPLAY_FMT_OCTAL
 -- Macro: GAL_TABLE_DISPLAY_FMT_HEX
 -- Macro: GAL_TABLE_DISPLAY_FMT_FIXED
 -- Macro: GAL_TABLE_DISPLAY_FMT_EXP
 -- Macro: GAL_TABLE_DISPLAY_FMT_GENERAL
     The display format used in C's ‘printf’ to display data of
     different types.  The ‘_STRING’ and ‘_DECIMAL’ are unique for
     printing strings and signed integers, they are mainly here for
     completeness.  However, unsigned integers and floating points can
     be displayed in multiple formats:

     Unsigned integer
          For unsigned integers, it is possible to choose from
          ‘_UDECIMAL’ (unsigned decimal), ‘_OCTAL’ (octal notation, for
          example, ‘125’ in decimal will be displayed as ‘175’), and
          ‘_HEX’ (hexadecimal notation, for example, ‘125’ in decimal
          will be displayed as ‘7D’).

     Floating point
          For floating point, it is possible to display the number in
          ‘_FLOAT’ (floating point, for example, ‘1500.345’), ‘_EXP’
          (exponential, for example, ‘1.500345e+03’), or ‘_GENERAL’
          which is the best of the two for the given number.

 -- Macro: GAL_TABLE_FORMAT_INVALID
 -- Macro: GAL_TABLE_FORMAT_TXT
 -- Macro: GAL_TABLE_FORMAT_AFITS
 -- Macro: GAL_TABLE_FORMAT_BFITS
     All the current acceptable table formats to Gnuastro.  The ‘AFITS’
     and ‘BFITS’ represent FITS ASCII tables and FITS Binary tables.
     You can use these anywhere you see the ‘tableformat’ variable.

 -- Macro: GAL_TABLE_SEARCH_INVALID
 -- Macro: GAL_TABLE_SEARCH_NAME
 -- Macro: GAL_TABLE_SEARCH_UNIT
 -- Macro: GAL_TABLE_SEARCH_COMMENT
     When the desired column is not a number, these values determine if
     the string to match, or regular expression to search, be in the
     _name_, _units_ or _comments_ of the column metadata.  These values
     should be used for the ‘searchin’ variables of the functions.

 -- Function:
          uint8_t
          gal_table_displayflt_from_str (char *string)
     Convert the input ‘string’ into one of the
     ‘GAL_TABLE_DISPLAY_FMT_FIXED’ (for fixed-point notation) or
     ‘GAL_TABLE_DISPLAY_FMT_EXP’ (for exponential notation).

 -- Function:
          char *
          gal_table_displayflt_to_str (uint8_t id)
     Convert the input identifier (one of the
     ‘GAL_TABLE_DISPLAY_FMT_FIXED’; for fixed-point notation, or
     ‘GAL_TABLE_DISPLAY_FMT_EXP’; for exponential notation) into a
     standard string that is used to identify them.

 -- Function:
          gal_data_t *
          gal_table_info (char *filename, char *hdu, gal_list_str_t
          *lines, size_t *numcols, size_t *numrows, int *tableformat)

     Store the information of each column of a table into an array of
     meta-data ‘gal_data_t’s.  In a metadata ‘gal_data_t’, the size
     elements are zero (‘ndim=size=0’ and ‘dsize=NULL’) but other
     relevant elements are filled).  See the end of this description for
     the exact components of each ‘gal_data_t’ that are filled.

     The returned array of ‘gal_data_t’s has ‘numcols’ datasets (one
     data structure for each column).  The number of rows in each
     dataset is stored in ‘numrows’ (in a table, all the columns have
     the same number of rows).  The format of the table (e.g., ASCII
     text file, or FITS binary or ASCII table) will be put in
     ‘tableformat’ (macros defined above).  If the ‘filename’ is not a
     FITS file, then ‘hdu’ will not be used (can be ‘NULL’).

     The input must be either a file (specified by ‘filename’) or a list
     of strings (‘lines’).  ‘lines’ is a list of strings with each node
     representing one line (including the new-line character), see *note
     List of strings::.  It will mostly be the output of
     ‘gal_txt_stdin_read’, which is used to read the program's input as
     separate lines from the standard input (see *note Text files::).
     Note that ‘filename’ and ‘lines’ are mutually exclusive and one of
     them must be ‘NULL’.

     In the output datasets, only the meta-data strings (column name,
     units and comments), will be allocated and set as shown below.
     This function is just for column information (meta-data), not
     column contents.

          *restrict array  ->  Blank value (if present, in col's own type).
                     type  ->  Type of column data.
                     ndim  ->  0
                   *dsize  ->  NULL
                     size  ->  0
                quietmmap  ->  ------------
                *mmapname  ->  ------------
               minmapsize  ->  Repeat (length of vector; 1 if not vector).
                     nwcs  ->  ------------
                     *wcs  ->  ------------
                     flag  ->  'GAL_TABLEINTERN_FLAG_*' macros.
                   status  ->  ------------
                    *name  ->  Column name.
                    *unit  ->  Column unit.
                 *comment  ->  Column comments.
                 disp_fmt  ->  'GAL_TABLE_DISPLAY_FMT' macros.
               disp_width  ->  Width of string columns.
           disp_precision  ->  ------------
                    *next  ->  Pointer to next column's metadata
                   *block  ->  ------------

 -- Function:
          void
          gal_table_print_info (gal_data_t *allcols, size_t numcols,
          size_t numrows, char *hdu_option_name)
     Print the column information for all the columns (output of
     ‘gal_table_info’) to standard output.  The output is in the same
     format as this command with Gnuastro Table program (see *note
     Invoking asttable::):
          $ asttable --info table.fits

 -- Function:
          gal_data_t *
          gal_table_read (char *filename, char *hdu, gal_list_str_t
          *lines, gal_list_str_t *cols, int searchin, int ignorecase,
          size_t numthreads, size_t minmapsize, int quietmmap, size_t
          *colmatch, char *hdu_option_name)

     Read the specified columns in a file (named ‘filename’), or list of
     strings (‘lines’) into a linked list of data structures.  If the
     file is FITS, then ‘hdu’ will also be used, otherwise, ‘hdu’ is
     ignored.  For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

     ‘lines’ is a list of strings with each node representing one line
     (including the new-line character), see *note List of strings::.
     It will mostly be the output of ‘gal_txt_stdin_read’, which is used
     to read the program's input as separate lines from the standard
     input (see *note Text files::).  Note that ‘filename’ and ‘lines’
     are mutually exclusive and one of them must be ‘NULL’.

     The information to search for columns should be specified by the
     ‘cols’ list of strings (see *note List of strings::).  The string
     in each node of the list may be a number, an exact match to a
     column name, or a regular expression (in GNU AWK format) enclosed
     in ‘/ /’.  The ‘searchin’ value must be one of the macros defined
     above.  If ‘cols’ is NULL, then this function will read the full
     table.  Also, the ‘ignorecase’ value should be 1 if you want to
     ignore the case of alphabetic characters while matching/searching
     column meta-data (see *note Input output options::).

     For FITS tables, each column will be read independently.  Therefore
     they will be read in ‘numthreads’ CPU threads to greatly speed up
     the reading when there are many columns and rows.  However, this
     only happens if CFITSIO was configured with ‘--enable-reentrant’.
     This test has been done at Gnuastro's configuration time; if so,
     ‘GAL_CONFIG_HAVE_FITS_IS_REENTRANT’ will have a value of 1,
     otherwise, it will have a value of 0.  For more on this macro, see
     *note Configuration information::).  Multi-threaded table reading
     is not currently applicable to other table formats (only for FITS
     tables).

     The output is an individually allocated list of datasets (see *note
     List of gal_data_t::) with the same order of the ‘cols’ list.  Note
     that one column node in the ‘cols’ list might give multiple columns
     (for example, from regular expressions), in this case, the order of
     output columns that correspond to that one input, are in order of
     the table (which column was read first).  So the first requested
     column is the first popped data structure and so on.

     if ‘colmatch!=NULL’, it is assumed to be an array that has at least
     the same number of elements as nodes in the ‘cols’ list.  The
     number of columns that matched each input column will be stored in
     each element.

 -- Function:
          gal_list_sizet_t *
          gal_table_list_of_indexs (gal_list_str_t *cols, gal_data_t
          *allcols, size_t numcols, int searchin, int ignorecase, char
          *filename, char *hdu, size_t *colmatch)
     Returns a list of indices (starting from 0) of the input columns
     that match the names/numbers given to ‘cols’.  This is a low-level
     operation which is called by ‘gal_table_read’ (described above),
     see there for more on each argument's description.  ‘allcols’ is
     the returned array of ‘gal_table_info’.

 -- Function:
          void
          gal_table_comments_add_intro (gal_list_str_t **comments, char
          *program_string, time_t *rawtime)
     Add some basic information to the list of ‘comments’.  This basic
     information includes the following information
        • If the program is run in a Git version controlled directory,
          Git's description is printed (see description under ‘COMMIT’
          in *note Output FITS files::).
        • The calendar time that is stored in ‘rawtime’ (‘time_t’ is C's
          calendar time format defined in ‘time.h’).  You can calculate
          the time in this format with the following expressions:
               time_t rawtime;
               time(&rawtime);
        • The name of your program in ‘program_string’.  If it is
          ‘NULL’, this line is ignored.

 -- Function:
          void
          gal_table_write (gal_data_t *cols, struct gal_fits_list_key_t
          **keylist, gal_list_str_t *comments, int tableformat, char
          *filename, char *extname, uint8_t colinfoinstdout, int
          freekeys)

     Write ‘cols’ (a list of datasets, see *note List of gal_data_t::)
     into a table stored in ‘filename’.  The format of the table can be
     determined with ‘tableformat’ that accepts the macros defined
     above.  When ‘filename==NULL’, the column information will be
     printed on the standard output (command-line).

     If ‘comments!=NULL’, the list of comments (see *note List of
     strings::) will also be printed into the output table.  When the
     output table is a plain text file, every node of ‘comments’ will be
     printed after a ‘#’ (so it can be considered as a comment) and in
     FITS table they will follow a ‘COMMENT’ keyword.

     If a file named ‘filename’ already exists, the operation depends on
     the type of output.  When ‘filename’ is a FITS file, the table will
     be added as a new extension after all existing extensions.  If
     ‘filename’ is a plain text file, this function will abort with an
     error.

     If ‘filename’ is a FITS file, the table extension will have the
     name ‘extname’.

     When ‘colinfoinstdout!=0’ and ‘filename==NULL’ (columns are printed
     in the standard output), the dataset metadata will also printed in
     the standard output.  When printing to the standard output, the
     column information can be piped into another program for further
     processing and thus the meta-data (lines starting with a ‘#’) must
     be ignored.  In such cases, you only print the column values by
     passing ‘0’ to ‘colinfoinstdout’.

 -- Function:
          void
          gal_table_write_log (gal_data_t *logll, char *program_string,
          time_t *rawtime, gal_list_str_t *comments, char *filename, int
          quiet, int format)
     Write the ‘logll’ list of datasets into a table in ‘filename’ (see
     *note List of gal_data_t::).  This function is just a wrapper
     around ‘gal_table_comments_add_intro’ and ‘gal_table_write’ (see
     above).  The ‘format’ should be one of the ‘GAL_TABLE_FORMAT_*’
     macros above.  If ‘quiet’ is non-zero, this function will print a
     message saying that the ‘filename’ has been created.

 -- Function:
          gal_data_t *
          gal_table_col_vector_extract (gal_data_t *vector,
          gal_list_sizet_t *indexs)
     Given the "vector" column ‘vector’ (which is assumed to be a 2D
     dataset), extract the tokens that are identified in the ‘indexs’
     list into a list of one dimensional datasets.  For more on vector
     columns in tables, see *note Vector columns::.

 -- Function:
          gal_data_t *
          gal_table_cols_to_vector (gal_data_t *list)
     Merge the one-dimensional datasets in the given list into one
     2-dimensional dataset that can be treated as a vector column.  All
     the input datasets have to have the same size and type.  For more
     on vector columns in tables, see *note Vector columns::.

 -- Function:
          void
          gal_table_sort (gal_data_t *table, gal_data_t *sortcol,
          uint8_t descending)
     Sort the given ‘table’ by the ‘sortcol’ column in ascending order
     (descending, if ‘descending’ is non-zero).  To sort only a single
     column in place, see ‘gal_statistics_sort_*’ of *note Statistical
     operations::.


File: gnuastro.info,  Node: FITS files,  Next: File input output,  Prev: Table input output,  Up: Gnuastro library

12.3.11 FITS files (‘fits.h’)
-----------------------------

The FITS format is the most common format to store data (images and
tables) in astronomy.  The CFITSIO library already provides a very good
low-level collection of functions for manipulating FITS data.  The
low-level nature of CFITSIO is defined for versatility and portability.
As a result, even a simple and basic operation, like reading an image or
table column into memory, will require a special sequence of CFITSIO
function calls which can be inconvenient and buggy to manage in separate
locations.  To ease this process, Gnuastro's library provides wrappers
for CFITSIO functions.  With these, it much easier to read, write, or
modify FITS file data, header keywords and extensions.  Hence, if you
feel these functions do not exactly do what you want, we strongly
recommend reading the CFITSIO manual to use its great features directly
(afterwards, send us your wrappers so we can include it here for others
to benefit also).

   All the functions and macros introduced in this section are declared
in ‘gnuastro/fits.h’.  When you include this header, you are also
including CFITSIO's ‘fitsio.h’ header.  So you do not need to explicitly
include ‘fitsio.h’ anymore and can freely use any of its macros or
functions in your code along with those discussed here.

* Menu:

* FITS macros errors filenames::  General macros, errors and checking names.
* CFITSIO and Gnuastro types::  Conversion between FITS and Gnuastro types.
* FITS HDUs::                   Opening and getting information about HDUs.
* FITS header keywords::        Reading and writing FITS header keywords.
* FITS arrays - images or cubes::  Reading and writing FITS images or cubes.
* FITS tables::                 Reading and writing FITS tables.


File: gnuastro.info,  Node: FITS macros errors filenames,  Next: CFITSIO and Gnuastro types,  Prev: FITS files,  Up: FITS files

12.3.11.1 FITS Macros, errors and filenames
...........................................

Some general constructs provided by Gnuastro's FITS handling functions
are discussed here.  In particular there are several useful functions
about FITS file names.

 -- Macro: GAL_FITS_MAX_NDIM
     The maximum number of dimensions a dataset can have in FITS format,
     according to the FITS standard this is 999.

 -- Function:
          void
          gal_fits_io_error (int status, char *message)
     If ‘status’ is non-zero, this function will print the CFITSIO error
     message corresponding to status, print ‘message’ (optional) in the
     next line and abort the program.  If ‘message==NULL’, it will print
     a default string after the CFITSIO error.

 -- Function:
          int
          gal_fits_name_is_fits (char *name)
     If the ‘name’ is an acceptable CFITSIO FITS filename return ‘1’
     (one), otherwise return ‘0’ (zero).  The currently acceptable FITS
     suffixes are ‘.fits’, ‘.fit’, ‘.fits.gz’, ‘.fits.Z’, ‘.imh’,
     ‘.fits.fz’.  IMH is the IRAF format which is acceptable to CFITSIO.

 -- Function:
          int
          gal_fits_suffix_is_fits (char *suffix)
     Similar to ‘gal_fits_name_is_fits’, but only for the suffix.  The
     suffix does not have to start with '<.>': this function will return
     ‘1’ (one) for both ‘fits’ and ‘.fits’.

 -- Function:
          int
          gal_fits_file_recognized (char *name)
     Return ‘1’ if the given file name (possibly including its contents)
     is a FITS file.  This is necessary when the contents of a FITS file
     do follow the FITS standard, but the file does not have a
     Gnuastro-recognized FITS suffix.  Therefore, it will first call
     ‘gal_fits_name_is_fits’, if the result is negative, then this
     function will attempt to open the file with CFITSIO and if it
     works, it will close it again and return 1.  In the process of
     opening the file, CFITSIO will just open the file, and no reading
     will take place, so it should have a minimal CPU footprint.

 -- Function:
          char *
          gal_fits_name_save_as_string (char *filename, char *hdu)
     If the name is a FITS name, then put a ‘(hdu: ...)’ after it and
     return the string.  If it is not a FITS file, just print the name,
     if ‘filename==NULL’, then return the string ‘stdin’.  Note that the
     output string's space is allocated.

     This function is useful when you want to report a random file to
     the user which may be FITS or not (for a FITS file, simply the
     filename is not enough, the HDU is also necessary).


File: gnuastro.info,  Node: CFITSIO and Gnuastro types,  Next: FITS HDUs,  Prev: FITS macros errors filenames,  Up: FITS files

12.3.11.2 CFITSIO and Gnuastro types
....................................

Both Gnuastro and CFITSIO have special and different identifiers for
each type that they accept.  Gnuastro's type identifiers are fully
described in *note Library data types:: and are usable for all kinds of
datasets (images, table columns, etc) as part of Gnuastro's *note
Generic data container::.  However, following the FITS standard, CFITSIO
has different identifiers for images and tables.  Following CFITSIO's
own convention, we will use ‘bitpix’ for image type identifiers and
‘datatype’ for its internal identifiers (and mainly used in tables).
The functions introduced in this section can be used to convert between
CFITSIO and Gnuastro's type identifiers.

   One important issue to consider is that CFITSIO's types are not fixed
width (for example, ‘long’ may be 32-bits or 64-bits on different
systems).  However, Gnuastro's types are defined by their width.  These
functions will use information on the host system to do the proper
conversion.  To have a portable (usable on different systems) code, is
thus recommended to use these functions and not to assume a fixed
correspondence between CFITSIO and Gnuastro's types.

 -- Function:
          uint8_t
          gal_fits_bitpix_to_type (int bitpix)
     Return the Gnuastro type identifier that corresponds to CFITSIO's
     ‘bitpix’ on this system.

 -- Function:
          int
          gal_fits_type_to_bitpix (uint8_t type)
     Return the CFITSIO ‘bitpix’ value that corresponds to Gnuastro's
     ‘type’.

 -- Function:
          char
          gal_fits_type_to_bin_tform (uint8_t type)
     Return the FITS standard binary table ‘TFORM’ character that
     corresponds to Gnuastro's ‘type’.

 -- Function:
          int
          gal_fits_type_to_datatype (uint8_t type)
     Return the CFITSIO ‘datatype’ that corresponds to Gnuastro's ‘type’
     on this machine.

 -- Function:
          uint8_t
          gal_fits_datatype_to_type (int datatype, int is_table_column)
     Return Gnuastro's type identifier that corresponds to the CFITSIO
     ‘datatype’.  Note that when dealing with CFITSIO's ‘TLONG’, the
     fixed width type differs between tables and images.  So if the
     corresponding dataset is a table column, put a non-zero value into
     ‘is_table_column’.


File: gnuastro.info,  Node: FITS HDUs,  Next: FITS header keywords,  Prev: CFITSIO and Gnuastro types,  Up: FITS files

12.3.11.3 FITS HDUs
...................

A FITS file can contain multiple HDUs/extensions.  The functions in this
section can be used to get basic information about the extensions or
open them.  Note that ‘fitsfile’ is defined in CFITSIO's ‘fitsio.h’
which is automatically included by Gnuastro's ‘gnuastro/fits.h’.

 -- Function:
          fitsfile *
          gal_fits_open_to_write (char *filename)
     If ‘filename’ exists, open it and return the ‘fitsfile’ pointer
     that corresponds to it.  If ‘filename’ does not exist, the file
     will be created which contains a blank first extension and the
     pointer to its next extension will be returned.

 -- Function:
          size_t
          gal_fits_hdu_num (char *filename)
     Return the number of HDUs/extensions in ‘filename’.

 -- Function:
          unsigned long
          gal_fits_hdu_datasum (char *filename, char *hdu, char
          *hdu_option_name)
     Return the ‘DATASUM’ of the given HDU in the given FITS file.  For
     more on ‘DATASUM’ in the FITS standard, see *note Keyword
     inspection and manipulation:: (under the ‘checksum’ component of
     ‘--write’).  For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

 -- Function:
          unsigned long
          gal_fits_hdu_datasum_encoded (char *filename, char *hdu, char
          *hdu_option_name)
     Similar to ‘gal_fits_hdu_datasum’, but the returned value is always
     a 16-character string following the encoding that is described in
     the FITS standard (primarily for the ‘CHECKSUM’ keyword, but can
     also be used for ‘DATASUM’.

 -- Function:
          unsigned long
          gal_fits_hdu_datasum_ptr (fitsfile *fptr)
     Return the ‘DATASUM’ of the already opened HDU in ‘fptr’.  For more
     on ‘DATASUM’ in the FITS standard, see *note Keyword inspection and
     manipulation:: (under the ‘checksum’ component of ‘--write’).

 -- Function:
          int
          gal_fits_hdu_format (char *filename, char *hdu, char
          *hdu_option_name)
     Return the format of the HDU as one of CFITSIO's recognized macros:
     ‘IMAGE_HDU’, ‘ASCII_TBL’, or ‘BINARY_TBL’.  For more on
     ‘hdu_option_name’ see the description of ‘gal_array_read’ in *note
     Array input output::.

 -- Function:
          int
          gal_fits_hdu_is_healpix (fitsfile *fptr)
     Return ‘1’ if the dataset may be a HEALpix grid and ‘0’ otherwise.
     Technically, it is considered to be a HEALPix if the HDU is not an
     ASCII table, and has the ‘NSIDE’, ‘FIRSTPIX’ and ‘LASTPIX’.

 -- Function:
          fitsfile *
          gal_fits_hdu_open (char *filename, char *hdu, int iomode, int
          exitonerror, char *hdu_option_name)
     Open the HDU/extension ‘hdu’ from ‘filename’ and return a pointer
     to CFITSIO's ‘fitsfile’.  ‘iomode’ determines how the FITS file
     will be opened using CFITSIO's macros: ‘READONLY’ or ‘READWRITE’.

     The string in ‘hdu’ will be appended to ‘filename’ in square
     brackets so CFITSIO only opens this extension.  You can use any
     formatting for the ‘hdu’ that is acceptable to CFITSIO. See the
     description under ‘--hdu’ in *note Input output options:: for more.

     If ‘exitonerror!=0’ and the given HDU cannot be opened for any
     reason, the function will exit the program, and print an
     informative message.  Otherwise, when the HDU cannot be opened, it
     will just return a NULL pointer.  For more on ‘hdu_option_name’ see
     the description of ‘gal_array_read’ in *note Array input output::.

 -- Function:
          fitsfile *
          gal_fits_hdu_open_format (char *filename, char *hdu, int
          img0_tab1, char *hdu_option_name)
     Open (in read-only format) the ‘hdu’ HDU/extension of ‘filename’ as
     an image or table.  When ‘img0_tab1’ is ‘0’(zero) but the HDU is a
     table, this function will abort with an error.  It will also abort
     with an error when ‘img0_tab1’ is ‘1’ (one), but the HDU is an
     image.  For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

     A FITS HDU may contain both tables or images.  When your program
     needs one of these formats, you can call this function so if the
     user provided the wrong HDU/file, it will abort and inform the user
     that the file/HDU is has the wrong format.


File: gnuastro.info,  Node: FITS header keywords,  Next: FITS arrays - images or cubes,  Prev: FITS HDUs,  Up: FITS files

12.3.11.4 FITS header keywords
..............................

Each FITS extension/HDU contains a raw dataset which can either be a
table or an image along with some header keywords.  The keywords can be
used to store meta-data about the actual dataset.  The functions in this
section describe Gnuastro's high-level functions for reading and writing
FITS keywords.  Similar to all Gnuastro's FITS-related functions, these
functions are all wrappers for CFITSIO's low-level functions.

   The necessary meta-data (header keywords) for a particular dataset
are commonly numerous, it is much more efficient to list them in one
variable and call the reading/writing functions once.  Hence the
functions in this section use linked lists, a thorough introduction to
them is given in *note Linked lists::.  To reading FITS keywords, these
functions use a list of Gnuastro's generic dataset format that is
discussed in *note List of gal_data_t::.  To write FITS keywords we
define the ‘gal_fits_list_key_t’ node that is defined below.

 -- Type (C struct): gal_fits_list_key_t
     Structure for writing FITS keywords.  This structure is used for
     one keyword and you do not need to set all elements.  With the
     ‘next’ element, you can link it to another keyword thus creating a
     linked list to add any number of keywords easily and at any step
     during your program (see *note Linked lists:: for an introduction
     on lists).  See the functions below for adding elements to the
     list.

          typedef struct gal_fits_list_key_t
          {
            int                        tfree; /* ==1, free title string.  */
            int                        kfree; /* ==1, free keyword name.  */
            int                        vfree; /* ==1, free keyword value. */
            int                        cfree; /* ==1, free comment.       */
            int                        ufree; /* ==1, free unit.          */
            uint8_t                     type; /* Keyword value type.      */
            char                      *title; /* !=NULL, only print title.*/
            char                    *keyname; /* Keyword Name.            */
            void                      *value; /* Keyword value.           */
            char                    *comment; /* Keyword comment.         */
            char                       *unit; /* Keyword unit.            */
            struct gal_fits_list_key_t *next; /* Pointer next keyword.    */
          } gal_fits_list_key_t;

 -- Function:
          int
          gal_fits_key_exists_fptr (fitsfile *fptr, char *keyname)
     Return 1 (true) if the opened FITS file pointer contains the
     requested keyword and 0 (false) otherwise.

 -- Function:
          void *
          gal_fits_key_img_blank (uint8_t type)
     Returns a pointer to an allocated space containing the value to the
     FITS ‘BLANK’ header keyword, when the input array has a type of
     ‘type’.  This is useful when you want to write the ‘BLANK’ keyword
     using CFITSIO's ‘fits_write_key’ function.

     According to the FITS standard: "If the ‘BSCALE’ and ‘BZERO’
     keywords do not have the default values of 1.0 and 0.0,
     respectively, then the value of the ‘BLANK’ keyword must equal the
     actual value in the FITS data array that is used to represent an
     undefined pixel and not the corresponding physical value".
     Therefore a special ‘BLANK’ value is needed for datasets containing
     signed 8-bit, unsigned 16-bit, unsigned 32-bit, and unsigned 64-bit
     integers (types that are defined with ‘BSCALE’ and ‘BZERO’ in the
     FITS standard).

     *Not usable when reading a dataset:* As quoted from the FITS
     standard above, the value returned by this function can only be
     generically used for the writing of the ‘BLANK’ keyword header.  It
     _must not_ be used as the blank pointer when when reading a FITS
     array using CFITSIO. When reading an array with CFITSIO, you can
     use ‘gal_blank_alloc_write’ to generate the necessary pointer.

 -- Function:
          void
          gal_fits_key_clean_str_value (char *string)
     Remove the single quotes and possible extra spaces around the
     keyword values that CFITSIO returns when reading a string keyword.
     CFITSIO does not remove the two single quotes around the string
     value of a keyword.  Hence the strings it reads are like: ‘'value
     '’, or ‘'some_very_long_value'’.  To use the value during your
     processing, it is commonly necessary to remove the single quotes
     (and possible extra spaces).  This function will do this within the
     allocated space of the string.

 -- Function:
          char *
          gal_fits_key_date_to_struct_tm (char *fitsdate, struct tm *tp)
     Parse ‘fitsdate’ as a FITS date format string (most generally:
     ‘YYYY-MM-DDThh:mm:ss.ddd...’) into the C library's broken-down time
     structure, or ‘struct tm’ (declared in ‘time.h’) and return a
     pointer to a newly allocated array for the sub-second part of the
     format (‘.ddd...’).  Therefore it needs to be freed afterwards (if
     it is not ‘NULL’) When there is no sub-second portion, this pointer
     will be ‘NULL’.

     This is a relatively low-level function, an easier function to use
     is ‘gal_fits_key_date_to_seconds’ which will return the sub-seconds
     as double precision floating point.

     Note that the FITS date format mentioned above is the most complete
     representation.  The following two formats are also acceptable:
     ‘YYYY-MM-DDThh:mm:ss’ and ‘YYYY-MM-DD’.  This option can also
     interpret the older FITS date format where only two characters are
     given to the year and the date format is reversed
     (‘DD/MM/YYThh:mm:ss.ddd...’).  In this case (following the GNU C
     Library), this option will make the following assumption: values 68
     to 99 correspond to the years 1969 to 1999, and values 0 to 68 as
     the years 2000 to 2068.

 -- Function:
          size_t
          gal_fits_key_date_to_seconds (char *fitsdate, char
          **subsecstr, double *subsec)
     Return the Unix epoch time (number of seconds that have passed
     since 00:00:00 Thursday, January 1st, 1970) corresponding to the
     FITS date format string ‘fitsdate’ (see description of
     ‘gal_fits_key_date_to_struct_tm’ above).  This function will return
     ‘GAL_BLANK_SIZE_T’ if the broken-down time could not be converted
     to seconds.

     The Unix epoch time is in units of seconds, but the FITS date
     format allows sub-second accuracy.  The last two arguments are for
     the optional sub-second portion.  If you do not want sub-second
     information, just set the second argument to ‘NULL’.

     If ‘fitsdate’ contains sub-second accuracy and ‘subsecstr!=NULL’,
     then the starting of the sub-second part's string is stored in
     ‘subsecstr’ (malloc'ed), and ‘subsec’ will be the corresponding
     numerical value (between 0 and 1, in double precision floating
     point).  So to avoid leaking memory, if a sub-second string is
     requested, it must be freed after calling this function.  When a
     sub-second string does not exist (and it is requested), then a
     value of ‘NULL’ and NaN will be written in ‘*subsecstr’ and
     ‘*subsec’ respectively.

     This is a very useful function for operations on the FITS date
     values, for example, sorting FITS files by their dates, or finding
     the time difference between two FITS files.  The advantage of
     working with the Unix epoch time is that you do not have to worry
     about calendar details (such as the number of days in different
     months or leap years).

 -- Function:
          void
          gal_fits_key_read_from_ptr (fitsfile *fptr, gal_data_t
          *keysll, int readcomment, int readunit)

     Read the list of keyword values from a FITS pointer.  The input
     should be a linked list of Gnuastro's generic data container
     (‘gal_data_t’).  Before calling this function, you just have to set
     the ‘name’, and optionally, the desired ‘type’ of the value of each
     keyword.  The given ‘name’ value will be directly passed to CFITSIO
     to read the desired keyword name.  This function will allocate
     space to keep the value.  If no pre-defined type is requested for a
     certain keyword's value, the smallest possible type to host the
     value will be found and used.  If ‘readcomment’ and ‘readunit’ are
     non-zero, this function will also try to read the possible comments
     and units of the keyword.

     Here is one example of using this function:

          /* Allocate an array of datasets. */
          gal_data_t *keysll=gal_data_array_calloc(N);

          /* Make the array usable as a list too (by setting `next'). */
          for(i=0;i<N-1;++i) keysll[i].next=&keysll[i+1];

          /* Fill the datasets with a `name' and a `type'. */
          keysll[0].name="NAME1";     keysll[0].type=GAL_TYPE_INT32;
          keysll[1].name="NAME2";     keysll[1].type=GAL_TYPE_STRING;
          ...
          ...

          /* Call this function. */
          gal_fits_key_read_from_ptr(fptr, keysll, 0, 0);

          /* Use the values as you like... */

          /* Free all the allocated spaces. Note that `name' was not
             allocated in this example, so we should explicitly set
             it to NULL before calling `gal_data_array_free'. */
          for(i=0;i<N;++i) keysll[i].name=NULL;
          gal_data_array_free(keysll, N, 1);

     If the ‘array’ pointer of each keyword's dataset is not ‘NULL’,
     then it is assumed that the space to keep the value has already
     been allocated.  If it is ‘NULL’, space will be allocated for the
     value by this function.

     Strings need special consideration: the reason is that generally,
     ‘gal_data_t’ needs to also allow for array of strings (as it
     supports arrays of integers for example).  Hence when reading a
     string value, two allocations may be done by this function (one if
     ‘array!=NULL’).

     Therefore, when using the values of strings after this function,
     ‘keysll[i].array’ must be interpreted as ‘char **’: one allocation
     for the pointer, one for the actual characters.  If you use
     something like the example, above you do not have to worry about
     the freeing, ‘gal_data_array_free’ will free both allocations.  So
     to read a string, one easy way would be the following:

          char *str, **strarray;
          strarr = keysll[i].array;
          str    = strarray[0];

     If CFITSIO is unable to read a keyword for any reason the ‘status’
     element of the respective ‘gal_data_t’ will be non-zero.  If it is
     zero, then the keyword was found and successfully read.  Otherwise,
     it is a CFITSIO status value.  You can use CFITSIO's error
     reporting tools or ‘gal_fits_io_error’ (see *note FITS macros
     errors filenames::) for reporting the reason of the failure.  A
     tip: when the keyword does not exist, CFITSIO's status value will
     be ‘KEY_NO_EXIST’.

     CFITSIO will start searching for the keywords from the last place
     in the header that it searched for a keyword.  So it is much more
     efficient if the order that you ask for keywords is based on the
     order they are stored in the header.

 -- Function:
          void
          gal_fits_key_read (char *filename, char *hdu, gal_data_t
          *keysll, int readcomment, int readunit, char *hdu_option_name)
     Same as ‘gal_fits_read_keywords_fptr’ (see above), but accepts the
     filename and HDU as input instead of an already opened CFITSIO
     ‘fitsfile’ pointer.

 -- Function:
          void
          gal_fits_key_list_add (gal_fits_list_key_t **list, uint8_t
          type, char *keyname, int kfree, void *value, int vfree, char
          *comment, int cfree, char *unit, int ufree)
     Add a keyword to the top of list of header keywords that need to be
     written into a FITS file.  In the end, the keywords will have to be
     freed, so it is important to know beforehand if they were allocated
     or not (hence the presence of the arguments ending in ‘free’).  If
     the space for the respective element is not allocated, set these
     arguments to ‘0’ (zero).

     You can call this function multiple times on a single list add
     several keys that will be written in one call to
     ‘gal_fits_key_write’ or ‘gal_fits_key_write_in_ptr’.  However, the
     resulting list will be a last-in-first-out list (for more on lists,
     see *note Linked lists::).  Hence, the written keys will have the
     inverse order of your calls to this function.  To avoid this
     problem, you can either use ‘gal_fits_key_list_add_end’ instead
     (which will add each key to the end of the list, not to the top
     like this function).  Alternatively, you can use
     ‘gal_fits_key_list_reverse’ after adding all the keys with this
     function.

     *Important note for strings*: the value should be the pointer to
     the string itself (‘char *’), not a pointer to a pointer (‘char
     **’).

 -- Function:
          void
          gal_fits_key_list_add_end (gal_fits_list_key_t **list, uint8_t
          type, char *keyname, int kfree, void *value, int vfree, char
          *comment, int cfree, char *unit, int ufree)
     Similar to ‘gal_fits_key_list_add’, but add the given keyword to
     the end of the list, see the description of ‘gal_fits_key_list_add’
     for more.  Use this function if you want the keywords to be written
     in the same order that you add nodes to the list of keywords.

 -- Function:
          void
          gal_fits_key_list_title_add (gal_fits_list_key_t **list, char
          *title, int tfree)
     Add a special "title" keyword (with the ‘title’ string) to the top
     of the keywords list.  If ‘cfree’ is non-zero, the space allocated
     for ‘comment’ will be freed immediately after writing the keyword
     (in another function).

 -- Function:
          void
          gal_fits_key_list_title_add_end (gal_fits_list_key_t **list,
          char *title, int tfree)
     Similar to ‘gal_fits_key_list_title_add’, but put the comments at
     the end of the list.

 -- Function:
          void
          gal_fits_key_list_fullcomment_add (gal_fits_list_key_t **list,
          char *comment, int fcfree)
     Add a ‘COMMENT’ keyword to the top of the keywords list.  If the
     comment is longer than 70 characters, CFITSIO will automatically
     break it into multiple ‘COMMENT’ keywords.  If ‘fcfree’ is
     non-zero, the space allocated for ‘comment’ will be freed
     immediately after writing the keyword (in another function).

 -- Function:
          void
          gal_fits_key_list_fullcomment_add_end (gal_fits_list_key_t
          **list, char *comment, int fcfree)
     Similar to ‘gal_fits_key_list_comment_add’, but put the comments at
     the end of the list.

 -- Function:
          void
          gal_fits_key_list_add_date (gal_fits_list_key_t **keylist,
          char *comment)
     Add a ‘DATE’ keyword to the input list of keywords containing the
     date this function was activated in the format of
     ‘YYYY-MM-DDThh:mm:ss’.  This function will also add a ‘DATEUTC’
     keyword that specifies if the date is in UTC or local time (this
     depends on CFITSIO being able to detect UTC in the running
     operating system or not).

     The comment of the keyword should also be specified as the second
     argument.  The comment is useful to inform users what this date
     refers to; for example the program starting time, its ending time,
     or etc.  For more, see the description under ‘DATE’ in *note Output
     FITS files::.

 -- Function:
          void
          gal_fits_key_list_add_software_versions (gal_fits_list_key_t
          **keylist)
     Add the version of Gnuastro *note Mandatory dependencies:: to the
     list of keywords.  Each software's keyword has the same name as the
     software itself (for example ‘GNUASTRO’ or ‘GSL’.  For the full
     list of software, see *note Output FITS files::.

 -- Function:
          void
          gal_fits_key_list_add_git_commit (gal_fits_list_key_t
          **keylist)
     If the optional libgit2 dependency is installed and your program is
     being run in a directory that is under version control, a ‘COMMIT’
     keyword will be added on the top of the list of keywords.  For
     more, see the description of ‘COMMIT’ in *note Output FITS files::.

 -- Function:
          void
          gal_fits_key_list_reverse (gal_fits_list_key_t **list)
     Reverse the input list of keywords.

 -- Function:
          void
          gal_fits_key_write_title_in_ptr (char *title, fitsfile *fptr)
     Add two lines of "title" keywords to the given CFITSIO ‘fptr’
     pointer.  The first line will be blank and the second will have the
     string in ‘title’ roughly in the middle of the line (a fixed
     distance from the start of the keyword line).  A title in the list
     of keywords helps in classifying the keywords into groups and
     inspecting them by eye.  If ‘title==NULL’, this function will not
     do anything.

 -- Function:
          void
          gal_fits_key_write_filename (char *keynamebase, char
          *filename, gal_fits_list_key_t **list, int top1end0, int
          quiet)
     Put ‘filename’ into the ‘gal_fits_list_key_t’ list (possibly broken
     up into multiple keywords) to later write into a HDU header.  The
     ‘keynamebase’ string will be appended with a ‘_N’ (N>0) and used as
     the keyword name.  If ‘top1end0!=0’, then the keywords containing
     the filename will be added to the top of the list.

     The FITS standard sets a maximum length of 69 characters for the
     string values of a keyword(1).  This creates problems with file
     names (which include directories) because file names/addresses can
     become longer than the maximum number of characters in a FITS
     keyword (around 70 characters).  Therefore, when ‘filename’ is
     longer than the maximum length of a FITS keyword value, this
     function will break it into several keywords (breaking up the
     string on directory separators).  So the full file/directory
     address (including directories) can be longer than 69 characters.
     However, if a single file or directory name (within a larger
     address) is longer than 69 characters, this function will truncate
     the name and print a warning.  If ‘quiet!=0’, then the warning will
     not be printed.

 -- Function:
          void
          gal_fits_key_write_wcsstr (fitsfile *fptr, struct wcsprm wcs,
          char *wcsstr, int nkeyrec)

     Write the WCS header string (produced with WCSLIB's ‘wcshdo’
     function) into the CFITSIO ‘fitsfile’ pointer.  ‘nkeyrec’ is the
     number of FITS header keywords in ‘wcsstr’.  This function will put
     a few blank keyword lines along with a comment ‘WCS information’
     before writing each keyword record.

 -- Function:
          void
          gal_fits_key_write (gal_fits_list_key_t *keylist, char
          *filename, char *hdu, char *hdu_option_name, int freekeys, int
          create_fits_not_exists)
     Write the list of keywords in ‘keylist’ into the ‘hdu’ extension of
     the file called ‘filename’.  If the file may not exist before this
     function is activated, set ‘create_fits_not_exists’ to non-zero and
     set the HDU to ‘"0"’.  If the keywords should be freed after they
     are written, set the ‘freekeys’ value to non-zero.  For more on
     ‘hdu_option_name’ see the description of ‘gal_array_read’ in *note
     Array input output::.

     The list nodes are meant to be dynamically allocated (because they
     will be freed after being written).  We thus recommend using the
     ‘gal_fits_key_list_add’ or ‘gal_fits_key_list_add_end’ to create
     and fill the list.  Below is one fully working example of using
     this function to write a keyword into an existing FITS file.

          #include <stdio.h>
          #include <stdlib.h>
          #include <gnuastro/fits.h>

          int main()
          {
            char *filename="test.fits";
            gal_fits_list_key_t *keylist=NULL;

            char *unit="unit";
            float value=123.456;
            char *keyname="MYKEY";
            char *comment="A good description of the key";
            gal_fits_key_list_add_end(&keylist, GAL_TYPE_FLOAT32, keyname, 0,
                                      &value, 0, comment, 0, unit, 0);
            gal_fits_key_list_title_add(&keylist, "Matching metadata", 0);
            gal_fits_key_write(keylist, filename, "1", "NONE", 1, 0);
            return EXIT_SUCCESS;
          }

 -- Function:
          void
          gal_fits_key_write_in_ptr (gal_fits_list_key_t *keylist,
          fitsfile *fptr, int freekeys)
     Write the list of keywords in ‘keylist’ into the given CFITSIO
     ‘fitsfile’ pointer and free keylist.  For more on the input
     ‘keylist’, see the description and example for
     ‘gal_fits_key_write’, above.

 -- Function:
          gal_list_str_t *
          gal_fits_with_keyvalue (gal_list_str_t *files, char *hdu, char
          *name, gal_list_str_t *values, char *hdu_option_name)
     Given a list of FITS file names (‘files’), a certain HDU (‘hdu’), a
     certain keyword name (‘name’), and a list of acceptable values
     (‘values’), return the subset of file names where the requested
     keyword name has one of the acceptable values.  For more on
     ‘hdu_option_name’ see the description of ‘gal_array_read’ in *note
     Array input output::.

 -- Function:
          gal_list_str_t *
          gal_fits_unique_keyvalues (gal_list_str_t *files, char *hdu,
          char *name, char *hdu_option_name)
     Given a list of FITS file names (‘files’), a certain HDU (‘hdu’), a
     certain keyword name (‘name’), return the list of unique values to
     that keyword name in all the files.  For more on ‘hdu_option_name’
     see the description of ‘gal_array_read’ in *note Array input
     output::.

   ---------- Footnotes ----------

   (1) The limit is actually 71 characters (which is the full 80
character length, subtracted by 8 for the keyword name and one character
for the <=>).  However, for strings, FITS also requires two single
quotes.


File: gnuastro.info,  Node: FITS arrays - images or cubes,  Next: FITS tables,  Prev: FITS header keywords,  Up: FITS files

12.3.11.5 FITS arrays (images)
..............................

Images (or multi-dimensional arrays in general) are one of the common
data formats that is stored in FITS files.  Only one image may be stored
in each FITS HDU/extension.  The functions described here can be used to
get the information of, read, or write images in FITS files.

 -- Function:
          void
          gal_fits_img_info (fitsfile *fptr, int *type, size_t *ndim,
          size_t **dsize, char **name, char **unit)
     Read the type (see *note Library data types::), number of
     dimensions, and size along each dimension of the CFITSIO ‘fitsfile’
     into the ‘type’, ‘ndim’, and ‘dsize’ pointers respectively.  If
     ‘name’ and ‘unit’ are not ‘NULL’ (point to a ‘char *’), then if the
     image has a name and units, the respective string will be put in
     these pointers.

 -- Function:
          size_t *
          gal_fits_img_info_dim (char *filename, char *hdu, size_t
          *ndim, char *hdu_option_name)
     Put the number of dimensions in the ‘hdu’ extension of ‘filename’
     in the space that ‘ndim’ points to and return the size of the
     dataset along each dimension as an allocated array with ‘*ndim’
     elements.  For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

 -- Function:
          gal_data_t *
          gal_fits_img_read (char *filename, char *hdu, size_t
          minmapsize, int quietmmap, char *hdu_option_name)
     Read the contents of the ‘hdu’ extension/HDU of ‘filename’ into a
     Gnuastro generic data container (see *note Generic data
     container::) and return it.  If the necessary space is larger than
     ‘minmapsize’, then do not keep the data in RAM, but in a file on
     the HDD/SSD. For more on ‘minmapsize’ and ‘quietmmap’ see the
     description under the same name in *note Generic data container::.
     For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

     Note that this function only reads the main data within the
     requested FITS extension, the WCS will not be read into the
     returned dataset.  To read the WCS, you can use ‘gal_wcs_read’
     function as shown below.  Afterwards, the ‘gal_data_free’ function
     will free both the dataset and any WCS structure (if there are
     any).
          data=gal_fits_img_read(filename, hdu, -1, 1, NULL);
          data->wcs=gal_wcs_read(filename, hdu, 0, 0, 0, &data->wcs->nwcs,
                                 NULL);

 -- Function:
          gal_data_t *
          gal_fits_img_read_to_type (char *inputname, char *inhdu,
          uint8_t type, size_t minmapsize, int quietmmap, char
          *hdu_option_name)
     Read the contents of the ‘hdu’ extension/HDU of ‘filename’ into a
     Gnuastro generic data container (see *note Generic data
     container::) of type ‘type’ and return it.

     This is just a wrapper around ‘gal_fits_img_read’ (to read the
     image/array of any type) and ‘gal_data_copy_to_new_type_free’ (to
     convert it to ‘type’ and free the initially read dataset).  See the
     description there for more.

 -- Function:
          gal_data_t *
          gal_fits_img_read_kernel (char *filename, char *hdu, size_t
          minmapsize, int quietmmap, char *hdu_option_name)
     Read the ‘hdu’ of ‘filename’ as a convolution kernel.  A
     convolution kernel must have an odd size along all dimensions, it
     must not have blank (NaN in floating point types) values and must
     be flipped around the center to make the proper convolution (see
     *note Convolution process::).  If there are blank values, this
     function will change the blank values to ‘0.0’.  If the input image
     does not have the other two requirements, this function will abort
     with an error describing the condition to the user.  The finally
     returned dataset will have a ‘float32’ type.

     For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

 -- Function:
          fitsfile *
          gal_fits_img_write_to_ptr (gal_data_t *input, char *filename,
          gal_fits_list_key_t *keylist, int freekeys)
     Write the ‘input’ dataset into a FITS file named ‘filename’ and
     return the corresponding CFITSIO ‘fitsfile’ pointer.  This function
     will not close ‘fitsfile’, so you can still add other extensions to
     it after this function or make other modifications.

     In case you want to add keywords into the HDU that contain the
     data, you can use the second two arguments (see the description of
     ‘gal_fits_key_write’).  These keywords will be written into the HDU
     before writing the data: when there are more than roughly 5
     keywords (assuming your dataset has WCS) and your dataset is large,
     this can result in significant optimization of the running time
     (because adding a keyword beyond the 36 key slots will cause the
     whole data to shift for another block of 36 keywords).

 -- Function:
          void
          gal_fits_img_write (gal_data_t *input, char *filename,
          gal_fits_list_key_t *keylist, int freekeys)
     Write the ‘input’ dataset into the FITS file named ‘filename’.
     Also add the list of header keywords (‘keylist’) to the newly
     created HDU/extension The list of keywords will be freed after
     writing into the HDU, if you need them later, keep a separate copy
     of the list before calling this function.

     For the importance of why it is better to add your keywords in this
     function (before writing the data) or after it, see the description
     of ‘gal_fits_img_write_to_ptr’.

 -- Function:
          void
          gal_fits_img_write_to_type (gal_data_t *input, char *filename,
          gal_fits_list_key_t *keylist, int type, int freekeys)
     Convert the ‘input’ dataset into ‘type’, then write it into the
     FITS file named ‘filename’.  Also add the ‘keylist’ keywords to the
     newly created HDU/extension along with your program's name
     (‘program_string’).  After the FITS file is written, this function
     will free the copied dataset (with type ‘type’) from memory.

     For the importance of why it is better to add your keywords in this
     function (before writing the data) or after it, see the description
     of ‘gal_fits_img_write_to_ptr’.  This is just a wrapper for the
     ‘gal_data_copy_to_new_type’ and ‘gal_fits_img_write’ functions.

 -- Function:
          void
          gal_fits_img_write_corr_wcs_str (gal_data_t *input, char
          *filename, char *wcsstr, int nkeyrec, double *crpix,
          gal_fits_list_key_t *keylist, int freekeys)
     Write the ‘input’ dataset into ‘filename’ using the ‘wcsstr’ while
     correcting the ‘CRPIX’ values.  For the importance of why it is
     better to add your keywords in this function (before writing the
     data) or after it, see the description of
     ‘gal_fits_img_write_to_ptr’.

     This function is mainly useful when you want to make FITS files in
     parallel (from one main WCS structure, with just a differing
     CRPIX), for more on the arguments, see the description of
     ‘gal_fits_img_write’.  This can happen in the following cases for
     example:

        • When a large number of FITS images (with WCS) need to be
          created in parallel, it can be much more efficient to write
          the header's WCS keywords once at first, write them in the
          FITS file, then just correct the CRPIX values.

        • WCSLIB's header writing function is not thread safe.  So when
          writing FITS images in parallel, we cannot write the header
          keywords in each thread.


File: gnuastro.info,  Node: FITS tables,  Prev: FITS arrays - images or cubes,  Up: FITS files

12.3.11.6 FITS tables
.....................

Tables are one of the common formats of data that is stored in FITS
files.  Only one table may be stored in each FITS HDU/extension, but
each table column must be viewed as a different dataset (with its own
name, units and numeric data type for example).  The only constraint of
the column datasets in a table is that they must be one-dimensional and
have the same number of elements as the other columns.  The functions
described here can be used to get the information of, read, or write
columns into FITS tables.

 -- Function:
          void
          gal_fits_tab_size (fitsfile *fitsptr, size_t *nrows, size_t
          *ncols)
     Read the number of rows and columns in the table within CFITSIO's
     ‘fitsptr’.

 -- Function:
          int
          gal_fits_tab_format (fitsfile *fitsptr)
     Return the format of the FITS table contained in CFITSIO's
     ‘fitsptr’.  Recall that FITS tables can be in binary or ASCII
     formats.  This function will return ‘GAL_TABLE_FORMAT_AFITS’ or
     ‘GAL_TABLE_FORMAT_BFITS’ (defined in *note Table input output::).
     If the ‘fitsptr’ is not a table, this function will abort the
     program with an error message informing the user of the problem.

 -- Function:
          gal_data_t *
          gal_fits_tab_info (char *filename, char *hdu, size_t *numcols,
          size_t *numrows, int *tableformat, char *hdu_option_name)
     Store the information of each column in ‘hdu’ of ‘filename’ into an
     array of data structures with ‘numcols’ elements (one data
     structure for each column) see *note Arrays of datasets::.  The
     total number of rows in the table is also put into the memory that
     ‘numrows’ points to.  The format of the table (e.g., FITS binary or
     ASCII table) will be put in ‘tableformat’ (macros defined in *note
     Table input output::).  For more on ‘hdu_option_name’ see the
     description of ‘gal_array_read’ in *note Array input output::.

     This function is just for column information.  Therefore it only
     stores meta-data like column name, units and comments.  No actual
     data (contents of the columns for example, the ‘array’ or ‘dsize’
     elements) will be allocated by this function.  This is a low-level
     function particular to reading tables in FITS format.  To be
     generic, it is recommended to use ‘gal_table_info’ which will allow
     getting information from a variety of table formats based on the
     filename (see *note Table input output::).

 -- Function:
          gal_data_t *
          gal_fits_tab_read (char *filename, char *hdu, size_t numrows,
          gal_data_t *colinfo, gal_list_sizet_t *indexll, size_t
          numthreads, size_t minmapsize, int quietmmap, char
          *hdu_option_name)
     Read the columns given in the list ‘indexll’ from a FITS table (in
     ‘filename’ and HDU/extension ‘hdu’) into the returned linked list
     of data structures, see *note List of size_t:: and *note List of
     gal_data_t::.  For more on ‘hdu_option_name’ see the description of
     ‘gal_array_read’ in *note Array input output::.

     Each column will be read independently, therefore they will be read
     in ‘numthreads’ CPU threads to greatly speed up the reading when
     there are many columns and rows.  However, this only happens if
     CFITSIO was configured with ‘--enable-reentrant’.  This test has
     been done at Gnuastro's configuration time; if so,
     ‘GAL_CONFIG_HAVE_FITS_IS_REENTRANT’ will have a value of 1,
     otherwise, it will have a value of 0.  For more on this macro, see
     *note Configuration information::).

     If the necessary space for each column is larger than ‘minmapsize’,
     do not keep it in the RAM, but in a file in the HDD/SSD. For more
     on ‘minmapsize’ and ‘quietmmap’, see the description under the same
     name in *note Generic data container::.

     Each column will have ‘numrows’ rows and ‘colinfo’ contains any
     further information about the columns (returned by
     ‘gal_fits_tab_info’, described above).  Note that this is a
     low-level function, so the output data linked list is the inverse
     of the input indexes linked list.  It is recommended to use
     ‘gal_table_read’ for generic reading of tables, see *note Table
     input output::.

 -- Function:
          void
          gal_fits_tab_write (gal_data_t *cols, gal_list_str_t
          *comments, int tableformat, char *filename, char *extname,
          gal_fits_list_key_t *keywords, int freekeys)
     Write the list of datasets in ‘cols’ (see *note List of
     gal_data_t::) as separate columns in a FITS table in ‘filename’.
     If ‘filename’ already exists then this function will write the
     table as a new extension called ‘extname’, after all existing ones.
     The format of the table (ASCII or binary) may be specified with the
     ‘tableformat’ (see *note Table input output::).  If
     ‘comments!=NULL’, each node of the list of strings will be written
     as a ‘COMMENT’ keywords in the output FITS file (see *note List of
     strings::.

     In case your table needs metadata keywords, you can use the
     ‘listkeys’ and ‘freekeys’.  For more on these, see the description
     of ‘gal_fits_key_write_in_ptr’.

     This is a low-level function for tables.  It is recommended to use
     ‘gal_table_write’ for generic writing of tables in a variety of
     formats, see *note Table input output::.

