Corrected AIC for X-ray model fitting

Akaike information criterion (AIC) is a popular general classical method for model comparison (see the book: Ivezić et al. 2020; or the original paper of Akaike: Akaike 1974 ).

The motivation for using AIC is simple, which is we need some tools to compare different models, i.e., which model is not that good and not substantially supported by the data.

What is AIC?

The AIC is defined as follows,

$$\mathrm{AIC}\equiv -2\ln (L^0(M))+2k,$$

where $L^0(M)$ is the maximum value of the likelihood of model $M$, $k$ is the number of parameters of the model $M$. The second term of $2k$ is a must to make AIC an unbiased estimator of Kullback–Leibler discrepancy, which quantifies the discrepancy between a selected model and the real mechanism (more details and definitions can be found in Cavanaugh & Neath 2019) Additionally, $2k$ can be simply regarded as a penalty term for the number of parameters. Adopting a more complicated model, i.e., the model with larger $k$, will usually result in a smaller $-2\ln (L^0(M))$ but increase the second term $2k$, and hence there should be a trade-off. The smaller the AIC, the better the model is, and the model with the smallest AIC is the best model. Note that it cannot guarantee the best model can describe the real mechanism well and it just performs better than any else adopted model.

What is corrected AIC?

For small samples, such as $N/k\lesssim 40$, where the $N$ is the number of data points, there should be another penalty term for the number of parameters (Sugiura 1978), which reads

$${\mathrm{AIC}}_c\equiv -2\ln (L^0(M))+2k+\frac{2k(k+1)}{N-k-1}=\mathrm{AIC}+\frac{2k(k+1)}{N-k-1}.$$

The last term is the second-order effect that should be taken into consideration when the sample size is small but will be negligible when $N$ is very large. A model with a number of parameters close to the number of data points will be rejected since the last term will tend to infinity when $k$ approaches $N-1$. If the last term is committed, the model with a large number may not be rejected for a small sample. Therefore, ${\mathrm{AIC}}_c$ is better than $\mathrm{AIC}$. Since $N/k\lesssim 40$ is usually satisfied in the context of X-ray model fitting, ${\mathrm{AIC}}_c$ will always be used.

How to calculate ${\mathrm{AIC}}_c$ when different statistics are adopted?

${\chi }^2$ statistics

For statistics based on Gaussian likelihood:

$$L(M)=\prod _i\frac{1}{\sqrt{2\pi }{\sigma }_i}\exp \left(\frac{-(D_i-M_i{)}^2}{2{\sigma }_i^2}\right),$$

where $L(M)$ is the likelihood for the model $M$, $D_i$ is the data in $i$th bin, $M_i$ denotes the model prediction in $i$th bin, and ${\sigma }_i$ is the error for the $i$th bin. ${\chi }^2$ is defined as follows,

$${\chi }^2\equiv \sum _i\frac{(D_i-M_i{)}^2}{{\sigma }_i^2},$$

which obeys the Chi-squared distribution. The relation of ${\chi }^2$ and $L(M)$ is straightforward,

$${\chi }^2=-2\ln L(M)+\mathrm{constant}.$$

Since only the relative value of ${\mathrm{AIC}}_c$ between different models is important, the first term is the minimal value of ${\chi }^2$ for a certain model, namely,

$${\mathrm{AIC}}_c={\chi }_{\min }^2+2k+\frac{2k(k+1)}{N-k-1}\mathrm{for}{\chi }^2\mathrm{statistics}.$$

C-statistics

For statistics based on Poisson likelihood:

$$L(M)=\prod _i\frac{M_i^{D_i}}{D_i!}\exp (-M_i)$$

Taking its logarithm and multiplying by $-2$, we can get

$$-2\ln L(M)=2\sum _i(M_i-D_i\ln M_i+\ln (D_i!)).$$

Omitting the factorial term, we can get the Cash-statistics (Cash 1979):

$$\tilde{C}=2\sum _i(M_i-D_i\ln M_i).$$

Approximating the factorial term by Stirling's formula, that is

$$\ln (D_i!)\approx D_i\ln D_i-D_i,$$

a modification of the original Cash-statistic, C-statistics, can be obtained as follows:

$$C=2\sum _i(M_i-D_i\ln M_i+D_i\ln D_i-D_i),$$

which is implemented in some popular fitting packages like XSPEC (Arnaud 1996), SHERPA (Freeman et al. 2001), and SPEX (Kaastra et al. 1996). $\tilde{C}$ is the same as $C$, up to a constant $\sum _i(D_i\ln D_i-D_i)$. $C$ is non-negative, $C$ is equal to $0$ if and only if all the $M_i$ are equal to $D_i$. Since the count rate is usually low for X-ray observation, it is better to use C-statistics fitting than ${\chi }^2$ fitting, i.e., getting the best-fit parameters by minimizing $C$ instead of ${\chi }^2$. The corresponding ${\mathrm{AIC}}_c$ is

$${\mathrm{AIC}}_c=C_{\min }+2k+\frac{2k(k+1)}{N-k-1}\mathrm{for}\mathrm{C}-\mathrm{statistics}.$$

As for the calculation of ${\mathrm{AIC}}_c$, the first term should be ${\chi }_{\min }^2$ if ${\chi }^2$ fitting is used, while it should be $C_{\min }$ if C-statistics fitting is implemented (Peca et al. 2023; Ng et al. 2022; Rogantini et al. 2019; Igoshev et al. 2018).

For a given model, its $\mathrm{\Delta }{\mathrm{AIC}}_c$ is defined as the relative difference of ${\mathrm{AIC}}_c$ between the minimum ${\mathrm{AIC}}_c$, which reads,

$$\mathrm{\Delta }{\mathrm{AIC}}_c\equiv {\mathrm{AIC}}_c-{\mathrm{AIC}}_{c,\min }.$$

The model with the minimum ${\mathrm{AIC}}_c$ is the most preferred model, but the rest models can be simply ruled out. Generally, the model with $\mathrm{\Delta }{\mathrm{AIC}}_c>2$ is not substantially supported by the observation data and may not be reserved (Burnham et al. 2002). The following table (Burnham et al. 2002) shows the level of empirical support for different $\mathrm{\Delta }{\mathrm{AIC}}_c$.

The model whose $\mathrm{\Delta }{\mathrm{AIC}}_c$ within 2 is substantially supported by the data, i.e., the model with $\mathrm{\Delta }{\mathrm{AIC}}_c$ within 2 is not significantly different from the best model and cannot be ruled out. However, the model whose $\mathrm{\Delta }{\mathrm{AIC}}_c$ is larger than 2 is considerably less supported by the data, i.e., the model with $\mathrm{\Delta }{\mathrm{AIC}}_c$ larger than 2 is probably not the best model and can be ruled out. The model whose $\mathrm{\Delta }{\mathrm{AIC}}_c$ is larger than 10 is essentially not supported by the data, you should discard such a model.

$\mathrm{\Delta }{\mathrm{AIC}}_c$	Level of Empirical Support
0-2	Substantial
4-7	Considerably less
>10	Essentially none

Attention

The corrected AIC can be used if you cannot rule out some models from the perspective of physics. Namely, corrected AIC just rules out the models from the perspective of statistics. If you can rule out some models from the perspective of physics, please rule out them first.