Image Source.
The third Systemic Challenge closed to entries on Friday, and I’ve gone through and evaluated the submitted fits. The results were very encouraging. Eight out of twenty-five submissions corresponded to both the correct orbital configuration and the correct number of planets in the underlying dynamical model.
For challenge 003, we looked to our own solar system for inspiration, and tapped the four Gallilean satellites of Jupiter. Eugenio writes:
The system is a scaled-up version of Jupiter and the four Galilean satellites. To generate the model, I first set the central mass to 1 solar mass. The (astrocentric) period of Callisto was set to 365.25 days, and I required that the mass and (astrocentric) period ratios in the system would remain the same. Here’s the resulting model (using Jacobi elements, with i~88 deg):
| Parameter | “Io” | “Europa” | “Ganymede” | “Callisto” |
|---|---|---|---|---|
| Period (days) | 38.77079 | 77.77920 | 156.65300 | 365.42094 |
| Mass (Jupiters) | 0.04926 | 0.02646 | 0.08175 | 0.05936 |
| Mean Anomaly (deg) | 99.453 | 50.772 | 285.591 | 47.538 |
| eccentricity | 0.003989 | 0.009792 | 0.001935 | 0.007547 |
| omega (deg) | 31.229 | 205.427 | 303.460 | 359.879 |
Among the eight entries that got both the total number of planets and their periods correct, there was a fair amount of variation among fits that had nearly equivalent values for the chi-square statistic. Chuck Smith (among others) turned in a configuration that bears a very strong resemblance to the actual input system. The four planets in his fit all have nearly circular orbits:
and the resulting radial velocity curve does a very good job of running through the data, with a chi-square value for the integrated fit equal to 1.1005:
A number of other users turned in very similar configurations.
Because of random measurement errors in the data, the true underlying planetary configuration will not necessarily provide the best fit to a given set of radial velocity observations. Often, a better fit can be found for a configuration that is different from the system that generated the data. Steve Undy, for example, achieved a slightly lower chi-square value for his fit by giving a very significant eccentricity to his “Europa”:
The winner of the contest, however, was Eric Diaz, who submitted a 6-planet fit that achieves an integrated chi-square value of 1.04. In addition to the four planets that are actually present in the model, Eric added small planets with periods of 1.06 days and 18.11 days. These objects soaked up some of the residual noise in the fit, allowing for a lower chi-square value, and a copy of the Sky and Telescope star atlas. Nice job Eric!
The contest raises some interesting issues. First, at what point should one stop adding planets to a fit? The chi-square statistic penalizes the inclusion of additional free parameters in a fit, but it’s clear that chi-square can nearly always be lowered by adding additional small bodies to the fit. Second, its very encouraging to see that subtle, but substantially non-interacting systems can be pulled out of radial velocity data sets. In this system, the masses of the planets are small enough so that their dynamical interactions with eachother are not significant over the time-frame that the system is observed. This is in stark contrast to systems such as GJ 876 and 55 Cancri where it is vital to take interactions into account (by fitting with the integrate button clicked on). Finally, I think that we’ll soon see examples of the 1:2:4 Laplace resonance as competitive fits within the existing catalog of radial velocity data sets on the systemic backend.
“… First, at what point should one stop adding planets to a fit? …”
I guess improving integrated chi-square from 1.1 to 1.04 can never be used to claim discovery of two (or even one) new planets. So I wonder in real discoveries, what’s the standard for claiming to discover a new (small or distant) planet in a system already with a couple of planets? Another question I have is, if you have, for example, an interacting system with 3 planets, could a fit of a non-interacting system with 4 (or more) degenerate with that of the real system? Or vice versus. Again, in the published claims on discovering additional planets using RV data, is it possible that similar degeneracies will happen?
Thanks.
wl-
There is a statistical way of determining whether adding extra planets is justifiably significant. The reduced chi-square is NOT enough. You must do an ***F-test*** between the chi-square values of, say, a 4-planet and a 6-planet solution. This is how it is done in the scientific community. I am positive that adding the last two planets is NOT statistically significant and that they are just fitting noise. (We also happen to know this is true because we also know the actual system parameters.) I made a comment on this earlier (http://oklo.org/?p=112) that has more details.
For real planetary systems, adding extra planets introduce gravitaitonal instabilities and will often cause the ‘integrated’ chi-squared to go down, as long as enough time has elapsed in the system. (In this systemic challenge, not enough time has gone by. Since Eric’s planets are so small, it would likely take several years before the gravitational interactions lead to a bad fit.)
In terms of degenerate systems, that is one of the great goals of the systemic project (in my opinion) and from this challenge system, we’ve already learned something. There have been some systems that have had problems with your question, where there is a difference between an interacting and non-interacting fit… oklo.org’s posts on mu Arae, for example, illustrate this difficulty. More than one scientist out there is spending much of thier life on these very problems (besides the systemic team, Eric Ford comes to mind).
In the end, we have to come to terms with the fact that our data will always have noise (not to mention systematic errors) and will always be sparse (not enough data). So, we use chi-squared and F-tests to get the best guesses of the true nature of these planetary systems… although the real distinguishing power lies in the human mind. (Which is why the systemic console needs users… a machine just isn’t smart enough to get its own good fits without decades of brute force computation.)
Hi Darin,
I agree completely. We’re working on putting an F-test (along with a bootstrap-based uncertainty estimator) into the console. We won’t have this completed for Systemic Jr (due out in a few weeks) but it should be implemented by the time the full Systemic simulation is released.
best,
Greg
Greg:
There is another problem that, I think, is important. It is the problem of missing data. The console, for example, does not consider missing data; that is, it considers that the data set is complete. This can be seen by giving the console only data for part of an orbit. It will try to fit an excentric planet to allow for the zeros in the orbit.
I think that the biggest improvement you can make in the console would be to solve this problem; that is, that the mathematics look at the data only as part of the orbit, ignoring the places where there is no data.
My two cents,
Best
Mike
Pingback: systemic - challenge 4