Hi Roger,
Here is a quick note since I haven't checked in in
a few days.
Been working hard.
Things are starting to look very interesting
indeed.
I have been working on two fronts.
The big news is that I have looked at the alternate
second data and there is a very clear correlation between the Stouffer z-sqrd
cumdev [and its Janus twin, the network variance] for two datasets with
alternating seconds of data. That is all odd seconds go to one set and the even
seconds go to the other. There is a strong correlation which is exactly what one
expects if an anomalous effect is responsible for the structure in the
data. [The significance depends on the numbers and I'm calculating
empirical distributions for my correlation function. So it will take a day or
two to have more precision.]
But it's better than that.
The correlation only comes through for the
post-9/11 data.
And it is coming from the same structure that we
see correlating with the poll results. So this alternate second result is
independent of the poll result and also supportive of it ( or visa vera)
Anyway it is starting to make a very nice story, since we now have
1. event experiment significant for an effect on
short timecsales and connecting to global scales.
2. alternate datasets showing significance on long
timescales.
3. poll results connecting long time behavior to
global events.
That's the short of it.
The attached html doc has some more details and
some more are to come.
The technical point of importance is that I've
devised a correlation test capable of detecting correlations in structure [as
opposed to the too limited usual correlation coeffcients that miss detailed
structure like peaks]. This is sketched in the doc.
The other front is the poll results. I'll use my
structure correlation test to look at the data/poll correlation quantitatively.
But another very interesting avenue is to study the correspondance by making a
mapping from one to the other. I fiddled around a bit and it looks possible, but
it will take some work. It would be a nice demo to show how you can generate the
structure in the GCP data by a simple transformation of the poll
data.
I think we have some red meat.
At this point we should reconsider the late
April date for a meeting. This could be the good moment to do it
and we could still make it if we move fast. Read over the doc and let me
know what you think.
best,
Peter