![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
15_the_circle) wrote2007-06-16 05:06 pm
trance state
[OT from cottage renovations]
so what the heck has been going on the past few days?
previously in this space I've written about this notion I've had of the potential use of railroad signal data as telemetry. it probably comes from my years at Intelsat back when it was the international treaty organisation that owned and operated the global communications satellite network (I use the past tense because a great deal of restructuring went on since I was there; the name persists but things are quite different these days).
the wires running along the tracks used by railroads for the signal system were called the code line so though the information now travels in a different way the web site where the details can be found is called www.codeline-telemetry.com.
for a proof-of-concept demonstration I have temporarily expanded the geographic scope of the application that checks regional signal data for anomalies. it's a completely insane volume of data. yesterday it pulled in signal indication messages from 742 railroad control points on four carriers across the eastern US and Canada. upstairs there are now four computers devoted to the project: two with radios for data collection, one for front ending ten such computers in the midwest and another that's aggregating all of the data into a logfile that will probably hit around 100Mb by midnight. although the two radio computers are just a few inches away in the rack it can't be bothered to talk to them directly so they are feeding an external aggregator in West Virginia into whose data stream their outputs are blended. another three such aggregators are hosted by collaborators in Florida and Virginia as well as another sixteen servers from which it is polling directly.
the machine doing all this is running flat out so I had to suspend the cron job that runs the daily logfile analysis; that gets started by hand on one of the downstairs computers. when it runs by itself this application is generally remarkably low in bandwidth consumption but for this test it has occasionally come close to saturating the DSL line out of here.
I'm hoping to run in this mode for a couple more days; if the moving parts can keep functioning through midnight on Monday that should suffice.
it was putting the pieces together to make this test possible that has been so distracting. not so much the computational and network resources; for the most part they were already in place though a certain amount of repurposing was called for. rather, it was building enough context into which the data can be dropped such that it can have a way to make sense.
it was years ago when I first learned about the danger of wandering into information space.
the way it works for me is that I have to be able to assimilate the entity relationships of whatever symbolic information I'm working with; it's all highly visual but once the mental data model is in place it becomes possible to manipulate it to find whatever it is I'm looking for.
the process is difficult to describe but suffice it to say that on seeing it a certain film made immediate sense to me.
I don't qualify as a genius, nor as a savant or a polymath.
but data structures have a meaning for me and working with them comes naturally -- but is accompanied by a danger that is real, very real: that of falling into them and not necessarily having a way back out.
don't ask me how I know this but the answer should be obvious.
I'll have more on this later, probably in a few days.
no subject
no subject
I had written, in part:
>
> a logfile that will probably hit around 100Mb by midnight
>
I was wrong. it was only 98-and-some-change-Mb.
so perhaps it isn't a completely insane amount of data: that 1+% margin is comforting in its own way.
no subject
I like all your uber-geek talk. However, I'm wondering if your upstairs room has walls plastered with newspaper clippings, .... John Nash.
no subject
>
> ... pretty soon I'll need a server just to host logfiles.
>
don't laugh; as you know perfectly well any good sized data centre has whole server farms, databases and SANs devoted to logfiles. all of them, in turn, generating their own logfiles, which is about the point where any sensible person gets a headache.
how a site or how a sysadmin handles logfiles is one of those quick but highly revealing indicators that tells you a lot of you need to know about them.
if it makes you feel any better, in this case the logfiles excreted by one process were fed to another as a primary data source. somehow a data file that big seems so much easier to deal with if it can be thought of as data rather than a log.
>
> I like all your uber-geek talk. However, I'm wondering if your upstairs
> room has walls plastered with newspaper clippings, .... John Nash.
>
uh, no. no clippings and no coloured strings. but it does have a rack with the aforementioned computers and radio gear (most of which will actually be finding new homes in places where better reception is better) and there is a red-white-and-green carved dragon hanging from a rafter. he took refuge upstairs when the refenestration work first started downstairs and seems to be happier up there though I have to be careful when making the bed or changing sheets because he has a loose paw that seems to fall off with little or no provocation.
no subject
Uh oh.