Friday, January 3, 2020

            I promised today would be something a bit more juicy, and maybe hinted that I myself would take on a greater
            form, of the late lyrical wordsmith, Biggie Smalls.  But alas, I'm not artist, and certainly don't have the
            skill and game that he had.  So maybe instead I'll gravitate to a spicy issue, that is more near and dear to
            my heart. To which, I will almost certainly do a dis-justice, but he, this is America, and I'm allowed to
            butcher things. That issue is, meteorological forecast models.

            I'll start by saying, meteorology is hard. I know what you're thinking, because I hear it all the time;
            "But Paul, how can it be hard if they get paid to be wrong 60% of the time?"  First off, fuck you.  If you
            honestly think that, eat shit and die, and don't read this blog anymore.  Second, expand your mind for
            just a second and consider the breadth of the problem Meteorologists are attempting to solve.  The earth,
            is really big.  And the atmosphere, encircles the entirety of it.  To boot, we sample a hilariously small
            portion of that enormous space, so at any point in time, the view we have of the atmosphere is partial.
            So what does that have to do with forecasting?  Well, if I told you to chart your way out of maze, but the
            map I gave you, only knew 5% of the map, and the maze was constantly changing, you'd probably struggle to
            find your way out.  That's the job of a forecaster.

            So, we have don't have a complete picture of the current state of the atmosphere at any given time.  So
            what do we have?  We have an incredibly detailed understanding of physics, dynamics, and how they work
            together on our planet to create weather. Since the onset of the "Meteorological era" (considered to begin
            largely with scientists like Carl-Gustaf Rossby in the early 1900s) we've understood a lot of the physics.
            Now obviously our understanding of that has improved over the years, but the fundamentals havn't changed.
            What has changed is computers, and our ability to numerical process the complex equations that describe
            the fluid flow in the atmosphere (yes the atmosphere is a fluid).

            Okay, so we've increased or ability to solve the equations with computers. Cool.  We've programmed those
            equations into what we call Forecast Models.  They take an input set of data, i.e. the current state of the
            atmosphere, and forecast (based on solving a bunch of those fundamentals equations) the future state of the
            atmosphere. Even if only one number in our initial state was wrong, that would introduce error, that over
            time through the future states, amplifies and could lead to a wildly erroneous state in the future. So
            that's problem number 1, lack of overall observations.

            The second problem, is simple.  The people who've spent their live studying maths, dynamics, and physics,
            are the meteorologists.  They understand the way the atmosphere works better than anyone else.  They are
            also usually really shitty programmers.  That's not meant to be offensive (sorry met friends, but it's true)
            it's just a fact.  Most humans are either okay at lots of things, or REALLY REALLY good at one thing.
            Meteorologists are REALLY good at meteorology.  They are usually okay at programming.  This, in my mind, has
            led us to a point where models are nowhere near what they could be.

            I once  had the honor of speaking directly with the Direct of the ECMWF (largely considered to be the
            premier forecasting agency in the world), who was visiting my senior capstone class.  During that talk, I
            asked him how many computer scientists (i.e. people who are REALLY REALLY good at programming) they
            employed. His response; 0.  They had hundreds of Meteorologists, 0 Computer Scientists.  That answer not
            only perplexed me, but honestly offended me (as a student at the time of both worlds).  Why would someone
            who specialized so deeply, and understood the deep specialized knowledge you get from that effort, assume
            that they could do a completely separate task, that also requires a specialist?

            To me it's akin to asking your plumber to do heart surgery on you.  Anyway, I've done a terrible job at
            making my point in time, and I'm out of time and really want to go to bed.  So maybe I'll re-visit this
            tomorrow.

            The take away here.  Forecasting groups (NWS, ECMWF) and various private agencies need to hire more
            computer scientists.  On a final note, the company I worked for at my first job,
            Weather Decision Technologies, did a fantastic job of this.  We had a very pleasant mix of both sciences
            on staff.  And we made some really cool products and advances because of that. SO ya WDT.