Identifying Hidden Structure (in Data) and Computing Knowledge

We live in the digital age. True.  The flow of data is impressive. Statistics inform us that each year we generate more data than the past generations did in decades. But the problem is that Information Technology (IT) has concentrated on simply automating old ways of thinking, creating bottlenecks and problems we didn't even imagine, and not really inventing new proccesses or approaches. There is little innovation going on in the IT arena. But one thing is certain, we are drowning in data. The problem is not simply storage (disk space is cheap). The big deal is how to extract workable knowledge out of all this data. But what is knowledge? What is a "body of knowledge"? Setting aside ontological hairsplitting, we could say that a body of knowledge is equivalent to a structured and dynamic set of inter-related rules. The rules can be crisp or fuzzy or both. But the key here is structure. Structure is the skeleton upon which a certain body of knowledge can be further expanded, refined, modified (this is why we say "dynamic"). One could say that structure forms the basis of a model or of a theory. Today, there exist many ways of extracting structure from data. Statistics is one way. Building models based on data is another. But because building models and mis-handling of statistics has contributed to the destruction of a big chunk of our economy, we have invented a new method of identifying structure in data - a model-free method, which if free of statistics and building models. A method which is "natural" and un-biased. Let's see an example of what we mean when we say "from data to knowledge". Consider a piece of ordinary data (such as that managed by accountants in any corporation or a bank, or by an investor): The data (only a portion is illustrated) has this structure, known as a Business Structure Map: What the map illustrates is the complete set of dependencies (we like the word "relations") between the business parameters. These relations are, de facto, rules. Rules of the type "if A increases then B decreases". So the above map does represent a body of knowledge. In this particular example it represents the knowledge of the functioning of a large multi-national software firm as reflected in the financials it publishes on a quarterly basis. In order to understand how to navigate such maps read here. It is easy and intuitive. The point now is this. When you make decisions based on data, such as the example illustrated above, what do you do about its structure? Do you take it into account? Probably not. Many managers do have a feeling of how their companies work. But intuition is one thing, science is another. In turbulence, intuition, even when backed by years of successful practice, can fail. We say that: There should be a Business Structure Map on the table during every Board Meeting The above map has around two dozen parameters and about a hundred rules. In our quarterly analysis of the Eurozone economy, we analyze a total of 648 parameters (24 parameters for each of the 27 member states). The corresponding Business Structure Map has approximately forty thousand rules! That gives an idea of the immense difficulty which fixing the EU will entail. www.ontonix.com