Here I want to present my new book on advanced algorithms for data-intensive applications named "Probabilistic Data Structures and Algorithms in Big Data Applications" (ISBN: 9783748190486). The detailed information about the book you can find at its webpage and below I give you some introduction to the topic this book is about.…

In the “Ecology of Metrics,” I wrote about “alignment” being a type of metric; alignment can measure the extent to which an organization’s supply or capacity is matched against the demands or needs of the market.  For instance, in a call centre, it would be highly desirable to have agents available to respond to calls at “precisely” the same time that clients are making calls.  If alignment is off even by only 15 to 30 seconds, impatient clients might hang up and never call again.  Similarly…

Fish schools, bird flocks, and bee swarms. These combinations of real-time biological systems can blend knowledge, exploration, and exploitation to unify intelligence and solve problems more efficiently. There’s no centralized control. These simple agents interact locally, within their environment, and new behaviors emerge from the group as a whole. In the world of…

I once posted about making use of narrative objects.  In this blog, I will be discussing an algorithm that supports the creation of these objects.  I call it my “Infereferencing Algorithm”: this term is most easily pronounced with a slight pause between “infer” and “referencing.”  I consider this a useful and widely applicable algorithm although I don’t believe it operates well in a relational database environment.  Instead, I use “mass data files”:  these contain unstructured lumps of…

At times it might be desirable to determine what storylines on a narrative database contain certain key elements.  In the example below, the object “Alien Allan” is used to compile a listing of titles on the database that are in thematic agreement.  But rather than simply provide the listing of titles, the algorithm tabulates the key elements contained in these titles.  Notice that the most important elements resulting from the search do not necessarily match the elements in the submitted…

When I was beginning my way in data science, I often faced the problem of choosing the most appropriate algorithm for my specific problem. If you’re like me, when you open some article about machine learning algorithms, you see dozens of detailed descriptions. The paradox is that they don’t ease the choice.

In this article, I will try to explain basic concepts and give some intuition of using different…

Random Forests algorithm has always fascinated me. I like how this algorithm can be easily explained to anyone without much hassle. One quick example, I use very frequently to explain the working of random forests is the way a company has multiple rounds of interview to hire a candidate. Let me elaborate. 

Say, you appeared for the position of Statistical analyst at WalmartLabs. Now like most of the companies, you don't just have one round of interview. You…

I was wondering how to approach this blog when I decided to toast some raisin-bread for breakfast.  Shortly after I started eating it, I began coughing.  I have shared this “type” of story with a few people.  I often continue eating something regardless of my exact circumstances - that is to say, oblivious to the consequences.  As I ate more of the raisin-bread and my coughing worsened, I eventually reached for my inhaler.  (I have been diagnosed with borderline asthma now by two doctors.) …

When discussing the use of algorithms, the issue of durability or portability has to be considered. For example, a stock trading algorithm might be used in a missile guidance system. The algorithm would have to operate on an abstract kinetic level rather than for a specific application. I have written in the past about using the same algorithm to study stocks, earthquakes, hurricanes, electro-cardiograms, and attempts at evasion - using my mouse in a game environment. Wouldn't an abstraction…

Last year, I wrote a blog on "mass data assignments." For readers that lack a prototype or application to handle data using mass data assignments, the topic probably seems a bit evasive. In this blog, I will be reinforcing and developing…

I made a recent discovery that I would like to share with the community. In my previous blog, I introduced a special algorithmic shell that distributes stocks based on their price movements (along the x-axis) and volume movements (y-axis). Using this shell, it is possible to visualize the trading behaviours of dozens of stocks simultaneously. I noticed one day that the stocks seemed to be lining up in formation. I decided to test the accuracy of my visual interpretation. Below I present the…

In my previous blog on the Hopscotch and Robots simulation environment, I discussed the use of structural data extracted from hypothetical and real-life organizational events. In the current blog, I will be briefly covering conceptual issues more focused on the nature of the structural data itself including its theoretical significance.

Structural data holds information about the relationship between events.…

When you use Twitter, how do you know when you are being presented with something credible instead of something totally bogus? The answer is, unless you spend a lot of time researching each tweet, you probably don’t. However, one thing is for certain, we rely on what we read on Twitter to be true.

Twitter is one of the fastest and most effective ways we disseminate news across our world. If this…

Thermometers and scales to measure weight appeared in retail outlets long ago. Blood pressure monitors perhaps came later. Pedometers and heart-rate monitors seem more recent - possibly closer to my time. I saw several devices while doing this blog intended to electronically record among other things hours of sleep; these are designed to be worn on the body all the time. A couple of weeks ago, I bought something to give the heart rate and blood oxygen saturation level. I consider it a real…

Many years ago, I attended a vocational college to learn skilled trade. I was taught about the behaviour of systems. I learned that after renovations to a house, the furnace might cycle on and off more frequently; this can leave some parts of the house too cold. A wood-burning stove or fireplace should be treated as a part of a system. Open doors and windows in the dwelling can cause exhaust from such appliances to enter living spaces. I realize that these particular examples of systems…

There was a television interview of a man who had to deal with severe breathing problems through much of his adult life.  He underwent various procedures although he found little relief.  In fact, the adverse impact to his body was long-lasting.  So he was surprised to discover how he was extremely allergic to his breakfast cereal, something that he ate almost without fail.  When he stopped eating the cereal, his breathing problems disappeared.  I am sure that many people would regard the…

Embodiment is comparable to the idea of an “ecosystemic” or “holistic” approach. In an ecosystem, each thing affects everything else. In light of the interrelationship, a person would not attempt to correct a problem by considering only a single piece of the puzzle. Instead, there is a need to bring together many aspects of the body. To understand embodiment, it is necessary to recognize how “the body” separates an organism from its environment; in a manner of speaking, the body represents…

In this blog, I share some images from an application called Storm.  I wrote the program many years ago.  Storm has the ability to generate 3-dimensional plumes from a stream of data.  It also has an unusual feature that allows the user to trade based on the kinetics - effectively eliminating the need to know about pricing.  At this time, I would like to draw a clear distinction between trading and investing.  I should also point out that I used Storm for recreational…

There are plenty of discussions about what data science is, what defines a data scientist, and how to position yourself as a …