As we all know, the dominant data tool in the world of metallurgical analysis is MS Excel. Spreadsheets are how data is first captured, communicated around, charted and analyzed, and stored long term. Our metallurgical files are filled with spreadsheets of various degrees of complexity.
It's easy to understand why. The data is right there. You can see it, change it, graph it, perform all kinds of calculations etc. Excel is truly the ubiquitous data tool for our industry. Yet at the same time Excel is limiting, and depending on your proficiency, rather frustrating. From the perspective of technical data analysis, it is also highly taxing on our most important commodity. Time.
Anyone who has tried to combine a number of large spreadsheets and perform an in-depth interactive investigation of what the data means will know this, regardless of your proficiency. So what is the alternative?
Metrix is providing one solution to this question.
Whilst fine for a few hundred rows and a few dozen columns, once you put a lot of data into Excel, say a few hundred thousand rows and several hundred columns, performance invariable suffers, either through slowdown or the spreadsheet becoming unresponsive.
When designing a system for multiple users, spreadsheets invariably suffer from problems due to changing data or overwriting someone else's work. Also, spreadsheets are single access leading to multiple copies with different data entered.
Typos, incorrect data, data entry errors are common with spreadsheets. Frequently, "placeholder" data is entered with the intent to update it later. Spreadsheets are often littered with inaccuracies and, as time passes, these become permanent.
Although some basic security features can be added to Excel spreadsheets, these cannot govern a multitude of people wishing to access the same data, to manage changes made, etc. Nor can they track when changes were made, and by whom.
When spreadsheets are used as databases, what happens when the same data is entered more than once? This can lead to conflicts in the data and confusion in subsequent processing and analysis.
Some data is fundamentally critical and losing it is simply not acceptable. Spreadsheets pose a risk of file corruption or accidental deletion leading to the loss of key data. Most people will have experienced loss of spreadsheet data at some point. This is mitigated when using a formal database.
There are many examples of commercial database software, but the most commonly known is the relational database, coupled with the standard Structured Query Language (SQL) – Examples include MySQL, MS Sequel Server and Oracle. Moving your data into an SQL database is fairly straightforward, but what next..?
With the data safely stored in a relational database, there is now a barrier to accessing that data. For those who don't write code, you now need an entire 'front end' infrastructure in order to see your data and interface with it. This can be a costly and time consuming exercise, and chances are you might end up with less functionality than you had with Excel.
Our 100+ years combined metallurgical consulting experience demonstrates that to truly understand a concentrator process or a large new project, your data quantity and complexity will necessitate a move to a database. Climbing over that barrier-to-access then presents a whole new vista of opportunity in data analytics, to reinvent how we analyze metallurgical data.
Data science is the combination of IT and programming, statistics, and strong Industry domain knowledge. This potent combination allows for a new paradigm in using data to deliver meaningful information to your business – In our case we want to help optimize your process
Metrix origins are in metallurgical consulting. We have decades of combined experience and have been involved in dozens of plant optimization studies and hundreds of development projects. Over this time we have also developed strong statistical approaches coupled with a more practical intuition of data interpretation.
Over the last 2 years, Metrix has been diving deep into the exciting world of computer data science and acquired a high caliber IT team with complementary professional skillsets. Starting from our domain knowledge as industry experts and consultants we knew exactly what we wanted to build. We called it dataSEA™.
Assembling data into a database is relatively straightforward, but how do we see the data and interact with it? Our answer is a custom developed, browser based, highly interactive single page dashboard, powered by D3.js (one of the most exciting data visualization tools in the world of Data Science). D3 is not so much a "chart library", as a "library for building chart libraries". Our custom metallurgical dashboards bring charts to life with interactivity and dynamic transitions. Further, our designs leverage our experience of how data metallurgical data needs to be examined and incorporates unique methods of data visualization developed from our abundant experiences as consultants. We provide a turn-key metallurgical exploration environment from day one.
Having all your data accessible in an interactive dashboard actually offers a brand new experience. Data exploration!
Before assembling a specific line of enquiry (as one typically would do in MS Excel) you might find great benefit in swimming around in the data, to see what you see, without any pre-existing ideas of what you are looking for. The speed and interactivity of our dashboard facilitates such exploration, wherein data of all types from any point in your projects history are combined and presented in a multitude of ways. This can lead to insights you never even considered..
Metrix dataSEA™ offers a turn-key template but also invites you to help customize your dashboard with your own interactive chart ideas. We offer ways to see and interact with data exactly how you want to.
dataSEA™ is the brainchild of Robert Thorpe
With modern computing power and the world of data science at our fingertips, Metrix offers far more than just data visualization. We are actively building a collection of data analytics scripts that, behind the scenes, investigate your entire data collection the way a senior consultant would, only in seconds rather than months..
Our 'back-end' analytics engine primarily utilizes the vast array of data science tools built in the Python programming language, with particular focus on Pandas and SciPy. These tools are used to create a 'virtual metallurgical brain' that critically assesses metallurgical data and reports back its findings. This leads to another new concept – Analyze the analysis!
For a simple example, imagine your concentrator performance over a 5-year period. You wish to use linear regression models to understand drivers of recovery on a shift-by-shift basis (this should resonate with most plant metallurgists). Almost universally you will get a highly noisy result and fairly poor correlations. Now consider automated regression analysis of every potential independent variable in your plant that could affect recovery, each cleaned of statistical outliers in a variety of ways and broken down by month, year and all data – Thousands of individual regression models. Now plot and navigate the R2 and p values. Do all of this in a few minutes. Now you are analyzing the analysis.
This is just the beginning. Within Metrix we have world class access to some of the finest minds in the industry within the fields of metallurgical testwork, flotation chemistry, flotation cell hydrodynamics, automated mineralogy and general optimization methodology.
Under the hood, Our dataSEA™ software can be understood as a collection of interweaving algorithms utilizing some of the most powerful data science tools currently available. Interestingly, much of our inspiration originates in the global banking and finance sectors.
We utilize the classic three-tier data application system, combining data visualization and business logic built upon a bedrock of SQL storage. We manage all of this; you just provide your data and then manipulate the dashboard.
Although SQL, Pandas and D3 are our primary tools, we embrace the whole ecosystem of the data science toolbox and are frequently looking for how the variety of available tools can help us with our primary objective: Enabling data analysis to be conducted in a fast, meaningful way resulting in actionable conclusions.
When compiling large datasets, especially from concentrators, we are never satisfied. Data is generated by the array of instrumentation every millisecond in some cases. To truly understand metallurgical performance of a concentrator, data from a variety of timescales must be seamlessly combined: the operational data produced every second by instruments, the metallurgical data typically measured in shift units, whereby assays and other various analyses are performed on samples, and lastly, and sat somewhat outside of time is the orebody itself, containing a plethora of metallurgical information locked away in the block model.
Our ultimate goal is to capture all of it, and assemble it all together in dataSEA™.
On the project development and testwork side, the focus of many companies is ever-improved geo-metallurgical models. Again, capturing the coded metallurgical data captured by geologist in the description of ore bodies is paramount and then elegantly brought together with the data resulting from metallurgical testing..
Such lofty goals take us on a path to yet more powerful tools – those of the big data and machine learning ecosystems. Far more than just distributed data storage, this toolkit makes it all possible... for those who know how.
If you are interested in going on this journey with us, please contact us to discuss your project..
Contact us and we will be back to you as soon as possible.
British Columbia, Canada