Continuous Intelligence

As part of the assignment, I would like to write something on the chosen topic, Continuous Intelligence. Continuous intelligence plays a major role in most digital business transformation projects. It is a growing part of enterprise analytics and BI strategies.

Definition

Continuous intelligence is a design pattern in which real-time analytics are integrated into a business operation, processing current and historical data to prescribe actions in response to business moments and other events. It provides decision automation or decision support. Continuous intelligence leverages multiple technologies such as augmented analytics, event stream processing, optimization, business relationship management (BRM), and machine learning (ML). The definition extracts from the Gartner Research.

What can you do with Continuous Intelligence?

Continuous intelligence enables companies to deliver better outcomes from a broad range of operational decisions since it involves more relevant, real-time data in decision-making algorithms. Individuals can make sense of extreme volumes of data in milliseconds, evaluating more alternatives in greater detail than humanly possible without access to real-time data and processing.

Gartner estimates that, within 3 years, more than 50% of all business initiatives require continuous intelligence, leveraging streaming data to enhance real-time decision-making.

Combining all these forms of artificial intelligence (AI) with continuous intelligence drawing from geospatial, real-time, and historical analytics can further enhance business ability to know where assets and people are at all times and help predict what might occur next.

Adding rules engines and programmatic logic to AI, location data enables organizations to automate many decisions that previously required human insights. From predictive maintenance based on actual driving conditions to decide the best next action to take with customers to improve loyalty, leading companies are decreasing costs and improving revenues to become more successful.

What are The Challenges?

What makes continuous intelligence difficult is feeding a business’s analytics systems with high volumes of real-time streaming data in a way that is robust, secure, and yet highly consumable. The ability to combine “always-on,” streaming data ingestion and integration with real-time complex event processing, enrichment with rules and optimization logic, and streaming analytics is key to enabling Continuous Intelligence.

Many data analytics organizations lack experience with Continuous Intelligence, or unsure how to start their Continuous Intelligence journey to keep up with growing business demand.

Continuous Intelligence requires the building of new capabilities, skills and technologies. The challenge for data and analytics leaders is to understand how these differ from existing practice.

Why Use Continuous Intelligence in DevOps/DataOps

If you are considering DevOps as a strategy to adopt continuous innovation, your data strategy has to evolve, too. Traditional BI has too many silos and too much human intervention to support your move to an agile system.

Up to this point, I would like to add that in my current project, some of my team members, who are in the agile system, try to implement the ETL (Extract, Transform, Load) processes by following agile methodology. Sometimes ago, I went to the agile workshop and I have forgotten some of the concepts. It is a good time to read them up again.

According to Open Data Science’s article entitled “Why Use Continuous Intelligence in DevOps/DataOps,” it wrote that businesses look out for continuous innovation. Those who do not may put out shoddy products. Your data strategy, therefore, has to be seamless, frictionless, and automated.

Artificial Intelligence

The article adds, “Artificial Intelligence is capable of continually combing data, looking for patterns as data updates. Continuous intelligence allows you to analyze this data accurately and in real-time. The other piece could be letting go of data wrangling. Until you have deployed Continuous Intelligence, data wrangling remains a huge and functional part of your data management plan.”

Gartner identifies six defining features of CI.

  1. Fast: Real-time insight keeps up with the pace of change in the modern age.
  2. Smart: The platform is capable of processing the type of data you get, not the type you wish you had.
  3. Automated: Human intervention is rife with mistakes and wastes your team’s time.
  4. Continuous: Real-time analytics requires a system that works around the clock.
  5. Embedded: It’s integral to your current system.
  6. Results-focused: It should go without saying, but data means nothing without insight. Your program should deliver those insights. Don’t forget the results in the search for more data.

Once you let go of batch processing and silos, moving towards an agile framework is a reality with CI.

Open Data Science

Your team has access to these insights to direct new inquiries and drive brainstorming, pivot during sprints, and reach a frictionless state in which data flows in and insights become the next iteration of a product or a new product altogether. “

With this information, I have a vision; I wish to move into Continuous Intelligence and bring this agile methodology into my project.

References:
https://www.striim.com/blog/2019/05/gartner-identifies-continuous-intelligence-as-top-10-trend-for-2019/
https://www.rtinsights.com/what-can-you-do-with-continuous-intelligence/
https://medium.com/@ODSC/why-use-continuous-intelligence-in-devops-dataops-b6bc0a448b7a

Assignment Topic: Continuous Intelligence

My assignment topic is about Continuous Intelligence, and I was told to refer to the Gartner Research. The Gartner Research is a global research and advisory firm providing information, advice, and tools for businesses in IT, finance, HR, customer service and support, legal and compliance, marketing, sales, and supply chain functions.

My lecturer advised us to refer to this website to complete my research paper. My school has a link to the Gartner Research papers available to all students.

A small introduction of what is Continuous Intelligence – Gartner identified Continuous Intelligence as one of the top 10 technology trends for data and analytics for 2019. 

In the website, the Gartner defines Continuous Intelligence as “a design pattern in which real-time analytics are integrated within a business operation, processing current and historical data to prescribe actions in response to events. It provides decision automation or decision support. Continuous intelligence leverages multiple technologies such as augmented analytics, event stream processing, optimization, business rule management, and Machine Learning.

Reference: https://www.striim.com/blog/2019/05/gartner-identifies-continuous-intelligence-as-top-10-trend-for-2019/

Fu Lin Bar & Kitchen

Today’s lunch is a little further from the working place; we traveled to Telok Ayer to try the yong tau foo. My colleague says this yong tau foo is different than the normal ones in the food court because of their unique sauces. Since I have not eaten yong tau foo for quite sometimes, I suggested trying there.

The Fulin Bar and Kitchen at Telok Ayer is very crowded, and it is not easy to find a place for a large group. So, I would suggest you go with a small group. Soon, after we got our seats, we queued to get our meal. The lunch menu goes by the basic six pieces of yong tau foo; you can add rice or noodle, and you can have it with soup based or dry sauce.

For myself, I ordered six pieces of my favorite yong tau foo and added a bowl of noodles. The stall offers a single type of noodles, and I do not see other types of noodles available, or the cashier did not ask what types of noodles I want to order. Besides that, it seems like the default version is dry.

Most of the yong tau foo is fried stuff and a handful of vegetables. Below is what I have picked. Well, I cannot even recognized them now 🙂 Mainly, they are brinjal, bitter gourd, beancurd puff, etc. Then it topped with their special sauce.

The dry noodle uses the same special sauce, which contains some minced mushrooms and minced pork. The sauce is not salty, tasted just right, but it is starchy and slightly oily. It is not a good pairing when I have both fried food and starchy sauce.

Address: 127 Telok Ayer St, Singapore 068596.

Database Model

During a check on the Database Engines ranking just now, I found that this link is listing the available database management systems according to their popularity. The website updates monthly and you able to see the current ranking, previous month ranking, and the ranking one year ago.

The picture above shows the list of the top 10 database management systems. Another exciting part of the website is the list of the database model.

Relational DBMS

Relational database management systems (RDBMS) support the relational or table-oriented data model. The schema of a table or the defines by the table name, fixed number of columns (attributes) with fixed data types. A record corresponds to a row in the table (entity) and consists of the values of each column. A relation thus consists of a set of uniform records, according to the website.

The normalization in the process of data modeling generates table schemas. There are some operations used to define a relationship. For example,

  • classical set operations (union, intersection, and difference)
  • Selection (selection of a subset of records according to certain filter criteria for the attribute values)
  • Projection (selecting a subset of attributes/columns of the table)
  • Join: special conjunction of multiple tables as a combination of the Cartesian product with selection and projection.

Document Stores

Document stores, also called document-oriented database systems, are characterized by their schema-free organization of data. According to the website that means,

  • Records do not need to have a uniform structure, i.e. different records may have different columns.
  • The types of values of individual columns can be different for each record.
  • Columns can have more than one value (arrays).
  • Records can have a nested structure.

Document stores often use internal notations, which can be processed directly in applications, mostly JSON. JSON documents, of course, can also be stored as pure text in key-value stores or relational database systems.

Key-value Stores

Key-value stores are probably the simplest form of database management systems. They can only store pairs of keys and values, as well as retrieve values when a key is known.

These simple systems usually are not adequate for complex applications. On the other hand, it is exactly this simplicity that makes such systems attractive in certain circumstances. For example, resource-efficient key-value stores that apply in the embedded systems or as high performance in-process databases.

Advanced Forms

An extended form of key-value stores is able to sort the keys, and thus enables range queries as well as ordered processing of keys. Many systems provide further extensions so that we see a fairly seamless transition to document stores and wide column stores.

Search Engines

Search engines are NoSQL database management systems dedicated to the search for data content. In addition to general optimization for this type of application, the specialization consists of typically offering the following features:

  • Support for complex search expressions
  • Full text search
  • Stemming (reducing inflected words to their stem)
  • Ranking and grouping of search results
  • Geospatial search
  • Distributed search for high scalability

Wide Column Stores

As mentioned above, the wide column stores, also called extensible record stores, store data in records with an ability to hold huge numbers of dynamic columns. Since the column names, as well as the record keys, are not fixed, and since a record can have billions of columns, wide column stores see as two-dimensional key-value stores.

The wide column stores share the characteristic of being schema-free with document stores. However, the implementation is very different. The wide column stores must not be confused with the column-oriented storage in some relational systems. The wide column stores is an internal concept for improving the performance of an RDBMS for OLAP (Online Analytical Processing) workloads and stores the data of a table, not record after record but column by column.

Graph DBMS

Graph DBMS, also called graph-oriented DBMS or graph database, represent data in graph structures as nodes and edges, which are relationships between nodes. Graph DBMS allows easy processing of data in that form, and a simple calculation of specific properties of the graph, such as the number of steps needed to get from one node to another node. Graph DBMS usually does not provide indexes on all nodes, direct access to nodes based on attribute values is not possible in these cases.

Time Series DBMS

A Time Series DBMS is a database management system that optimizes handling time series data; for example, each entry associated with a timestamp.

Time Series DBMS is designed to efficiently collect, store, and query various time series with high transaction volumes. Although the management of the time series data can be the same as other categories of DBMS (from key-value stores to relational systems), the specific challenges often require specialized systems.

I hope the information extracted from the website is able to help us understand the differences between the database models.

References: https://db-engines.com/en/ranking

Life in this November

It has been a while since the last write-up in my blog. My work has been piling up, and I am busy with my school assignments too. This month has been quite impressive, both work and study. I did not realize that my school has signed up a premium package for all their students to use the Grammarly. I wanted to purchase their premium package before this, but I did not proceed (even with 40% discounts) because I stopped writing for the technical documents.

It came to me quite suddenly, and they decided not to renew the contract. It was a great two months of experience writing the technical guides and documents of a product. This experience improved my English and writing skills. I am going to continue writing for myself.

I learned what passive voice writing and active voice writing method are. From not performing well to getting feedback, ‘I am there, but I take a longer route to reach.’ However, I found it quite relieved because I wanted to concentrate on my studies and assignments. This month, there are three assignments, I have completed one of them, two more to be completed by 1st and 2nd week of Dec, just before the school break.

As for work, I will be concentrated on my role to get things standardized. I have been working on data models, standard codes, etc. I met many people in the last two months, some of them I managed to catch up again with them on different occasions such as meetings, while others are during the team bonding sessions or townhall meetups.

I will try to continue writing until the end of the year. Hopefully, you can give me some feedback about what I shall share more for next year. If you have any feedback, please write to me in this Google form.

Lai Wah Restaurant

Last Thursday, I went out for dinner at Lai Wah Restaurant. The restaurant located at Bendemeer Road. It was recommended by my friend to visit here for Cantonese cuisine. Together with two other ladies, we headed to the restaurant at 7pm after I received my freelance payment and completed the formality. 

We ordered three dishes and a plate of noodles to share. Two of the dishes were my favorite; the yam ring and pork ribs. We asked for the recommendations from the waitress. In my assumption, she simply suggested us to order the salted egg crispy squid. She added the salted egg crispy squids were delicious and unique. Well, I was not convinced, but we were fine to try it. Instead of going for white rice, we ordered one of their fried noodles that turned out to be fried hor fun with venison.

Overall, the dishes were just all right. It was not the best Cantonese cuisine so far. However, it able to maintain some of the traditional Cantonese dishes such as the yam ring, consider pretty good. The yam ring’s texture was a little hard, but the taste was quite close to those I tried in Kuala Lumpur, Malaysia. 

The pork rib is a common dish in Cantonese restaurants. Some restaurants try to suit some customers’ requirements and change the pork rib to pork chop with the same cooking style.

The fried hor fun with venison looked great on other people’s photos when I googled it from my mobile phone. Another waiter suggested us to try it because I mentioned that I wanted to try fried hor fun. When the waiter served the fried hor fun with venison, I felt a little disappointed. The fried hor fun flooded with starchy black sauce; it tasted normal. I will not be recommending it to you for trying.

Lastly, the salted egg crispy squid can give a miss too. 

My friend told me that this restaurant is an old brand in Singapore. The restaurant packed with people, especially during the Chinese New Year. I do wish to revisit the restaurant to try other dishes, some of my favorite Cantonese dishes. If you have tried it before this, do recommend some dishes that you will like me to try.

Address: Lai Wah Restaurant, Block 44 Bendemeer Road, Singapore.

MongoDB: Importing csv files with mongoimport

Someone asked for my help to upload some .csv files to the MongoDB database and backup the database before sending the file to the next person. I completed the task with the command below. It imports a .csv file to the selected database and collection by specifying the file type, location and whether the file has a headerline. It runs for both Linux and Windows’ machines using the terminal or command prompt.
mongoimport -d mydb -c things --type csv --file locations.csv --headerline