A Comprehensive Guide to Data Management for Businesses
The use of data management and analytics is on the rise as more and more companies realize the enormous impact this can have on their businesses. In fact, an IDG Enterprise study reveals that “organizations are seeing exponential growth in the amount of data managed with an expected increase of 76% within the next 12-18 months.”
In order to leverage data for your business effectively though, you have to first develop a clear understanding of what data is and how you can efficiently make the most out of it. We have come up with this ultimate guide to data management in order to help you out.
TABLE OF CONTENTS
CHAPTER 1: What Is Data Management
What is data?
Data is all around us. At this day and age, there exists an enormous amount of data available for everyone to use. In fact, according to an IBM study, “Every day, we create 2.5 quintillion bytes of data—so much that 90% of the data in the world today has been created in the last two years alone.”
Undoubtedly, you’ve also heard of the term “big data”. Lisa Arthur, in an article on Forbes.com, defines big data simply as “a collection of data from traditional and digital sources inside and outside your company that represents a source for ongoing discovery and analysis.”
Other industry experts, such as Doug Laney of Gartner, also defines big data in terms of the three V’s: Volume, Velocity and Variety.
Volume refers to the amount of data that is available and is collected. It can be attributed to several factors, such as unstructured data streaming, and the increasing amounts of sensor and machine-to-machine data collection.
Velocity refers to the speed at which data streams in. Most organizations these days are said to experience difficulty responding to data velocity because data is streaming in at unprecedented speed.
Variety refers to the type of data. Because data comes in a variety of types, organizations find it difficult to manage and merge all this data efficiently. The two major types include Structured and Unstructured data.
The industry analysts at SAS added two more attributes to the equation: Variability and Complexity.
Variability has to do with the consistency of data flows, which depends on the increasing velocities and varieties of data. Data loads can be hard to manage because they can vary daily, seasonally or during times when an event triggers peak data loads.
Complexity is an attribute brought about by data coming from multiple sources. It is essential not only to collect as much of them as possible, but also to connect and correlate their relationships as well as to transform them across all systems—which proves to be challenging for most companies.
Types of Data
It is important to distinguish between Structured and Unstructured Data because both are very different from each other, yet both should also work together in order for any big data operation to be effective.
Unstructured data are information that are not organized in a predefined manner, such as those coming from social media and other forms such as Word documents, PDFs, audio files, presentations, videos, images, messaging, and more. On the other hand, structured data refers to those that can be organized easily, such as machine generate data (sensory data, point-of-sale data, web server logs) and human generate data, which are inputted in a computer (age, zip code, gender and the like).
The TCS Big Data Global Trend Study of 2013 states that “Nearly half the data (49%) is unstructured or semi-structured, while 51% is structured.” Unstructured data, when simply inputted into a table, will not be helpful to an organization. It is therefore necessary to further process this data and connect them with their structured data counterpart in order to derive better insight from the data set.
Data Management and Its Benefits
Data management is defined as the “administrative process by which the required data is acquired, validated, stored, protected, and processed, and by which its accessibility, reliability, and timeliness is ensured to satisfy the needs of the data users.”
Data collection is essential for any organization, but simply collecting huge amounts of data will not be helpful to you. What counts is what you do with that data. Proper collection, validation, storage, protection and processing of data is what will give you the information you will need in order to make accurate and timely decisions that will greatly benefit your business.
Organizations can reap many benefits from data management and Marc Vael summed them up into 5 key areas in his article on Computer Weekly.
- Product development
- Market development
- Operational efficiency
- Customer experience and loyalty
- Market demand predictions
How Data Management Works
Data management is primarily handled at the data center, whose main focus is to ensure that data can be accessed by users when they need it while at the same time maintaining security and protection. Data centers can be located inhouse or commissioned through data management services. There are four pillars of a data management operation and they include storage provisioning, data protection, data replication, and recover operations.
In his article for Computer World, Chris Poelker explains each pillar as follows:
Provisioning refers to data storage in including functions such as storage management, I/O performance management, data tiering, capacity management, access protocol management, RAID management, object management, metadata management, access management, I/O troubleshooting, and security.
Protection is required to ensure the availability and integrity of data. It includes making sure that the storage and protection technology used is appropriate for the data type and structure.
Replication is necessary to ensure the availability of data even during instances when technology changes or something in the system goes wrong. Some areas covered by this include networking, capacity management, recovery service levels, business continuity, and more.
Recovery is important to make sure that business operations continue even if something breaks. It “involves the entire stack, from user access to the application itself, to where and how the data is stored.” This includes data service level agreements, data retention policy management, data archiving, and facilities management among others.
CHAPTER 2: Ensure the Success of Your Data Management
Data Management Best Practices
Data management has gone a long way from simply commissioning data entry services. Ensure the success of your data management efforts by learning from the industry’s best practices.
Study business requirements first.
The process must begin by first understanding what your business requires from the data analysis. By doing this, you will know exactly what you need to do and how you need to do it. Make sure that what you decide to do is aligned to specific business goals.
Think of big data as a business decision.
Data management is not a concern of your IT department alone. It is an investment that will affect the entire organization and so approaching it using a business perspective will increase the likelihood of its success.
Evaluate data before engaging in big data analytics.
Chances are, your business has already accumulated huge amounts of data. Some of them may not be useful at all. Before engaging in any data analytics operation, first evaluate the data that you have and determine which ones to retain and which ones can be discarded. This will reduce the amount of time and effort needed to manage data.
Address skills shortage.
There’s a growing demand for jobs that involve data management. A Gartner research shows that “By 2015, 4.4 million IT jobs globally will be created to support big data, generating 1.9 million IT jobs in the United States.” However, there is also a growing shortage of talent and this is something that must be addressed. Standardizing big data efforts and developing an IT governance program will help overcome the issue.
Remember that your data management system will be accessed by different types of people ranging from operational workers, analysts and even senior executives. Therefore, make sure that your system is easy to understand and use in order to match user expectations. This helps encourage adoption of the tools and ensure the success of the project as well.
Have a maintenance plan.
Any system will require continuous updates and maintenance. However, keep in mind that changes can and will occur as data volumes rise and as users demand more from the system. This means that, when analyzing which data management services to avail, you need to choose one that supports iterative development so that it can adapt to the changing requirements of your business over time.
There is no doubt that data management concerns a growing number of businesses these days. A study shows that “companies are intensifying their efforts to derive value through big data initiatives with nearly half (49%) of respondents already implementing big data projects or are in the process of doing so in the future.” Companies are either investing in developing or buying software applications, additional server hardware, as well as hiring staff to handle their big data efforts.
It can be an overwhelming task. However, there are already quite a number of technologies and data management services available these days to enable organizations to make the most of big data and big data analytics.
CHAPTER 3: Data Security: Keep Your Business Safe
No one can dispute the fact that data security is the backbone of any well-maintained data management system. Data security must be prioritized by any organization to enable it to function properly and for operations to flow efficiently. It also provides stockholders and executive teams peace of mind of knowing that the information they’ve stored in their servers will not be easily exploited by hackers or cyber-criminals.
More and more companies are putting a sizable share of their financial resources behind data security, which indicates an increased awareness in its important role in any business enterprise. Consider these facts:
- Security (36%), cloud computing (31%) and mobile devices (28%) are the top 3 initiatives IT executives are planning to have their organizations focus on over the next 12 months. (Source: 2015 State of the Network Study, Technology Adoption Trends & Their Impact on the Network)
- 42% of IT decision makers are planning to increase spending on cloud computing in 2015 (Source: Computerworld’s 2015 Forecast Predicts Security, Cloud Computing And Analytics Will Lead IT Spending.)
- Forrester Research concluded that Big Data continues to grow, saying that the average organization will grow their data by 50% in the coming year
- overall corporate data will grow by 94%; database systems will grow by 97%, and server backups for disaster recovery and continuity will expand by 89%.
Today’s organizations are increasingly dependent on computer systems and software for storing data. Data management services are therefore expected to offer superior cyber-security systems to be included in their suite of services for outsourcing clients. It’s important to recognize that mistakes are inevitable; it is something that data management companies will always have to deal with, regardless of whether these were man-made, accidental, or mistakes caused by system errors such as viruses or malware.
While security breaches cannot be eradicated completely, untoward incidents can be considerably reduced; mistakes can be amended, identified, analyzed, and then eventually chalked up to experience. Documenting such mishaps are done in an effort to minimize or even eradicate the same mistakes from being committed again in the future.
Here is a compilation of cases studies and real-life examples of various companies as well as how they’ve dealt with data security—from some that have been unfortunate victims of sophisticated data breaches to those that have put policies in place to help make their data systems less vulnerable to unwanted incidents.
As the second largest health insurer in the United States, the recent cyber attack on Anthem Inc. is said to be the largest data breach in recent history. On January 29, 2015, approximately 80 million names on its database along with sensitive personal information were compromised. Social security numbers, addresses, medical ID numbers, email addresses, employment and medical histories, even income information are exposed and possible made vulnerable to being exploited by cyber criminals.
Crisis management experts weighed in on how Anthem handled the security breach. Many have lauded their proactive communications efforts, and some have expressed their opinions on what can be gleaned from this recent data security breach.
Andrea Bonime-Blanc, CEO, GEC Risk Advisory: “Anthem’s immediate corporate response to its cyber breach crisis appears to be close to a textbook case of effective immediate crisis management and preparedness. First, Anthem actually discovered the breach themselves–they weren’t extorted by the hackers or outed by the media or others. This is good reputation risk management.
“Second, Anthem immediately advised federal authorities of the breach and hired reputable cyber consultants to deal with immediate damage control. This, too, denotes the existence of internal preparedness. Third, although applicable regulations appear to allow for a 60-day reporting window, Anthem decided to publicly announce its crisis within days of its first discovery. While perhaps risky, such a move can provide Anthem with longer-term reputation enhancement with key stakeholders, restoring and building trust and customer loyalty over time.
“Fourth, the company provided clear and coherent messaging of what happened–down to the kind of information that might have been compromised–in easily available and clearly written materials, including special instructions and a website for the occasion. Fifth, the CEO letter is an effective letter, addressing the concerns of key stakeholders (employees, customers, regulators and investigators) and providing them with immediate resources. In the letter, the CEO also apologizes and brilliantly shows empathy with his customers and employees by referencing the fact that his personal data was stolen as well.
“The only downside that I can discern from what has been reported doesn’t have to do with Anthem’s crisis response but more with its risk preparedness regarding the apparent lack of encryption on the data that was stolen. However, this is more of a risk management issue that Anthem and its executives and board will now surely be focusing on as they begin build stronger cyber resilience.”
Daniel Diermeier, dean, Harris School of Public Policy, University of Chicago: “As in many major business crises, companies need to respond quickly and focus on reassuring customers and restoring trust. This requires an authentic response that is transparent, competent, committed and shows real concern for the fears and frustrations of customers, even if such fears may be overstated.
“Anthem’s message from the CEO was appropriate and personal, but various questions–whether customers were notified sufficiently quickly, and some confusion on whether customers need to sign up for identity protection or if it is provided automatically–led to some doubts about the company’s competent handling of the crisis.
“Customers will not be forgiving in cases of lapses or missteps. Such heated customer reactions in the context of hacking incidents may be puzzling, even upsetting to executives. After all, companies such as Anthem were the victims of sophisticated, organized criminals. Shouldn’t the public have some sympathy for the company that has been victimized? In the case of Anthem’s data breach that means that management will be solely evaluated on how well it takes care of customers whose data has been compromised, whether the company is ultimately responsible for the breach or not.”
Although Desjardins Bank of Quebec, Canada has not been hit by a cyber attack in the same magnitude as Anthem’s or other high profile cases, the bank’s management and communications team has decided to face the issue head-on by addressing the issue with its clients.
In her website, Melissa Agnes pointed out that Desjardins has adopted what she calls a “very important and advantageous strategy” and continues by asking, “Why not leverage this proactive effort to help you position your organization as a leader and to help you build trusting and credible relationships with your stakeholders? This can prove to be a very strategic crisis preparedness strategy.”
Agnes points out that she received a newsletter from the bank with the following message: “Mobile devices are attracting more and more interest by fraudsters. Check out our infographic to find out what to do to protect your mobile device … and your identity!”
By sending this helpful information to its clients, Desjardins has done something “proactive and useful to help their stakeholders think of this risk and have provided them with useful solutions.” She pointed out that they “[show] their stakeholders that they care enough about their protection to provide them with helpful information in a trendy, share-able format, positioning their organization as a voice of trust and credibility on this issue. Think of it, people who read this newsletter are thinking to themselves “Good to know that they’re thinking of this. My banking information must be good and secure with them.” It’s a psychological effect that can help to give you the benefit of the doubt in a crisis – though of course, you (and they) need to follow through with actually making sure you’re doing what people expect.”
In October 2013, more than 5,000 credit cards stored in Staysure.co.uk’s database were accessed and used fraudulently by hackers, who were able to remotely view and modify the sensitive information stored in servers. Staysure, a holiday insurance company that is based in the United Kingdom, was issued a £175,000 Fine for its failure to keep their customer’s information secure.
Over 10,000 of their customers’ personal information became vulnerable to attack, including credit card and medical details. The CVV security numbers were also accessible despite industry rules against gathering and storing such data from customers. Investigations revealed that Staysure violated the Data Protection Act because it had no policies in place to review and modernize their IT security systems and even failed to update the data system which could had prevented the hack not once, but twice.
Clearly, security was not a priority for the company, where sensitive data was left dangling for as long as five years, made particularly vulnerable to cyber criminals who took advantage of the weak infrastructure to pounce and exploit the stored data.
This attack in 2014 showed that over one billion data records were breached in the UK, a whopping 78% increase from the previous year—with identity theft being the most common type—accounting for over half of the total reported.
Said to the biggest retail hack in US history, Target’s 2013 security breach involved having 40 million credit and debit card accounts divulged to fraudsters, and 70 million customers’ personal information, including email addresses, names, phone numbers, revealed and open to identity theft or sold to the black market.
Two years later, the effects of this high profile cyber attack can still be felt:
- a $10 million fine that Target must pay affected customers
- the CEO at the time of the attack, Greg Steinhafel, stepped down in May 2014
- the closing down of all 133 stores in Canada (resulting in having 17,000 workers laid off)
- 1,700 employees laid off from its headquarters and 1,400 open positions removed.
Yet it has been said that Target was not remiss in updating its data security systems. Six months before the attack, Target had installed a $1.6 million malware detection tool made by the computer security firm FireEye, a software company that has the CIA and the Pentagon among its customers. Security specialists had the ability monitor the site round the clock, with security operations centers located in Bangalore ready and able to detect any anomalies.
They had alarm systems in place to alert the Minneapolis headquarters in case of any security breach. When the malware was detected, the security procedures proceeded without a hitch, with the Bangalore team flagging and alerting their US counterparts of an impending cyber attack. It was alleged that the Minneapolis team did not react soon enough and did nothing to stop the hackers from exploiting the data.
Target chairman’s response was an emailed statement, where Chief Executive Officer Gregg Steinhafel issued the following: “Target was certified as meeting the standard for the payment card industry (PCI) in September 2013. Nonetheless, we suffered a data breach. As a result, we are conducting an end-to-end review of our people, processes and technology to understand our opportunities to improve data security and are committed to learning from this experience. While we are still in the midst of an ongoing investigation, we have already taken significant steps, including beginning the overhaul of our information security structure and the acceleration of our transition to chip-enabled cards. However, as the investigation is not complete, we don’t believe it’s constructive to engage in speculation without the benefit of the final analysis.”
With data security breaches becoming increasingly common in today’s wired, interconnected world, many organizations are implementing steps to avert cyber attacks and increasing security measures. Some technology companies are also adding security protocols into an already relatively secure system, such as Google and Twitter’s latest two-factor authentication.
There’s also the introduction of Blackphone, a totally encrypted, air-gapped mobile phone, and WhatsApp adding encryption to its text messaging. Box’s Chief Trust Officer Justin Somaini said that “security as a business model has arrived.”
Indeed, security is now front and center in many organization’s concerns; the days where it is included almost as an after thought by a data management company is fading fast.
CHAPTER 4: How Data Quality Can Increase Your Profit
Gathering data is the first step to properly managing it, but it doesn’t end there. There must be system in place that makes sure that the quality of the information is correct, relevant, and useful. Otherwise, collating data becomes a waste of time.
It can also be a costly missed opportunity. A study conducted by Experian Data Quality shows that outstanding data quality has a direct correlation to an increase in company profits. Their research also concluded with the following points:
- “Companies that have enjoyed a significant rise in profits over the past 12 months manage their data quality in the same fashion: with ownership resting with a single executive.
- The lion’s share (92%) of companies find some element of managing data challenging; the most common culprit being to address data quality issues before they have a negative effect on business.
- Eighty-eight percent of companies implement a data-quality tool, but the average U.S. organization still claims that one third of its data is inaccurate.”
However, while everyone recognizes that optimizing an organization’s data management strategy offers many benefits, few take the necessary steps to address it properly. The same Experian Data Quality study found that “78% of U.S. companies could stand to improve their level of data management sophistication.”
Which is a shame, considering how companies who adopt a data management approach that prioritizes quality are amply rewarded not just with accurate information, but with a more stable bottom line. To address this, Thomas Schutz, SVP and GM of Experian Data Quality, said that companies must invest in improving their standards, and they can do this by “hiring the right people and focusing them on centralizing processes and installing preventative software.”
Data Quality and the Human Factor
They say that “to err is human”, which is a cliché that applies not only to philosophical rumination, but to the issue of data quality entry as well. A study made by Experian QAS found that “65 percent of organizations cite human error as the main cause of data problems”. It is also one of the thirteen causes of data quality problems that was identified in an academic article written by Arkady Maydanchik titled Data Quality Assessment. As someone who has extensively dealt with the issue of data management, he has this to say on the realities of human error:
“Despite high automation, much data is (and will always be!) typed into the databases by people through various forms and interfaces. The most common source of data inaccuracy is that the person manually entering the data just makes a mistake… Good data entry forms and instructions somewhat mitigate data entry problems. In an ideal fantasy world, data entry is as easy to the user as possible: fields are labeled and organized clearly, data entry repetitions are eliminated, and data is not required when it is not yet available or is already forgotten. The reality of data entry, however, is not that rosy (and probably won’t be for years to come). Thus we must accept that manual data entry will always remain a significant cause of data problems.”
Once we’ve accepted the realities of mistakes caused by human errors as well as other factors, it’s time to move forward and adopt a proactive approach toward minimizing not just the mistakes committed, but the consequences that such errors might potentially have. Ideally, there shouldn’t be a significant impact to normal operations; it should have little to no effect to company profits and the bottom line.
The Experian QAS study has outlined these four steps to helping organizations eliminate the incidence of human error:
You must first understand how data flows into the system. Doing so allows you to prioritize high volume channels or platforms that have too many data-quality errors. Find answers to these questions to help you know the next steps to take: Is consumer data collected at point of sale, or captured via online forms? Do your own staff type in the data, or are clients made to do so?
Take the time to explain why inputting correct and accurate data has a huge impact on how the information is used in the business. Most of the time, those who are tasked to manually enter data do not see the big picture, and are unaware of how a seemingly minor typographical error can affect how your company makes decisions related to launching marketing campaigns. Once the data-entry employees are educated of how their actions form a ripple effect on the business, errors may significantly be reduced.
There are numerous software solutions and apps that can help prevent inaccuracies in date entry. Prioritize what type of data has the greatest impact on the business and on the bottom line, and utilize these as needed.
Even though data may have been entered accurately the first time around, databases changed regularly; updates are periodically added or removed from system. Don’t be complacent about the “cleanliness” of your data. Do a regular cleansing of your data so as to allow you review information and make sure that data quality is up to par.
Companies that offer data management services must take the necessary steps to see to it that quality is a priority; it’s a continuous process of identifying errors, checking for accuracy, mitigating mistakes, and reiterating the best practices that are being successfully utilized by top performers in the industry.
CHAPTER 5: Finding Your Perfect Data Management Team
To manage data effectively and efficiently, it’s important to hire a competent team of professionals who know their roles very well. Clearly defining roles from the outset will reduce confusion and conflict. It is a collaborative effort, which is why having each member know exactly what their roles are will eliminate guess work and provide outstanding results.
What roles are needed to help you manage your data requirements effectively and efficiently? We’ve enumerated some of the key functions, a few of which might be fulfilled by a single individual. For example, quality assurance staff could also be in charge of training and development responsibilities. In some cases, both roles might even be part of the job description of one manager, particularly for data sets that do not require frequent classroom-style training or regular updates.
Here’s what you need to run a well-oiled data management team.
The supervisor oversees the overall operation of the data management team, which include the following tasks:
- Acts as the team leader
- Mentoring date entry staff
- Supervising the interview process and recording daily activities
- Ensuring data quality
- Managing human resource issues
- Sending progress reports to clients
- Assign roles with staff that have the proper skill sets
- Monitor results
- Establish data quality objectives
Skills and attributes:
- Ability to work in teams and motivate staff
- Organized and efficient in implementing projects
- Ability to mobilize a team to work efficiently
- Collaborate with internal and external teams
- Able to establish, monitor, and implement corrective measures as needed
- Thorough understanding of the objectives of assigned projects
- Detail oriented
- Systematic work experience
Data entry staff encode or enter data in the database. They are recruited to enter, validate, and check the data.
Skills and Attributes:
- Good oral and written communication skills
- Attention to detail
- Accurate keyboard or typing skills
- Computer literate, or familiar with the system being used
- Methodological and tidy work habits
- Ability to follow instructions consistently but raise concerns when appropriate
- Interact efficiently with others to achieve accurate results
Data analysts are assigned to undertake a descriptive analysis of the data gathered. They do exploratory analyses, which include calculating estimations, producing graphs and tables for reports, and assisting in report preparation.
Skills and Attributes:
- Must have a science or computing background
- Competent in using computer or technical systems
- Strong analytical and objective inclination
- Detail oriented
- Able to produce accurate reports in a timely manner
This role is in charge of establishing data quality objectives and prioritizes the accuracy of reports and work output.
Skills and Attributes:
- Develops a training plan
- Identifies areas for improvement and implements corrective action as needed
- Establishes quality objectives
- Checks for compliance to quality objectives
- Conducts training and development programs
- Provide training for new hires
While each member of the data management staff must have the proper training to enable them to perform their own individual tasks, they must also recognize the fact that they are part of an intricate whole.
Working in silos without understanding how each moving part contributes to the final outcome might work temporarily, but it is unsustainable if you want a winning team. Regular training and continuously measuring results against quality standards must also be done to succeed in data management.
Download our Comprehensive Guide to Data Management for Businesses and learn how to leverage data for your business effectively
Lisa Arthur, What Is Big Data?
SAS Software, What Is Big Data?
Chris Poelker, 4 pillars of data management
Michele Nemschoff, A Quick Guide to Structured and Unstructured Data
Marc Vael, How to manage big data and reap the benefits
IDG Enterprise, Research Reports Big Data
IBM, What is big data?
Tata Consultancy Services, The Emerging Big Returns on Big Data
Ash Ashutosh, Best Practices For Managing Big Data
Sushil Pramanick, 10 Big Data Implementation Best Practices
Lindsay Wise, Five first steps to creating an effective ‘big data’ analytics program
Gartner, Gartner Says Big Data Creates Big Jobs: 4.4 Million IT Jobs Globally to Support Big Data By 2015
PWC, Integrated reporting
DirectMarketingNews, Data Quality Is Lacking
Dan Bieler, How Data Can Enable Business Disruption: Traditional Retailers Must Take Note Of The Sharing Economy
Tower Data , 4 Steps to Eliminating Human Error in Big Data
Focus on Training, £175,000 Fine for Data Breach by UK Holiday Insurance Company
Kiley Nichols, 5 Things Every Business Should Know about Data Privacy and Security in 2015
Data One, Define roles and assign responsibilities for data management
WHO, Section 2: Roles and Responsibilities
Jessica Banks, Define Cross-Functional Team Roles In Master Data Management Strategies
Read more here:: A Comprehensive Guide to Data Management for Businesses