Architecting Big Data Solutions

Architecting Big Data

In this article I would like to outline, at a high level, the overall thought process for “architecting” big data solutions. The process of developing an architecture is a very personal thing. I don’t propose to say that this is the only way – this is simply a framework from which you could launch your own effort.

Architecting Big Data

Architecting Big Data

Stage 1: Determine the Business Goal

The starting point for every architecture must be the business.

Some of the questions that should be answered are:
“What business goal or goals am I trying to achieve with this initiative?” This question must give you the problem that you are trying to solve in the language of the business. Something like “What are people saying on the internet about product X in the last 30 days.” presents a clear gold that can be acted on.

“How will the solution data, process, and systems integrate into or change the business process? By defining these characteristics we can determine the effect the solution will have, any inconsistencies or conflicts with existing business processes. This activity will generate additional clarity for all involved.

“Have I confirmed that the proposed change is required, beneficial, and consumable by the business?” One of the questions that we don’t ask enough is if the change is going to benefit the company or is it just change for the sake of change. Many times we can develop a solution that an excellent technical solution but does not fit with the organization. Benefit generation and Organizational fit provide solutions that cause less resistance to change.

Aligning your design efforts with the business is paramount to success. With Big Data it is doubly important in that the data and the analysis are significantly more impacting, especially when they will be directly influencing executive thinking. When we use Big Data to generate analytic data, executive dashboards or decision support systems we are presenting or executive leaders with decision making tools we must never take this privilege lightly.

Business alignment of IT projects creates a more efficient IT operation, better return on investment, reduced risk, and better clarity of vision.

IT operations are more efficient when aligned with business goals since only the projects that return business value are undertaken reducing wasted effort and erosion of the business’ trust. Trust takes a moment to lose and a lifetime to regain.

Return on investment is critical in demonstrating that IT understands the business drivers and is being a good steward of the business’ investment. We must always strive to demonstrate our understanding and unwavering support for the business.

IT can reduce business risk by thoroughly validating the risk factors for a solution and exposing them clearly to the business. Many businesses will accept these risks if the perceived payoff is large enough. When IT does not expose these risks to business we are being irresponsible and eroding trust.

Stage 2: Determine the Source and Type of Data

Now that we have a clear understanding of the business goals and how the business will utilize the data, we are ready to take a look at the source and type of data that you will be utilizing.

The source and type of data will drive the architectural decisions around integration, access, security, and extract-transform-load (ETL). We must determine how we will get the data, how to handle the data and how to secure the data to have a clear vision of how we will design the proposed solution. This activity will also provide us with an understanding of what is possible and not possible.

For example, if we are mining social media for comments on the company’s products we will have a much different design and implementation than if we were to collect data from the company’s web site clicks.

Stage 3: Develop & Document the Proposed Solution

Now that we have a clear understanding of the Business Goals and the Data we will be working with we can begin to develop the solution in line with the business goals, organizational capabilities, and enterprise standards.

Every aspect of the proposed solution should be tied to the business goals that we are supporting. This linkage will provide a strong indication to the business that we are thinking about their needs and now this system will support those needs.

As architects, system designers and business analysts we need to consider the organizational structure that we are designing solutions for so that we make good decisions that fit. For example a designer specifying a solution using Java in a dot net shop is likely to be taken out and shot. (well maybe not)

One of the places that we as designers can add a lot of value during the architecture process is when we are thinking through the architecture. When an innovative thought hits and we can predict some future requirement, issue or opportunity. We can present those in the architecture for discussion with the business. Everyone wins even if the lightning bolt was really a wet noodle at least we have thoroughly thought through the idea, communicated it and rationally included or excluded it from the solution documentation.

Conformance with Enterprise Standards presents another opportunity to support the operational efficiency of the business by seeking reuse and intellectual capital development. If we follow the enterprise standards and along the way develop additional capabilities that other units can use I have invested in the company for the future.

Documentation is something that must be done intelligently. We can definitely stall a project if we try to create too much documentation. Too little documentation will leave ambiguity in the solution that may come back to bite us. My general rule of thumb is to document to the point where ambiguity is removed and clarity of business and technical purpose are achieved.

Architecting Big Data

Architecting Big Data

Stage 4: Validate the Proposed Solution

This stage is critically important. This is the time that we are both validating the appropriateness of the proposed solution to the business and selling the solution to those who are not currently on board.

Call a meeting with the stakeholders and present the solution to them in their language. Demonstrate how this meets the business goals provided. Describe the risks and any mitigation strategies that we have developed for those risks. Clear up any lingering ambiguity that may still be present and get them input.

This activity will help the project to gain traction and garner buy in from your business stakeholders. Generating a win / win situation is the only way to drive a project to a successful solution.

Stage 5: Design the Final Solution

Now all of the input has been collected we are in a position to get down to the details of how we are going to technically implement the solution. We need to complete the systems and technology architecture design and documentation. Develop a team profile with disciplines, skills and experiences to assist with team selection and staffing. We need to work with the Project Manager to create a project and staffing plan. And finally work with the Implementation Teams to get the solution build, implemented and tested.

Also posted on SmartData Collective

Big Data

Over the past couple of years Big Data has really become a buzzword in the business and IT landscape. I thought that I would take some time to distill the hype down into something that we as business executives can consume. My hope is that this will provide some thought provoking context for your thinking on Big Data and how you might be able to take advantage of this hot topic.

What is Big Data?

Many believe that Big Data is the same as Business Intelligence or simply a very large volume of data. Although those are generally included in the definition of Big Data I would suggest that Big Data is much more than this.

Big Data is about what you do with the data the the actual data, the source of the data and the type of the data.

Big Data
Big data has become a catchphrase for voluminous data, business intelligence, streaming data, unstructured data and on and on. These sources are accurate but don’t really help us to understand how to apply Big Data to our business decisions.

Our best approach to this overused generic term is to pick an area to focus on that returns business value to us and explore that prior to widening the adoption. Focus is always going to pay larger dividends than the “shotgun approach”.

Many companies are starting with Big Data as Business Intelligence, Analytics or Reporting. These are good starting points but miss the true intent and value of Big Data.

Big Data Opportunities

Using the definition that I propose above we can see how the source and type of data will greatly impact the opportunities available to us in the use and application of these resources.

For example, if you have sensor data from the production line that can be captured analyzed and pumped into a decision support system you could be generating a significant differentiator for your company.

Another example could be that you collect anecdotal data from the internet relating to public perception and comment on your products and services and quantify those into performance indicators you again differentiate your company from the pack.

Big Data and Data Science opportunities abound and I believe that only our imagination and innovation constrain the potential uses.

The Business Value of Design

Throughout my career I have experienced various “philosophies” in planning and design of IT projects. Some choose the fire fighter philosophy, while others choose the “agile as an excuse for insufficient design”. On the other end of the scale I have found those that won’t move without every aspect of the system elaborated 6 different ways.

I tend to use the “design where ambiguity exists” philosophy. To be sure I have spent much of my career in design and strategy roles so I am a HUGE proponent of design. That said I always try to design with the business objectives in mind and document with a purpose.

For the purpose of this document I am using the word design in a very general sense. Design comes in various clothes like Business Process Design, Software Design, Software Architecture and Enterprise Architecture. Each of these aspects of design provide their own opportunities for returning value to the business.

There will be many different schools of thought here and opinions will abound this article presents mine.

Characteristics of a Good Design

So up to this point I have discussed the philosophy of design but how does design provide business value?  How can something that is seemingly just a cost return tangible value to the business?

Before we get to the business value of design we should understand the characteristics of a good design.

A Good Design is Clear
When designing a solution it is imperative that the resulting design is clear and unambiguous. The design should be described in the language of the consumer not necessarily the designer.

For example you have a software system design where the designer is a software architect or developer and the consumer is a line of business expert. A clear design would be written in the language of the business with as little techno gorp as possible.

The design should be easy to understand with the express purpose of clearing any ambiguity and elaborating on any areas of potential conflict. If there are various ways of executing on a specific feature, function, process or task the end state design should pick one and detail the selected option clearly.

A Good Design is Thoughtful
The primary job of a design is to consider a problem domain and present a solution that is a well reasoned. There are almost always many different ways to solve and design problem so the job of the designer is to weigh as many options as feasible and present a reasoned approach, the design.

The thoughtfulness of the design will be manifest in the care that the designer takes in considering the business needs, consumer’s viewpoint and the presentation.

The elaboration of the design should have both a clear directive tone supported by rationale of why the option was selected rather than one of the alternates. Don’t make your consumer wonder why.

A Good Design is Thorough
Thoroughness is the trademark of a good design. That does not mean that you necessarily have to consider every conceivable option, path or permutation but you should consider those that fit within the guiding principles that you have developed. Some alternatives deserve little more than a quick glance some require in-depth analysis. If you are going to do an in-depth analysis the amount of coverage you give it in the resulting design artifact should be related to the suitability of the alternative to your objective.

If you have multiple suitable alternatives that have been thoughtfully and thoroughly considered the rationale should be expressed in the design artifact so your consumer has enough support to understand why you selected the option you did. Don’t make your consumer do their own research.

A Good Design is Balanced
Balancing your design and the resulting artifact will be one of the toughest jobs for the designer. What goes in? what doesn’t? How much or how little detail to include? What tone to use? What biases are at play? are all questions that the designer must answer to present a balanced design.

Presenting a balanced design only comes through experience and doing the hard work of critical thinking and objectively looking at the design and resulting artifact. Don’t put your consumer off by an unbalanced design.

Business Value 1: A Clear Target

In giving the business a clear target for the problem domain presented in a thoughtful, thorough and balanced way you save the company money. The value of this aspect is often missed by the “fire fighter” mentality. They say it costs too much to spend time designing. I say if you can’t afford to design you can’t afford to do the project.

Clarity induces innovation. When the problem domain is clearly understood and the subject matter experts can see the whole picture innovation is just around the corner. I can’t tell you how many times as soon as I documented a process and started discussing it with the team how the innovation juices begin to flow. People literally get excited about the opportunity to make a difference.

An investment in the initial design will pay dividends for the life of the resulting artifact.

Business Value 2: Reduce waste

With a Clear Target the business reduces wasted effort, resources and time thus saving a tremendous amount of money and frustration. How many times have we started a project without a proper design and then paid for it every day of the project? How many dead ends have been explored at an expense to the business that could have been avoided? How many times have we had to live with a sub-optimal solution because, “we have already invested so much in this direction”?

I could go on endlessly here but the point is that you will pay something now or pay a lot more later but you will pay. Determine the budget for the design and execute on it. Design always pays dividends.

Business Value 3: Trust

Presenting a design as I am outlining in this article cultivates trust and respect from the business, peers and your consumer community. Each person that is part of the process can feel like they were heard and contributed to the overall bottom line of the business. The business value here is immeasurable.

This one is really a win-win in that both the design team and the business benefit through a stronger corporate community. Operating from a position of trust and respect based on experience geometrically accelerates the business values espoused in this article.

Business Value 4: Quality

A designed solution is generally a quality solution. The quality aspect of the designed artifact is lasting and pleasurable. The designed artifact just does what it is supposed to do the way it is supposed to do it. When change comes, and it always does, the designed artifact is easier to adapt due to the clarity exposed in the design.

A quality solution is generally less expensive to support, change and maintain.

I hope that I have presented a system and method of generating an adaptable design philosophy that you can use.

The resulting design should present the object of the design in a way that someone new to the project, with the business background, could understand it and contribute to a business discussion.

Sorry this article became longer than intended, sadly could have been much longer, but I hope that you enjoyed it and benefit from it.


Asynchronis Pattern (Part 4)

Part 4

The Event-based Asynchronous Pattern makes available the advantages of multithreaded applications while hiding many of the complex issues inherent in multithreaded design.

Using a class that supports this pattern can allow you to:

  • Perform time-consuming tasks, such as downloads and database operations, “in the background,” without interrupting your application.
  • Execute multiple operations simultaneously, receiving notifications when each completes.
  • Wait for resources to become available without stopping (“hanging”) your application.
  • Communicate with pending asynchronous operations using the familiar events-and-delegates model. For more information on using event handlers and delegates, see Events and Delegates.

By using the Asynch pattern in the LightSwitch HTML client the developer can build responsive user interfaces that are capable of doing a lot of work. Typically we use a BackgroundWorker component when building simple multi-threaded applications but as we start to build more complex solutions we will want to build a set of methods, generally analogous to the ones working on the current thread, to handle the computing. The biggest challenge that I have found in doing this kind of work is the lack of ability to update across threads. With some ingenuity you can overcome this nicely. LightSwitch HTML client solves many of those issues for you as it generates the application.

In the HTML client, we chose to expose promise objects to represent asynchronous work. Promise objects provide the most flexible set of options when working in an asynchronous environment and make it easier to structure code in its logical order of execution.

There are a number of promise object implementations available. We chose to utilize the WinJS implementation of promise objects as it was one of the more complete offerings and provided technical alignment with the Windows 8 development experience.

Additional Reading

Event-based Asynchronous Pattern Overview on MSDN
Using Promise objects with LightSwitch