Wednesday 19 March 2014

Stream Computing for Real-time Analytics: Overview, Integration and Use Cases



 
Stream computing delivers real-time analytic processing on constantly changing data in motion. It enables descriptive and predictive analytics to support real time decisions. Stream computing allows you to capture and analyze all data - all the time, just in time. Relational databases and warehouses find information stored on disk. Streams analyses data before you store it. Key points here are -

 

1) Stream is the right capability when the primary big data challenge is analyze data that is in motion (Velocity) – because the business imperative requires a real-time response/action based on analyzing the data or the data is very large and want to more cost-effectively filter and remove data before moving into your data warehouse or Hadoop system.  It can handle continuous or bursty streams of data – millions of events per second with microsecond latency.
2) Streams can process any type of data (Variety) – audio, video, network logs, sensors, social media such as Twitter, in addition to structured data.
3) And, Streams is designed to scale to process any size of data from Terabytes to Zetabytes per day

Stream computing changes where, when and how much data you can analyze. Store less, analyze more, and make better decisions, faster with stream computing. The benefits of streaming analytics are immediately obvious.  Dramatic cost savings by analyzing data and only storing what is necessary.  The ability to detect and make real-time decisions, results in customer retention to detect fraud to cross-selling a product. 

IBM InfoSphere Streams for Stream Computing

IBM InfoSphere Streams is an advanced analytic platform that allows user-developed applications to quickly ingest, analyze and correlate information as it arrives from real-time sources. InfoSphere Streams is designed to handle very high data throughput rates, up to millions of events per second. A market leader in providing sophisticated analytics for IoT, IBM received the 2013 Ventana Research award for Operational Intelligence in the IT Innovation category for InfoSphere Streams.

Core highlights are -

  • Perform advanced real-time analytics on data in motion
  • Rapidly ingest, correlate and continuously analyze a massive volume and variety of structured and unstructured streaming data as it arrives from thousands of sources
  • Make real-time predictions and discoveries as data arrives
  • Visualize data easily with drag-and-drop development tools
  • Detect and respond to critical events immediately
  • Learn and update models for future analysis and trend prediction with cognitive computing

InfoSphere Streams helps you:
  • Analyze data in motion—provides sub-millisecond response times, allowing you to view information and events as they unfold. Tools facilitate sophisticated analytics, such as geospatial, voice, image and text, and also update models on the fly.
  • Simplify development of streaming applications—uses an Eclipse-based integrated development environment (IDE). Developers are able to easily and rapidly build applications and connect to new data sources. Drag-and-drop editors, wizards, visualization tools, and runtime monitoring and debuggers are available.
  • Extend the value of existing systems—integrates with your applications, and supports both structured and unstructured data sources. The supporting infrastructure adapt to rapidly changing data formats, types and messaging protocols. It also read from and writes to a vast number of data sources. A massively parallel architecture is designed to deliver unlimited compute potential.
 
IBM Infosphere Streams capabilities are designed to work together and with existing bigdata & analytics applications such as BI and predictive analytics.  Here’s an example scenario:

1)     Historic data is stored in the DB/warehouse (DB2, Infosphere Warehouse, Informix, Oracle, solidDb, MySQL, SQLServer, Netezza etc.) where interesting patterns are detected using database toolkit operators, such as the pattern of credit card transactions that would indicate possible fraud. Support for XML allows developers to fuse a broader range of traditional and untraditional data.

2)     IBM SPSS leverages IBM SPSS Modeler to develop and build predictive models, and then deploy them using the SPSS Scoring Operator. The PMML models are then imported into InfoSphere Streams Studio to generate Streams programs that are executed to score the incoming records in real time without suspending InfoSphere Streams applications.

3)     Additional data sources such as RFID tags, blogs, or other information might be used to improve the confidence levels of the scoring algorithms.

4)     These measures can be sent to Dashboards like IBM Cognos Real Time Monitoring or business process management (BPM) systems to trigger business processes to take immediate action as required.

5)     IBM InfoSphere BigInsights lets you store streaming data in an enterprise-class Hadoop environment for additional analysis or historic retention. InfoSphere Streams and InfoSphere BigInsights use the same advanced text analytics capabilities to simplify natural language processing applications for both data in motion and data at rest. In addition, InfoSphere BigInsights can be used to augment streaming sources with contextual information, and users can visualize InfoSphere Streams data in the InfoSphere BigInsights console.

6)     Streams real-time analytics can be integrated with ETL solutions like IBM DataStage helps get more timely results and offload some analytics load from the warehouse. IBM InfoSphere DataStage helps users perform deep analysis and gain additional insight using contextual and source data from other parts of the infrastructure.

7)     Messaging queues allow InfoSphere Streams to receive data from or send data to IBM WebSphere MQ, IBM MessageSight and Java Messaging System (JMS) offerings.

8)  IBM InfoSphere Data Explorer enables users to visualize InfoSphere Streams data in the InfoSphere Data Explorer CXO dashboard and add streaming data to the InfoSphere Data Explorer index. 
 
Stream computing use cases
When companies can analyze ALL of their available data, rather than a subset, they gain a powerful advantage over their competition. Many customers are seeing tangible ROI using IBM Stream solutions to address their big data challenges:
  • Healthcare: 20% decrease in patient mortality by analyzing streaming patient data
  • Telco: 92% decrease in processing time by analyzing networking and call data
  • Utilities: 99% improved accuracy in placing power generation resources by analyzing 2.8 petabytes of untapped data
Below are few cross-industry scenarios best suitable for stream computing –

1)     Know Everything about your Customers
·         Social media customer sentiment analysis
·         Promotion optimization
·         Segmentation
·         Customer profitability
·         Click-stream analysis
·         CDR processing
·         Multi-channel interaction analysis
·         Loyalty program analytics
·         Churn prediction

2)     Innovate New Products at Speed and Scale
·         Social Media - Product/brand Sentiment analysis
·         Brand strategy
·         Market analysis
·         RFID tracking & analysis
·         Transaction analysis to create insight-based product/service offerings

3)     Instant Awareness of Risk and Fraud - Lower risk, detect fraud and monitor cyber security in real time. Augment and enhance cyber security and intelligence analysis platforms with big data technologies to process and analyze new types (e.g. social media, emails, sensors) and sources of under-leveraged data to significantly improve intelligence, security and law enforcement insight.
·         Multimodal surveillance
·         Cyber security
·         Fraud modeling & detection
·         Risk modeling & management
·         Regulatory reporting

4)     Exploit Instrumented Assets
·         Network analytics
·         Asset management and predictive issue resolution
·         Website analytics
·         IT log analysis

5)     Run Zero Latency Operations
·         Smart Grid/meter management
·         Distribution load forecasting
·         Sales reporting
·         Inventory & merchandising optimization
·         Options trading
·         ICU patient monitoring
·         Disease surveillance
·         Transportation network optimization
·         Store performance
·         Environmental analysis
·         Experimental research

Here’s few usecases in industries to get an idea about the breadth of possibilities that stream technology along with other bigdata products can offer. To explore more details, click on the industry title below.
 

Automotive

  • Data warehouse optimization
  • Predictive asset optimization
  • Connected vehicle
  • Actionable customer insight

Banking

  • Optimize offers and cross sell
  • Contact center efficiency and problem resolution
  • Payment fraud detection and investigation
  • Counterparty credit risk management

Consumer Products

  • Optimized promotions effectiveness
  • Micro-market campaign management
  • Real-time demand forecast

Energy and Utilities

  • Distribution load forecasting and scheduling
  • Create targeted customer offerings
  • Condition-based maintenance
  • Enable customer energy management
  • Smart meter analytics

Government

Geospatial analysis requires complex mathematics such as set theory and geospatial geometry. It is used for location intelligence and location-based services for security and surveillance, geographic information systems, traffic patterns and more. The city of Dublin, Ireland, uses InfoSphere Streams to analyze 50 bus locations per second for its fleet of roughly 1,000 buses. Check it out - http://www-01.ibm.com/software/success/cssdb.nsf/CS/RNAE-9C9PN5?OpenDocument&Site=software&cty=en_us
  • Threat prediction and prevention
  • Social program fraud, waste and errors
  • Tax compliance - fraud and abuse
  • Crime prediction and prevention

Healthcare

  • Measure and act on population health
  • Engage consumers in their healthcare
  • Health monitoring and intervention

Insurance

Knowing the order of events can have profound impacts, for example in predicting the path of a natural disaster or picking the next best stock trade. InfoSphere Streams helps insurance companies plan for natural disasters and enables real-time public alerts. It also performs real-time analysis of sensor data collected from the Hudson River, one of the most instrumented bodies of water in the world. Check this out - https://www.youtube.com/watch?v=y3CZQOtVx6s&list=PLA98824D75176BAEB&index=18
  • Claims fraud detection
  • Next best action and customer retention
  • Catastrophe risk modeling
  • Usage-based insurance
  • Portfolio management
  • Producer optimization

Oil & Gas

  • Advanced condition monitoring
  • Drilling surveillance & optimization
  • Production surveillance & optimization

Retail

  • Merchandise optimization
  • Actionable customer insight

Telecommunications

Telecommunications service providers continue to experience a huge growth in smartphone and mobile device use. Growing text and data usage creates a deluge of context- and time-sensitive data. InfoSphere Streams enables telecommunications providers to analyze billions of call data records per day to detect fraud, ensure high asset utilization and create accurate customer profiles for heightened customer service and retention. Using InfoSphere Streams, Sprint reduced storage costs by 90 percent. Check this out - https://www.youtube.com/watch?v=eg8KSLAZ2HM&feature=player_embedded
  • Pro-active call center
  • Smarter campaigns
  • Network analytics
  • Location-based services

Travel & Transportation

  • Customer analytics and loyalty marketing
  • Capacity & pricing optimization
  • Predictive maintenance optimization
 Reference:

39 comments:

  1. Cognos Enterprise Planning Cognos Enterprise Planning Training "Cognos Enterprise Planning Online Training
    Send ur Enquiry to contact@21cssindia.com
    Introduction to Enterprise Planning & Business Intelligence
    Enterprise Planning - Business Intelligence Overview" more… Online Training- Corporate Training- IT Support U Can Reach Us On +917386622889 - +919000444287 http://www.21cssindia.com/courses/cognos-enterprise-planning-online-training-85.html

    ReplyDelete
  2. Security Intelligence Solution provides one-click access to a comprehensive forensic trail and analytics in the same solution to simplify and accelerate threat discovery and incident investigation. To know more, visit Hadoop Training Bangalore

    ReplyDelete
  3. Today, the demand of a trained and professional business analytics has witnessed a massive growth. Business analytics training in this field is the most important thing to meet the requirement

    ReplyDelete
  4. Great Post, I read it well. very useful info Thank you so much for sharing this one really well defined all peaceful information...Network Analytics Market Report | Enterprise Social Software (ESS) Market Report

    ReplyDelete
  5. Great thoughts you got there, believe I may possibly try just some of it throughout my daily life.


    Retail Management Software

    ReplyDelete
  6. It seems you are so busy in last month. The detail you shared about your work and it is really impressive that's why i am waiting for your post because i get the new ideas over here and you really write so well.
    digital marketing training in tambaram

    digital marketing training in annanagar

    ReplyDelete
  7. Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.
    full stack developer training in annanagar

    full stack developer training in tambaram

    full stack developer training in velachery

    ReplyDelete
  8. This blog is the general information for the feature. You got a good work for these blog.We have a developing our creative content of this mind.Thank you for this blog. This for very interesting and useful.
    python training institute in chennai
    python training in Bangalore
    python training in pune

    ReplyDelete
  9. Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Blueprism training in tambaram

    Blueprism training in annanagar

    Blueprism training in velachery

    ReplyDelete
  10. Good Post, I am a big believer in posting comments on sites to let the blog writers know that they ve added something advantageous to the world wide web.
    java training in chennai | java training in bangalore


    java training in tambaram | java training in velachery

    ReplyDelete
  11. Were a gaggle of volunteers as well as starting off a brand new gumption within a community. Your blog furnished us precious details to be effective on. You've got completed any amazing work!
    Java training in Chennai | Java training in Omr

    Oracle training in Chennai

    Java training in Chennai | Java training in Annanagar

    Java training in Chennai | Java training institute in Chennai | Java course in Chennai

    ReplyDelete
  12. Needed to compose one simple word yet thanks for the suggestions that you are contributed here, please do keep updating us...
    Hadoop Training in Chennai | Hadoop Online Training

    ReplyDelete
  13. Excellant post!!!. The strategy you have posted on this technology helped me to get into the next level and had lot of information in it.
    Devops Training in Bangalore
    Best Devops Training in pune
    Microsoft azure training in Bangalore
    Power bi training in Chennai

    ReplyDelete
  14. I would like to thank you for the efforts you have made in writing this article. I am hoping the same best work from you in the future as well. In fact your creative writing abilities has inspired me to start my own BlogEngine blog now. Really the blogging is spreading its wings rapidly. Your write up is a fine example of it.
    Data science training in bangalore

    ReplyDelete
  15. Very nice post here and thanks for it .I always like and such a super contents of these post.
    Excellent and very cool idea and great content of different kinds of the valuable information's.

    Java training in Bangalore



    ReplyDelete
  16. I am really very happy to find this particular site. I just wanted to say thank you for this huge read!! I absolutely enjoying every petite bit of it and I have you bookmarked to test out new substance you post.
    Microsoft Azure online training
    Selenium online training
    Java online training
    Java Script online training
    Share Point online training

    ReplyDelete
  17. Thanks for sharing a useful information.. we have learnt so much information from your blog.... oracle training in chennai

    ReplyDelete
  18. I read this post two times, I like it so much, please try to keep posting & Let me introduce other material that may be good for our community.

    ReplyDelete
  19. I read this post two times, I like it so much, please try to keep posting & Let me introduce other material that may be good for our community. data science courses

    ReplyDelete
  20. Nice & Informative Blog !
    QuickBooks is an easy-to-use accounting software that helps you manage all the operations of businesses. In case you want immediate help for QuickBooks issues, call us on Quickbooks Customer Service Number 1-855-550-7546.

    ReplyDelete
  21. Nice & Informative Blog !
    you may encounter various issues in QuickBooks that can create an unwanted interruption in your work. To alter such problems, call us at QuickBooks Customer Service Number and get immediate technical services for QuickBooks in less time.

    ReplyDelete
  22. Wow! Such an amazing and helpful post this is. I really really love it. I hope that you continue to do your work like this in the future also.

    Big Data Hadoop Online Training
    Online Training For Big Data

    ReplyDelete
  23. I see some amazingly important and kept up to length of your strength searching for in your on the site
    data scientist course in hyderabad

    ReplyDelete
  24. So, you have made it through the many obstacles of everyday life and are now a truly happy baby boomer. What are some things that you do in order to keep the same feeling and make it through these tough times? In this article I am going to list five things that I do every day that helps me to stay really happy:Buy tinder accounts

    ReplyDelete
  25. Hey! Lovely blog. Your blog contains all the details and information related to the topic. In case you are a QuickBooks user, here is good news for you. You may encounter any error like QuickBooks Error, visit at QuickBooks Customer Support Number for quick help.

    ReplyDelete
  26. Thank you for taking the time to write this essay. It has proven to be really useful to me. Your essay was enjoyable to read. Continue to update your blog.
    please visit my site: Buygenmeds
    One of the most well-known and well-known little blue pills for the treatment of erectile dysfunction is Cenforce 100 mg tablets.

    ReplyDelete
  27. This comment has been removed by the author.

    ReplyDelete
  28. Very interesting and informative post. thanks for this nice post.
    Vidalista black 80 has truly transformed my intimate experiences. It has provided me with a renewed sense of confidence and vitality, allowing me to fully embrace moments of passion and connection with my partner. The effects are remarkable, and the sense of fulfillment it brings is unparalleled. I highly recommend Vidalista Black 80 to anyone seeking to revitalize their intimate life.

    ReplyDelete