What is Apache Spark ?
Apache Spark is a cluster computing technology and mainly designed for fast computation. Spark Based on MapReduce and used to more type of computation process which includes more queries and stream processing. Spark is an memory cluster computing that increases processing speed of Hadoop Applications.
Benefits of Apache Spark
- Apache Spark cluster computing technology has in memory feature.
- Using Apache Spark you can get high data processing speed which is nearly about 100x & 10x faster for memory and disk.
- Apache Spark provides 80 high-level operators which can be used develop parallel application easily.
- The best part of Apache Spark is it supports multiple programming languages like Java, Scala, Python, R.
- Spark is the best option when comparing with Hadoop with respect to cost, since in Hadoop large data center and storage is required to store the Big Data.
Course Features
- Duration60 hours
- Skill levelAll level
- Batch Strength15
- AssessmentsYes
- Mock InterviewsYes
- Resume BuildingYes
- PlacementsYes
- Flexible TimingYes
- Fee InstallmentsYes
- LanguageTamil/English
Section 1: Introduction to Scala for Apache Spark
Learning Objectives - In this module, you will understand the basics of Scala that are required for programming Spark applications. You can learn about the basic constructs of Scala such as variable types, control structures, collections, and more.
- What is Scala?
- Why Scala for Spark?
- Scala in other frameworks
- Introduction to Scala REPL
- Basic Scala operations
- Variable types in Scala
- Control Structures in Scala
- For each loop
- Functions, Procedures, Collections in Scala-Array
- ArrayBuffer
- Map, Tuples, Lists, and more.
Section 2: OOPS and Functional Programming in Scala
Learning Objectives - In this module, you will learn about object oriented programming and functional programming techniques in Scala.
- Class in Scala
- Getters and Setters
- Custom Getters and Setters
- Properties with only Getters
- Auxiliary Constructor
- Primary Constructor
- Singletons
- Companion Objects
- Extending a Class
- Overriding Methods
- Traits as Interfaces
- Layered Traits
- Functional Programming
- Higher Order Functions
- Anonymous Functions and more.
Section 3: Introduction to Big Data and Apache Spark
Learning Objectives - In this module, you will understand what is big data, challenges associated with it and the different frameworks available. The module also includes a first-hand introduction to Spark.
- Introduction to big data
- Challenges with big data
- Batch Vs. Real Time big data analytics
- Batch Analytics - Hadoop Ecosystem Overview
- Real-time Analytics Options
- Streaming Data- Spark
- In-memory data- Spark
- What is Spark?
- Spark Ecosystem
- Modes of Spark
- Spark installation demo
- Overview of Spark on a cluster Spark Standalone cluster Spark Web UI.
Section 4: Spark Common Operations
Learning Objectives - In this module, you will learn how to invoke Spark Shell and use it for various common operations.
- Invoking Spark Shell
- Creating the Spark Context
- Loading a file in Shell
- Performing basic Operations on files in Spark Shell
- Overview of SBT
- Building a Spark project with SBT
- Running Spark project with SBT
- Local mode
- Spark mode
- Caching overview
- Distributed Persistence
Section 5: Playing with RDDs
Learning Objectives - In this module, you will learn one of the fundamental building blocks of Spark - RDDs and related manipulations for implementing business logic.
- RDDs
- Transformations in RDD
- Actions in RDD
- Loading data in RDD
- Saving data through RDD
- Key-Value Pair RDD
- MapReduce and Pair RDD Operations
- Spark and Hadoop Integration-HDFS
- Spark and Hadoop Integration-Yarn Handling Sequence Files and Partitioner.
Section 6: Spark Streaming and MLlib
Learning Objectives - In this module, you will learn about the major APIs that Spark offers. You will get an opportunity to work on Spark streaming which makes it easy to build scalable fault-tolerant streaming applications, MLlib which is Spark’s machine learning library.
- Spark Streaming Architecture
- First Spark Streaming Program
- Transformations in Spark Streaming
- Fault tolerance in Spark Streaming
- Checkpointing
- Parallelism level
- Machine learning with Spark
- Data types
- Algorithms– statistics
- Classification and regression
- Clustering
- Collaborative filtering
Section 7: GraphX, SparkSQL and Performance Tuning in Spark
Learning Objectives - In this module, you will learn about Spark SQL that is used to process structured data with SQL queries, graph analysis with Spark, GraphX for graphs and graph-parallel computation. You will also get a chance to learn the various ways to optimize performance in Spark.
- Analyze Hive and Spark SQL architecture
- SQLContext in Spark SQL
- Working with DataFrames
- Implementing an example for Spark SQL
- Integrating hive and Spark SQL
- Support for JSON and Parquet File Formats
- Implement data visualization in Spark
- Loading of data
- Hive queries through Spark
- Testing tips in Scala
- Performance tuning tips in Spark
- Shared variables: Broadcast Variables
- Shared Variables: Accumulators.
Section 8: A complete project on Apache Spark
Learning Objectives - In this module, you will get an opportunity to work on a live Spark project where you can implement the learnings from previous modules hands-on, and solve a real-time use case.Problem Statement: Design a system to replay the real time replay of transactions in HDFS using Spark. Technologies used:
- Spark Streaming
- Kafka (for messaging)
- HDFS (for storage)
- Core Spark API (for aggregation)
You will be going through detailed 2 to 3 months of Apache Spark Hands-on training
- Detailed instructor led sessions to help you become a proficient Expert in Apache Spark.
- Build a Apache Spark professional portfolio by working on hands on assignments and projects.
- Personalised mentorship from professionals working in leading companies.
- Lifetime access to download able Apache Spark course materials, interview questions and project resources.
Credo Systemz - Velachery, Chennai
Call Us +91 9884412301
Credo Systemz - OMR, Chennai
Call Us +91 9600112302
Can’t find a batch you were looking for?
