package com.ercancan.spark
import org.apache.log4j.{Level, Logger}
import org.apache.spark.{SparkConf, SparkContext}
/**
* Spark word count example
* @author Ercan Can
*/
object WordCount {
//Main function which implements the primary logic of SPARC job
def main(args: Array[String]) {
/*
Windows binaries for Hadoop versions
Apache Spark requires the executable file winutils.exe to function correctly on the
Windows Operating System when running against a non-Windows cluster.
*/
System.setProperty("hadoop.home.dir", "C:\\winutils")
Logger.getLogger("org").setLevel(Level.ERROR)
/*
Create the SPARK configuration. This is entry point to SPARK applicaion.
Our application named as "wordcount". We're going to run our application on an embedded SPARK instance on our
local box which cold use up to two core cpu.
*/
val conf = new SparkConf().setAppName("wordcount").setMaster("local[2]")
val sc = new SparkContext(conf)
/*
Load word count article as an RDD
*/
val lines = sc.textFile("inputs/wordcount.text")
/*
Split the article into seperate words using whitespace as the delimiter
*/
val words = lines.flatMap(line => line.split(" "))
/*
Calculate the occurence of each word and printed out the results
*/
val wordCounts = words.countByValue()
for ((word, count) <- wordCounts) println(word + " : " + count)
}
}
Want GitHub Copilot-level code assistance — but fully offline, air-gapped, and private? This guide walks you through setting up Continue (a VSCode extension) with Ollama (a local model runner) so you get chat, autocomplete, and inline editing powered entirely by models running on your own machine. No internet required at runtime. Table of Contents What Are Continue and Ollama? Prerequisites & Hardware Requirements Choosing the Right Models Step 1 — Installing Ollama & Pulling Models Step 2 — Installing Continue in VSCode (Offline) Step 3 — Configuring config.yaml Step 4 — Disabling Telemetry Step 5 — Usage & Shortcuts Troubleshooting What Are Continue and Ollama? Continue is an open-source VSCode (and JetBrains) extension that adds AI-powered coding features directly into your editor — think chat, inline edits, and tab autocomplete. By default it can connect to Claude, GPT-4, or any API. But it also supports local providers , which is exactly what we want here. Ollama i...
Comments
Post a Comment