-
Notifications
You must be signed in to change notification settings - Fork 708
Run in Intellij IDEA
oleksii iepishkin edited this page Jan 15, 2014
·
1 revision
You can run your job from IDEA locally. Run -> Edit configurations -> New Application
Option | Value |
---|---|
Main class | com.twitter.scalding.Tool |
VM options | -XX:MaxPermSize=512M -Xmx1024M |
Program arguments | job.class.name --hdfs --param1 value --param2 value --input local-path |
Working directory | ~/projects/scalding-jobs |
Use classpath of module | first-party |
Note that --output is optional usually. When not specified we write to NullSource which prints to stdout.
Here is sample scalding job:
package com.sample
import com.twitter.scalding._
import com.twitter.scalding.Tsv
class SampleJob(args: Args) extends Job(args) {
val input = args("input")
val output = args.getOrElse("output", null)
val results = Tsv(input).read
//todo do something here
if (output != null)
results.write(Tsv(output))
else
results.debug.write(NullSource)
}
- Scaladocs
- Getting Started
- Type-safe API Reference
- SQL to Scalding
- Building Bigger Platforms With Scalding
- Scalding Sources
- Scalding-Commons
- Rosetta Code
- Fields-based API Reference (deprecated)
- Scalding: Powerful & Concise MapReduce Programming
- Scalding lecture for UC Berkeley's Analyzing Big Data with Twitter class
- Scalding REPL with Eclipse Scala Worksheets
- Scalding with CDH3U2 in a Maven project
- Running your Scalding jobs in Eclipse
- Running your Scalding jobs in IDEA intellij
- Running Scalding jobs on EMR
- Running Scalding with HBase support: Scalding HBase wiki
- Using the distributed cache
- Unit Testing Scalding Jobs
- TDD for Scalding
- Using counters
- Scalding for the impatient
- Movie Recommendations and more in MapReduce and Scalding
- Generating Recommendations with MapReduce and Scalding
- Poker collusion detection with Mahout and Scalding
- Portfolio Management in Scalding
- Find the Fastest Growing County in US, 1969-2011, using Scalding
- Mod-4 matrix arithmetic with Scalding and Algebird
- Dean Wampler's Scalding Workshop
- Typesafe's Activator for Scalding