Skip to content

Commit

Permalink
deploy: e511a78
Browse files Browse the repository at this point in the history
  • Loading branch information
pomadchin committed Mar 23, 2024
1 parent a32622b commit 5713b56
Show file tree
Hide file tree
Showing 7 changed files with 22 additions and 22 deletions.
4 changes: 2 additions & 2 deletions Cats.html
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ <h2 id="effect-suspension-in-typed-datasets" class="section"><a class="anchor-li
</span><span class="identifier">count</span><span> &lt;- </span><span class="identifier">typedDs</span><span>.</span><span class="identifier">count</span><span>[</span><span class="type-name">Action</span><span>]()
} </span><span class="keyword">yield</span><span> (</span><span class="identifier">sample</span><span>, </span><span class="identifier">count</span><span>)
</span><span class="comment">// result: Action[(Seq[(Int, String)], Long)] = Kleisli(
// cats.data.Kleisli$$Lambda$15452/0x0000000804388840@3cd334a6
// cats.data.Kleisli$$Lambda$12732/0x0000000803795840@7e1db744
// )</span></code></pre>
<p>As with <code>Job</code>, note that nothing has been run yet. The effect has been properly suspended. To
run our program, we must first supply the <code>SparkSession</code> to the <code>ReaderT</code> layer and then
Expand All @@ -193,7 +193,7 @@ <h3 id="convenience-methods-for-modifying-spark-thread-local-variables" class="s
}
} </span><span class="keyword">yield</span><span> </span><span class="identifier">r</span><span>
</span><span class="comment">// resultWithDescription: Action[(Seq[(Int, String)], Long)] = Kleisli(
// cats.data.Kleisli$$$Lambda$16915/0x0000000804969840@41084deb
// cats.data.Kleisli$$$Lambda$14206/0x0000000803d5a840@42e7ca37
// )
</span><span>
</span><span class="identifier">resultWithDescription</span><span>.</span><span class="identifier">run</span><span>(</span><span class="identifier">spark</span><span>).</span><span class="identifier">unsafeRunSync</span><span>()
Expand Down
4 changes: 2 additions & 2 deletions FeatureOverview.html
Original file line number Diff line number Diff line change
Expand Up @@ -688,8 +688,8 @@ <h2 id="sorting-columns" class="section"><a class="anchor-link left" href="#sort
</span><span class="comment">// +----+-------+--------+--------+
// |city|surface| price|bedrooms|
// +----+-------+--------+--------+
// |Lyon| 45|133000.0| 1|
// |Lyon| 83|200000.0| 2|
// |Lyon| 45|133000.0| 1|
// +----+-------+--------+--------+
// only showing top 2 rows
//</span></code></pre>
Expand All @@ -716,7 +716,7 @@ <h2 id="user-defined-functions" class="section"><a class="anchor-link left" href
</span><span class="comment">// priceModifier: (String, Double) =&gt; Double = &lt;function2&gt;
</span><span>
</span><span class="keyword">val</span><span> </span><span class="identifier">udf</span><span> = </span><span class="identifier">aptTypedDs</span><span>.</span><span class="identifier">makeUDF</span><span>(</span><span class="identifier">priceModifier</span><span>)
</span><span class="comment">// udf: (frameless.TypedColumn[Apartment, String], frameless.TypedColumn[Apartment, Double]) =&gt; frameless.TypedColumn[Apartment, Double] = frameless.functions.Udf$$Lambda$18022/0x0000000804d24840@40258c22
</span><span class="comment">// udf: (frameless.TypedColumn[Apartment, String], frameless.TypedColumn[Apartment, Double]) =&gt; frameless.TypedColumn[Apartment, Double] = frameless.functions.Udf$$Lambda$15277/0x00000008041bc840@704c2976
</span><span>
</span><span class="keyword">val</span><span> </span><span class="identifier">aptds</span><span> = </span><span class="identifier">aptTypedDs</span><span> </span><span class="comment">// For shorter expressions
// aptds: TypedDataset[Apartment] = [city: string, surface: int ... 2 more fields]
Expand Down
8 changes: 4 additions & 4 deletions Injection.html
Original file line number Diff line number Diff line change
Expand Up @@ -144,7 +144,7 @@ <h2 id="example" class="section"><a class="anchor-link left" href="#example"><i
</span><span class="comment">// people: Seq[Person] = List(
// Person(
// 42,
// java.util.GregorianCalendar[time=1709406839758,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id=&quot;Etc/UTC&quot;,offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2024,MONTH=2,WEEK_OF_YEAR=9,WEEK_OF_MONTH=1,DAY_OF_MONTH=2,DAY_OF_YEAR=62,DAY_OF_WEEK=7,DAY_OF_WEEK_IN_MONTH=1,AM_PM=1,HOUR=7,HOUR_OF_DAY=19,MINUTE=13,SECOND=59,MILLISECOND=758,ZONE_OFFSET=0,DST_OFFSET=0]
// java.util.GregorianCalendar[time=1711202985788,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id=&quot;Etc/UTC&quot;,offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2024,MONTH=2,WEEK_OF_YEAR=12,WEEK_OF_MONTH=4,DAY_OF_MONTH=23,DAY_OF_YEAR=83,DAY_OF_WEEK=7,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=2,HOUR_OF_DAY=14,MINUTE=9,SECOND=45,MILLISECOND=788,ZONE_OFFSET=0,DST_OFFSET=0]
// )
// )</span></code></pre>
<p>And an instance of a <code>TypedDataset</code>:</p>
Expand All @@ -167,7 +167,7 @@ <h2 id="example" class="section"><a class="anchor-link left" href="#example"><i
</span><span class="identifier">cal</span><span>
}
}
</span><span class="comment">// calendarToLongInjection: AnyRef with Injection[Calendar, Long] = repl.MdocSession$MdocApp0$$anon$1@64e44cda</span></code></pre>
</span><span class="comment">// calendarToLongInjection: AnyRef with Injection[Calendar, Long] = repl.MdocSession$MdocApp0$$anon$1@70c61909</span></code></pre>
<p>We can be less verbose using the <code>Injection.apply</code> function:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">import</span><span> </span><span class="identifier">frameless</span><span>.</span><span class="identifier">_</span><span>

Expand All @@ -180,7 +180,7 @@ <h2 id="example" class="section"><a class="anchor-link left" href="#example"><i
</span><span class="identifier">cal</span><span>.</span><span class="identifier">setTime</span><span>(</span><span class="keyword">new</span><span> </span><span class="identifier">java</span><span>.</span><span class="identifier">util</span><span>.</span><span class="type-name">Date</span><span>(</span><span class="identifier">l</span><span>))
</span><span class="identifier">cal</span><span>
})
</span><span class="comment">// calendarToLongInjection: Injection[Calendar, Long] = frameless.Injection$$anon$1@4fa0456f</span></code></pre>
</span><span class="comment">// calendarToLongInjection: Injection[Calendar, Long] = frameless.Injection$$anon$1@7d4fe061</span></code></pre>
<p>Now we can create our <code>TypedDataset</code>:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">val</span><span> </span><span class="identifier">personDS</span><span> = </span><span class="type-name">TypedDataset</span><span>.</span><span class="identifier">create</span><span>(</span><span class="identifier">people</span><span>)
</span><span class="comment">// personDS: TypedDataset[Person] = [age: int, birthday: bigint]</span></code></pre>
Expand Down Expand Up @@ -214,7 +214,7 @@ <h2 id="another-example" class="section"><a class="anchor-link left" href="#anot
</span><span class="keyword">case</span><span> </span><span class="number-literal">2</span><span> =&gt; </span><span class="type-name">Female</span><span>
</span><span class="keyword">case</span><span> </span><span class="number-literal">3</span><span> =&gt; </span><span class="type-name">Other</span><span>
})
</span><span class="comment">// genderToInt: Injection[Gender, Int] = frameless.Injection$$anon$1@29c0334e</span></code></pre>
</span><span class="comment">// genderToInt: Injection[Gender, Int] = frameless.Injection$$anon$1@6647fa98</span></code></pre>
<p>And now we can create our <code>TypedDataset</code>:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">val</span><span> </span><span class="identifier">personDS</span><span> = </span><span class="type-name">TypedDataset</span><span>.</span><span class="identifier">create</span><span>(</span><span class="identifier">people</span><span>)
</span><span class="comment">// personDS: TypedDataset[Person] = [age: int, gender: int]</span></code></pre>
Expand Down
4 changes: 2 additions & 2 deletions Job.html
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ <h1 id="job-a" class="title">Job[A]</h1>
</span><span class="identifier">count</span><span> &lt;- </span><span class="identifier">ds</span><span>.</span><span class="identifier">count</span><span>()
</span><span class="identifier">sample</span><span> &lt;- </span><span class="identifier">ds</span><span>.</span><span class="identifier">take</span><span>((</span><span class="identifier">count</span><span>/</span><span class="number-literal">5</span><span>).</span><span class="identifier">toInt</span><span>)
} </span><span class="keyword">yield</span><span> </span><span class="identifier">sample</span><span>
</span><span class="comment">// countAndTakeJob: frameless.Job[Seq[Int]] = frameless.Job$$anon$3@46287670
</span><span class="comment">// countAndTakeJob: frameless.Job[Seq[Int]] = frameless.Job$$anon$3@4920885a
</span><span>
</span><span class="identifier">countAndTakeJob</span><span>.</span><span class="identifier">run</span><span>()
</span><span class="comment">// res1: Seq[Int] = WrappedArray(1, 2, 3, 4)</span></code></pre>
Expand All @@ -167,7 +167,7 @@ <h1 id="job-a" class="title">Job[A]</h1>
</span><span class="keyword">def</span><span> </span><span class="declaration-name">computeMinOfSample</span><span>(</span><span class="identifier">sample</span><span>: </span><span class="type-name">Job</span><span>[</span><span class="type-name">Seq</span><span>[</span><span class="type-name">Int</span><span>]]): </span><span class="type-name">Job</span><span>[</span><span class="type-name">Int</span><span>] = </span><span class="identifier">sample</span><span>.</span><span class="identifier">map</span><span>(</span><span class="identifier">_</span><span>.</span><span class="identifier">min</span><span>)

</span><span class="keyword">val</span><span> </span><span class="identifier">finalJob</span><span> = </span><span class="identifier">computeMinOfSample</span><span>(</span><span class="identifier">countAndTakeJob</span><span>)
</span><span class="comment">// finalJob: Job[Int] = frameless.Job$$anon$2@702f39fc</span></code></pre>
</span><span class="comment">// finalJob: Job[Int] = frameless.Job$$anon$2@52d2f575</span></code></pre>
<p>Now we can execute this new job by specifying a <a href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.SparkContext@setJobGroup(groupId:String,description:String,interruptOnCancel:Boolean):Unit">group-id</a> and a description.
This allows the programmer to see this information on the Spark UI and help track, say,
performance issues.</p>
Expand Down
2 changes: 1 addition & 1 deletion TypedDatasetVsSparkDataset.html
Original file line number Diff line number Diff line change
Expand Up @@ -159,8 +159,8 @@ <h1 id="comparing-typeddatasets-with-spark-s-datasets" class="title">Comparing T
</span><span class="comment">// +---+---+
// | i| j|
// +---+---+
// | 10| W|
// | 1| Q|
// | 10| W|
// |100| E|
// +---+---+
//</span></code></pre>
Expand Down
2 changes: 1 addition & 1 deletion TypedEncoder.html
Original file line number Diff line number Diff line change
Expand Up @@ -212,7 +212,7 @@ <h1 id="typed-encoders-in-frameless" class="title">Typed Encoders in Frameless</
</span><span class="comment">// ds: TypedDataset[Foo] = [i: int, b: struct&lt;d: double, s: string&gt;]
</span><span>
</span><span class="identifier">ds</span><span>.</span><span class="identifier">collect</span><span>()
</span><span class="comment">// res3: frameless.Job[Seq[Foo]] = frameless.Job$$anon$4@7a07589c</span></code></pre>
</span><span class="comment">// res3: frameless.Job[Seq[Foo]] = frameless.Job$$anon$4@759e2669</span></code></pre>
<p>But any non-encodable in the case class hierarchy will be detected at compile time:</p>
<pre class="keep-together pdf epub"><code class="nohighlight"><span class="keyword">case</span><span> </span><span class="keyword">class</span><span> </span><span class="type-name">BarDate</span><span>(</span><span class="identifier">d</span><span>: </span><span class="type-name">Double</span><span>, </span><span class="identifier">s</span><span>: </span><span class="type-name">String</span><span>, </span><span class="identifier">t</span><span>: </span><span class="identifier">java</span><span>.</span><span class="identifier">util</span><span>.</span><span class="type-name">Calendar</span><span>)
</span><span class="keyword">case</span><span> </span><span class="keyword">class</span><span> </span><span class="type-name">FooDate</span><span>(</span><span class="identifier">i</span><span>: </span><span class="type-name">Int</span><span>, </span><span class="identifier">b</span><span>: </span><span class="type-name">BarDate</span><span>)</span></code></pre>
Expand Down
Loading

0 comments on commit 5713b56

Please sign in to comment.