fix #538 - make /opt/spark/work-dir writable by gid 0

[erikerlandson]

What changes were proposed in this pull request?

make /opt/spark/work-dir writable by gid 0 to operate better in anonymous uid scenarios

(Please fill in changes proposed in this fix)

How was this patch tested?

I'm using CI infra to fully test image creation and exercising via the unit/integration tests

[erikerlandson]

Note, altering the permissions of /opt/spark/work-dir appears to be sufficient, not all of $SPARK_HOME as I assumed originally when I wrote up #538

[erikerlandson]

Tangentially, I fixed a missing && after mkdir -p /opt/spark/work-dir - which makes me wonder what that RUN complex was actually doing

[erikerlandson]

@foxish ptal - also curious what you think about the missing && that was there

[foxish]

Do we need this to go into the upstream as well? apache#19717 has the spark-base docker-file within the PR.

[foxish]

I think for people looking to customize the image using ours as base, this might be an issue - to have the owner be root. I had a discussion on the thread with Mridul in apache#19717 (comment). Assuming that it'll be root might be tenuous when someone uses PodSecurityPolicy to enforce a certain user.

cc @liyinan926