[SPARK-58559][PYTHON] Package all of sbin in PySpark classic distribution

[nchammas]

What changes were proposed in this pull request?

Package everything in sbin/ when building a PySpark Classic distribution.

Why are the changes needed?

Some sbin scripts were first added to PySpark in #23715. There was some disagreement then about whether PySpark should include these scripts, mainly because some committers at the time thought PySpark should be a client-only distribution.

In the intervening years, the Spark Connect effort has created a true client-only distribution of Spark in contrast to the "heavier" PySpark Classic which includes all of Spark's assembly JARs.

PySpark Classic has included some sbin scripts since 3.0.0, and #56907 recently added the Connect server scripts. #56907, however, neglected to add the corresponding directives to MANIFEST.in.

Since #57452 clarifies the intended use of the Python distributions of Spark -- specifically, that starting a full cluster is not supported, regardless of whether it's possible -- I believe it's conceptually simpler to just package all of sbin. That would, for example, prevent the kind of gap identified in #56907.

Does this PR introduce any user-facing change?

Yes, it packages additional sbin scripts in the PySpark classic distribution.

How was this patch tested?

Distributions are not tested thoroughly. #57645 adds a dedicated distribution validation script. I think we should discuss there any testing we would like to add for this.

Was this patch authored or co-authored using generative AI tooling?

No.

[nchammas]

cc @gaogaotiantian @HyukjinKwon @ericm-db

[HyukjinKwon]

I think the problem was that we wanted to avoid for users to use pip installed pyspark to start a cluster or sth. what scripts do we include?

[nchammas]

Yes, we wanted to prevent users from using PySpark to start a standalone cluster. I tried to address this with a clearer README so that we can just package everything. In other words, I think it's easier to control this via our support policy instead of via "hard" packaging changes.

Switching to * means we will package sbin/start-{thriftserver, master, worker}.sh in addition to the existing scripts we are already packaging today. These newly packaged scripts can also be called by the new CLI if/when we merge it.