diff --git a/software/gluten/README.md b/software/gluten/README.md index ea00c9f..e9876a8 100644 --- a/software/gluten/README.md +++ b/software/gluten/README.md @@ -9,18 +9,18 @@ Click to learn more about [Gluten](https://github.com/apache/incubator-gluten) a # Contents -[Parameter Tuning](#gluten-related-Parameter-Tuning) -[Intel® Optimizations](#intel-optimizations) -[Performance Results](#performance-results) +- [Parameter Tuning](#gluten-related-Parameter-Tuning) +- [Intel® Optimizations](#intel-optimizations) +- [Performance Results](#performance-results) # Gluten related Parameter Tuning -| Parameters | Description | Recommend Setting | +| Parameters | Description | Recommended Setting | | ---------- | ----------- | ----------------- | | spark.executor.instances | |Depends on total cores per host and related to cores setting | | spark.executor.cores | | Recommend to allocate 4 to 8 cores per executor. Please note that using an odd number of cores is not advised on systems with Simultaneous Multithreading(SMT) enabled| -| spark.executor.memory | | At least 0.5GB-1GB per core. Please note too small heap memory size could trigger GC and hurt the performance.| +| spark.executor.memory | | Recommend to allocate at least 0.5GB-1GB per core. Please note too small heap memory size could trigger GC and hurt the performance.| | spark.executor.memoryOverhead | | Keep a few memory size for memoryOverhead | | spark.memory.offHeap.enabled | Gluten uses offHeap memory for the native engine. | true | | spark.memory.offHeap.size | Set up offHeap memory size | In fully offload cases, please allocate as much memory as possible for offHeap. | @@ -40,7 +40,7 @@ Click to learn more about [Gluten](https://github.com/apache/incubator-gluten) a | spark.gluten.sql.columnar.physicalJoinOptimizationLevel | | 18. Intend to fallback to JVM engine when consequent join happen like q72 in TPC-DS. | | spark.gluten.sql.columnar.physicalJoinOptimizeEnable | | true | | spark.gluten.sql.columnar.numaBinding | | Set true for bare metal and false for Cloud environment. | -| spark.gluten.sql.columnar.coreRange | | This parameters only works when numaBinding is enabled. Please use numactl tool to find the detailed NUMA topology then set the number to align with the NUMA topology in the Intel® processor. | +| spark.gluten.sql.columnar.coreRange | | This parameter only works when numaBinding is enabled. Please use numactl tool to find the detailed NUMA topology then set the number to align with the NUMA topology in the Intel® processor. | ## Differences between Spark and Gluten in the Memory setting In Spark, the total executor memory is calculated as: @@ -86,9 +86,15 @@ Please refer to [Velox HBM](https://apache.github.io/incubator-gluten/get-starte # Performance Results +- [Performance Results for Intel® Xeon® 6 with Performance Cores](#performance-results-for-intel-xeon-6-with-performance-cores) +- [Performance Results for Intel® Xeon® 6 with Efficiency Cores](#performance-results-for-intel-xeon-6-with-efficiency-cores) + ## Performance Results for Intel® Xeon® 6 with Performance Cores -## Intel® Xeon® 6960P vs. 5th Generation Intel® Xeon® 8592+ +- [6th Generation Intel® Xeon® 6960P vs. 5th Generation Intel® Xeon® 8592+](#6th-generation-intel-xeon-6960p-vs-5th-generation-intel-xeon-8592) +- [6th Generation Intel® Xeon® 6985P-C vs. 4th Generation Intel® Xeon® Platinum 8481C In Google Cloud](#6th-generation-intel-xeon-6985p-c-vs-4th-generation-intel-xeon-platinum-8481c-in-google-cloud) + +## 6th Generation Intel® Xeon® 6960P vs. 5th Generation Intel® Xeon® 8592+ @@ -118,12 +124,40 @@ Gluten on GNR: 1-node, 2x Intel(R) Xeon(R) 6960P, 72 cores, HT On, Turbo On, NUM Results may vary. +## 6th Generation Intel® Xeon® 6985P-C vs. 4th Generation Intel® Xeon® Platinum 8481C In Google Cloud + + + +### 6th Generation Xeon® compared to 4th Generation Xeon® with Gluten + Velox In Google Cloud +- **1.49×** speedup for **TPCDS**-Like SF3TB +- **1.25×** perf/$ improvement for **TPCDS**-Like SF3TB + + + +### Gluten + Velox vs. Spark 3.5.2 In Google Cloud +- **2.13×** speedup for **TPCDS**-Like SF3TB on 4th Generation Intel® Xeon® Platinum 8481C in Google Cloud (c3-standard-88-lssd) +- **2.22×** speedup **TPCDS**-Like SF3TB on 6th Generation Intel® Xeon® 6985P-C in Google Cloud (c4-standard-96-lssd) + +### Details +Testing Date: Performance results are based on testing by Intel as of 1 Aug 2025 and may not reflect all publicly available security updates. + +Gluten on SPR: 1-node, 1x Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz, 44 cores, c3-standard-88-lssd VM, HT On, Turbo Off, NUMA 1, Total Memory 352GB, BIOS Google, 1x Compute Engine Virtual Ethernet [gVNIC], 1x 375G nvme_card10, 1x 375G nvme_card8, 1x 375G nvme_card12, 1x 512G nvme_card-pd, 1x 375G nvme_card2, 1x 375G nvme_card5, 1x 375G nvme_card11, 1x 375G nvme_card15, 1x 375G nvme_card14, 1x 375G nvme_card0, 1x 375G nvme_card3, 1x 375G nvme_card13, 1x 375G nvme_card1, 1x 375G nvme_card4, 1x 375G nvme_card7, 1x 375G nvme_card9, 1x 375G nvme_card6, Ubuntu 24.04.2 LTS, 6.14.0-1011-gcp, TPC-DS Like SF3T, JDK 1.8, GCC 11, Spark 3.5.2, Hadoop 3.3.5, score=TPC-DS Like SF3T 2786 sec + +Gluten on GNR: 1-node, 1x Intel(R) Xeon(R) 6985P-C CPU @ 2.30GHz, 48 cores, c4-standard-96-lssd VM, HT On, Turbo Off, NUMA 2, Total Memory 360GB, BIOS Google, 1x Compute Engine Virtual Ethernet [gVNIC], 1x 375G nvme_card14, 1x 375G nvme_card1, 1x 375G nvme_card3, 1x 375G nvme_card0, 1x 375G nvme_card4, 1x 375G nvme_card8, 1x 375G nvme_card15, 1x 512G nvme_card-pd, 1x 375G nvme_card2, 1x 375G nvme_card5, 1x 375G nvme_card9, 1x 375G nvme_card11, 1x 375G nvme_card6, 1x 375G nvme_card7, 1x 375G nvme_card10, 1x 375G nvme_card12, 1x 375G nvme_card13, Ubuntu 24.04.2 LTS, 6.14.0-1012-gcp, TPC-DS Like SF3T, JDK 1.8, GCC 11, Spark 3.5.2, Hadoop 3.3.5, score=TPC-DS Like SF3T 1871 sec + +OSS Spark on SPR: 1-node, 1x Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz, 44 cores, c3-standard-88-lssd VM, HT On, Turbo Off, NUMA 1, Total Memory 352GB, BIOS Google, 1x Compute Engine Virtual Ethernet [gVNIC], 1x 375G nvme_card10, 1x 375G nvme_card8, 1x 375G nvme_card12, 1x 512G nvme_card-pd, 1x 375G nvme_card2, 1x 375G nvme_card5, 1x 375G nvme_card11, 1x 375G nvme_card15, 1x 375G nvme_card14, 1x 375G nvme_card0, 1x 375G nvme_card3, 1x 375G nvme_card13, 1x 375G nvme_card1, 1x 375G nvme_card4, 1x 375G nvme_card7, 1x 375G nvme_card9, 1x 375G nvme_card6, Ubuntu 24.04.2 LTS, 6.14.0-1011-gcp, TPC-DS Like SF3T, JDK 1.8, GCC 11, Spark 3.5.2, Hadoop 3.3.5, score=TPC-DS Like SF3T 5926 sec + +OSS Spark on GNR: 1-node, 1x Intel(R) Xeon(R) 6985P-C CPU @ 2.30GHz, 48 cores, c4-standard-96-lssd VM, HT On, Turbo Off, NUMA 2, Total Memory 360GB, BIOS Google, 1x Compute Engine Virtual Ethernet [gVNIC], 1x 375G nvme_card14, 1x 375G nvme_card1, 1x 375G nvme_card3, 1x 375G nvme_card0, 1x 375G nvme_card4, 1x 375G nvme_card8, 1x 375G nvme_card15, 1x 512G nvme_card-pd, 1x 375G nvme_card2, 1x 375G nvme_card5, 1x 375G nvme_card9, 1x 375G nvme_card11, 1x 375G nvme_card6, 1x 375G nvme_card7, 1x 375G nvme_card10, 1x 375G nvme_card12, 1x 375G nvme_card13, Ubuntu 24.04.2 LTS, 6.14.0-1012-gcp, TPC-DS Like SF3T, JDK 1.8, GCC 11, Spark 3.5.2, Hadoop 3.3.5, score=TPC-DS Like SF3T 4162 sec + +Results may vary. + ### Intel® Xeon® 6 with Performance Cores Summary The combination of **Intel® Xeon® 6 with Performance Cores** and **Gluten + Velox** demonstrates consistent performance advantages making it a strong candidate for high-performance data processing workloads. ## Performance Results for Intel® Xeon® 6 with Efficiency Cores +- [6th Generation Intel® Xeon® 6780E vs. 3rd Generation Intel® Xeon®](#6th-generation-intel-xeon-6780e-vs-3rd-generation-intel-xeon) -### Intel® Xeon® 6780E vs. 3rd Generation Intel® Xeon® +### 6th Generation Intel® Xeon® 6780E vs. 3rd Generation Intel® Xeon® diff --git a/software/gluten/images/gluten-gen-perf.png b/software/gluten/images/gluten-gen-perf.png new file mode 100644 index 0000000..24f59e7 Binary files /dev/null and b/software/gluten/images/gluten-gen-perf.png differ diff --git a/software/gluten/images/oss-spark-gluten-gen-perf.png b/software/gluten/images/oss-spark-gluten-gen-perf.png new file mode 100644 index 0000000..a282f37 Binary files /dev/null and b/software/gluten/images/oss-spark-gluten-gen-perf.png differ diff --git a/software/spark/README.md b/software/spark/README.md index 82db9d1..f96f416 100644 --- a/software/spark/README.md +++ b/software/spark/README.md @@ -21,11 +21,12 @@ focused on TPC-DS, please reference - [Storage Configuration](#storage-configuration) - [Network Configuration](#network-configuration) - [Software Tuning](#software-tuning) - - [Linux Kernel Optimization Settings](#linux-kernel-optimization-settings) - - [Configure `ulimit`](#configure-ulimit) - - [Configure Transparent Huge Pages](#configure-transparent-huge-pages) + - [Linux Optimization Settings](#linux-optimization-settings) + - [Configure `limits.conf`](#configure-limitsconf) + - [Configure Huge Pages](#configure-huge-pages) - [Spark Parameter Tuning](#spark-parameter-tuning) - [Memory Tuning](#memory-tuning) +- [Performance Results](#performance-results) # References @@ -124,7 +125,7 @@ good candidates for experimentation in your deployment. For information on these knobs, please review the [Spark documentation](https://spark.apache.org/docs/latest/configuration.html). -| Parameters | Description & Recommend Setting | +| Parameters | Description & Recommended Setting | |:-------|:--------------------------------| | spark.executor.cores | The number of cores to use on each executor.

**Recommended:** Recommend to allocate 4 to 8 cores per executor. Please note that using an odd number of cores is not advised on systems with Simultaneous Multithreading (SMT) enabled | | spark.executor.instances | The number of executors to launch for this session.

**Recommended:** Divide the output of `nproc` by the number of cores per executor (`spark.executor.cores`) | @@ -156,9 +157,34 @@ some experimentation. For information on these knobs, please review the [Spark documentation](https://spark.apache.org/docs/latest/configuration.html). -| Parameters | Description & Recommend Setting | +| Parameters | Description & Recommended Setting | |:-----------|:--------------------------------| | spark.executor.memoryOverhead | Memory that accounts for things like VM overheads, interned strings, other native overheads, etc.

**Recommended:** 1g | | spark.executor.memory | Amount of memory to use per executor process

**Recommended:** 35% of total memory, divided by number of executors | | spark.memory.offHeap.enabled | Determines if Spark will use off-heap memory for certain operations

**Recommended:** true | -| spark.memory.offHeap.size | The absolute amount of memory which can be used for off-heap allocation

**Recommended:** 45% of total memory, divided by number of executors | \ No newline at end of file +| spark.memory.offHeap.size | The absolute amount of memory which can be used for off-heap allocation

**Recommended:** 45% of total memory, divided by number of executors | + +# Performance Results + +[Performance Results for Intel® Xeon® 6 with Performance Cores](#performance-results-for-intel-xeon-6-with-performance-cores) + +## Performance Results for Intel® Xeon® 6 with Performance Cores +- [6th Generation Intel® Xeon® 6985P-C vs. 4th Generation Intel® Xeon® Platinum 8481C In Google Cloud](#6th-generation-intel-xeon-6985p-c-vs-4th-generation-intel-xeon-platinum-8481c-in-google-cloud) + +## 6th Generation Intel® Xeon® 6985P-C vs. 4th Generation Intel® Xeon® Platinum 8481C In Google Cloud + + + +### 6th Generation Xeon® compared to 4th Generation Xeon® with Spark 3.5.2 In Google Cloud +- **1.42×** speedup for **TPCDS**-Like SF3TB +- **1.19×** perf/$ improvement for **TPCDS**-Like SF3TB +- For 6th Generation Xeon® performance compared to 4th Generation Xeon® in perfomance Google Cloud using Apache Gluten, reference the [Gluten article](../gluten/README.md#6th-generation-intel-xeon-6985p-c-vs-4th-generation-intel-xeon-platinum-8481c-in-google-cloud) in Optimization Zone. + +### Details +Testing Date: Performance results are based on testing by Intel as of 1 Aug 2025 and may not reflect all publicly available security updates. + +OSS Spark on SPR: 1-node, 1x Intel(R) Xeon(R) Platinum 8481C CPU @ 2.70GHz, 44 cores, c3-standard-88-lssd VM, HT On, Turbo Off, NUMA 1, Total Memory 352GB, BIOS Google, 1x Compute Engine Virtual Ethernet [gVNIC], 1x 375G nvme_card10, 1x 375G nvme_card8, 1x 375G nvme_card12, 1x 512G nvme_card-pd, 1x 375G nvme_card2, 1x 375G nvme_card5, 1x 375G nvme_card11, 1x 375G nvme_card15, 1x 375G nvme_card14, 1x 375G nvme_card0, 1x 375G nvme_card3, 1x 375G nvme_card13, 1x 375G nvme_card1, 1x 375G nvme_card4, 1x 375G nvme_card7, 1x 375G nvme_card9, 1x 375G nvme_card6, Ubuntu 24.04.2 LTS, 6.14.0-1011-gcp, TPC-DS Like SF3T, JDK 1.8, GCC 11, Spark 3.5.2, Hadoop 3.3.5, score=TPC-DS Like SF3T 5926 sec + +OSS Spark on GNR: 1-node, 1x Intel(R) Xeon(R) 6985P-C CPU @ 2.30GHz, 48 cores, c4-standard-96-lssd VM, HT On, Turbo Off, NUMA 2, Total Memory 360GB, BIOS Google, 1x Compute Engine Virtual Ethernet [gVNIC], 1x 375G nvme_card14, 1x 375G nvme_card1, 1x 375G nvme_card3, 1x 375G nvme_card0, 1x 375G nvme_card4, 1x 375G nvme_card8, 1x 375G nvme_card15, 1x 512G nvme_card-pd, 1x 375G nvme_card2, 1x 375G nvme_card5, 1x 375G nvme_card9, 1x 375G nvme_card11, 1x 375G nvme_card6, 1x 375G nvme_card7, 1x 375G nvme_card10, 1x 375G nvme_card12, 1x 375G nvme_card13, Ubuntu 24.04.2 LTS, 6.14.0-1012-gcp, TPC-DS Like SF3T, JDK 1.8, GCC 11, Spark 3.5.2, Hadoop 3.3.5, score=TPC-DS Like SF3T 4162 sec + +Results may vary. \ No newline at end of file diff --git a/software/spark/images/oss-spark-gen-perf.png b/software/spark/images/oss-spark-gen-perf.png new file mode 100644 index 0000000..0f820d6 Binary files /dev/null and b/software/spark/images/oss-spark-gen-perf.png differ