Skip to content

Drop storm-sql-* modules #8174

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
rzo1 merged 3 commits into from
Aug 9, 2025
Merged

Drop storm-sql-* modules #8174

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
f3671ef
Draft: Drop storm-sql-* modules
rzo1 Jul 15, 2025
253ed63
Draft: Drop storm-sql-* modules
rzo1 Jul 15, 2025
a423143
Merge branch 'master' into draft-drop-storm-sql
rzo1 Aug 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 0 additions & 13 deletions DEPENDENCY-LICENSES
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,6 @@ List of third-party dependencies grouped by their license type.

* Kryo (com.esotericsoftware:kryo:5.6.2 - https://github.com/EsotericSoftware/kryo/kryo)
* MinLog (com.esotericsoftware:minlog:1.3.1 - https://github.com/EsotericSoftware/minlog)
* Protocol Buffers [Core] (com.google.protobuf:protobuf-java:3.3.0 - https://developers.google.com/protocol-buffers/protobuf-java/)
* ReflectASM (com.esotericsoftware:reflectasm:1.11.9 - https://github.com/EsotericSoftware/reflectasm)

AL 2.0, GPL v2, MPL 2.0
Expand All @@ -20,10 +19,8 @@ List of third-party dependencies grouped by their license type.
Apache License, Version 2.0

* ActiveMQ :: Client (org.apache.activemq:activemq-client:6.1.7 - http://activemq.apache.org/activemq-client)
* Aggregate Designer Algorithm (net.hydromatic:aggdesigner-algorithm:6.0 - http://github.com/julianhyde/aggdesigner/aggdesigner-algorithm)
* Annotations for Metrics (io.dropwizard.metrics:metrics-annotation:4.2.33 - https://metrics.dropwizard.io/metrics-annotation)
* Apache Avro (org.apache.avro:avro:1.12.0 - https://avro.apache.org)
* Apache Calcite Avatica (org.apache.calcite.avatica:avatica-core:1.11.0 - https://calcite.apache.org/avatica/avatica-core)
* Apache Commons BeanUtils (commons-beanutils:commons-beanutils:1.9.4 - https://commons.apache.org/proper/commons-beanutils/)
* Apache Commons CLI (commons-cli:commons-cli:1.9.0 - https://commons.apache.org/proper/commons-cli/)
* Apache Commons Codec (commons-codec:commons-codec:1.18.0 - https://commons.apache.org/proper/commons-codec/)
Expand All @@ -32,7 +29,6 @@ List of third-party dependencies grouped by their license type.
* Apache Commons Compress (org.apache.commons:commons-compress:1.27.1 - https://commons.apache.org/proper/commons-compress/)
* Apache Commons Configuration (org.apache.commons:commons-configuration2:2.12.0 - https://commons.apache.org/proper/commons-configuration/)
* Apache Commons Crypto (org.apache.commons:commons-crypto:1.1.0 - https://commons.apache.org/proper/commons-crypto/)
* Apache Commons CSV (org.apache.commons:commons-csv:1.14.0 - https://commons.apache.org/proper/commons-csv/)
* Apache Commons Exec (org.apache.commons:commons-exec:1.5.0 - https://commons.apache.org/proper/commons-exec/)
* Apache Commons FileUpload (commons-fileupload:commons-fileupload:1.6.0 - https://commons.apache.org/proper/commons-fileupload/)
* Apache Commons IO (commons-io:commons-io:2.20.0 - https://commons.apache.org/proper/commons-io/)
Expand Down Expand Up @@ -85,16 +81,11 @@ List of third-party dependencies grouped by their license type.
* AutoService (com.google.auto.service:auto-service-annotations:1.1.1 - https://github.com/google/auto/tree/main/service)
* AutoService Processor (com.google.auto.service:auto-service:1.1.1 - https://github.com/google/auto/tree/main/service)
* Caffeine cache (com.github.ben-manes.caffeine:caffeine:3.2.2 - https://github.com/ben-manes/caffeine)
* Calcite Core (org.apache.calcite:calcite-core:1.16.0 - https://calcite.apache.org/calcite-core)
* Calcite Linq4j (org.apache.calcite:calcite-linq4j:1.16.0 - https://calcite.apache.org/calcite-linq4j)
* CDI APIs (javax.enterprise:cdi-api:1.0 - http://www.seamframework.org/Weld/cdi-api)
* chill-java (com.twitter:chill-java:0.9.5 - https://github.com/twitter/chill)
* ClassMate (com.fasterxml:classmate:1.7.0 - https://github.com/FasterXML/java-classmate)
* com.helger:profiler (com.helger:profiler:1.1.1 - https://github.com/phax/profiler)
* com.yahoo.datasketches:memory (com.yahoo.datasketches:memory:0.9.0 - https://datasketches.github.io/memory/)
* com.yahoo.datasketches:sketches-core (com.yahoo.datasketches:sketches-core:0.9.0 - https://datasketches.github.io/sketches-core/)
* Commons Lang (commons-lang:commons-lang:2.6 - http://commons.apache.org/lang/)
* Commons Logging (commons-logging:commons-logging:1.1.3 - http://commons.apache.org/proper/commons-logging/)
* Curator Client (org.apache.curator:curator-client:5.9.0 - https://curator.apache.org/curator-client)
* Curator Framework (org.apache.curator:curator-framework:5.9.0 - https://curator.apache.org/curator-framework)
* Curator Recipes (org.apache.curator:curator-recipes:5.9.0 - https://curator.apache.org/curator-recipes)
Expand All @@ -113,8 +104,6 @@ List of third-party dependencies grouped by their license type.
* Dropwizard Utility Classes (io.dropwizard:dropwizard-util:4.0.14 - http://www.dropwizard.io/4.0.14/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-util)
* Dropwizard Validation Support (io.dropwizard:dropwizard-validation:4.0.14 - http://www.dropwizard.io/4.0.14/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-validation)
* error-prone annotations (com.google.errorprone:error_prone_annotations:2.40.0 - https://errorprone.info/error_prone_annotations)
* Esri Geometry API for Java (com.esri.geometry:esri-geometry-api:2.0.0 - https://github.com/Esri/geometry-api-java)
* FindBugs-jsr305 (com.google.code.findbugs:jsr305:3.0.1 - http://findbugs.sourceforge.net/)
* FindBugs-jsr305 (com.google.code.findbugs:jsr305:3.0.2 - http://findbugs.sourceforge.net/)
* Google Guice - Core Library (com.google.inject:guice:4.2.1 - https://github.com/google/guice/guice)
* Graphite Integration for Metrics (io.dropwizard.metrics:metrics-graphite:4.2.33 - https://metrics.dropwizard.io/metrics-graphite)
Expand Down Expand Up @@ -346,9 +335,7 @@ List of third-party dependencies grouped by their license type.
* asm (org.ow2.asm:asm:9.8 - http://asm.ow2.io/)
* asm-commons (org.ow2.asm:asm-commons:9.8 - http://asm.ow2.io/)
* asm-tree (org.ow2.asm:asm-tree:9.8 - http://asm.ow2.io/)
* Commons Compiler (org.codehaus.janino:commons-compiler:2.7.6 - http://docs.codehaus.org/display/JANINO/Home/commons-compiler)
* dnsjava (dnsjava:dnsjava:3.6.1 - https://github.com/dnsjava/dnsjava)
* Janino (org.codehaus.janino:janino:2.7.6 - http://docs.codehaus.org/display/JANINO/Home/janino)
* Protocol Buffer Java API (com.google.protobuf:protobuf-java:2.5.0 - http://code.google.com/p/protobuf)

BSD License
Expand Down
13 changes: 0 additions & 13 deletions LICENSE-binary
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -647,7 +647,6 @@ The license texts of these dependencies can be found in the licenses directory.

* Kryo (com.esotericsoftware:kryo:5.6.2 - https://github.com/EsotericSoftware/kryo/kryo)
* MinLog (com.esotericsoftware:minlog:1.3.1 - https://github.com/EsotericSoftware/minlog)
* Protocol Buffers [Core] (com.google.protobuf:protobuf-java:3.3.0 - https://developers.google.com/protocol-buffers/protobuf-java/)
* ReflectASM (com.esotericsoftware:reflectasm:1.11.9 - https://github.com/EsotericSoftware/reflectasm)

Apache-2.0
Expand All @@ -669,9 +668,7 @@ The license texts of these dependencies can be found in the licenses directory.

Apache License, Version 2.0

* Aggregate Designer Algorithm (net.hydromatic:aggdesigner-algorithm:6.0 - http://github.com/julianhyde/aggdesigner/aggdesigner-algorithm)
* Annotations for Metrics (io.dropwizard.metrics:metrics-annotation:4.2.33 - https://metrics.dropwizard.io/metrics-annotation)
* Apache Calcite Avatica (org.apache.calcite.avatica:avatica-core:1.11.0 - https://calcite.apache.org/avatica/avatica-core)
* Apache Commons BeanUtils (commons-beanutils:commons-beanutils:1.9.4 - https://commons.apache.org/proper/commons-beanutils/)
* Apache Commons CLI (commons-cli:commons-cli:1.9.0 - http://commons.apache.org/proper/commons-cli/)
* Apache Commons Codec (commons-codec:commons-codec:1.18.0 - http://commons.apache.org/proper/commons-codec/)
Expand All @@ -680,7 +677,6 @@ The license texts of these dependencies can be found in the licenses directory.
* Apache Commons Compress (org.apache.commons:commons-compress:1.27.1 - https://commons.apache.org/proper/commons-compress/)
* Apache Commons Configuration (org.apache.commons:commons-configuration2:2.12.0 - https://commons.apache.org/proper/commons-configuration/)
* Apache Commons Crypto (org.apache.commons:commons-crypto:1.1.0 - https://commons.apache.org/proper/commons-crypto/)
* Apache Commons CSV (org.apache.commons:commons-csv:1.14.0 - http://commons.apache.org/proper/commons-csv/)
* Apache Commons Exec (org.apache.commons:commons-exec:1.5.0 - http://commons.apache.org/proper/commons-exec/)
* Apache Commons IO (commons-io:commons-io:2.20.0 - https://commons.apache.org/proper/commons-io/)
* Apache Commons Logging (commons-logging:commons-logging:1.2 - http://commons.apache.org/proper/commons-logging/)
Expand Down Expand Up @@ -717,15 +713,10 @@ The license texts of these dependencies can be found in the licenses directory.
* Apache ZooKeeper - Server (org.apache.zookeeper:zookeeper:3.9.3 - http://zookeeper.apache.org/zookeeper)
* ASM based accessors helper used by json-smart (net.minidev:accessors-smart:2.5.2 - https://urielch.github.io/)
* Caffeine cache (com.github.ben-manes.caffeine:caffeine:3.2.2 - https://github.com/ben-manes/caffeine)
* Calcite Core (org.apache.calcite:calcite-core:1.16.0 - https://calcite.apache.org/calcite-core)
* Calcite Linq4j (org.apache.calcite:calcite-linq4j:1.16.0 - https://calcite.apache.org/calcite-linq4j)
* chill-java (com.twitter:chill-java:0.9.5 - https://github.com/twitter/chill)
* ClassMate (com.fasterxml:classmate:1.7.0 - https://github.com/FasterXML/java-classmate)
* com.helger:profiler (com.helger:profiler:1.1.1 - https://github.com/phax/profiler)
* com.yahoo.datasketches:memory (com.yahoo.datasketches:memory:0.9.0 - https://datasketches.github.io/memory/)
* com.yahoo.datasketches:sketches-core (com.yahoo.datasketches:sketches-core:0.9.0 - https://datasketches.github.io/sketches-core/)
* Commons Lang (commons-lang:commons-lang:2.6 - http://commons.apache.org/lang/)
* Commons Logging (commons-logging:commons-logging:1.1.3 - http://commons.apache.org/proper/commons-logging/)
* Commons Math (org.apache.commons:commons-math3:3.6.1 - http://commons.apache.org/math/)
* Curator Client (org.apache.curator:curator-client:5.9.0 - https://curator.apache.org/curator-client)
* Curator Framework (org.apache.curator:curator-framework:5.9.0 - https://curator.apache.org/curator-framework)
Expand All @@ -745,8 +736,6 @@ The license texts of these dependencies can be found in the licenses directory.
* Dropwizard Utility Classes (io.dropwizard:dropwizard-util:4.0.14 - http://www.dropwizard.io/4.0.14/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-util)
* Dropwizard Validation Support (io.dropwizard:dropwizard-validation:4.0.14 - http://www.dropwizard.io/4.0.14/dropwizard-bom/dropwizard-dependencies/dropwizard-parent/dropwizard-validation)
* error-prone annotations (com.google.errorprone:error_prone_annotations:2.40.0 - https://errorprone.info/error_prone_annotations)
* Esri Geometry API for Java (com.esri.geometry:esri-geometry-api:2.0.0 - https://github.com/Esri/geometry-api-java)
* FindBugs-jsr305 (com.google.code.findbugs:jsr305:3.0.1 - http://findbugs.sourceforge.net/)
* FindBugs-jsr305 (com.google.code.findbugs:jsr305:3.0.2 - http://findbugs.sourceforge.net/)
* Graphite Integration for Metrics (io.dropwizard.metrics:metrics-graphite:4.2.33 - https://metrics.dropwizard.io/metrics-graphite)
* Gson (com.google.code.gson:gson:2.13.1 - https://github.com/google/gson)
Expand Down Expand Up @@ -927,8 +916,6 @@ The license texts of these dependencies can be found in the licenses directory.

BSD 3-Clause License

* Commons Compiler (org.codehaus.janino:commons-compiler:2.7.6 - http://docs.codehaus.org/display/JANINO/Home/commons-compiler)
* Janino (org.codehaus.janino:janino:2.7.6 - http://docs.codehaus.org/display/JANINO/Home/janino)
* Protocol Buffer Java API (com.google.protobuf:protobuf-java:2.5.0 - http://code.google.com/p/protobuf)

BSD License
Expand Down
65 changes: 0 additions & 65 deletions bin/storm.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -315,7 +315,6 @@ def initialize_main_command():
initialize_localconfvalue_subcommand(subparsers)
initialize_remoteconfvalue_subcommand(subparsers)
initialize_local_subcommand(subparsers)
initialize_sql_subcommand(subparsers)
initialize_kill_subcommand(subparsers)
initialize_upload_credentials_subcommand(subparsers)
initialize_blobstore_subcommand(subparsers)
Expand Down Expand Up @@ -563,31 +562,6 @@ def initialize_upload_credentials_subcommand(subparsers):
sub_parser.set_defaults(func=upload_credentials)
add_common_options(sub_parser)


def initialize_sql_subcommand(subparsers):
command_help = """Compiles the SQL statements into a Trident topology and submits it to Storm.
If user activates explain mode, SQL Runner analyzes each query statement
and shows query plan instead of submitting topology.
"""

sub_parser = subparsers.add_parser("sql", help=command_help, formatter_class=SortingHelpFormatter)

add_client_jar_options(sub_parser)

sub_parser.add_argument("sql_file", metavar="sql-file")

group = sub_parser.add_mutually_exclusive_group(required=True)

group.add_argument(
"topology_name", metavar="topology-name",nargs='?'
)

group.add_argument("--explain", action="store_true", help="activate explain mode")

sub_parser.set_defaults(func=sql)
add_common_options(sub_parser, main_args=False)


def initialize_blobstore_subcommand(subparsers):
sub_parser = subparsers.add_parser("blobstore", formatter_class=SortingHelpFormatter)
command_help = """
Expand Down Expand Up @@ -1087,45 +1061,6 @@ def local(args):
"org.apache.storm.LocalCluster", args,
client=False, daemon=False, extrajvmopts=extrajvmopts)


def sql(args):
local_jars = [arg for arg in args.jars.split(",") if arg]

artifact_to_file_jars = resolve_dependencies(
args.artifacts, args.artifactRepositories,
args.mavenLocalRepositoryDirectory, args.proxyUrl,
args.proxyUsername, args.proxyPassword
)

# include storm-sql-runtime jar(s) to local jar list
# --jars doesn't support wildcard so it should call get_jars_full
sql_runtime_jars = get_jars_full(os.path.join(STORM_TOOLS_LIB_DIR, "sql", "runtime"))
local_jars.extend(sql_runtime_jars)

extra_jars = [USER_CONF_DIR, STORM_BIN_DIR]
extra_jars.extend(local_jars)
extra_jars.extend(artifact_to_file_jars.values())

# include this for running StormSqlRunner, but not for generated topology
sql_core_jars = get_wildcard_dir(os.path.join(STORM_TOOLS_LIB_DIR, "sql", "core"))
extra_jars.extend(sql_core_jars)

if args.explain:
sql_args = ["--file", args.sql_file, "--explain"]
else:
sql_args = ["--file", args.sql_file, "--topology", args.topology_name]

exec_storm_class(
"org.apache.storm.sql.StormSqlRunner", storm_config_opts=args.storm_config_opts,
jvmtype="-client",
extrajars=extra_jars,
main_class_args=sql_args,
daemon=False,
jvmopts=["-Dstorm.dependency.jars=" + ",".join(local_jars)] +
["-Dstorm.dependency.artifacts=" + json.dumps(artifact_to_file_jars)],
overriding_conf_file=args.config)


def kill(args):
exec_storm_class(
"org.apache.storm.command.KillTopology",
Expand Down
1 change: 0 additions & 1 deletion docs/Implementation-docs.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,4 +11,3 @@ This section of the wiki is dedicated to explaining how Storm is implemented. Yo
- [Acking framework implementation](Acking-framework-implementation.html)
- [Metrics](Metrics.html)
- [Nimbus HA](nimbus-ha-design.html)
- [Storm SQL](storm-sql-internal.html)
2 changes: 1 addition & 1 deletion docs/State-checkpointing.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -53,7 +53,7 @@ can be changed by setting the storm config `topology.state.checkpoint.interval.m
5. For state persistence, use a state provider that supports persistence by setting the `topology.state.provider` in the
storm config. E.g. for using Redis based key-value state implementation set `topology.state.provider: org.apache.storm.redis.state.RedisKeyValueStateProvider`
in storm.yaml. The provider implementation jar should be in the class path, which in this case means adding `storm-redis`
to dependency of your topology, or adding `--artifacts "org.apache.storm:storm-sql-redis:<storm-version>"` when submitting your topology with `storm jar`.
to dependency of your topology.
6. The state provider properties can be overridden by setting `topology.state.provider.config`. For Redis state this is a
json config with the following properties.

Expand Down
12 changes: 0 additions & 12 deletions docs/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,18 +38,6 @@ We're also notifying it via annotating classes with marker interface `@Interface

* [Streams API](Stream-API.html)

#### SQL

The Storm SQL integration allows users to run SQL queries over streaming data in Storm.

NOTE: Storm SQL is an `experimental` feature, so the internals of Storm SQL and supported features are subject to change.
But small change will not affect the user experience. We will notify the user when breaking UX change is introduced.

* [Storm SQL overview](storm-sql.html)
* [Storm SQL example](storm-sql-example.html)
* [Storm SQL reference](storm-sql-reference.html)
* [Storm SQL internal](storm-sql-internal.html)

#### Flux

* [Flux Data Driven Topology Builder](flux.html)
Expand Down
Loading