为具有瞬态scala依赖性的spring-xd项目优化gradle(Optimize gradle for spring-xd project with transient scala dependency)
我正在开发一个项目,其中包含一系列Spring XD的模块子项目,这些子项目碰巧对碰巧使用Scala的非模块子项目具有瞬态依赖性:
ext { springXdVersion = '1.1.0.RELEASE' moduleProjects = subprojects.findAll { project -> project.path.startsWith(':modules.')} javaProjects = subprojects - (moduleProjects + nonJavaProjects) } configure(moduleProjects) { moduleProject -> apply plugin: 'spring-xd-module' } project('core-dependency') { apply plugin: 'scala' // configuration/dependencies } project('modules.source.example') { dependencies { provided(":core-dependency") } } // More modules bearing resemblance to modules.source.example
核心依赖关系最终设置在xd-container的类路径中,并以这种方式在运行时提供给模块。
不幸的是,似乎对于使用它的每个模块,核心依赖性都会被重新编译(由于它也包含了scala编译,因此特别昂贵)。 这导致构建在30分钟以后运行,我想改进。 有没有办法缩短构建时间? 理想情况下,我不想重新编译核心依赖,但我不确定如何实现这一点,考虑到bootRepackage似乎负责为每个模块触发它。 我也尝试过其他技巧,例如并行性,但到目前为止,这样做只能冻结我的系统。 我正在使用gradle 2.1。
我应该注意,gradle配置文件报告指出,对于每个模块,大部分时间沉没都在configureModule步骤中,根据spring-xd repo,它看起来像这样:
project.task('configureModule') << { project.configurations.provided.resolvedConfiguration.firstLevelModuleDependencies.each { excludeTransitiveDependencies(project, it) } }
I am working on a project that contains a series of module subprojects for spring XD which happen to have a transient dependency on a non-module subproject that happens to use Scala:
ext { springXdVersion = '1.1.0.RELEASE' moduleProjects = subprojects.findAll { project -> project.path.startsWith(':modules.')} javaProjects = subprojects - (moduleProjects + nonJavaProjects) } configure(moduleProjects) { moduleProject -> apply plugin: 'spring-xd-module' } project('core-dependency') { apply plugin: 'scala' // configuration/dependencies } project('modules.source.example') { dependencies { provided(":core-dependency") } } // More modules bearing resemblance to modules.source.example
Core-dependency is ultimately set up to be in the classpath of our xd-container, and is provided to the modules at runtime in this manner.
Unfortunately, it seems that for each module that uses it, the core-dependency gets recompiled (which is particularly expensive since it includes a scala compile, too). This causes builds to run north of 30 minutes, which I'd like to improve upon. Is there a way to bring down the build time? Ideally, I'd like to not have to recompile core-dependency, but I'm not sure how to approach accomplishing this, considering that bootRepackage seems to be responsible for triggering it for each module. I have also tried other tricks, such as parallelism, but doing this has only managed to freeze my system so far. I am using gradle 2.1.
I should note that a gradle profile report indicates that for each module, most of the time sunk is in the configureModule step, which, according to the spring-xd repo, looks like this:
project.task('configureModule') << { project.configurations.provided.resolvedConfiguration.firstLevelModuleDependencies.each { excludeTransitiveDependencies(project, it) } }
最满意答案
scala依赖性来自SpringXD中的
Spark streaming
集成。 我们正在努力消除对spring-xd-dirt
依赖性,并将其从模块中提供:https://jira.spring.io/browse/XD-2857
您依赖的具体非模块子项目是什么? 如果它是
spring-xd-module
,那么你可以尝试1.2.0.M1,我们已经将spring-xd-module
的spark依赖项移动到了spring-xd-dirt
。During the configureModule step, all projects are evaluated for transitive provided dependencies, which core-dependency was. The slowdown was caused by the sheer number of dependencies that core-dependency relies on, and consquently requires scanning. Since we want to avoid having core-dependency scanned by configureModule because it is too time-expensive, and we know it needs to be excluded from module fatjars, the appropriate course of action is to remove core-dependency from the provided configuration, and simply exclude it from the fat jar ourselves.
To accomplish this, the gradle build script is modified as follows:
ext { springXdVersion = '1.1.0.RELEASE' moduleProjects = subprojects.findAll { project -> project.path.startsWith(':modules.')} javaProjects = subprojects - (moduleProjects + nonJavaProjects) } configure(moduleProjects) { moduleProject -> apply plugin: 'spring-xd-module' configurations{ core compile.extendsFrom(core) } configurations.exported.exclude module: 'core-dependency' } project('core-dependency') { apply plugin: 'scala' // configuration/dependencies } project('modules.source.example') { dependencies { core project(":core-dependency") } } // More modules bearing resemblance to modules.source.example
The "core" configuration is essentially a second "provided" configuration, which will not get picked up by the configureModule task, avoiding the time wasting evaluation upon it. It is also excluded from the "exported" configuration, whose contents determines what goes into the fat jar that bootRepackage builds.
相关问答
更多-
为什么Gradle只包含来自我的库的具有项目依赖性的类(Why does Gradle only include classes from my library with project dependency)[2022-11-10]
好的,这是我的错。 我的库的build.gradle被配置为只包含jar文件中的源文件。 以下是不正确的 Gradle代码,会给你和我一样的问题。 task jar(type: Jar) { from android.sourceSets.main.java } 这个答案显示了如何修复jar文件的创建。 这是丑陋的,但它似乎工作。 Okay, this was my fault. My library's build.gradle was configured to only include the ... -
您尚未声明要使用哪个Scala版本,如文档和示例所示。 (错误消息在即将发布的Gradle 1.4中得到修复。) repositories { mavenCentral() } dependencies { scalaTools "org.scala-lang:scala-compiler:2.9.2" compile "org.scala-lang:scala-library:2.9.2" } You haven't declared which Scala version t ...
-
您将在流定义中使用url和httpMethod属性 http-client --url=http://... --httpMethod=GET 您还必须设置mappedRequestHeaders以传递已在上游设置的任何自定义标头。 --mappedRequestHeaders=HTTP_REQUEST_HEADERS,myHeader1,myHeader2 但是,如果你想使用uri变量...... url=http://somehost/foo/{bar}/{baz} 您将需要一个自定义的http-c ...
-
Spring XD项目站点文档是在Spring XD GitHub项目的gh-pages分支中维护的GitHub页面; 可以通过正常的拉取请求提交更正。 The Spring XD project site documentation are GitHub Pages maintained in the gh-pages branch of the Spring XD GitHub project; corrections can be submitted via normal pull-requests. ...
-
使用“简单”拆分器(而不是XPath拆分器 - XD当前没有XPathSplitter模块)时,需要在表达式中使用#xpath函数 。 When using a "simple" splitter (rather than an XPath Splitter - XD currently doesn't have an XPathSplitter module), you need to use the #xpath function in the expression.
-
你确定你在这个包下有两个sql文件: 组织/ springframework的/ XD /批号/架构 Are you sure you have both the sql files under this package: org/springframework/xd/batch/schema
-
为具有瞬态scala依赖性的spring-xd项目优化gradle(Optimize gradle for spring-xd project with transient scala dependency)[2023-03-21]
scala依赖性来自SpringXD中的Spark streaming集成。 我们正在努力消除对spring-xd-dirt依赖性,并将其从模块中提供: https://jira.spring.io/browse/XD-2857 您依赖的具体非模块子项目是什么? 如果它是spring-xd-module ,那么你可以尝试1.2.0.M1,我们已经将spring-xd-module的spark依赖项移动到了spring-xd-dirt 。 During the configureModule step, al ... -
一旦销毁了流,您就可以在hdfs sink中看到这些文件。 2.另外,Rollover:即使流处于活动状态,一旦存储的数据大小超过1G(默认值),Spring XD会将1G内容翻转为HDFS文件并创建一个新的tmp文件并将当前的时间戳值存储在其中。 谢谢S.Satish You can see the files in hdfs sink once you destroy the stream. 2.Also, Rollover: Even when the stream is alive, once th ...
-
我在过去遇到过这个问题,当我包含module:时,有效的是module:明确地与小组一起。 像这样: exclude group: 'commons-math3', module: 'commons-math3' 如果您使用配置来排除,则不必在所有*上执行,您可以在本地项目编译时执行以下操作: compile.exclude group: 'commons-math3', module: 'commons-math3' I've had issues with this in the past, wha ...
-
我们遇到了同样的问题。 如果您将存储库从jcenter()更改为mavenCentral(),则应解决该问题。 它也可以有两个存储库。 We experienced the same issue. If you change your repository to mavenCentral() from jcenter(), it should resolve the issue. Its also possible to have both repositories.