玩转Solr源码之(三)Solr源码Deploy

发表于 | | 阅读数
字数统计: 3.8k 字 | 阅读时长 ≈ 18 分钟

玩转Solr源码之—Solr源码Deploy

这篇文章是玩转Solr源码系列的第三篇，紧接着上一篇的源码Deug。如果你已经修改好了源码，并且调试ok，想要将源码打包后分发给同时或者其他项目引用，那么这边文章一定能够帮助你。

不到万不得己，别直接改源代码

一般而言，如果将Solr/Elasticsearch应用于企业级开发，Solr原生的功能或多或少都不能100%的满足企业的业务需求(客户的需求都是比较变态的),这也使得我们有的时候不得不通过修改源代码的方式来达到我们的目的，改源码难，管理源代码更难，毕竟几万甚至几十万的行的Solr源码，你只动了其中一个类中的某一个方法，结果为了管理，你需要专门搞个git仓库，然后还要定义好版本号，放到nexus仓库给其他组员或者其他项目引用，改代码可能花了你一个小时，但是管理这些代码可能需要让你忙活一个上午。所以不到万不得已尽量通过继承/实现等方式来修改源码，这样在版本升级的时候，工作量会小很多。否则的话，干过的人都知道很痛苦，特别是上一个改了源码的人已经离职了。
uwD7dI.jpg

从源码到Nexus

笔者所在的公司Java开发使用的构建和依赖管理工具是Maven,仓库的话使用的是自建的nexus,我相信这也是大部分公司的模式，当然也有用Gradle，这里不比较两个工具的好坏，能完成开发，高效率并且使用的顺手的都是好工具。笔者这里以maven+ nexus为例。

安装Nexus

大部分情况下，nexus仓库这种工作都是交给运维去搞的，一般开发只需要知道如何使用就行，如果你对安装nexus没有任何兴趣，那么这部分可以选择跳过。
为了方便演示，这里笔者使用了docker进行安装和部署nexus，流程比较简单，几行命令就行了,这里以nexus2为例子.

1
2
docker pull sonatype/nexus
docker run -d -p 8081:8081 --name nexus-oss sonatype/nexus

打开浏览器输入http://127.0.0.1:8081/nexus就可以看到nexuys界面啦，默认的登陆用户名和密码为admin,admin123
Nexus2

使用ant生成maven artifacts并上传

Solr中的源码已经支持生成maven artifacets并且上传到自定义的仓库中，简单的命令为

1
2
3
ant -Dversion=my-special-version -Dm2.repository.id=my-repo-id \
          -Dm2.repository.url=https://example.org/my/repo \
          generate-maven-artifacts

这里解释下参数的含义

变量名 解释
my-special-version 指定版本号，这个大家根据需要可以自定义，但是还是需要一定的命名规范（下面会讲到）
my-repo-id 表示nexus仓库的id，名称一般都是自定义的
m2.repository.url 表示nexuys仓库的地址，笔者这里的地址就是http://127.0.0.1:8081/nexus/content/repositories/releases/

具体的可以参考文档Lucene/Solr Maven build instructions。
到这里大家基本上已经知道如是将修改过后的solr源代码打包并且上传maven了吧，废话不多说，直接上命令

1
2
3
4
5
6
7
8
[fql@localhost solr-7.7.2]$ ant -Dversion=TEST -Dm2.repository.id=nexus -Dm2.repository.url=http://127.0.0.1:8081/nexus/content/repositories/releases/ generate-maven-artifacts
Buildfile: /home/fql/IdeaProjects/solr-7.7.2/build.xml

BUILD FAILED
/home/fql/IdeaProjects/solr-7.7.2/build.xml:21: The following error occurred while executing this line:
/home/fql/IdeaProjects/solr-7.7.2/lucene/common-build.xml:63: If you pass -Dversion=... to set a release version, it must match "7.7.2", optionally followed by a suffix (e.g., "-SNAPSHOT").

Total time: 0 seconds

擦，<一顿操作猛如虎，定睛一看原地杵>,原来version的名称海必须匹配7.7.2,好吧，再来

1
2
3
4
5
6
7
8
9
10
11
[fql@localhost solr-7.7.2]$ ant -Dversion=7.7.2-TEST -Dm2.repository.id=nexus -Dm2.repository.url=http://127.0.0.1:8081/nexus/content/repositories/releases/ generate-maven-artifacts
Buildfile: /home/fql/IdeaProjects/solr-7.7.2/build.xml
...
...
[artifact:deploy] Error deploying artifact 'org.apache.lucene:lucene-solr-grandparent:pom': Error deploying artifact: Failed to transfer file: http://127.0.0.1:8081/nexus/content/repositories/releases/org/apache/lucene/lucene-solr-grandparent/7.7.2-TEST/lucene-solr-grandparent-7.7.2-TEST.pom. Return code is: 401

BUILD FAILED
/home/fql/IdeaProjects/solr-7.7.2/build.xml:209: The following error occurred while executing this line:
/home/fql/IdeaProjects/solr-7.7.2/lucene/build.xml:404: The following error occurred while executing this line:
/home/fql/IdeaProjects/solr-7.7.2/lucene/common-build.xml:656: Error deploying artifact 'org.apache.lucene:lucene-solr-grandparent:pom': Error deploying artifact: Failed to transfer file: http://127.0.0.1:8081/nexus/content/repositories/releases/org/apache/lucene/lucene-solr-grandparent/7.7.2-TEST/lucene-solr-grandparent-7.7.2-TEST.pom. Return code is: 401
Total time: 8 minutes 53 seconds

Return code is: 401,貌似权限不对，确实命令中没有传任何用户名和密码的信息，这里通过generate-maven-artifacts定位到solr-7.7.2/lucene/common-build.xml 文件中的m2-deploy(第646行)，并且在其下面的remoteRepository添加

1
2
<authentication username="admin" password="admin123"/>
<!-- admin 和admin123 就是nexus的默认用户名和密码-->

重新执行命令我们就可以看到项目已经成功的上传到了本地的nexus(看起来一行代码搞定的问题，确实让你琢磨了一个上午)

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[fql@localhost solr-7.7.2]$ ant -Dversion=7.7.2-TEST -Dm2.repository.id=nexus -Dm2.repository.url=http://192.168.34.128:8081/nexus/content/repositories/releases/ generate-maven-artifacts
...
...
[artifact:deploy] [INFO] Uploading repository metadata for: 'artifact org.apache.solr:solr-velocity'
[artifact:deploy] Uploading: org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST-sources.jar to repository nexus at http://192.168.34.128:8081/nexus/content/repositories/releases/
[artifact:deploy] Transferring 31K from nexus
[artifact:deploy] Uploaded 31K
[artifact:deploy] Uploading: org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST-javadoc.jar to repository nexus at http://192.168.34.128:8081/nexus/content/repositories/releases/
[artifact:deploy] Transferring 64K from nexus
[artifact:deploy] Uploaded 64K
[artifact:install] [INFO] Installing /home/fql/IdeaProjects/solr-7.7.2/solr/build/solr.tgz.unpacked/solr-7.7.2-TEST/dist/solr-velocity-7.7.2-TEST.jar to /home/fql/.m2/repository/org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST.jar
[artifact:install] [INFO] Installing /home/fql/IdeaProjects/solr-7.7.2/solr/build/contrib/solr-velocity/solr-velocity-7.7.2-TEST-src.jar to /home/fql/.m2/repository/org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST-sources.jar
[artifact:install] [INFO] Installing /home/fql/IdeaProjects/solr-7.7.2/solr/build/contrib/solr-velocity/solr-velocity-7.7.2-TEST-javadoc.jar to /home/fql/.m2/repository/org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST-javadoc.jar
[artifact:install] [INFO] Installing /home/fql/IdeaProjects/solr-7.7.2/solr/build/contrib/solr-velocity/solr-velocity-7.7.2-TEST-src.jar to /home/fql/.m2/repository/org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST-sources.jar
[artifact:install] [INFO] Installing /home/fql/IdeaProjects/solr-7.7.2/solr/build/contrib/solr-velocity/solr-velocity-7.7.2-TEST-javadoc.jar to /home/fql/.m2/repository/org/apache/solr/solr-velocity/7.7.2-TEST/solr-velocity-7.7.2-TEST-javadoc.jar
BUILD SUCCESSFUL
Total time: 6 minutes 37 seconds

打开浏览器http://127.0.0.1:8081/nexus/content/repositories/releases/org/apache/lucene/ 我们就可以看见solr相关的jar包pom等信息已经上传成功
nexus_solr

爬坑小技巧

看似简单的工作，实际操作起来整的耗费了很长的时间，关键是关于authentication的那部分google了好久发现没一个人提。为了防止大家再次爬坑，这里给大家分享两个知识点/小技巧

  • 解决ant idea 下载比较慢的问题
    大家在导入Solr源码项目的时候可能已经发现了，(ant idea)整个过程非常的慢,笔者更是等了一个下午才构建好，正常的开发中我们不可能等这么久，这里可能有人说使用maven默认的仓库在海外导致的，我们使用aliyun的仓库就可行了(笔者尝试了，不行，不知道为什么阿里云下载的时候没有http状态码，导致ivy认为仓库不可用)，这里笔者使用了腾讯云的仓库(http://mirrors.cloud.tencent.com/nexus/repository/maven-public/),企鹅貌似比阿里守规范。。,具体的改动点为，修改**solr-7.7.2/lucene/default-nested-ivy-settings.xml**配置文件，添加自定义resolver (+表示在源文件中增加一行，-表示删除源文件中的某一行)
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    <resolvers>
     +   <ibiblio name="tecent" root="http://mirrors.cloud.tencent.com/nexus/repository/maven-public/"  m2compatible="true" />
        <ibiblio name="sonatype-releases" root="https://oss.sonatype.org/content/repositories/releases" m2compatible="true" />
        <ibiblio name="maven.restlet.com" root="https://maven.restlet.com" m2compatible="true" />
        <ibiblio name="releases.cloudera.com" root="https://repository.cloudera.com/cloudera/libs-release-local" m2compatible="true" />
    ...
    ...
    -    <resolver ref="main"/>
    +    <resolver ref="tecent"/>
        <resolver ref="maven.restlet.com" />
        <resolver ref="sonatype-releases" />
        <resolver ref="releases.cloudera.com"/>
        </chain>
      </resolvers>

备份并删除~/.ivy2/cache,否则会使用缓存，重新执行命令(ant idea)之后你会发现还是会卡在resolve,但是下载jar包的速度变快了，笔者google了半天发现没有答案，猜想可能是多个resolve 导致的，尝试注释其他的机器resolver,具体的改动为

1
2
3
4
5
6
7
8
9
10
...
-    <resolver ref="main"/>
+    <resolver ref="tecent"/>
+   <!--
    <resolver ref="maven.restlet.com" />
    <resolver ref="sonatype-releases" />
    <resolver ref="releases.cloudera.com"/>
+   -->
    </chain>
  </resolvers>

删除~/.ivy2/cache ,重新执行命令(ant idea)之后，果然速度快了很多,至此，慢的问题我们也解决了，简直💯。(使用默认的话执行需要40分钟，改进之后20分钟即可)

总结

到这里玩转Solr源码系列就完全结束了，虽然内容不多，但是全是干货(毕竟是笔者一步一个坑爬过来的)。如果你对该系列有补充或者更好的意见可以联系Email:fengqingleiyue@163.com
最后来碗鸡汤:<你能走多远,取决于你填坑能力有多强>
uwX5Zj.jpg

2020/02 解决maven http 501的错误

最近发现之前一直没有问题的solr-7.7.2的build和部署最近出现了问题，具体的表现为:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
[artifact:install-provider] Installing provider: org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7:runtime
[artifact:install-provider] Downloading: org/apache/maven/wagon/wagon-ssh/1.0-beta-7/wagon-ssh-1.0-beta-7.pom from repository central at http://repo1.maven.org/maven2
[artifact:install-provider] Error transferring file: Server returned HTTP response code: 501 for URL: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh/1.0-beta-7/wagon-ssh-1.0-beta-7.pom
[artifact:install-provider] [WARNING] Unable to get resource 'org.apache.maven.wagon:wagon-ssh:pom:1.0-beta-7' from repository central (http://repo1.maven.org/maven2): Error transferring file: Server returned HTTP response code: 501 for URL: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh/1.0-beta-7/wagon-ssh-1.0-beta-7.pom
[artifact:install-provider] Downloading: org/apache/maven/wagon/wagon-ssh/1.0-beta-7/wagon-ssh-1.0-beta-7.jar from repository central at http://repo1.maven.org/maven2
[artifact:install-provider] Error transferring file: Server returned HTTP response code: 501 for URL: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh/1.0-beta-7/wagon-ssh-1.0-beta-7.jar
[artifact:install-provider] [WARNING] Unable to get resource 'org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7' from repository central (http://repo1.maven.org/maven2): Error transferring file: Server returned HTTP response code: 501 for URL: http://repo1.maven.org/maven2/org/apache/maven/wagon/wagon-ssh/1.0-beta-7/wagon-ssh-1.0-beta-7.jar
[artifact:install-provider] An error has occurred while processing the Maven artifact tasks.
[artifact:install-provider]  Diagnosis:
[artifact:install-provider]
[artifact:install-provider] Error downloading wagon provider from the remote repository: Missing:
[artifact:install-provider] ----------
[artifact:install-provider] 1) org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7
[artifact:install-provider]
[artifact:install-provider]   Try downloading the file manually from the project website.
[artifact:install-provider]
[artifact:install-provider]   Then, install it using the command:
[artifact:install-provider]       mvn install:install-file -DgroupId=org.apache.maven.wagon -DartifactId=wagon-ssh -Dversion=1.0-beta-7 -Dpackaging=jar -Dfile=/path/to/file
[artifact:install-provider]
[artifact:install-provider]   Alternatively, if you host your own repository you can deploy the file there:
[artifact:install-provider]       mvn deploy:deploy-file -DgroupId=org.apache.maven.wagon -DartifactId=wagon-ssh -Dversion=1.0-beta-7 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
[artifact:install-provider]
[artifact:install-provider]   Path to dependency:
[artifact:install-provider]   	1) unspecified:unspecified:jar:0.0
[artifact:install-provider]   	2) org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7
[artifact:install-provider]
[artifact:install-provider] ----------
[artifact:install-provider] 1 required artifact is missing.
[artifact:install-provider]
[artifact:install-provider] for artifact:
[artifact:install-provider]   unspecified:unspecified:jar:0.0
[artifact:install-provider]
[artifact:install-provider] from the specified remote repositories:
[artifact:install-provider]   central (http://repo1.maven.org/maven2)
[artifact:install-provider]
[artifact:install-provider]

BUILD FAILED
/home/search/source_code_test/project-search-engine-solr-7.7.2/build.xml:209: The following error occurred while executing this line:
/home/search/source_code_test/project-search-engine-solr-7.7.2/lucene/build.xml:404: The following error occurred while executing this line:
/home/search/source_code_test/project-search-engine-solr-7.7.2/lucene/common-build.xml:654: Error downloading wagon provider from the remote repository: Missing:
----------
1) org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7

  Try downloading the file manually from the project website.

  Then, install it using the command:
      mvn install:install-file -DgroupId=org.apache.maven.wagon -DartifactId=wagon-ssh -Dversion=1.0-beta-7 -Dpackaging=jar -Dfile=/path/to/file

  Alternatively, if you host your own repository you can deploy the file there:
      mvn deploy:deploy-file -DgroupId=org.apache.maven.wagon -DartifactId=wagon-ssh -Dversion=1.0-beta-7 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]

  Path to dependency:
  	1) unspecified:unspecified:jar:0.0
  	2) org.apache.maven.wagon:wagon-ssh:jar:1.0-beta-7

----------
1 required artifact is missing.

for artifact:
  unspecified:unspecified:jar:0.0

from the specified remote repositories:
  central (http://repo1.maven.org/maven2)

具体的原因是因为在2020-01-15号maven的仓库只支持https了 (https://support.sonatype.com/hc/en-us/articles/360041287334),解决这个问题的方案比较简单，既然是**central**仓库需要使用https，那么只要在maven的配置中加入

1
2
3
4
5
<mirror>
    <id>maven-https</id>
    <mirrorOf>central</mirrorOf>
    <url>https://repo1.maven.org/maven2</url>
</mirror>

就可以解决了，但是重新执行脚本(ant generate-maven-artifacts)之后发现之前的错误没有了，但是新的错误又来了:

1
2
3
4
5
6
7
8
[artifact:pom] Downloading: org/apache/apache/13/apache-13.pom from repository central at http://repo1.maven.org/maven2
[artifact:pom] Error transferring file: Server returned HTTP response code: 501 for URL: http://repo1.maven.org/maven2/org/apache/apache/13/apache-13.pom
[artifact:pom] [WARNING] Unable to get resource 'org.apache:apache:pom:13' from repository central (http://repo1.maven.org/maven2): Error transferring file: Server returned HTTP response code: 501 for URL: http://repo1.maven.org/maven2/org/apache/apache/13/apache-13.pom
[artifact:pom] An error has occurred while processing the Maven artifact tasks.
[artifact:pom]  Diagnosis:
[artifact:pom]
[artifact:pom] Unable to initialize POM pom.xml: Cannot find parent: org.apache:apache for project: org.apache.lucene:lucene-solr-grandparent:pom:7.7.2-Patsnap for project org.apache.lucene:lucene-solr-grandparent:pom:7.7.2-Patsnap
[artifact:pom] Unable to download the artifact from any repository

这个非常错误是非常奇怪的，因为明明已经将central的仓库从http改成了https但是下载的还是走的http,
这是因为maven还有一个repositories,好吧，加上

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
<profile>
    <id>maven-https</id>
    <activation>
        <activeByDefault>true</activeByDefault>
    </activation>
    <repositories>
        <repository>
            <id>central</id>
            <url>https://repo1.maven.org/maven2</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </repository>
    </repositories>
    <pluginRepositories>
        <pluginRepository>
            <id>central</id>
            <url>https://repo1.maven.org/maven2</url>
            <snapshots>
                <enabled>false</enabled>
            </snapshots>
        </pluginRepository>
    </pluginRepositories>
</profile>

加上之后发现问题解决，可以进行打包发布了。