Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

修改mysql 行记录数据,adapter 有监测到数据变更,但并没有同步到es #2284

Closed
Ecli2h opened this issue Oct 10, 2019 · 8 comments

Comments

@Ecli2h
Copy link

Ecli2h commented Oct 10, 2019

我找到一个和我问题一样的issue
@agapple 大佬是说没有开启row模式,我检查我的数据库是开启的,但是还是同样的问题,修改mysql 行记录数据,adapter 有监测到数据变更,但并没有同步到es,日志没有报错

2019-10-10 10:15:31.814 [Thread-26] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to connect destination: example <=============
2019-10-10 10:15:31.816 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterLoader - Start adapter for canal instance: example succeed
2019-10-10 10:15:36.705 [main] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterService - ## the canal client adapters are running now ......
2019-10-10 10:15:36.721 [main] INFO  org.apache.coyote.http11.Http11NioProtocol - Starting ProtocolHandler ["http-nio-8081"]
2019-10-10 10:15:36.724 [main] INFO  org.apache.tomcat.util.net.NioSelectorPool - Using a shared selector for servlet write/read
2019-10-10 10:15:36.749 [main] INFO  o.s.boot.web.embedded.tomcat.TomcatWebServer - Tomcat started on port(s): 8081 (http) with context path ''
2019-10-10 10:15:36.753 [main] INFO  c.a.otter.canal.adapter.launcher.CanalAdapterApplication - Started CanalAdapterApplication in 54.174 seconds (JVM running for 57.087)
2019-10-10 10:15:36.762 [Thread-26] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Start to subscribe destination: example <=============
2019-10-10 10:15:36.765 [Thread-26] INFO  c.a.o.canal.adapter.launcher.loader.CanalAdapterWorker - =============> Subscribe destination: example succeed <=============
2019-10-10 10:15:50.622 [pool-2-thread-1] INFO  c.a.o.canal.client.adapter.logger.LoggerAdapterExample - DML: {"data":[{"INFOID":"04B7F026-F8A7-4419-A9BA-C807A4601D58","RESNAME":"对讲机7","RESCODE":"201909291010444780","TITLE":"","RESTYPE":1,"COPYRIGHT":"广州欧科信息技术股份有限公司","DESCRIPT":"","PUBUSER":"","PUBDT":null,"INFOSRC":19,"RECSTATUS":0,"AUDITUSER":"","AUDITDT":null,"AUDITIDEA":"","PHOTOGRAPUSER":"","PHOTOGRAPPLACE":"","PHOTOGRAPDT":"2016-07-22 10:21:32","DEVICE":"Canon EOS 5D Mark II","HELPER":"","MAKE":"","CREATOR":"1","CTIME":1569723288000,"EDITOR":"","ETIME":null,"BROWSECOUNT":0,"COLLECTCOUNT":0,"DOWNCOUNT":0,"REMARK":"","RESCLASS":""}],"database":"prodplus","destination":"example","es":1570673749000,"groupId":null,"isDdl":false,"old":[{"RESNAME":"对讲机2"}],"pkNames":["INFOID"],"sql":"","table":"dr_resinfo","ts":1570673750309,"type":"UPDATE"}

@Ecli2h
Copy link
Author

Ecli2h commented Oct 10, 2019

@rewerma @agapple 请问有什么办法定位到问题出现在哪么?

@Teamo512
Copy link

Teamo512 commented Nov 2, 2019

和你一样的情况,日志上没报错,也有数据变更记录,但es上就是没数据。。我一开始用的es7.1,后来换成es6.x,也还是一样的问题,,端口换成9200和9300都没用,es上始终没数据写进去。。。。

@Ecli2h
Copy link
Author

Ecli2h commented Nov 7, 2019

和你一样的情况,日志上没报错,也有数据变更记录,但es上就是没数据。。我一开始用的es7.1,后来换成es6.x,也还是一样的问题,,端口换成9200和9300都没用,es上始终没数据写进去。。。。

我解决是改了源码,是大小写的问题导致的

@Curisepl
Copy link

@Ecli2h 我也是一样的问题,请问您是改了哪块代码

@Ecli2h
Copy link
Author

Ecli2h commented Nov 26, 2019

@Ecli2h 我也是一样的问题,请问您是改了哪块代码

canal\client-adapter\elasticsearch\src\main\java\com\alibaba\otter\canal\client\adapter\es\support\ESSyncUtil.java
这个类里面有几处获取ID,对应代码里面的idVal为空,在获取字段名的时候转成大写就好了

@chshzh2000
Copy link

1.1.5版本源码的问题,下载源码,修改client-adapter/escore下的pom文件,将 com.alibabadruid包加上 provided配置,然后canal重新打包,再在canal-canal-1.1.5\client-adapter\es7x\target\目录下找到client-adapter.es7x-1.1.5-jar-with-dependencies.jar,将这个文件复制到/usr/local/canal_adapter/plugin下,重启adapter即可

@byebai95
Copy link

byebai95 commented Jul 6, 2021

同样的问题 ,看到楼上的讨论,我也试试这个方式

@u19900101
Copy link

这里有解决方法,https://github.com/alibaba/canal/issues/3144,我最开始是看漏了adapter日志中的报错信息,Canal Adapter com.alibaba.druid.pool.DruidDataSource cannot be cast to com.alibaba.druid.pool.DruidDataSource,看看是不是有这个报错

@Ecli2h Ecli2h closed this as completed Nov 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants