How to connect ElasticSearch Index(Using Logstash with MySQL to feed the data) to Java Rest High Level Client(Spring Boot)

huangapple
未分类评论43阅读模式
英文:

How to connect ElasticSearch Index(Using Logstash with MySQL to feed the data) to Java Rest High Level Client(Spring Boot)

问题

以下是您提供的内容的翻译：

我能够通过Java High Level Rest Client连接到ElasticSearch索引，但我想了解以下内容：

  1. 使用POSTMAN手动创建了名为"students"的索引，使用PUT /students。
  2. 使用POSTMAN手动设置了字段，使用PUT /students/_mappings。
  3. 使用Logstash将数据从MySQL连续地传输到索引。
  4. 如何从Spring Boot应用程序连接到我们创建的这个特定索引，并在该索引中搜索文档？（注意：如果我使用Java Rest High Level Client自动创建索引，则可以连接到索引，但如果我使用Logstash并分离索引的创建，则似乎无法从索引中检索任何记录）。

这是我的Logstash配置文件：

  1. input {
  2.   jdbc {
  3.     clean_run => true
  4.     jdbc_driver_library => "/../mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar"
  5.     jdbc_driver_class => "com.mysql.jdbc.driver"
  6.     jdbc_connection_string => "jdbc:mysql://localhost:3306/exampledb?autoReconnect=true&characterEncoding=UTF-8&verifyServerCertificate=false&useSSL=false&requireSSL=false"
  7.     jdbc_user => "user"
  8.     jdbc_password => "pass"
  9.     schedule => "* * * * *"
  10.     statement => "SELECT STUDENTID, FIRSTNAME, LASTNAME, AGE, PHONE, CITY FROM student WHERE STUDENTID > :sql_last_value;"
  11.     use_column_value => true
  12.     tracking_column => "studentid"
  13.   }
  14. }
  16. output {
  17.   elasticsearch {
  18.     hosts => ["localhost:9200"]
  19.     index => "students"
  20.   }
  21.   stdout {
  22.     codec => rubydebug
  23.   }
  24. }

这是Student.class：

  1. @NoArgsConstructor
  2. @AllArgsConstructor
  3. @Getter
  4. @Setter
  5. @ToString
  6. @EqualsAndHashCode
  7. @Data
  8. @Document(indexName = "students", type = "students", shards = 1)
  9. public class Student {
  11.   @Id
  12.   private long studentid;
  14.   private String firstname;
  16.   private String lastname;
  18.   private int age;
  20.   private String phone;
  22.   private String city;
  23. }

这是我从中使用ElasticSearch查询的StudentDAO：

  1. public List<Student> getAllStudents(String term, String order, String value) {
  2.   QueryBuilder query;
  4.   query = QueryBuilders.boolQuery()
  5.     .should(QueryBuilders.matchAllQuery())
  6.     .should(QueryBuilders.queryStringQuery("*" + term + "*")
  7.       .lenient(true)
  8.       .field("studentid")
  9.       .field("firstname")
  10.       .field("lastname")
  11.       .field("age")
  12.       .field("phone")
  13.       .field("city"));
  15.   NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder()
  16.     .withQuery(query)
  17.     .withPageable(PageRequest.of(0, 1000))
  18.     .build();
  20.   return elasticsearchTemplate.queryForList(nativeSearchQuery, Student.class);
  21. }

这是PUT /students/_mappings：

  1. {
  2.   "properties": {
  3.     "@timestamp": {
  4.       "type": "date"
  5.     },
  6.     "@version": {
  7.       "type": "keyword",
  8.       "fields": {
  9.         "keyword": {
  10.           "type": "keyword",
  11.           "ignore_above": 256
  12.         }
  13.       }
  14.     },
  15.     "FIRSTNAME": {
  16.       "type": "keyword"
  17.     },
  18.     "LASTNAME": {
  19.       "type": "keyword"
  20.     },
  21.     "PHONE": {
  22.       "type": "keyword"
  23.     },
  24.     "AGE": {
  25.       "type": "keyword"
  26.     },
  27.     "CITY": {
  28.       "type": "keyword"
  29.     },
  30.     "STUDENTID": {
  31.       "type": "long"
  32.     }
  33.   }
  34. }
英文:

I am able to connect to ElasticSearch Indexes when created through the Java High Level Rest Client but I want to know this:

  1. Created Index(students) manually using PUT /students in POSTMAN
  2. Set the Fields manually using PUT /students/_mappings in POSTMAN
  3. Feeding the Data from MySQL to the Index using Logstash continuously
  4. How to connect to this specific index that we created and search for documents in this index from the Spring boot application? (Note: Without using Logstash, I am able to connect to the index if I create it automatically using the Java Rest High Level Client, but if I use the Logstash and separate the Index creation, then it doesn't seem like its retrieving any records from the index).

This is my Logstash config file

  1. input{
  2. jdbc{
  3. clean_run =&gt; true
  4.     jdbc_driver_library =&gt; &quot;/../mysql-connector-java-8.0.20/mysql-connector-java-8.0.20.jar&quot;
  5.     jdbc_driver_class =&gt; &quot;com.mysql.jdbc.driver&quot;
  6.     jdbc_connection_string =&gt; &quot;jdbc:mysql://localhost:3306/exampledb?autoReconnect=true&amp;characterEncoding=UTF-8&amp;verifyServerCertificate=false&amp;useSSL=false&amp;requireSSL=false&quot;
  7.     jdbc_user =&gt; &quot;user&quot;
  8.     jdbc_password =&gt; &quot;pass&quot;
  9.     schedule =&gt; &quot;* * * * *&quot;
  10.     statement =&gt; &quot;SELECT STUDENTID, FIRSTNAME, LASTNAME, AGE, PHONE, CITY FROM student WHERE STUDENTID &gt; :sql_last_value;&quot;
  11.     use_column_value =&gt; true
  12.     tracking_column =&gt; &quot;studentid&quot;
  13.     }
  14.     }
  15.     output{
  16.         elasticsearch{
  17.             hosts=&gt;[&quot;localhost:9200&quot;]
  18.             index=&gt;&quot;students&quot;
  19.             }
  20.             stdout{
  21.             codec =&gt; rubydebug
  22.             }
  23.     }

This is Student.class

  1. @NoArgsConstructor
  2. @AllArgsConstructor
  3. @Getter
  4. @Setter
  5. @ToString
  6. @EqualsAndHashCode
  7. @Data
  8. @Document(indexName = &quot;students&quot;, type = &quot;students&quot;, shards = 1)
  9. public class Student {
  11.    @Id
  12.    private long studentid;
  14.    private String firstname;
  16.    private String lastname;
  18.    private int age;
  20.    private String phone;
  22.    private String city;
  23. }

This is the StudentDAO from where I am using the ElasticSearch query.

  1.  public List&lt;Student&gt; getAllStudents(String term, String order, String value) {
  3.       QueryBuilder query;
  5.       query = QueryBuilders.boolQuery()
  6.                            .should(
  7.                                  QueryBuilders.matchAllQuery()
  9.                            )
  10.                            .should(QueryBuilders.queryStringQuery(&quot;*&quot; + term + &quot;*&quot;).
  11.                                  lenient(true).field(&quot;studentid&quot;)
  12.                                                 .field(&quot;firstname&quot;)
  13.                                                 .field(&quot;lastname&quot;)
  14.                                                 .field(&quot;age&quot;)
  15.                                                 .field(&quot;phone&quot;)
  16.                                                 .field(&quot;city&quot;));
  19.       NativeSearchQuery nativeSearchQuery = new NativeSearchQueryBuilder().withQuery(query)
  20.                                                                           .withPageable(PageRequest.of(0,
  21.                                                                                 1000))
  22.                                                                           .build();
  24.       return elasticsearchTemplate.queryForList(nativeSearchQuery, Student.class);
  25.    }

This is the PUT /students/_mappings

  1. {
  2. 	&quot;properties&quot;: {
  3.         &quot;@timestamp&quot;: {
  4.           &quot;type&quot;: &quot;date&quot;
  5.         },
  6.         &quot;@version&quot;: {
  7.           &quot;type&quot;: &quot;keyword&quot;,
  8.           &quot;fields&quot;: {
  9.             &quot;keyword&quot;: {
  10.               &quot;type&quot;: &quot;keyword&quot;,
  11.               &quot;ignore_above&quot;: 256
  12.             }
  13.           }
  14.         },
  15.         &quot;FIRSTNAME&quot;: {
  16.           &quot;type&quot;: &quot;keyword&quot;
  17.         },
  18.         &quot;LASTNAME&quot;: {
  19.           &quot;type&quot;: &quot;keyword&quot;
  20.         },
  21.         &quot;PHONE&quot;: {
  22.           &quot;type&quot;: &quot;keyword&quot;
  23.         },
  24.         &quot;AGE&quot;: {
  25.           &quot;type&quot;: &quot;keyword&quot;
  26.         },
  27.         &quot;CITY&quot;: {
  28.           &quot;type&quot;: &quot;keyword&quot;
  29.         },
  30.         &quot;STUDENTID&quot;: {
  31.           &quot;type&quot;: &quot;long&quot;
  32.         }
  33.       }
  34.     }

huangapple
  • 本文由 发表于 2020年7月25日 12:10:59
  • 转载请务必保留本文链接：https://java.coder-hub.com/63084234.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定