Property 'num' not found on type java.lang.String....

本文介绍了一种常见的JSP中使用<c:forEach>标签迭代List时出现的属性未找到错误,并给出了正确的代码示例。

摘要生成于 C知道 ,由 DeepSeek-R1 满血版支持, 前往体验 >

list在jsp迭代读取时,我出现了Property ‘num’ not found on type java.lang.String….的问题,在网上找了发现是我的获取写错了。原本我写的是<c:forEach items="users" var="user">,应该改成<c:forEach items="${requestScope.users}" var="user">

 <c:forEach items="${requestScope.users}" var="user">
                <tr>

                    <td width="100px" align="center">
                        <label>${user.date}</label>
                    </td>
                    <td width="100px" align="center">
                        <label>${user.num}</label>
                    </td>           

参考链接https://www.cnblogs.com/liguo-ji/p/6061421.html

boolean existsByNumder(String numder);可不可以不写@Query org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'dingdanService': Unsatisfied dependency expressed through field 'dingdanRepository'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dingdanRepository' defined in com.kucun.dataDo.DingdanRepository defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration: Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Failed to create query for method public abstract boolean com.kucun.dataDo.DingdanRepository.existsByNumder(java.lang.String)! No property numder found for type Dingdan! Did you mean 'number'? at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:660) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:640) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessProperties(AutowiredAnnotationBeanPostProcessor.java:399) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1425) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:897) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:879) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:551) ~[spring-context-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:143) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:755) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:747) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:402) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.SpringApplication.run(SpringApplication.java:312) ~[spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:173) [spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:153) [spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.boot.web.servlet.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:95) [spring-boot-2.3.12.RELEASE.jar:2.3.12.RELEASE] at org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:172) [spring-web-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5128) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.37] at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:717) [catalina.jar:9.0.37] at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:690) [catalina.jar:9.0.37] at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705) [catalina.jar:9.0.37] at org.apache.catalina.startup.HostConfig.deployDirectory(HostConfig.java:1133) [catalina.jar:9.0.37] at org.apache.catalina.startup.HostConfig$DeployDirectory.run(HostConfig.java:1866) [catalina.jar:9.0.37] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_331] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_331] at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.37] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) [na:1.8.0_331] at org.apache.catalina.startup.HostConfig.deployDirectories(HostConfig.java:1045) [catalina.jar:9.0.37] at org.apache.catalina.startup.HostConfig.deployApps(HostConfig.java:429) [catalina.jar:9.0.37] at org.apache.catalina.startup.HostConfig.start(HostConfig.java:1576) [catalina.jar:9.0.37] at org.apache.catalina.startup.HostConfig.lifecycleEvent(HostConfig.java:309) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.fireLifecycleEvent(LifecycleBase.java:123) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.setStateInternal(LifecycleBase.java:423) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.setState(LifecycleBase.java:366) [catalina.jar:9.0.37] at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:936) [catalina.jar:9.0.37] at org.apache.catalina.core.StandardHost.startInternal(StandardHost.java:841) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.37] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1384) [catalina.jar:9.0.37] at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1374) [catalina.jar:9.0.37] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_331] at org.apache.tomcat.util.threads.InlineExecutorService.execute(InlineExecutorService.java:75) [tomcat-util.jar:9.0.37] at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:134) [na:1.8.0_331] at org.apache.catalina.core.ContainerBase.startInternal(ContainerBase.java:909) [catalina.jar:9.0.37] at org.apache.catalina.core.StandardEngine.startInternal(StandardEngine.java:262) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.37] at org.apache.catalina.core.StandardService.startInternal(StandardService.java:421) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.37] at org.apache.catalina.core.StandardServer.startInternal(StandardServer.java:930) [catalina.jar:9.0.37] at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183) [catalina.jar:9.0.37] at org.apache.catalina.startup.Catalina.start(Catalina.java:738) [catalina.jar:9.0.37] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_331] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_331] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_331] at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_331] at org.apache.catalina.startup.Bootstrap.start(Bootstrap.java:342) [bootstrap.jar:9.0.37] at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:473) [bootstrap.jar:9.0.37] Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'dingdanRepository' defined in com.kucun.dataDo.DingdanRepository defined in @EnableJpaRepositories declared on JpaRepositoriesRegistrar.EnableJpaRepositoriesConfiguration: Invocation of init method failed; nested exception is java.lang.IllegalArgumentException: Failed to create query for method public abstract boolean com.kucun.dataDo.DingdanRepository.existsByNumder(java.lang.String)! No property numder found for type Dingdan! Did you mean 'number'? at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1799) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:594) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:516) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:324) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:322) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:202) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.config.DependencyDescriptor.resolveCandidate(DependencyDescriptor.java:276) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1307) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:1227) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.resolveFieldValue(AutowiredAnnotationBeanPostProcessor.java:657) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] ... 62 common frames omitted Caused by: java.lang.IllegalArgumentException: Failed to create query for method public abstract boolean com.kucun.dataDo.DingdanRepository.existsByNumder(java.lang.String)! No property numder found for type Dingdan! Did you mean 'number'? at org.springframework.data.jpa.repository.query.PartTreeJpaQuery.<init>(PartTreeJpaQuery.java:96) ~[spring-data-jpa-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.jpa.repository.query.JpaQueryLookupStrategy$CreateQueryLookupStrategy.resolveQuery(JpaQueryLookupStrategy.java:107) ~[spring-data-jpa-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.jpa.repository.query.JpaQueryLookupStrategy$CreateIfNotFoundQueryLookupStrategy.resolveQuery(JpaQueryLookupStrategy.java:218) ~[spring-data-jpa-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.jpa.repository.query.JpaQueryLookupStrategy$AbstractQueryLookupStrategy.resolveQuery(JpaQueryLookupStrategy.java:81) ~[spring-data-jpa-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.lookupQuery(QueryExecutorMethodInterceptor.java:99) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.lambda$mapMethodsToQuery$1(QueryExecutorMethodInterceptor.java:92) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_331] at java.util.Iterator.forEachRemaining(Iterator.java:116) ~[na:1.8.0_331] at java.util.Collections$UnmodifiableCollection$1.forEachRemaining(Collections.java:1051) ~[na:1.8.0_331] at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_331] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_331] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_331] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.mapMethodsToQuery(QueryExecutorMethodInterceptor.java:94) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.lambda$new$0(QueryExecutorMethodInterceptor.java:84) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at java.util.Optional.map(Optional.java:215) ~[na:1.8.0_331] at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.<init>(QueryExecutorMethodInterceptor.java:84) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:332) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.lambda$afterPropertiesSet$4(RepositoryFactoryBeanSupport.java:294) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.util.Lazy.getNullable(Lazy.java:211) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.util.Lazy.get(Lazy.java:95) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.afterPropertiesSet(RepositoryFactoryBeanSupport.java:300) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.jpa.repository.support.JpaRepositoryFactoryBean.afterPropertiesSet(JpaRepositoryFactoryBean.java:144) ~[spring-data-jpa-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1858) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1795) ~[spring-beans-5.2.15.RELEASE.jar:5.2.15.RELEASE] ... 72 common frames omitted Caused by: org.springframework.data.mapping.PropertyReferenceException: No property numder found for type Dingdan! Did you mean 'number'? at org.springframework.data.mapping.PropertyPath.<init>(PropertyPath.java:94) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.mapping.PropertyPath.create(PropertyPath.java:382) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.mapping.PropertyPath.create(PropertyPath.java:358) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.mapping.PropertyPath.lambda$from$0(PropertyPath.java:311) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at java.util.concurrent.ConcurrentMap.computeIfAbsent(ConcurrentMap.java:324) ~[na:1.8.0_331] at org.springframework.data.mapping.PropertyPath.from(PropertyPath.java:293) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.mapping.PropertyPath.from(PropertyPath.java:276) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.query.parser.Part.<init>(Part.java:82) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.query.parser.PartTree$OrPart.lambda$new$0(PartTree.java:251) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_331] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_331] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_331] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_331] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_331] at org.springframework.data.repository.query.parser.PartTree$OrPart.<init>(PartTree.java:252) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.query.parser.PartTree$Predicate.lambda$new$0(PartTree.java:381) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193) ~[na:1.8.0_331] at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175) ~[na:1.8.0_331] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) ~[na:1.8.0_331] at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) ~[na:1.8.0_331] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[na:1.8.0_331] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499) ~[na:1.8.0_331] at org.springframework.data.repository.query.parser.PartTree$Predicate.<init>(PartTree.java:382) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.repository.query.parser.PartTree.<init>(PartTree.java:97) ~[spring-data-commons-2.3.9.RELEASE.jar:2.3.9.RELEASE] at org.springframework.data.jpa.repository.query.PartTreeJpaQuery.<init>(PartTreeJpaQuery.java:89) ~[spring-data-jpa-2.3.9.RELEASE.jar:2.3.9.RELEASE] ... 98 common frames omitted
06-07
from pyflink.datastream import StreamExecutionEnvironment from pyflink.table import StreamTableEnvironment, EnvironmentSettings, DataTypes from pyflink.table.expressions import col from pyflink.table.udf import udf import os # 定义处理CDC操作的UDF @udf(result_type=DataTypes.STRING()) def process_cdc_op(op): """将CDC操作转换为可读的标签""" return { 'I': 'INSERT', 'U': 'UPDATE', 'D': 'DELETE' }.get(op, 'UNKNOWN') def main(): env = StreamExecutionEnvironment.get_execution_environment() env.set_parallelism(1) env.enable_checkpointing(5000) # 添加必要的连接器JAR包 flink_home = os.getenv('FLINK_HOME', '/opt/flink') env.add_jars( f"file://{flink_home}/lib/flink-connector-kafka-1.17.1.jar", f"file://{flink_home}/lib/flink-connector-jdbc-1.17.1.jar", f"file://{flink_home}/lib/flink-sql-connector-hive-3.1.2_2.12-1.16.3.jar", # 注意Scala版本 f"file://{flink_home}/lib/mysql-connector-java-8.0.28.jar", f"file://{flink_home}/lib/hive-exec-3.1.2.jar", ) settings = EnvironmentSettings.new_instance().in_streaming_mode().build() t_env = StreamTableEnvironment.create(env, environment_settings=settings) # 注册UDF t_env.create_temporary_function("process_cdc_op", process_cdc_op) # 注册Hive Catalog t_env.execute_sql(""" CREATE CATALOG hive_catalog WITH ( 'type' = 'hive', 'hive-conf-dir' = '/opt/hive/conf', 'hive.metastore.uris' = 'thrift://master:9083', 'default-database' = 'user_meal' ) """) t_env.use_catalog("hive_catalog") # 1. 创建Kafka源表 t_env.execute_sql(""" CREATE TABLE kafka_user_meal ( id STRING, review STRING, rating DOUBLE, review_time STRING, user_id STRING, meal_id STRING, op STRING, -- CDC操作类型(I/U/D) ts AS TO_TIMESTAMP(FROM_UNIXTIME(CAST(review_time AS BIGINT))), WATERMARK FOR ts AS ts - INTERVAL '5' SECOND, PRIMARY KEY (id) NOT ENFORCED ) WITH ( 'connector' = 'kafka', 'topic' = 'cleaned-user-meal-reviews', 'properties.bootstrap.servers' = 'master:9092,slave01:9092,slave02:9092', 'properties.group.id' = 'flink-cdc-group', 'scan.startup.mode' = 'latest-offset', 'format' = 'json', 'json.ignore-parse-errors' = 'true' ) """) # 2. 创建Hive表(存储原始CDC操作) t_env.execute_sql(""" CREATE TABLE IF NOT EXISTS hive_user_meal_cdc ( id STRING, review STRING, rating DOUBLE, review_time STRING, user_id STRING, meal_id STRING, operation_type STRING, -- 转换后的操作类型 operation_ts TIMESTAMP(3), op STRING -- 原始操作类型 ) PARTITIONED BY (op) -- 按操作类型分区 TBLPROPERTIES ( 'sink.partition-commit.policy.kind' = 'metastore,success-file', 'auto-compaction' = 'true' ) """) # 3. 创建MySQL表(存储最新状态) t_env.execute_sql(""" CREATE TABLE mysql_user_meal ( id STRING PRIMARY KEY NOT ENFORCED, review STRING, rating DOUBLE, review_time TIMESTAMP(3), user_id STRING, meal_id STRING, last_operation STRING, update_time TIMESTAMP(3) ) WITH ( 'connector' = 'jdbc', 'url' = 'jdbc:mysql://mysql-host:3306/user_meal', 'table-name' = 'user_meal_reviews', 'username' = 'root', 'password' = '5266', 'driver' = 'com.mysql.cj.jdbc.Driver', 'sink.buffer-flush.max-rows' = '100', 'sink.buffer-flush.interval' = '5s' ) """) # 4. 写入Hive(存储所有CDC操作) t_env.execute_sql(""" INSERT INTO hive_user_meal_cdc SELECT id, review, rating, review_time, user_id, meal_id, process_cdc_op(op) AS operation_type, -- 使用UDF转换操作类型 ts AS operation_ts, op FROM kafka_user_meal """) # 5. 从Hive读取最新状态并写入MySQL # 使用Flink的CDC特性获取最新状态 latest_data = t_env.sql_query(""" SELECT id, LAST_VALUE(review) AS review, LAST_VALUE(rating) AS rating, MAX(ts) AS review_time, LAST_VALUE(user_id) AS user_id, LAST_VALUE(meal_id) AS meal_id, LAST_VALUE(process_cdc_op(op)) AS last_operation, CURRENT_TIMESTAMP AS update_time FROM ( SELECT *, ROW_NUMBER() OVER (PARTITION BY id ORDER BY ts DESC) AS row_num FROM kafka_user_meal WHERE op <> 'D' -- 排除删除操作 ) WHERE row_num = 1 """) # 创建临时视图 t_env.create_temporary_view("latest_user_meal", latest_data) # 写入MySQL t_env.execute_sql(""" INSERT INTO mysql_user_meal SELECT * FROM latest_user_meal """) # 执行任务 env.execute("Flink CDC to Hive and MySQL Pipeline") if __name__ == '__main__': main() /home/hadoop/桌面/pyflink/bin/python3.8 /home/hadoop/PycharmProjects/SparkProject/src/flinkCDC.py Traceback (most recent call last): File "/home/hadoop/PycharmProjects/SparkProject/src/flinkCDC.py", line 169, in <module> main() File "/home/hadoop/PycharmProjects/SparkProject/src/flinkCDC.py", line 42, in main t_env.execute_sql(""" File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/pyflink/table/table_environment.py", line 837, in execute_sql return TableResult(self._j_tenv.executeSql(stmt)) File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/py4j/java_gateway.py", line 1322, in __call__ return_value = get_return_value( File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/pyflink/util/exceptions.py", line 146, in deco return f(*a, **kw) File "/home/hadoop/桌面/pyflink/lib/python3.8/site-packages/py4j/protocol.py", line 326, in get_return_value raise Py4JJavaError( py4j.protocol.Py4JJavaError: An error occurred while calling o44.executeSql. : org.apache.flink.table.api.ValidationException: Unable to create catalog 'hive_catalog'. Catalog options are: 'default-database'='user_meal' 'hive-conf-dir'='/opt/hive/conf' 'hive.metastore.uris'='thrift://master:9083' 'type'='hive' at org.apache.flink.table.factories.FactoryUtil.createCatalog(FactoryUtil.java:439) at org.apache.flink.table.api.internal.TableEnvironmentImpl.createCatalog(TableEnvironmentImpl.java:1468) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeInternal(TableEnvironmentImpl.java:1214) at org.apache.flink.table.api.internal.TableEnvironmentImpl.executeSql(TableEnvironmentImpl.java:765) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.flink.api.python.shaded.py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at org.apache.flink.api.python.shaded.py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:374) at org.apache.flink.api.python.shaded.py4j.Gateway.invoke(Gateway.java:282) at org.apache.flink.api.python.shaded.py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at org.apache.flink.api.python.shaded.py4j.commands.CallCommand.execute(CallCommand.java:79) at org.apache.flink.api.python.shaded.py4j.GatewayConnection.run(GatewayConnection.java:238) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.flink.table.api.ValidationException: Unsupported options found for 'hive'. Unsupported options: hive.metastore.uris Supported options: default-database hadoop-conf-dir hive-conf-dir hive-version property-version at org.apache.flink.table.factories.FactoryUtil.validateUnconsumedKeys(FactoryUtil.java:632) at org.apache.flink.table.factories.FactoryUtil$FactoryHelper.validate(FactoryUtil.java:931) at org.apache.flink.table.catalog.hive.factories.HiveCatalogFactory.createCatalog(HiveCatalogFactory.java:69) at org.apache.flink.table.factories.FactoryUtil.createCatalog(FactoryUtil.java:436) ... 14 more
最新发布
06-19
评论
添加红包

请填写红包祝福语或标题

红包个数最小为10个

红包金额最低5元

当前余额3.43前往充值 >
需支付:10.00
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明:

1.余额是钱包充值的虚拟货币,按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载,可以购买VIP、付费专栏及课程。

余额充值