Failed to fetch definition for table - java.lang.IllegalArgumentException: No enum constant com.google.cloud.bigquery.LegacySQLTypeName.NUMERIC

[peelandsee]

Hi!

let me thank you for the amazing project!
I'm just testing the BigQuery connector, but I'm struggling in exposing a couple of test tables.

I'm getting this error:

15:31:37.551 [application-contexts.metadataRefresh-42] WARN com.thumbtack.becquerel.datasources.bigquery.BqService  - Failed to fetch definition for table {datasetId=salesforce, projectId=data-test-xxxxx, tableId=test1}
java.lang.IllegalArgumentException: No enum constant com.google.cloud.bigquery.LegacySQLTypeName.NUMERIC

with a table that contains a NUMERIC field:

the table is ignored in the exposed OData schema.

may you know how this could be fixed?

thanks for your efforti!

full error:

15:31:36.222 [application-contexts.metadataRefresh-23] INFO com.thumbtack.becquerel.datasources.bigquery.BqService  - Starting metadata refresh for BigQuery service bq.
15:31:37.551 [application-contexts.metadataRefresh-42] WARN com.thumbtack.becquerel.datasources.bigquery.BqService  - Failed to fetch definition for table {datasetId=salesforce, projectId=data-test-xxxxx, tableId=test1}
java.lang.IllegalArgumentException: No enum constant com.google.cloud.bigquery.LegacySQLTypeName.NUMERIC
        at java.lang.Enum.valueOf(Enum.java:238)
        at com.google.cloud.bigquery.LegacySQLTypeName.valueOf(LegacySQLTypeName.java:25)
        at com.google.cloud.bigquery.Field.fromPb(Field.java:473)
        at com.google.cloud.bigquery.Field$1.apply(Field.java:46)
        at com.google.cloud.bigquery.Field$1.apply(Field.java:43)
        at com.google.common.collect.Lists$TransformingRandomAccessList$1.transform(Lists.java:640)
        at com.google.common.collect.TransformedIterator.next(TransformedIterator.java:48)
        at java.util.AbstractCollection.toArray(AbstractCollection.java:141)
        at java.util.ArrayList.<init>(ArrayList.java:178)
        at com.google.common.collect.Lists.newArrayList(Lists.java:146)
        at com.google.cloud.bigquery.Schema$Builder.setFields(Schema.java:86)
        at com.google.cloud.bigquery.Schema.of(Schema.java:189)
        at com.google.cloud.bigquery.Schema.fromPb(Schema.java:200)
        at com.google.cloud.bigquery.TableDefinition$Builder.<init>(TableDefinition.java:88)
        at com.google.cloud.bigquery.StandardTableDefinition$Builder.<init>(StandardTableDefinition.java:172)
        at com.google.cloud.bigquery.StandardTableDefinition$Builder.<init>(StandardTableDefinition.java:149)
        at com.google.cloud.bigquery.StandardTableDefinition.fromPb(StandardTableDefinition.java:404)
        at com.google.cloud.bigquery.TableDefinition.fromPb(TableDefinition.java:198)
        at com.google.cloud.bigquery.TableInfo$BuilderImpl.<init>(TableInfo.java:190)
        at com.google.cloud.bigquery.Table.fromPb(Table.java:591)
        at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:398)
        at com.thumbtack.becquerel.datasources.bigquery.BqService$$anonfun$fetchDefinitionWithSchema$1$$anonfun$apply$15.apply(BqService.scala:276)
        at com.thumbtack.becquerel.datasources.bigquery.BqService$$anonfun$fetchDefinitionWithSchema$1$$anonfun$apply$15.apply(BqService.scala:275)
        at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
        at scala.concurrent.package$.blocking(package.scala:123)
        at com.thumbtack.becquerel.datasources.bigquery.BqService$$anonfun$fetchDefinitionWithSchema$1.apply(BqService.scala:275)
        at com.thumbtack.becquerel.datasources.bigquery.BqService$$anonfun$fetchDefinitionWithSchema$1.apply(BqService.scala:275)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.liftedTree1$1(Future.scala:24)
        at scala.concurrent.impl.Future$PromiseCompletingRunnable.run(Future.scala:24)
        at com.thumbtack.becquerel.util.MDCPropagatingDispatcher$$anon$1$$anon$2.run(MDCPropagatingDispatcher.scala:98)
        at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:39)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

[SteelPangolin]

Hey, I'm the original author of Becquerel. Sorry for the turnaround time; I left the company that maintains it a few months back, and didn't see this until today. In the event you're still trying to solve this, Becquerel was written before BigQuery added the NUMERIC and BIGNUMERIC types, so it doesn't know about them. You'll likely need to:

• update the BQ client library in build.sbt (the current version, 2.1.6, is three years newer)
• create a BqNumericMapper in BqValueMappers.scala to map NUMERIC fields and their values to a suitable OData EDM type for the field (probably Decimal) and a suitable JVM type for the value (Java's BigDecimal through the fieldValue.getNumericValue accessor)
• extend the pattern match in BqFieldMapper to handle NUMERIC with that new BqNumericMapper

I'm not sure how this needs to work for parameterized NUMERIC, but worry about that once you have the regular one working, and only if you actually use parameterized types.

[peelandsee]

Hey @SteelPangolin, thanks for your followup and the clear path you're showing to implement the new type support.

Unfortunately I'm not working anymore on this topic and I'll be unable to address this issue.
I keep the issue open in the case someone wants to pick your suggestions and work on it.

thanks again,