Table creation / deletion oddities when writing from Telegraf | Requires Investigation

[peterbarnett03]

An odd scenario where deleting a DB seems to work, but then when recreating it, tag issues are created. This is primarily seen on Telegraf writing on 1.33, and doesn't appear when writing line protocol from the CLI (on macOS). So needs investigation in several areas.

Steps to reproduce:

1. Create DB cpu
2. Run SHOW DATABASES (influxQL) > The cpu DB has been created
3. Delete DB cpu
4. Run SHOW DATABASES > The cpu DB has been deleted
5. Start Telegraf in v2 mode, attempt to write to cpu DB > Say's it's writing, but doesn't, and doesn't create DB or table
6. Start Telegraf in v1 mode, attempt to write to cpu DB > Fails on write, error says we're trying to add a tag that doesn't exist | Telegraf in v2 continues to say it's writing, but doesn't
7. Run SHOW DATABASES > The cpu DB has returned, with no data tables, but contains system tables

Expected behaviour:
When Telegraf says it's writing, it should create a new DB, a new table, write the data, and have no tag issues, since it's a fresh new database and table.

Actual behaviour:
InfluxDB is saying that there is an attempt to write a new tag, as if there is an old table it's trying to write to and can't, and therefore it doesn't write to a new table at all (but it does create the DB).

Seems like the system:

1. Gets a write request
2. Creates the new DB
3. Checks if there's a table
4. Discovers a table, can't write a tag to it
5. Complains

It should never discover a table. Perhaps due to soft delete?

Environment info:

System: Linux ARM
Storage: Memory
Telegraf: v1.33

Logs:
error_message: "Detected a new tag 'label' in write. The tag set is immutable on first write to the table."

[pauldix]

I think Telegraf is excess noise here. We should be able to reproduce using just curl against the v1 or v2 write APIs.

[M617263Kaplan]

telegraf.txt

Provided config file for telegraf.

[M617263Kaplan]

1. influxdb3 serve -vv --host-id=local03 --object-store=memory
2. influxdb3 query --dbname=servers --lang=influxql "SHOW DATABASES"

[M617263Kaplan]

[M617263Kaplan]

Last step --- Telegraf is writing all metrics to metrics.out. now we write them back in.

1. influxdb3 write --dbname=cpu4 --file /tmp/metrics.out
   

[M617263Kaplan]

Telegraf 1.30 appears to work but 1.33 might be an issue,

[M617263Kaplan]

[M617263Kaplan]

Mac M2 Telgraf sending data to OSS 3 running in ESXi Linux Ubuntu 20.4
Virtualbox running Linux Ubuntu 20.4. Telegraf ARM processor sending data to OSS 3 running in ESXi Linux Ubuntu 20.4
Esxi Linux Ubuntu 20.4 Intel telegraf writing to the local Database
influxdb3 serve --host-id=local1231v2 -v --object-store=memory
The problem:
Only some of the Databases are created.
+------------------------------+
| iox::database |
+------------------------------+
| esx_linux__v1_Intel_telegraf |
| mac_v1_arm_telegraf |
| virtualbox_v1_arm_telegraf |
| virtualbox_v2_arm_telegraf |
+------------------------------+
As you can see, only Virtualbox Linux can accept V1 and V2 writes.
Telegraf 1.33
The attached Telegraf conf is identical on all platforms.
I can force V1 and V2 to write into the Database from all hosts by using the export to file and then writing the data back into the Database.
And the last problem --- :headwall: --- As in the EAR.
Only one database shows Tables and will return query data.
influxdb3 query --dbname=virtualbox_v1_arm_telegraf "SHOW TABLES"
+---------------+--------------------+----------------------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+----------------------------+------------+
| public | system | last_caches | BASE TABLE |
| public | system | meta_caches | BASE TABLE |
| public | system | parquet_files | BASE TABLE |
| public | system | processing_engine_plugins | BASE TABLE |
| public | system | processing_engine_triggers | BASE TABLE |
| public | system | queries | BASE TABLE |
| public | information_schema | tables | VIEW |
| public | information_schema | views | VIEW |
| public | information_schema | columns | VIEW |
| public | information_schema | df_settings | VIEW |
| public | information_schema | schemata | VIEW |
+---------------+--------------------+----------------------------+------------+
android@ossalpha:/etc/telegraf$ influxdb3 query --dbname=virtualbox_v2_arm_telegraf "SHOW TABLES"
+---------------+--------------------+----------------------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+----------------------------+------------+
| public | system | last_caches | BASE TABLE |
| public | system | meta_caches | BASE TABLE |
| public | system | parquet_files | BASE TABLE |
| public | system | processing_engine_plugins | BASE TABLE |
| public | system | processing_engine_triggers | BASE TABLE |
| public | system | queries | BASE TABLE |
| public | information_schema | tables | VIEW |
| public | information_schema | views | VIEW |
| public | information_schema | columns | VIEW |
| public | information_schema | df_settings | VIEW |
| public | information_schema | schemata | VIEW |
+---------------+--------------------+----------------------------+------------+
android@ossalpha:/etc/telegraf$ influxdb3 query --dbname=mac_v1_arm_telegraf "SHOW TABLES"
+---------------+--------------------+----------------------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+----------------------------+------------+
| public | iox | swap | BASE TABLE |
| public | iox | filestat | BASE TABLE |
| public | iox | mem | BASE TABLE |
| public | iox | cpu | BASE TABLE |
| public | iox | system | BASE TABLE |
| public | iox | disk | BASE TABLE |
| public | iox | diskio | BASE TABLE |
| public | iox | net_response | BASE TABLE |
| public | iox | ping | BASE TABLE |
| public | iox | processes | BASE TABLE |
| public | iox | netstat | BASE TABLE |
| public | system | last_caches | BASE TABLE |
| public | system | meta_caches | BASE TABLE |
| public | system | parquet_files | BASE TABLE |
| public | system | processing_engine_plugins | BASE TABLE |
| public | system | processing_engine_triggers | BASE TABLE |
| public | system | queries | BASE TABLE |
| public | information_schema | tables | VIEW |
| public | information_schema | views | VIEW |
| public | information_schema | columns | VIEW |
| public | information_schema | df_settings | VIEW |
| public | information_schema | schemata | VIEW |
+---------------+--------------------+----------------------------+------------+
android@ossalpha:/etc/telegraf$ influxdb3 query --dbname=esx_linux__v1_Intel_telegraf "SHOW TABLES"
+---------------+--------------------+----------------------------+------------+
| table_catalog | table_schema | table_name | table_type |
+---------------+--------------------+----------------------------+------------+
| public | system | last_caches | BASE TABLE |
| public | system | meta_caches | BASE TABLE |
| public | system | parquet_files | BASE TABLE |
| public | system | processing_engine_plugins | BASE TABLE |
| public | system | processing_engine_triggers | BASE TABLE |
| public | system | queries | BASE TABLE |
| public | information_schema | tables | VIEW |
| public | information_schema | views | VIEW |
| public | information_schema | columns | VIEW |
| public | information_schema | df_settings | VIEW |
| public | information_schema | schemata | VIEW |
+---------------+--------------------+----------------------------+------------+ (edited)
Binary

[M617263Kaplan]

telegraf.conf.txt