Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix "Filter internal and test users" when personsOnEventsMode = PersonsOnEventsMode.PERSON_ID_NO_OVERRIDE_PROPERTIES_ON_EVENTS #27132

Open
danielbachhuber opened this issue Dec 23, 2024 · 1 comment · May be fixed by #27067
Assignees
Labels
bug Something isn't working right feature/experimentation Feature Tag: Experimentation

Comments

@danielbachhuber
Copy link
Contributor

When using a data warehouse table with experiments, "Filter internal and test users" fails with this SQL error when personsOnEventsMode = PersonsOnEventsMode.PERSON_ID_NO_OVERRIDE_PROPERTIES_ON_EVENTS:

posthog.errors.CHQueryErrorUnknownIdentifier: Unknown expression or function identifier 'mat_pp_email' in scope

Here's an example of the full failing query from #27067:

SELECT
    ifNull(sum(accurateCastOrNull(e.usage, 'Float64')), 0) AS total,
    toStartOfDay(toDateTime(e.ds, 'UTC')) AS day_start,
    ifNull(nullIf(toString(e__events.`properties___$feature/test-experiment`), ''), '$$_posthog_breakdown_null_$$') AS breakdown_value
FROM
    (SELECT *
     FROM s3('http://host.docker.internal:19000/posthog/test_storage_bucket-posthog.hogql.datawarehouse.trendquery/*.parquet',
             'object_storage_root_user',
             '[HIDDEN]',
             'Parquet',
             '`ds` Date, `id` String, `usage` Int64, `userid` String')) AS e
    ASOF LEFT JOIN
    (SELECT
        replaceRegexpAll(nullIf(nullIf(JSONExtractRaw(events.properties, '$feature/test-experiment'), ''), 'null'), '^"|"$', '') AS `properties___$feature/test-experiment`,
        replaceRegexpAll(nullIf(nullIf(JSONExtractRaw(events.properties, '$host'), ''), 'null'), '^"|"$', '') AS `properties___$host`,
        events.event AS event,
        replaceRegexpAll(nullIf(nullIf(JSONExtractRaw(events.properties, '$user_id'), ''), 'null'), '^"|"$', '') AS `e__events___properties___$user_id`,
        toTimeZone(events.timestamp, 'UTC') AS timestamp,
        events.distinct_id AS distinct_id,
        events.properties AS properties
     FROM events
     WHERE (events.team_id = 2185)
        AND (event = '$feature_flag_called')
        AND (timestamp >= assumeNotNull(parseDateTime64BestEffortOrNull('2023-01-01 00:00:00', 6, 'UTC')))
        AND (timestamp <= assumeNotNull(parseDateTime64BestEffortOrNull('2025-01-06 13:07:18', 6, 'UTC')))) AS e__events
    ON (e__events.event = '$feature_flag_called')
        AND (e.userid = e__events.`e__events___properties___$user_id`)
        AND (e.ds >= e__events.timestamp)
WHERE
    ifNull(toDateTime(e.ds, 'UTC') >= assumeNotNull(parseDateTime64BestEffortOrNull('2023-01-01 00:00:00', 6, 'UTC')), 0)
    AND ifNull(toDateTime(e.ds, 'UTC') <= assumeNotNull(parseDateTime64BestEffortOrNull('2025-01-06 13:07:18', 6, 'UTC')), 0)
    AND ifNull(nullIf(nullIf(mat_pp_email, ''), 'null') NOT ILIKE '%@posthog.com%', 1)
    AND ifNull(NOT match(toString(e__events.`properties___$host`), '^(localhost|127\\.0\\.0\\.1)($|:)'), 1)
    AND (ifNull(e__events.event = '$feature_flag_called', 0)
        AND (ifNull(e__events.`properties___$feature/test-experiment` = 'control', 0)
            OR ifNull(e__events.`properties___$feature/test-experiment` = 'test', 0)))
GROUP BY
    day_start,
    breakdown_value

The underlying problem is that e__events__person.properties___email is resolving to mat_pp_email, but there's no mat_pp_email materialized column on the data warehouse table. The query should resolve to e__events.mat_pp_email.

@danielbachhuber danielbachhuber added bug Something isn't working right feature/experimentation Feature Tag: Experimentation labels Dec 23, 2024
@danielbachhuber
Copy link
Contributor Author

Query failure comes from

field = field_type.resolve_database_field(self.context)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working right feature/experimentation Feature Tag: Experimentation
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant