Clarify "File" Source behavior

[DmitryLukyanov]

I have very simple vector configuration (windows OS):

api:
  enabled: true

acknowledgements:
  enabled: true

data_dir: ./data

sources:
  fileIn:
	type: "file"
	include: 
	  - ./logs/*.log
	internal_metrics:
	  include_file_tag: true

transforms:
  msg_parser:
	type: remap
	inputs: [fileIn]
	source: |
	  # send everything to sinks
	  .=.

sinks:

  http:
	type: "http"
	method: post
	inputs:
	  - "msg_parser"
	uri: http://localhost:7000/api/MyFunction
	tls:
	  verify_certificate: false
	encoding:
	  codec: "json"
	request:
	  retry_attempts: 3

which I run with the following command:

vector `
    -c .\vector.yaml `
    --require-healthy true `
    --color always `
    --no-graceful-shutdown-limit `
    --watch-config `
    --verbose `

Let's say I have a log file with 10000 lines (each line is guid\r\n). When I run vector command, it looks like it processes only first "batch" (in my case first 52 lines):

The last output lines in the shell is:

    ..
	2025-01-01T20:42:56.006974Z DEBUG sink{component_kind="sink" component_id=http component_type=http}:request{request_id=1}:http: vector::internal_events::http_client: HTTP response. status=200 OK version=HTTP/1.1 headers={"content-type": "text/plain; charset=utf-8", "date": "Wed, 01 Jan 2025 20:42:55 GMT", "server": "Kestrel", "transfer-encoding": "chunked"} body=[unknown]
	2025-01-01T20:42:56.059867Z DEBUG hyper::proto::h1::conn: incoming body completed
	2025-01-01T20:42:56.070865Z DEBUG hyper::client::pool: pooling idle connection for ("http", localhost:7000)
	2025-01-01T20:42:56.078576Z DEBUG sink{component_kind="sink" component_id=http component_type=http}: vector::topology::builder: Sink finished normally.
	2025-01-01T20:42:56.127379Z DEBUG source{component_kind="source" component_id=fileIn component_type=file}: vector::topology::builder: Source finished normally.
	PS C:\VECTOR_TOOL_PATH>

to process a next batch, I need to launch the vector -c .. command again (checkpoints.json each time is updated accordingly).

So my questions are:

1. Is there a way to process the whole file until the end without recalling vector -c .. command multiple time?
2. How to understand how many lines will be placed into a single batch? It looks like it's completely random (sizes are like 52, 500+, 447..)

[jszwedko]

Hi @DmitryLukyanov !

• Is there a way to process the whole file until the end without recalling vector -c .. command multiple time?

This should be the default behavior. Vector will attempt to read all data from the last checkpoint and send it (this may mean multiple batches). If you aren't seeing this, which would be surprising given how widely used the file source is, granted more widely on Linux than Windows, would you mind opening a bug report with a reproduction case?

• How to understand how many lines will be placed into a single batch? It looks like it's completely random (sizes are like 52, 500+, 447..)

The default batch settings for the http sink are documented here: https://vector.dev/docs/reference/configuration/sinks/http/#batch. In particular, the default batch timeout is 1s so it'll batch data every 1s by default.

[DmitryLukyanov]

Thanks @jszwedko ,

The default batch settings for the http sink are documented here: https://vector.dev/docs/reference/configuration/sinks/http/#batch. In particular, the default batch timeout is 1s so it'll batch data every 1s by default.

I saw it, I mainly surprised that the first batch is often (always?) small comparing with other. For log file with 1000 records like:

	____1_72112c2f-7a53-48eb-ba97-a32b5af86735
	____2_fa23ee39-3d55-4710-9cd6-8e287f832346
	____3_d5841f3f-01b3-41b9-a65a-e2d606dc725a
	____4_69c05d23-dad7-4d5a-bcd8-5361ce311567
	____5_fdf3668e-a997-410d-8953-831defc2f3bb
	____6_07591eef-29f2-47a0-a5ce-38f2667da5e8
	____7_7f0900ae-00f8-492f-87f3-c8ea23544c75
	____8_732d2854-cb4e-4218-a9c0-a60f2550bd58
        ...
        ____1000_#############################

log lines in batches for few runs:

[1-47, 48-370, 371-830, 831-1000]
[1-47, 48-462, 463-1000]
[1-47, 48-462, 463-968, 969-1000]

would you mind opening a bug report with a reproduction case?

please see it here #22108