Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..............X.............................................] duration:
19.317111288s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:16:14 UTC)
orbiter warning: context deadline exceeded
Duration: 19.317132547s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
13 out of 60 is failed (threshold=3) [..X.X..........X..........X.X.......X........X...X.X.X...XXX] duration:
30.00085192s trace:
https://jaeger.dev.storj.io/trace/790f8f85937efaa5
FAIL (2026-01-13 11:15:54 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000874645s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...............X............................................] duration:
26.35187986s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:15:43 UTC)
orbiter warning: context deadline exceeded
Duration: 26.351906459s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001131979s trace:
https://jaeger.dev.storj.io/trace/dc2fc1ae2b7b6e1
FAIL (2026-01-13 11:15:23 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001154901s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
12 out of 60 is failed (threshold=3) [...X.X..........X..........X.X.......X........X...X.X.X...XX] duration:
30.001035286s trace:
https://jaeger.dev.storj.io/trace/349434337a6115d8
FAIL (2026-01-13 11:15:05 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001046408s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [................X...........................................] duration:
20.178299466s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:15:00 UTC)
orbiter warning: context deadline exceeded
Duration: 20.17831741s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
11 out of 60 is failed (threshold=3) [....X.X..........X..........X.X.......X........X...X.X.X...X] duration:
30.001659968s trace:
https://jaeger.dev.storj.io/trace/3c65b6151dbde8c1
FAIL (2026-01-13 11:14:35 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001681029s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00148209s trace:
https://jaeger.dev.storj.io/trace/23357b55a9a012
FAIL (2026-01-13 11:14:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001504422s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.................X..........................................] duration:
19.818744045s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:14:06 UTC)
orbiter warning: context deadline exceeded
Duration: 19.818760957s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000745485s trace:
https://jaeger.dev.storj.io/trace/5e1358068060855c
FAIL (2026-01-13 11:13:58 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000831648s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.....X.X..........X..........X.X.......X........X...X.X.X...] duration:
27.795717464s trace:
https://jaeger.dev.storj.io/trace/7a5ae7dc904a6aa8
FAIL (2026-01-13 11:13:53 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 27.795741821s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000868006s trace:
https://jaeger.dev.storj.io/trace/ddfc01aa58338c9
FAIL (2026-01-13 11:13:38 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000884166s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
5.686868082s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:13:34 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.686893051s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [..................X.........................................] duration:
22.806699013s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:13:25 UTC)
orbiter warning: context deadline exceeded
Duration: 22.806725854s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
5.754410619s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:12:58 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.75446327s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [......X.X..........X..........X.X.......X........X...X.X.X..] duration:
23.167941633s trace:
https://jaeger.dev.storj.io/trace/7a5ae7dc904a6aa8
FAIL (2026-01-13 11:12:41 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 23.167954588s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.000753384s trace:
https://jaeger.dev.storj.io/trace/f79b9b9f3c79946
FAIL (2026-01-13 11:12:38 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000776427s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [...................X........................................] duration:
18.775348835s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:12:29 UTC)
orbiter warning: context deadline exceeded
Duration: 18.77537247s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
5.676851791s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:12:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.676879945s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001483489s trace:
https://jaeger.dev.storj.io/trace/38e092e4923911bf
FAIL (2026-01-13 11:12:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001502315s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.......X.X..........X..........X.X.......X........X...X.X.X.] duration:
24.347997505s trace:
https://jaeger.dev.storj.io/trace/7a5ae7dc904a6aa8
FAIL (2026-01-13 11:12:07 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 24.348014255s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [....................X.......................................] duration:
18.577033229s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:11:59 UTC)
orbiter warning: context deadline exceeded
Duration: 18.577059929s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001217916s trace:
https://jaeger.dev.storj.io/trace/2835c25a5f34b764
FAIL (2026-01-13 11:11:47 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001242632s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [........X.X..........X..........X.X.......X........X...X.X.X] duration:
30.001240075s trace:
https://jaeger.dev.storj.io/trace/7a5ae7dc904a6aa8
FAIL (2026-01-13 11:11:42 UTC)
orbiter warning: operation error S3: DeleteObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.001251106s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001234124s trace:
https://jaeger.dev.storj.io/trace/786ad80e03268796
FAIL (2026-01-13 11:11:32 UTC)
orbiter warning: context deadline exceeded
Duration: 5.001256337s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.....................X......................................] duration:
17.357174705s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:11:27 UTC)
orbiter warning: context deadline exceeded
Duration: 17.357199162s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
5.764403753s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:11:17 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.76442944s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [X........X.X..........X..........X.X.......X........X...X.X.] duration:
24.845833086s trace:
https://jaeger.dev.storj.io/trace/695c8fccbbe9b345
FAIL (2026-01-13 11:11:12 UTC)
orbiter warning: context deadline exceeded
Duration: 24.845846191s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.X........X.X..........X..........X.X.......X........X...X.X] duration:
30.000738455s trace:
https://jaeger.dev.storj.io/trace/695c8fccbbe9b345
FAIL (2026-01-13 11:10:47 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000766869s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [......................X.....................................] duration:
26.333693342s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:10:31 UTC)
orbiter warning: context deadline exceeded
Duration: 26.333737586s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001260313s trace:
https://jaeger.dev.storj.io/trace/69c04917f7a4b4fe
FAIL (2026-01-13 11:10:28 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001291883s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
5.758363448s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:10:19 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.758388645s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
5.722029257s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:09:45 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.722053922s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [X...........................................................] duration:
6.343142888s trace:
https://jaeger.dev.storj.io/trace/47e2785e526cc2e9
FAIL (2026-01-13 11:09:42 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 6.343173475s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001855268s trace:
https://jaeger.dev.storj.io/trace/5325527cdd1d49b9
FAIL (2026-01-13 11:09:41 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001890215s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [X.X........X.X..........X..........X.X.......X........X...X.] duration:
19.164920649s trace:
https://jaeger.dev.storj.io/trace/54c4d6555b0dd702
FAIL (2026-01-13 11:09:39 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 19.164943443s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.......................X....................................] duration:
17.273454762s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:09:12 UTC)
orbiter warning: context deadline exceeded
Duration: 17.273481754s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.X..........................................................] duration:
9.09631406s trace:
https://jaeger.dev.storj.io/trace/47e2785e526cc2e9
FAIL (2026-01-13 11:09:09 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 9.096341202s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
5.668155983s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:09:02 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.668180569s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.0010273s trace:
https://jaeger.dev.storj.io/trace/36cfe874e53506bc
FAIL (2026-01-13 11:08:53 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001050435s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [..X.........................................................] duration:
9.652424694s trace:
https://jaeger.dev.storj.io/trace/47e2785e526cc2e9
FAIL (2026-01-13 11:08:41 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 9.652505919s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.00177655s trace:
https://jaeger.dev.storj.io/trace/619282d205f37277
FAIL (2026-01-13 11:08:33 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001801158s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
28.276474886s trace:
https://jaeger.dev.storj.io/trace/47e2785e526cc2e9
FAIL (2026-01-13 11:08:19 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 28.276499794s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
10 out of 60 is failed (threshold=3) [.X.X........X.X..........X..........X.X.......X........X...X] duration:
10.680036848s trace:
https://jaeger.dev.storj.io/trace/54c4d6555b0dd702
FAIL (2026-01-13 11:08:17 UTC)
orbiter warning: Post "https://auth.us-select-1.storjshare.io/v1/access": net/http: TLS handshake timeout
Duration: 10.680061845s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
59 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX.XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX] duration:
5.001319967s trace:
https://jaeger.dev.storj.io/trace/5be1a60fbed670b7
FAIL (2026-01-13 11:08:13 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00134232s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [........................X...................................] duration:
17.826740035s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:08:03 UTC)
orbiter warning: context deadline exceeded
Duration: 17.826778579s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
5.732451819s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:07:56 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.732542371s
core
uplink
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
5.72178423s trace:
https://jaeger.dev.storj.io/trace/26edf54e939e970
FAIL (2026-01-13 11:07:40 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
an 13 10:27:58.408186422 (930h58m58.279269457s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 13 10:27:58.646330692 (930h58m58.517413727s) [seg_pos={0 0}, upload=814840]: Begin upload segment...
Jan 13 10:27:58.646391258 (930h58m58.517474303s) [seg_pos={0 0}, upload=814840]: Done begin upload segment.
Jan 13 10:27:58.646454538 (930h58m58.517537573s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.646473925 (930h58m58.51755696s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647050132 (930h58m58.518133167s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.647393596 (930h58m58.518476641s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648088799 (930h58m58.519171834s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.648541381 (930h58m58.519624426s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.649132316 (930h58m58.520215351s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650817144 (930h58m58.521900179s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.650935891 (930h58m58.522018926s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652370643 (930h58m58.523453678s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.652656386 (930h58m58.523739431s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.653934761 (930h58m58.525017796s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.654305877 (930h58m58.525388912s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.655277527 (930h58m58.526360572s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.658532194 (930h58m58.529615229s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:27:58.661697051 (930h58m58.532780086s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Uploading piece...
Jan 13 10:28:00.490393026 (930h59m0.361476071s) [upload=814840]: Next returned nil segment
Jan 13 10:28:00.490410380 (930h59m0.361493425s) [upload=814840]: Waiting for error group managing segments...
Jan 13 10:28:00.787228480 (930h59m0.658311515s) [noise=true , address=77.90.31.16:4735 , piece_id=NJAJCJVIY3AQW6343NUPRDQO5PC3AUQ6SK5HUYEQVXQW2MBDZ5SA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.792526608 (930h59m0.663609654s) [noise=true , address=145.79.227.19:4077 , piece_id=37PG6IGDB23X35D3BU4AHTZAFOVWNC25EKMU5U5K3M2X2OFXPRPQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.801640662 (930h59m0.672723697s) [noise=true , address=145.79.227.5:4221 , piece_id=KS7FJTYX67KZDH2J2RBP7BKY6YZDNSGBO3D7EFINRJKLPPCKKZ5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.805972771 (930h59m0.677055806s) [noise=true , address=145.79.227.4:4185 , piece_id=WNPO7CX5KPS6RVMTLLWL6BLD2QQDY6KVMDHORD4JA4ZHZJQULFCQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.816426246 (930h59m0.687509291s) [noise=true , address=145.79.227.51:4411 , piece_id=TH6EIG5APQSOAMN3ARGL74IU6YZFFNVUSVVSWM525HZIC6TAJ6VA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.819742430 (930h59m0.690825475s) [noise=true , address=145.79.227.48:4302 , piece_id=SKZR6QJ6YPPGVPDXURZMEOX4BIF4O3AGVD5ZEVYP7Z7Y5Q35AM5A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.826633591 (930h59m0.697716626s) [noise=true , address=145.79.227.21:4473 , piece_id=26L46PQ3JAEYDBMXPOGUSJLSCONN6Z3HX6TOYDMRMTAI2O2DVBIQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.831071451 (930h59m0.702154496s) [noise=true , address=145.79.227.44:4193 , piece_id=CVU7DBSS4VVSLYWGAESJ7R3DSYNTO3XOQKTGVZPFETF3MG4DBWWQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.883814340 (930h59m0.754897375s) [noise=true , address=145.79.227.35:4483 , piece_id=QAFNOD6VAC35OSVQ6Y2NPA4SZL432UYCJDVSFP3DNAGHD6UMWZUQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.898714861 (930h59m0.769797896s) [noise=true , address=145.79.227.38:4979 , piece_id=EDQMWT2QOHOAETJQIORDKXVORXF7LBHCVERA7NGU2N4I7TBQIFFA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:00.912698137 (930h59m0.783781182s) [noise=true , address=145.79.227.14:4839 , piece_id=O3FNWQVLUGIBHPSAXY34W3SMK5DS2T2PWKAHG4RUJAWDJXJOSDYQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.011874529 (930h59m0.882957574s) [noise=true , address=145.79.227.52:4446 , piece_id=LOZMZHIB72OFW4FCAAZ3ICJ4DVUNFZNT26BZZ244VJA53FTUFUZA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545340891 (930h59m1.416423946s) [noise=true , address=77.90.31.21:4880 , piece_id=R7COZDY42N3ET2MA7TBNH6STQX4HNKLPIYUJ5KSZBNEVSMAHZ56A, seg_pos={0 0}, upload=814840]: Done uploading piece. err: <nil>
Jan 13 10:28:01.545362893 (930h59m1.416445928s) [seg_pos={0 0}, upload=814840]: Segment reached optimal threshold of 13 pieces.
Jan 13 10:28:01.545551462 (930h59m1.416634497s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1JVvtSnmC6UL5tNSGeJg27797Z4T1DBsMLqVche429yRKvZZyG): protocol: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37832->45.91.116.14:4190: use of closed network connection
Jan 13 10:28:01.545560739 (930h59m1.416643784s) [noise=true , address=45.91.116.14:4190 , piece_id=7PV5GC24U6MPJYOF5G3HUZROKC2ZFQDE2TE7EPCQLNXLRHESI6IQ, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545573574 (930h59m1.416656619s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1KhBWMerZe9tUmzdVEYkBb6NmJ25dGP26Ea7CLjNJT5gdAeiah): protocol: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:34902->145.79.227.26:4654: use of closed network connection
Jan 13 10:28:01.545579155 (930h59m1.41666219s) [noise=true , address=145.79.227.26:4654 , piece_id=3ZDKMFRF6WBKNABL74723UNCR4UWPO3MHJ6GK6ILVFKSIPBS2EMA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545669947 (930h59m1.416752992s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12T7G85LguKGbxuMDp5u7q1j3TWc1ERBH6YgynJhE5NzzcyuMom): protocol: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:37788->145.79.227.11:4730: use of closed network connection
Jan 13 10:28:01.545676480 (930h59m1.416759525s) [noise=true , address=145.79.227.11:4730 , piece_id=CH4OVACXUIFE3HC5SWJLCSAVC6CE5LD2XJBYWCIGAY6MRSYFAWXA, seg_pos={0 0}, upload=814840]: Long tail cancellation due to optimal threshold
Jan 13 10:28:01.545768705 (930h59m1.41685175s) [seg_pos={0 0}, upload=814840]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 13 10:28:01.545932858 (930h59m1.417015903s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 13 10:28:01.545941504 (930h59m1.41702455s) [upload=814840]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 13 10:28:01.830463272 (930h59m1.701546327s) [upload=814840]: Done uploading segments...
Jan 13 10:28:01.830492568 (930h59m1.701575623s) [upload=814840]: Upload finished. err: <nil>
Duration: 5.721945918s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
9 out of 60 is failed (threshold=3) [..X.X........X.X..........X..........X.X.......X........X...] duration:
22.296593457s trace:
https://jaeger.dev.storj.io/trace/2bbd0d70cfae568
FAIL (2026-01-13 11:07:35 UTC)
orbiter warning: context deadline exceeded
Duration: 22.296622502s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
1 out of 60 is failed (threshold=3) [.........................X..................................] duration:
16.30939509s trace:
https://jaeger.dev.storj.io/trace/4043e5ecf3cbdbeb
FAIL (2026-01-13 11:07:13 UTC)
orbiter warning: context deadline exceeded
Duration: 16.309421328s
core
uplink
us-select-1
us1-placement-20
|