Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.X] duration:
5.000705591s trace:
https://jaeger.dev.storj.io/trace/745868f00d473319
FAIL (2026-01-12 14:14:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000726951s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [.X..X.X..............XX..X............X...X...X...X.........] duration:
8.656531649s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:14:26 UTC)
orbiter warning: context deadline exceeded
Duration: 8.656558471s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................] duration:
30.000606037s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:14:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000624802s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..] duration:
23.44631507s trace:
https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3
FAIL (2026-01-12 14:14:18 UTC)
orbiter warning: context deadline exceeded
Duration: 23.446331761s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......] duration:
5.749428663s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:14:13 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.749470512s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.] duration:
4.155179845s trace:
https://jaeger.dev.storj.io/trace/301c56ec948f5d75
FAIL (2026-01-12 14:14:05 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.155205856s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX.] duration:
26.247074043s trace:
https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3
FAIL (2026-01-12 14:13:55 UTC)
orbiter warning: context deadline exceeded
Duration: 26.247098449s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....] duration:
5.855454615s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:13:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.855481125s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX] duration:
5.001605902s trace:
https://jaeger.dev.storj.io/trace/301c56ec948f5d75
FAIL (2026-01-12 14:13:44 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00162046s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................] duration:
8.623478517s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:13:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.623515809s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [..X..X.X..............XX..X............X...X...X...X........] duration:
20.757974297s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:13:26 UTC)
orbiter warning: context deadline exceeded
Duration: 20.758002291s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....] duration:
5.801296371s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:13:20 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.801326629s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX] duration:
30.00077818s trace:
https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3
FAIL (2026-01-12 14:13:05 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000802936s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...] duration:
5.839819873s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:13:00 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.839846363s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.X] duration:
5.001160322s trace:
https://jaeger.dev.storj.io/trace/474d86d177a81dd9
FAIL (2026-01-12 14:12:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001177343s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [...X..X.X..............XX..X............X...X...X...X.......] duration:
26.313499954s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:12:24 UTC)
orbiter warning: context deadline exceeded
Duration: 26.313518869s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.] duration:
4.133250954s trace:
https://jaeger.dev.storj.io/trace/1d8b7a245f50d7b0
FAIL (2026-01-12 14:12:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.133274239s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................] duration:
7.61583797s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:12:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.615866794s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...X] duration:
30.001765819s trace:
https://jaeger.dev.storj.io/trace/1b93517304fda245
FAIL (2026-01-12 14:12:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001790816s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [....X..X.X..............XX..X............X...X...X...X......] duration:
32.264495392s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:11:58 UTC)
orbiter warning: context deadline exceeded
Duration: 32.26452059s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..] duration:
5.892815012s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:11:50 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.892895604s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX] duration:
5.000800301s trace:
https://jaeger.dev.storj.io/trace/1d8b7a245f50d7b0
FAIL (2026-01-12 14:11:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000824506s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...] duration:
11.530062106s trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:11:33 UTC)
orbiter warning: context deadline exceeded
Duration: 11.530088906s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................] duration:
6.074232205s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:11:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.07426153s
edge
gateway
us-select-1
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.] duration:
12.475006627s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:10:57 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 12.475025191s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXX] duration:
5.000711199s trace:
https://jaeger.dev.storj.io/trace/34942d9b407cfc57
FAIL (2026-01-12 14:10:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000771514s
core
uplink
us-select-1
us1-placement-20
|
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X] duration:
30.003097197s trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:10:44 UTC)
orbiter warning:
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773 , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329 , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410 , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120 , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150 , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015 , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010 , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940 , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226 , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804 , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293 , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114 , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909 , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121 , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840 , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509 , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 30.003120382s
core
uplink
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [X.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX..] duration:
26.764268714s trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:10:38 UTC)
orbiter warning: context deadline exceeded
Duration: 26.764301066s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [.....X..X.X..............XX..X............X...X...X...X.....] duration:
19.549049075s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:10:32 UTC)
orbiter warning: context deadline exceeded
Duration: 19.549178301s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XX] duration:
5.001908943s trace:
https://jaeger.dev.storj.io/trace/1978a372cd0fdf99
FAIL (2026-01-12 14:10:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001934502s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................] duration:
15.407194876s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:10:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.407216376s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..X] duration:
5.000472963s trace:
https://jaeger.dev.storj.io/trace/130f2c92c624998
FAIL (2026-01-12 14:09:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00049268s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [X.....X..X.X..............XX..X............X...X...X...X....] duration:
10.035085191s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:09:48 UTC)
orbiter warning: context deadline exceeded
Duration: 10.035114497s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [.X.....X..X.X..............XX..X............X...X...X...X...] duration:
24.372352382s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:09:36 UTC)
orbiter warning: context deadline exceeded
Duration: 24.372366909s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................] duration:
7.541529701s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:09:22 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.54155005s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX.] duration:
20.864895844s trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:09:18 UTC)
orbiter warning: context deadline exceeded
Duration: 20.864920902s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [..X.....X..X.X..............XX..X............X...X...X...X..] duration:
34.21347984s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:09:11 UTC)
orbiter warning: context deadline exceeded
Duration: 34.2135014s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..] duration:
4.878046335s trace:
https://jaeger.dev.storj.io/trace/14fff1b25636a1ec
FAIL (2026-01-12 14:08:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.8780697s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................] duration:
13.273918492s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:08:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.273949591s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [...X.....X..X.X..............XX..X............X...X...X...X.] duration:
18.103630388s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:08:19 UTC)
orbiter warning: context deadline exceeded
Duration: 18.103644295s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
33 out of 60 is failed (threshold=3) [.XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX] duration:
30.000571947s trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:08:16 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000592366s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X.] duration:
4.301902302s trace:
https://jaeger.dev.storj.io/trace/14fff1b25636a1ec
FAIL (2026-01-12 14:08:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.301929915s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
11 out of 60 is failed (threshold=3) [....X.....X..X.X..............XX..X............X...X...X...X] duration:
1m0.001559494s trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:08:01 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001836222s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
32 out of 60 is failed (threshold=3) [..XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...X] duration:
30.001233765s trace:
https://jaeger.dev.storj.io/trace/6da172feac770390
FAIL (2026-01-12 14:07:27 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001264433s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X] duration:
5.001359219s trace:
https://jaeger.dev.storj.io/trace/14fff1b25636a1ec
FAIL (2026-01-12 14:07:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001382212s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [X.........X.................................................] duration:
8.948318239s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:07:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.948346894s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)antiblip:
31 out of 60 is failed (threshold=3) [...XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...] duration:
7.322595896s trace:
https://jaeger.dev.storj.io/trace/6482acd26cf47fea
FAIL (2026-01-12 14:06:32 UTC)
orbiter warning: context deadline exceeded
Duration: 7.322616615s
edge
gateway
us-select-1
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.] duration:
4.460747824s trace:
https://jaeger.dev.storj.io/trace/2995f64c02db1c02
FAIL (2026-01-12 14:06:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.460773081s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)antiblip:
10 out of 60 is failed (threshold=3) [.....X.....X..X.X..............XX..X............X...X...X...] duration:
9.163777461s trace:
https://jaeger.dev.storj.io/trace/17a40fac4325fdf3
FAIL (2026-01-12 14:06:30 UTC)
orbiter warning: context deadline exceeded
Duration: 9.163807538s
core
uplink
us-select-1
us1-placement-20
|
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)antiblip:
2 out of 60 is failed (threshold=3) [.X.........X................................................] duration:
7.501269694s trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:06:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.501294652s
edge
gateway
us-select-1
|