Orbiter: EU/US/AP | Code | Config | Playbook | Edge Deployment | Escalation Procedure

Orbiter Check History

Showing 50 of 460752 reports (Page 451 of 9216)
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.X]
duration:
5.000705591s
trace:
https://jaeger.dev.storj.io/trace/745868f00d473319
FAIL (2026-01-12 14:14:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000726951s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
10 out of 60 is failed (threshold=3) [.X..X.X..............XX..X............X...X...X...X.........]
duration:
8.656531649s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:14:26 UTC)
orbiter warning: context deadline exceeded
Duration: 8.656558471s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [...X........................................................]
duration:
30.000606037s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:14:19 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 30.000624802s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
31 out of 60 is failed (threshold=3) [.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX..]
duration:
23.44631507s
trace:
https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3
FAIL (2026-01-12 14:14:18 UTC)
orbiter warning: context deadline exceeded
Duration: 23.446331761s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.....................................................X......]
duration:
5.749428663s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:14:13 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.749470512s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
48 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX.]
duration:
4.155179845s
trace:
https://jaeger.dev.storj.io/trace/301c56ec948f5d75
FAIL (2026-01-12 14:14:05 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.155205856s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
32 out of 60 is failed (threshold=3) [X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX.]
duration:
26.247074043s
trace:
https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3
FAIL (2026-01-12 14:13:55 UTC)
orbiter warning: context deadline exceeded
Duration: 26.247098449s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [......................................................X.....]
duration:
5.855454615s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:13:50 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.855481125s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.XX]
duration:
5.001605902s
trace:
https://jaeger.dev.storj.io/trace/301c56ec948f5d75
FAIL (2026-01-12 14:13:44 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00162046s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [....X.......................................................]
duration:
8.623478517s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:13:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.623515809s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
10 out of 60 is failed (threshold=3) [..X..X.X..............XX..X............X...X...X...X........]
duration:
20.757974297s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:13:26 UTC)
orbiter warning: context deadline exceeded
Duration: 20.758002291s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.......................................................X....]
duration:
5.801296371s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:13:20 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.801326629s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
32 out of 60 is failed (threshold=3) [.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...XX]
duration:
30.00077818s
trace:
https://jaeger.dev.storj.io/trace/2ef8330bda1f08b3
FAIL (2026-01-12 14:13:05 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000802936s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [........................................................X...]
duration:
5.839819873s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:13:00 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.839846363s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.X]
duration:
5.001160322s
trace:
https://jaeger.dev.storj.io/trace/474d86d177a81dd9
FAIL (2026-01-12 14:12:46 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001177343s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
10 out of 60 is failed (threshold=3) [...X..X.X..............XX..X............X...X...X...X.......]
duration:
26.313499954s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:12:24 UTC)
orbiter warning: context deadline exceeded
Duration: 26.313518869s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
49 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX.]
duration:
4.133250954s
trace:
https://jaeger.dev.storj.io/trace/1d8b7a245f50d7b0
FAIL (2026-01-12 14:12:16 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.133274239s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [.....X......................................................]
duration:
7.61583797s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:12:16 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.615866794s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
32 out of 60 is failed (threshold=3) [X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...X]
duration:
30.001765819s
trace:
https://jaeger.dev.storj.io/trace/1b93517304fda245
FAIL (2026-01-12 14:12:13 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001790816s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
10 out of 60 is failed (threshold=3) [....X..X.X..............XX..X............X...X...X...X......]
duration:
32.264495392s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:11:58 UTC)
orbiter warning: context deadline exceeded
Duration: 32.26452059s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [.........................................................X..]
duration:
5.892815012s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:11:50 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 5.892895604s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXXX]
duration:
5.000800301s
trace:
https://jaeger.dev.storj.io/trace/1d8b7a245f50d7b0
FAIL (2026-01-12 14:11:37 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000824506s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
31 out of 60 is failed (threshold=3) [.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX...]
duration:
11.530062106s
trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:11:33 UTC)
orbiter warning: context deadline exceeded
Duration: 11.530088906s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [......X.....................................................]
duration:
6.074232205s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:11:04 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 6.07426153s edge gateway us-select-1
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [..........................................................X.]
duration:
12.475006627s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:10:57 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 12.475025191s core uplink us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XXX]
duration:
5.000711199s
trace:
https://jaeger.dev.storj.io/trace/34942d9b407cfc57
FAIL (2026-01-12 14:10:48 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.000771514s core uplink us-select-1 us1-placement-20
checkers.Uplink{Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "uplink-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (uplink.us_select_1.remote)
antiblip:
1 out of 60 is failed (threshold=3) [...........................................................X]
duration:
30.003097197s
trace:
https://jaeger.dev.storj.io/trace/62457ac9390c0b68
FAIL (2026-01-12 14:10:44 UTC)
orbiter warning: 
operation: sync2.Copy
======ERROR======
uplink: metaclient: context deadline exceeded
======LOGS======
 12 14:10:16.214443221 (910h41m16.085526256s) [upload=797081]: Starting upload
Jan 12 14:10:16.214501773 (910h41m16.085584808s) [upload=797081]: Done starting upload
Jan 12 14:10:16.214529335 (910h41m16.08561237s) [upload=797081]: Uploading segments...
Jan 12 14:10:16.214820539 (910h41m16.085903575s) [upload=797081]: Got next segment. Inline: false
Jan 12 14:10:16.214857710 (910h41m16.085940755s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectBegin
Jan 12 14:10:16.214862800 (910h41m16.085945835s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentBegin
Jan 12 14:10:16.725378405 (910h41m16.59646144s) [seg_pos={0 0}, upload=797081]: Begin upload segment...
Jan 12 14:10:16.725422480 (910h41m16.596505515s) [seg_pos={0 0}, upload=797081]: Done begin upload segment.
Jan 12 14:10:16.725445774 (910h41m16.596528809s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.725501781 (910h41m16.596584816s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726116811 (910h41m16.597199846s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726412845 (910h41m16.59749588s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.726772159 (910h41m16.597855194s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727073553 (910h41m16.598156598s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727623851 (910h41m16.598706886s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.727995899 (910h41m16.599078934s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.729145138 (910h41m16.600228183s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.730429523 (910h41m16.601512558s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.732089985 (910h41m16.60317302s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.734072910 (910h41m16.605155945s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.735553440 (910h41m16.606636475s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.736428867 (910h41m16.607511902s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.737513312 (910h41m16.608596357s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:16.738969535 (910h41m16.61005257s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Uploading piece...
Jan 12 14:10:18.481875246 (910h41m18.352958281s) [upload=797081]: Next returned nil segment
Jan 12 14:10:18.481893851 (910h41m18.352976896s) [upload=797081]: Waiting for error group managing segments...
Jan 12 14:10:18.865443483 (910h41m18.736526518s) [noise=true , address=77.90.31.17:4773     , piece_id=5GONMY42UZABMNKH7VEKP4DNQRKHQRDPWDQIFXS2TU3ZD4OGKUHA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.874831919 (910h41m18.745914954s) [noise=true , address=145.79.227.8:4329    , piece_id=OV53BYUU5J4JJRP6RCS6LFCGAFM4GVVXQQ7EOWNY7S2L7BTC6M3A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.912617466 (910h41m18.783700501s) [noise=true , address=145.79.227.51:4410   , piece_id=JHH7QYI3SEC6UIRPAN2KVPXVP3I5BM2IUDC5COS27VVS3DWIXMIQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.972954513 (910h41m18.844037548s) [noise=true , address=45.91.116.11:4120    , piece_id=3FJMTDZDXLLJH7CZC62LKO3M6HQLJWASXIR3TVFSWHGZC4JRHJNA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.977638962 (910h41m18.848722007s) [noise=true , address=145.79.227.3:4150    , piece_id=UYVQ2ZSQEO6YMEYD6N36AASIVFLERJ7FZDVAGPU37W7FCTGVTLRA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:18.996512569 (910h41m18.867595604s) [noise=true , address=145.79.227.39:4015   , piece_id=6SVVXMZDSR6NSTW62KAQ43SD6DNZ3NPL7GIVGALAN7IHHEWPN3UQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.004938513 (910h41m18.876021568s) [noise=true , address=45.91.116.7:4010     , piece_id=PF5Y7RDLRPNHWUKM2WBN7I4GLEWUDC4OVFTDNZ52TF2OCNBZO5MQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.011100415 (910h41m18.882183461s) [noise=true , address=45.91.116.4:4940     , piece_id=YHCUZPDIZXCKIQZMXDMEM2UFMDWKTP4ZBZH3WD4YKQDKLCV3FPWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.068340406 (910h41m18.939423451s) [noise=true , address=45.91.116.12:4226    , piece_id=LW2SXKY7MVKXVUQPCFSVIHJWSZOJENBJ5S7MBJWJQNKT5WCEWL4A, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.251481592 (910h41m19.122564637s) [noise=true , address=145.79.227.13:4804   , piece_id=HCPA472DYKBR2FOFLWOF43TWZXZL7JJIGFMKSPI2PNO7EZG7WUPA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.563768072 (910h41m19.434851107s) [noise=true , address=145.79.227.7:4293    , piece_id=GHVSTBWK44UQ3GKKEHGGNFT3IJAIXVARD3HBL7XJQ67MCMST5UWQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.581405435 (910h41m19.45248848s) [noise=true , address=145.79.227.2:4114    , piece_id=CQMEYBMBI32LEQES66BCXMRUROHIO3367MANXQW6RYTS72TTFI4Q, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646368661 (910h41m19.517451696s) [noise=true , address=145.79.227.16:4909   , piece_id=25Z7ECFJQBG3K3UJKVL3UQNKCZUTPKCF3QLMSLWEHFL4QTGBXQZA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: <nil>
Jan 12 14:10:19.646388719 (910h41m19.517471754s) [seg_pos={0 0}, upload=797081]: Segment reached optimal threshold of 13 pieces.
Jan 12 14:10:19.646563071 (910h41m19.517646106s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1QJBWV6xLXS7DcX8d26be79z1DH3r27oi4EcuiwVwR5Dn8JEGh): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646570135 (910h41m19.51765318s) [noise=true , address=145.79.227.42:4121   , piece_id=Q2ADXCPCMQVJEZKYSOFFAFF3U2IZNJA3H3IALYETZVOPSOGWJ4BQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.646875135 (910h41m19.51795817s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:12crcXpNe9NgeXTsUpooLSBgrSUFbRxbSbanLa39w2Hr5mGFww): protocol: expected piece hash; context canceled; EOF
Jan 12 14:10:19.646901976 (910h41m19.517985011s) [noise=true , address=145.79.227.14:4840   , piece_id=L7Q5VDDJQ7QMOJDJYTA5TE6V57UMRL2WSRBFT3QRQXTYB5FM36OA, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647079855 (910h41m19.5181629s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Done uploading piece. err: context canceled; ecclient: upload cut due to slow connection (node:1229E58YirF37sZU6hfZGrDNFSxQDYsWSo4x3mG9BoYsNsLNtjy): protocol: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection; piecestore: piecestore close: write tcp 188.166.225.170:51734->145.79.227.22:4509: use of closed network connection
Jan 12 14:10:19.647093291 (910h41m19.518176326s) [noise=true , address=145.79.227.22:4509   , piece_id=5BXZPLQ3EVX7MFVQBIUDEU37U7OU734U3Z6PT6LCVQ56ZA7G26DQ, seg_pos={0 0}, upload=797081]: Long tail cancellation due to optimal threshold
Jan 12 14:10:19.647125863 (910h41m19.518208908s) [seg_pos={0 0}, upload=797081]: Done waiting for segment. successful: 13 optimal: 13 errs: <nil>
Jan 12 14:10:19.647203160 (910h41m19.518286195s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_SegmentCommit
Jan 12 14:10:19.647208149 (910h41m19.518291194s) [upload=797081]: Flush batch item: *pb.BatchRequestItem_ObjectCommit
Jan 12 14:10:19.967827495 (910h41m19.838910541s) [upload=797081]: Done uploading segments...
Jan 12 14:10:19.967846733 (910h41m19.838929768s) [upload=797081]: Upload finished. err: <nil>
Duration: 30.003120382s core uplink us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
32 out of 60 is failed (threshold=3) [X.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX..]
duration:
26.764268714s
trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:10:38 UTC)
orbiter warning: context deadline exceeded
Duration: 26.764301066s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
10 out of 60 is failed (threshold=3) [.....X..X.X..............XX..X............X...X...X...X.....]
duration:
19.549049075s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:10:32 UTC)
orbiter warning: context deadline exceeded
Duration: 19.549178301s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..XX]
duration:
5.001908943s
trace:
https://jaeger.dev.storj.io/trace/1978a372cd0fdf99
FAIL (2026-01-12 14:10:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001934502s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [.......X....................................................]
duration:
15.407194876s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:10:25 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 15.407216376s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..X]
duration:
5.000472963s
trace:
https://jaeger.dev.storj.io/trace/130f2c92c624998
FAIL (2026-01-12 14:09:51 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.00049268s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
11 out of 60 is failed (threshold=3) [X.....X..X.X..............XX..X............X...X...X...X....]
duration:
10.035085191s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:09:48 UTC)
orbiter warning: context deadline exceeded
Duration: 10.035114497s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
11 out of 60 is failed (threshold=3) [.X.....X..X.X..............XX..X............X...X...X...X...]
duration:
24.372352382s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:09:36 UTC)
orbiter warning: context deadline exceeded
Duration: 24.372366909s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [........X...................................................]
duration:
7.541529701s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:09:22 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.54155005s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
33 out of 60 is failed (threshold=3) [XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX.]
duration:
20.864895844s
trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:09:18 UTC)
orbiter warning: context deadline exceeded
Duration: 20.864920902s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
11 out of 60 is failed (threshold=3) [..X.....X..X.X..............XX..X............X...X...X...X..]
duration:
34.21347984s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:09:11 UTC)
orbiter warning: context deadline exceeded
Duration: 34.2135014s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
50 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X..]
duration:
4.878046335s
trace:
https://jaeger.dev.storj.io/trace/14fff1b25636a1ec
FAIL (2026-01-12 14:08:57 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.8780697s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
1 out of 60 is failed (threshold=3) [.........X..................................................]
duration:
13.273918492s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:08:27 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 13.273949591s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
11 out of 60 is failed (threshold=3) [...X.....X..X.X..............XX..X............X...X...X...X.]
duration:
18.103630388s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:08:19 UTC)
orbiter warning: context deadline exceeded
Duration: 18.103644295s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
33 out of 60 is failed (threshold=3) [.XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...XX]
duration:
30.000571947s
trace:
https://jaeger.dev.storj.io/trace/2e66a8e9764e4194
FAIL (2026-01-12 14:08:16 UTC)
orbiter warning: context deadline exceeded
Duration: 30.000592366s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
51 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X.]
duration:
4.301902302s
trace:
https://jaeger.dev.storj.io/trace/14fff1b25636a1ec
FAIL (2026-01-12 14:08:03 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.301929915s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
11 out of 60 is failed (threshold=3) [....X.....X..X.X..............XX..X............X...X...X...X]
duration:
1m0.001559494s
trace:
https://jaeger.dev.storj.io/trace/deb3adabf9d5a90
FAIL (2026-01-12 14:08:01 UTC)
orbiter warning: context deadline exceeded
Duration: 1m0.001836222s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
32 out of 60 is failed (threshold=3) [..XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...X]
duration:
30.001233765s
trace:
https://jaeger.dev.storj.io/trace/6da172feac770390
FAIL (2026-01-12 14:07:27 UTC)
orbiter warning: context deadline exceeded
Duration: 30.001264433s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.X]
duration:
5.001359219s
trace:
https://jaeger.dev.storj.io/trace/14fff1b25636a1ec
FAIL (2026-01-12 14:07:23 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 5.001382212s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
2 out of 60 is failed (threshold=3) [X.........X.................................................]
duration:
8.948318239s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:07:21 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 8.948346894s edge gateway us-select-1
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us_select_1.remote)
antiblip:
31 out of 60 is failed (threshold=3) [...XX.X.X.....XX.XXX...XXXXXXX.XX....X.X.XXX.X.XXX.XX.X.X...]
duration:
7.322595896s
trace:
https://jaeger.dev.storj.io/trace/6482acd26cf47fea
FAIL (2026-01-12 14:06:32 UTC)
orbiter warning: context deadline exceeded
Duration: 7.322616615s edge gateway us-select-1
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/50kb", Size: 50.0 KiB} failure (antiblip window 6/60): (gateway.us1.placement_20_thumbnails)
antiblip:
52 out of 60 is failed (threshold=6) [XXXXXXXXXXXXXXXXXXXXXXXXXXXXX..XX.XXXXXX.X.XX.XXXXXX.XXXXXX.]
duration:
4.460747824s
trace:
https://jaeger.dev.storj.io/trace/2995f64c02db1c02
FAIL (2026-01-12 14:06:31 UTC)
orbiter warning: operation error S3: PutObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 4.460773081s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.de2eae4e.storjapi.io/", AuthURL: "https://auth.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-placement-20", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/4M", Size: 4.0 MiB} failure (antiblip window 3/60): (gateway.us1.placement_20)
antiblip:
10 out of 60 is failed (threshold=3) [.....X.....X..X.X..............XX..X............X...X...X...]
duration:
9.163777461s
trace:
https://jaeger.dev.storj.io/trace/17a40fac4325fdf3
FAIL (2026-01-12 14:06:30 UTC)
orbiter warning: context deadline exceeded
Duration: 9.163807538s core uplink us-select-1 us1-placement-20
checkers.Gateway{"https://s3.us-select-1.storjapi.io/", AuthURL: "https://auth.us-select-1.storjshare.io/", Satellite: "12EayRS2V1kEsWESU9QMRseFhdxYxKicsiFmxrsLZHeLUtdps3S@us1.storj.io:7777", Bucket: "uptime-checks-us-select", Key: "gateway-tests/from_do_prod_ops_orbiter_ap_0/1K", Size: 1.0 KiB} failure (antiblip window 3/60): (gateway.us1.inline)
antiblip:
2 out of 60 is failed (threshold=3) [.X.........X................................................]
duration:
7.501269694s
trace:
https://jaeger.dev.storj.io/trace/5ec1a2058ce3325
FAIL (2026-01-12 14:06:29 UTC)
orbiter warning: operation error S3: GetObject, https response error StatusCode: 0, RequestID: , HostID: , canceled, context deadline exceeded
Duration: 7.501294652s edge gateway us-select-1